[go: up one dir, main page]

DEV Community

#robots

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
Should your link checker obey robots.txt? Let's argue about it
linkbruno profile

Should your link checker obey robots.txt? Let's argue about it

#discuss #webdev #crawling #robots
Comments
6 min read
The Five Paths Our robots.txt Names: Two Are Not on This Host, and Three Carry No Canonical
onepassapps profile

The Five Paths Our robots.txt Names: Two Are Not on This Host, and Three Carry No Canonical

#webdev #seo #robots #sitemap
Comments
3 min read
Two robots.txt Files, One Product: The App Host Has Rules, the API Host Has 1,248 Bytes of Licence and No Directives
onepassapps profile

Two robots.txt Files, One Product: The App Host Has Rules, the API Host Has 1,248 Bytes of Licence and No Directives

#webdev #seo #documentation #robots
Comments
3 min read
How the top 1,000 .jp sites treat AI bots: weekly robots.txt census
theta-migurei profile

How the top 1,000 .jp sites treat AI bots: weekly robots.txt census

#robots #ai #seo #webdev
Comments
5 min read
One robots.txt Rule to Opt Out of AI Training: What Developers Need to Know (2027)
hafizwaleedahmed5 profile

One robots.txt Rule to Opt Out of AI Training: What Developers Need to Know (2027)

#seo #webdev #ai #robots
Comments 3
2 min read
Humanoid Robots in 2026: How Close Are They to Useful Everyday Work?
the_daily_flare profile

Humanoid Robots in 2026: How Close Are They to Useful Everyday Work?

#robotics #ai #robots
Comments
8 min read
GPTBot in robots.txt: the hosting toggle developers need to check
henrikaberg profile

GPTBot in robots.txt: the hosting toggle developers need to check

#robots #ai #cloudflare #webdev
1
Comments
3 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.