How does robots.txt control AI crawlers?
robots.txt is a plain text file in the root of your site that tells crawlers which paths they may fetch. Rules are written in groups that start with User-agent lines. A crawler first looks for the group that carries its own name; if there is none, it follows the default group marked with an asterisk (*). Within a group, the longest rule matching the path wins, and Allow wins a tie.
These rules became a standard with RFC 9309 in 2022. Most well-known crawlers follow the standard, but robots.txt is a request, not access control. To stop requests that ignore the rules, you need additional measures on your server or CDN. This tool also applies the interpretations Google documents: groups naming the same crawler are merged, and content after 500 KiB is ignored.
