Skip to content
GEO & AI visibilityFree

Robots.txt AI Crawler Checker

Enter your domain to see what your robots.txt rules mean for each AI crawler, allowed or blocked, which line makes the decision, and which rules we suggest.

Enter a domain or the full URL of a page.

If left empty, the path of the URL you entered is checked, or the home page (/) for a bare domain.

Results will appear here.

Suggested robots.txt rules

Choose your preferences and add the rules below to your robots.txt. If you already have groups for these crawlers, remove them first, because groups with the same name are merged.

Suggested robots.txt rules

Your pages can only be cited as sources in AI answers if these crawlers are allowed. Blocking them lowers the chance of being cited in those products.

Allowing them means your content may be used to train future models. Blocking them limits that and, according to the providers' documentation, does not affect their AI search crawlers. Both choices are valid; decide based on your content and licensing policy.

# AI search and user-request crawlers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Amazonbot
User-agent: DuckAssistBot
Allow: /

# Model training crawlers and control tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /

Googlebot, Bingbot, Applebot are left out of these rules; search engine crawlers keep following your existing rules.

We do not store the URL you enter or the results.

How to use it

  1. Enter your domain or the URL of a page; optionally add a separate path.

  2. Press Analyze; robots.txt is downloaded and evaluated for 18 crawlers one by one.

  3. Choose your preferences, then copy the suggested rules into your robots.txt.

How does robots.txt control AI crawlers?

robots.txt is a plain text file in the root of your site that tells crawlers which paths they may fetch. Rules are written in groups that start with User-agent lines. A crawler first looks for the group that carries its own name; if there is none, it follows the default group marked with an asterisk (*). Within a group, the longest rule matching the path wins, and Allow wins a tie.

These rules became a standard with RFC 9309 in 2022. Most well-known crawlers follow the standard, but robots.txt is a request, not access control. To stop requests that ignore the rules, you need additional measures on your server or CDN. This tool also applies the interpretations Google documents: groups naming the same crawler are merged, and content after 500 KiB is ignored.

Crawler types: search, user request and training

AI companies use different crawler names for different jobs. You need to know the difference to make the right call:

  • AI search crawlers: build the index behind AI search products and make it possible for answers to cite you as a source.
  • User-request fetchers: fetch a page on the spot when a user asks a question or shares a link. Some providers state that robots.txt rules may not apply to these requests.
  • Model training crawlers: collect content for training future models. Google and Apple manage this choice not with a separate crawler but with control tokens used in robots.txt.
  • Search engine crawlers: feed the regular search index, which AI features inside search, such as Google AI Overviews, also rely on.

Block or allow?

There is no single right answer. If you want your content to appear as a source in AI answers, you need to allow search and user-request crawlers. Training crawlers are a separate decision: allowing them means agreeing that your content may be used in future models; blocking them limits that and, according to the providers' documentation, does not affect search visibility.

The suggested rules take care of one detail: once you give a crawler its own named group, it stops reading the default (*) group. That is why the tool copies your existing default restrictions into the groups you allow, so closed areas such as an admin panel do not become exposed. Once you have decided, add the rules to your file and run the tool again to verify the result.

What happens if robots.txt cannot be reached?

If the file returns a 4xx code such as 404, crawlers assume there are no restrictions and may crawl the whole site. For server errors such as 5xx, RFC 9309 asks crawlers to treat the site as fully disallowed, and Google temporarily stops crawling in that case. The tool shows these two situations separately. If the robots.txt URL returns an HTML page instead of plain text, your rules are most likely not being read, and the report flags this as a warning.

Frequently asked questions

Can I block ChatGPT with robots.txt?

Partly. OpenAI uses separate crawler names for training, search and user requests, and you can manage each one separately in robots.txt. However, OpenAI states that robots.txt rules may not apply to requests a user initiates. If you want a complete block, you need additional rules on your server or CDN, because robots.txt is not access control.

If I block Google's AI training token, will I disappear from search?

No, you stay in search. Google's training control token is not a separate crawler; your pages are still crawled by Google's search crawler. According to Google, the token does not affect inclusion or ranking in search results; it only governs use of the content in Gemini models. AI Overviews rely on the search index, so they are controlled by search directives instead.

What happens if I have no robots.txt file?

If there is no file and the server returns 404, all crawlers may crawl the whole site, and that is not an error. Publishing a robots.txt is still an easy way to declare your sitemap and close areas such as an admin panel. What matters is that the URL never returns a 5xx error; otherwise crawlers may treat the whole site as disallowed.

How soon do rule changes take effect?

Not immediately, because crawlers cache robots.txt. Google generally caches the file for up to 24 hours, and other providers may use different durations. This tool, however, downloads the file live on every check, so you can verify your change right away. Give crawlers some time to pick up the new rules.

Why is a crawler evaluated against the default group instead of its own rule?

Because your robots.txt has no group that carries that crawler's name. In that case the crawler follows the default group marked with an asterisk (*), and the “Default (*) group” label in the table shows this. Conversely, once you give a crawler its own group, it no longer reads the default group, so you need to repeat important restrictions there.

Next step

Let's fix the findings on your site together

The tools show where the gaps are; applying the fixes to your site is a separate job. Book a free initial consultation with our team for SEO, GEO and AI integration.