Skip to main content

Free

Robots.txt Tester and AI Crawler Checker

Paste a robots.txt and see which search and AI bots can fetch a path, which rule decides it, and why. Generate a clean block for AI crawlers. Matching stays in your browser. An optional server fetch can load a live robots.txt when the site blocks the browser. Bot list last checked 25 October 2026.

Robots.txt tester

Paste a file, drop a .txt, or try a direct fetch. Matching stays in your browser.

Last checked: 25 October 2026

Fetch from a domain

Best effort only. With the box unticked, the browser asks for https://{domain}/robots.txt. Most sites block that read.

Robots.txt

0 B

Drop a .txt file onto the editor, or paste. Nothing is uploaded to a server.

Path(s) to test

Bots

Search
AI training
AI search
AI user-triggered
AI control tokens
Custom

18 allowed, 0 blocked.

Paste a robots.txt, load the example, or fetch from a domain to fill the matrix.

How robots.txt matching works

RFC 9309 treats the file as groups. A group is one or more user-agent lines followed by allow and disallow rules. Fields are case-insensitive. # starts a comment.

The tester picks the most specific matching group for the bot’s product token. If several groups name the same agent, their rules are combined. If none match, it uses the * group. If there is no * group either, the path is allowed.

Among matching rules, the longest path wins, counted in characters, including * and $. On a tie, allow wins. * matches any characters. $ anchors the end of the path. Percent-encoding is normalised before comparison. An empty disallow: means no restriction. /robots.txt itself is always allowed.

Worked example

User-agent: *
Disallow: /checkout/
Allow: /checkout/help$
  • /checkout/basket is blocked by the longer prefix /checkout/.
  • /checkout/help is allowed because /checkout/help$ is longer than /checkout/.
  • /checkout/help/x is blocked. The $ anchor stops the allow rule matching extra path segments.

Which AI crawlers should I block?

Training, search and user-triggered bots are separate programmes. A rule for one token does not cover the others. Blocking a training crawler asks the vendor not to use future crawls for foundation models. Blocking a search crawler can keep the site out of that product’s answers. Blocking a user-triggered fetcher can stop a live visit when a person asks the product to open a page.

There is no single correct policy. Publishers who want citations often allow search bots and restrict training bots. Publishers who want fewer machine visits restrict both. This tool shows the effect of each choice. It does not tell you which vendors will honour the file.

Related tools on the tools hub stay in the browser as well. An llms.txt builder and a sitemap validator are planned for the same directory when they exist.

Google-Extended explained

Google-Extended is a control token, not a crawler with its own user-agent string. Googlebot still fetches the page. If a robots.txt group disallows Google-Extended, Google says that crawled content is left out of Gemini training and out of grounding in Gemini Apps and Vertex AI. The token is not used as a Google Search ranking signal, and it does not remove the URL from Search.

Applebot-Extended works in a similar way for Apple foundation models. Applebot can still crawl for Spotlight, Siri and Safari after Applebot-Extended is disallowed.

Frequently asked questions

Does robots.txt stop AI training?

Only if the crawler honours the file, and only for the tokens you name. GPTBot, ClaudeBot, Google-Extended and similar tokens are the usual training or control switches. A Disallow is a request. It does not encrypt the page, hide it from people, or undo copies that were taken earlier.

Is GPTBot the same as ChatGPT-User?

No. OpenAI documents them separately. GPTBot is the training crawler. OAI-SearchBot is used for ChatGPT search. ChatGPT-User is used for some user-triggered fetches. A rule for one token does not apply to the others.

Does Google still have a robots.txt tester?

Google retired the robots.txt tester in Search Console. This page is a private replacement. Matching stays in your browser. It follows RFC 9309, with Google’s documented fallback from Googlebot-Image, Googlebot-News and Googlebot-Video to the googlebot group.

Does robots.txt stop a page being indexed?

No. robots.txt can ask a crawler not to fetch a URL. Search engines can still index a URL from links on other pages. To keep a page out of search results, use a noindex robots meta tag or HTTP header on a URL the crawler is allowed to fetch.

Where does robots.txt go?

At the host root, for example https://www.example.co.uk/robots.txt. Each host needs its own file. www and the apex are different hosts. Subfolders cannot supply a robots.txt for the rest of the site.

Is my robots.txt uploaded anywhere?

Paste, typing, file drop and clipboard reads stay in this browser. Fetching a live file is a direct request from your browser to https://{domain}/robots.txt unless you tick Fetch via aamservices.uk, which loads that same path through our server. Share links store the text in the URL hash on your device.

Does blocking Google-Extended remove a site from Google Search?

No. Google documents Google-Extended as a control token, not a crawler. Googlebot still crawls. Disallowing Google-Extended opts out of Gemini training and grounding. It is not used as a Google Search ranking signal.