Introducing Our Free Robots.txt Tester and AI Crawler Checker

If you've ever changed a robots.txt file and then crossed your fingers, you're not alone. One wrong line can stop Google crawling your product pages. One missing line can leave AI crawlers free to read everything.

The short answer: our free Robots.txt Tester and AI Crawler Checker lets you paste a robots.txt file, test any path against Googlebot, GPTBot, ClaudeBot and other search and AI crawlers, and see the exact rule that decides each result. It also builds a clean block of AI crawler rules for you. It's free, there's no account needed, and anything you paste stays in your browser.

Here's why we built it, how it works and how to use it.

Why we built a robots.txt tester

For years, the standard way to check a robots.txt file was Google's own tester in Search Console. You pasted your rules, typed a URL and it told you whether Googlebot could crawl it.

In November 2023 Google announced it was retiring that tester and replacing it with a robots.txt report in Search Console. The report is useful. It shows which robots.txt files Google found for your site, when it last fetched them and any errors. But it doesn't let you try out new rules before they go live, and it only covers Google.

Google's own help page now says that if you want to test a rule against a file that isn't on the web yet, you should use a third-party robots.txt tester. So we built one.

Why AI crawlers changed the question

The other reason is AI. A few years ago, most site owners only cared about Googlebot and Bingbot. Now the question we hear is "is GPTBot blocked on my site?", closely followed by "should it be?"

The big AI companies don't use a single crawler. They split them by job, and each one has its own user-agent name in robots.txt:

• OpenAI: GPTBot crawls content that may be used to train its models. OAI-SearchBot is used to show sites in ChatGPT search. ChatGPT-User visits pages when a person asks ChatGPT to, and OpenAI says robots.txt rules may not apply to those user-triggered visits.
• Anthropic: ClaudeBot collects content that may be used for training. Claude-SearchBot helps with search results. Claude-User fetches pages when someone asks Claude a question.
• Google: Google-Extended isn't a separate crawler at all. It's a control token. Blocking it tells Google not to use your content for training Gemini models or for grounding in Gemini, but Google says it doesn't affect your inclusion or ranking in Google Search.

That split matters. Blocking GPTBot asks OpenAI not to train on your content. Blocking OAI-SearchBot can stop you appearing in ChatGPT search answers. Many shops want the first and not the second. If you get the names wrong, you can end up blocking the wrong thing, or nothing at all.

If you're thinking about how AI tools will find and use your shop, our post on agentic commerce for UK shops goes into more detail.

What the tool does

• Tests paths against search and AI bots: pick bots from Search (Googlebot, Bingbot, DuckDuckBot, Applebot), AI training, AI search and user-triggered groups, or type in any custom user-agent token.
• Shows the deciding rule: click any result to open the "Why this cell" panel. It shows the matched group, the winning rule and the line that decides it, and highlights that line in your file. If a bot has no group of its own, it tells you it fell back to *.
• Bulk testing: switch on Bulk and test up to 500 paths at once, one per line.
• Lint checks: it flags rules that sit before any User-agent line, paths that don't start with / or *, Disallow: / under * (which blocks your whole site for any bot without its own group), relative sitemap URLs, and lines Google ignores such as crawl-delay, host and noindex. It also warns if the file is over 500 KiB, because Google ignores anything after that.
• AI rules generator: set each AI vendor to Allow, Block or Block training only, and it writes a commented robots.txt block you can copy. "Merge into my file" replaces just those bots' groups and shows you the changes before applying them.
• Shareable tests: "Copy link" stores your test in the link itself, so you can send it to a colleague without anything being saved on a server.

The matching follows the robots.txt standard (RFC 9309). The longest matching rule wins, and on a tie Allow wins. It also follows Google's documented behaviour, for example Googlebot-Image falling back to the Googlebot group when it has none of its own.

How to use it

1. Open the Robots.txt Tester.
2. Add your robots.txt. Paste it in, drop a .txt file onto the editor, or click "Load example" to see how it works first.
3. Or fetch it from your domain. Type your domain in "Fetch from a domain" and click Fetch. If the site blocks browsers from reading its robots.txt, you'll see a message saying so. Tick "Fetch via aamservices.uk" and fetch again, or use "Open robots.txt" and "Paste from clipboard" to paste the file in.
4. Enter a path to test, for example /checkout/basket?x=1. Turn on Bulk if you want to check a list.
5. Choose your bots. Use All, None or AI only, or tick them one by one.
6. Read the results. Each cell says Allowed or Blocked. Click one to open "Why this cell", which shows the "Matched group", the winning rule and the line in your file that decides it.
7. Fix and recheck. Edit the file in place and the results update as you type. When you're happy, copy it back to your site.

Once the new file is live, Google's robots.txt report in Search Console will show when Google has fetched it.

What stays in your browser

Like our other free tools, the testing happens on your device. What you paste, type, drop or read from the clipboard never leaves your browser. There's no account needed and nothing to install.

Fetching a domain is different. By default, your browser requests the domain's robots.txt directly. If you tick "Fetch via aamservices.uk", the request goes through our server instead, which helps with sites that block browsers. Your pasted content is never sent. You can choose whether to remember your last input, and if you do, it's saved in your browser only.

Robots.txt is not the whole story

Robots.txt asks crawlers not to fetch pages. It doesn't stop a page being indexed if other sites link to it, and it isn't a security tool. If a page must stay out of search results, use a noindex tag. If it must stay private, put it behind a login.

It also relies on crawlers respecting it. OpenAI, Anthropic and Google all document how their crawlers follow it, but the tester can only tell you what your file asks for, not who will honour it.

If you want a wider look at how your site is crawled and found, start with our guide to SEO basics for new websites.

FAQ

Does Google still have a robots.txt tester?

Not the old one. Google announced its retirement in November 2023 and replaced it with a robots.txt report in Search Console. To check whether a live URL is blocked for Google, you can use URL Inspection. To test new rules before they go live, Google suggests a third-party tester like this one.

Is GPTBot the same as ChatGPT-User?

No. GPTBot crawls content that may be used to train OpenAI's models. ChatGPT-User visits a page when a person asks ChatGPT to, and OpenAI says robots.txt rules may not apply to it. OAI-SearchBot is the one that decides whether you can appear in ChatGPT search.

Does robots.txt stop AI training?

It sends the signal. OpenAI, Anthropic and Google all say that blocking GPTBot, ClaudeBot or Google-Extended tells them not to use your content for training. It can't force a crawler that ignores the file.

Will blocking Google-Extended hurt my Google rankings?

Google says no. Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal.

Does robots.txt stop a page being indexed?

No. A blocked page can still appear in search if other pages link to it. Use a noindex tag for that, and don't block the page in robots.txt, or Google won't see the tag.

Where does robots.txt go?

At the root of each host, for example https://www.example.co.uk/robots.txt. A file in a subfolder doesn't count, and each subdomain needs its own.

Is my robots.txt uploaded anywhere?

No. Anything you paste or type stays in your browser, and there's no account to create. The only time our server is involved is if you tick "Fetch via aamservices.uk" to fetch a domain's public robots.txt.

How do I see why a bot is blocked?

Click its cell in the results. The "Why this cell" panel shows the "Matched group", the rule that won and the line that decided it, and highlights that line in your file.

Try the Robots.txt Tester and AI Crawler Checker and stop guessing which bots your file lets in. If you'd like help with crawling, indexing or AI search on your own site, get in touch. We're happy to talk it through.