AI Crawler Checker

What it does

This checker reads a robots.txt file and tells you, for 27 AI user agents, whether the file allows or blocks them at the root of your site. Bots are grouped by purpose: model training, AI search and citations, and assistants that fetch a page live when a user asks. Each row shows the owner and a verdict. The rules are evaluated the way the robots.txt standard describes: a group that names the bot beats the * group, the longest matching path wins, Allow wins a tie, and * and $ wildcards are supported.

How to use it

1. To check a live site, type a domain or URL in "Domain or URL" and click "Fetch robots.txt & Check". The tool requests /robots.txt on that host through a public CORS proxy, copies the file into the text box and shows the report.
2. If the fetch fails, open the robots.txt link the tool gives you, copy the file, and paste it into "Or paste robots.txt content".
3. To test a draft, paste it into the same box and click "Check Pasted robots.txt". That path never leaves your browser.

Reading the verdicts

A "not named" tag means the file never mentions that bot, so its verdict comes from the * group, or it is allowed by default. The summary counts how many bots are blocked and how many your file names directly, and warns when you block some AI bots but leave others unnamed.

The tool tests the path "/" only, so a bot marked Allowed may still be shut out of a folder like /blog/. If one bot is named in several groups, their rules are merged before the check, as RFC 9309 requires. Some tokens, such as Google-Extended and Applebot-Extended, are control names you set in robots.txt rather than separate crawlers. And robots.txt is a request: this check cannot see firewall or CDN blocks, and it cannot tell you whether a bot obeys the file.

Related guide: The AI Crawler Map: Every Bot Reading Your Site in 2026