Free tool
robots.txt tester
Load a site's robots.txt or paste your own, choose a crawler, and list the URLs to test. Each URL shows whether that crawler may fetch it and the exact rule and line that decides, using the matching rules of RFC 9309 that Google and Bing follow.
How the rules are applied
The crawler's own group wins
A crawler follows the group that names it (User-agent: GPTBot). Only when no group names it does it follow User-agent: *. Groups that name the same crawler are combined.
The longest match decides
Among the rules that match a path, the longest one applies. When an Allow and a Disallow are equally long, Allow wins. No matching rule means the URL is allowed.
Wildcards
* matches any run of characters and a final $ anchors the end, so Disallow: /*.pdf$ blocks PDFs but not /file.pdf?download=1.
What robots.txt does not do
It asks crawlers not to fetch URLs; it does not hide them. A blocked page can still be indexed from links, and the file does not stop anyone who ignores it.
Questions
How do I block AI crawlers but keep search engines?
Add a group for each AI crawler with Disallow: / (for example User-agent: GPTBot, ClaudeBot, CCBot, Google-Extended) and leave the * group as it is. Test each name here before you publish.
Is crawl-delay respected?
Bing and some other crawlers read it; Googlebot ignores it. The tester shows the delay that applies to the chosen crawler.
Is the file I paste sent anywhere?
No. Testing runs in your browser. Only "Load robots.txt" asks our server to fetch the public file from the site.
More free tools
-
Website data checker
Enter a URL to see the JSON-LD, embedded JSON, feeds, sitemap, robots.txt rules and llms.txt a crawler finds there.
-
JSON to CSV converter
Paste an API response or NDJSON; nested fields become columns and you download a spreadsheet-ready CSV.
-
JSON Schema generator
Paste sample records to get a JSON Schema, TypeScript interfaces and Pydantic models that cover every record.
-
Sitemap checker
Find a site's sitemap and check its URLs, child sitemaps, duplicates, escaping and lastmod dates against the sitemap protocol.
-
llms.txt generator
Enter a site to get a draft llms.txt from its home page, sitemap and page titles, ready to edit and publish.
-
Shopify store to CSV
Enter a Shopify store or collection to download its public products as CSV, one row per product or per variant.
Need the data itself, on a schedule?
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.