CleanedWeb

Free tool

robots.txt tester

Load a site's robots.txt or paste your own, choose a crawler, and list the URLs to test. Each URL shows whether that crawler may fetch it and the exact rule and line that decides, using the matching rules of RFC 9309 that Google and Bing follow.

Or paste or edit the file below; the results update as you type.

How the rules are applied

The crawler's own group wins

A crawler follows the group that names it (User-agent: GPTBot). Only when no group names it does it follow User-agent: *. Groups that name the same crawler are combined.

The longest match decides

Among the rules that match a path, the longest one applies. When an Allow and a Disallow are equally long, Allow wins. No matching rule means the URL is allowed.

Wildcards

* matches any run of characters and a final $ anchors the end, so Disallow: /*.pdf$ blocks PDFs but not /file.pdf?download=1.

What robots.txt does not do

It asks crawlers not to fetch URLs; it does not hide them. A blocked page can still be indexed from links, and the file does not stop anyone who ignores it.

Questions

How do I block AI crawlers but keep search engines?

Add a group for each AI crawler with Disallow: / (for example User-agent: GPTBot, ClaudeBot, CCBot, Google-Extended) and leave the * group as it is. Test each name here before you publish.

Is crawl-delay respected?

Bing and some other crawlers read it; Googlebot ignores it. The tester shows the delay that applies to the chosen crawler.

Is the file I paste sent anywhere?

No. Testing runs in your browser. Only "Load robots.txt" asks our server to fetch the public file from the site.

More free tools

All free tools

Need the data itself, on a schedule?

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.