CleanedWeb

Free tool

llms.txt generator

llms.txt is a Markdown file at a site's root that tells language models what the site is and which pages matter. Enter your site to get a draft built from its home page, sitemap and page titles, edit it here, and publish it at /llms.txt.

Reads the home page and up to 12 pages from the sitemap. Takes a few seconds.

What goes in llms.txt

A name and a summary

The file starts with # Site name and a one-paragraph summary as a blockquote. The draft takes them from your home page's title and description.

Sections of links

Each ## Section lists pages as - [Title](URL): note. The draft groups pages by their first path segment, such as Docs or Blog.

Keep it short

List the pages that explain your product, pricing, documentation and policies. Leave out tag pages, archives and anything a model does not need to answer questions about you.

Publishing

Save the file as llms.txt at your site root and serve it as plain text. Some sites also publish Markdown copies of their pages and link those instead.

Questions

Do AI assistants read llms.txt?

It is a proposed convention (llmstxt.org), not a standard. Some agents and tools read it; others do not. It costs little to publish and gives agents a clean map of your site.

Why are some pages missing from the draft?

The draft takes at most 12 pages, shallow paths first, and skips pages that are marked noindex, have no title, or could not be loaded. Add any page by hand.

Is the draft stored?

No. It is generated on request and kept for ten minutes so a repeated request does not fetch your site again.

More free tools

All free tools

Need the data itself, on a schedule?

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.