Free tool
llms.txt generator
llms.txt is a Markdown file at a site's root that tells language models what the site is and which pages matter. Enter your site to get a draft built from its home page, sitemap and page titles, edit it here, and publish it at /llms.txt.
What goes in llms.txt
A name and a summary
The file starts with # Site name and a one-paragraph summary as a blockquote. The draft takes them from your home page's title and description.
Sections of links
Each ## Section lists pages as - [Title](URL): note. The draft groups pages by their first path segment, such as Docs or Blog.
Keep it short
List the pages that explain your product, pricing, documentation and policies. Leave out tag pages, archives and anything a model does not need to answer questions about you.
Publishing
Save the file as llms.txt at your site root and serve it as plain text. Some sites also publish Markdown copies of their pages and link those instead.
Questions
Do AI assistants read llms.txt?
It is a proposed convention (llmstxt.org), not a standard. Some agents and tools read it; others do not. It costs little to publish and gives agents a clean map of your site.
Why are some pages missing from the draft?
The draft takes at most 12 pages, shallow paths first, and skips pages that are marked noindex, have no title, or could not be loaded. Add any page by hand.
Is the draft stored?
No. It is generated on request and kept for ten minutes so a repeated request does not fetch your site again.
More free tools
-
Website data checker
Enter a URL to see the JSON-LD, embedded JSON, feeds, sitemap, robots.txt rules and llms.txt a crawler finds there.
-
JSON to CSV converter
Paste an API response or NDJSON; nested fields become columns and you download a spreadsheet-ready CSV.
-
JSON Schema generator
Paste sample records to get a JSON Schema, TypeScript interfaces and Pydantic models that cover every record.
-
robots.txt tester
Load or paste a robots.txt, pick a crawler and see whether each URL is allowed, with the rule and line that decides it.
-
Sitemap checker
Find a site's sitemap and check its URLs, child sitemaps, duplicates, escaping and lastmod dates against the sitemap protocol.
-
Shopify store to CSV
Enter a Shopify store or collection to download its public products as CSV, one row per product or per variant.
Need the data itself, on a schedule?
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.