Glossary
llms.txt
llms.txt is a proposed convention: a Markdown file at /llms.txt that gives language models and AI agents a short, curated guide to a website, with links to its most useful pages. It was proposed in 2024 and is not an internet standard.
Format
# Example Store
> Desk lamps and lighting, shipped from Rotterdam.
## Products
- [Desk lamps](https://www.example.com/lamps.md): every lamp model with specs
## Help
- [Shipping and returns](https://www.example.com/shipping.md): delivery times and return policy
The proposal asks for a single H1 with the site name, a short quoted summary, then sections of links with a line of description each. Some sites also publish a larger llms-full.txt that includes the page content itself.
What it does and does not do
- It is a guide, not a rule. Unlike robots.txt, it does not say what may be fetched. It says where the good pages are.
- It is not a sitemap. A sitemap lists every URL for crawlers; llms.txt is a short, hand-picked list written for a reader with limited context.
- No one has to read it. Adoption varies between AI vendors and tools, and none of them guarantees that the file changes what a model says or finds. Treat it as low-cost help, not a ranking lever.
If you collect data, an llms.txt is an easy list of the pages a site considers its main ones, and its links often point to clean Markdown versions of pages. If you publish one, keep it short, keep it current, and link to pages that stand on their own.
In CleanedWeb
This site publishes /llms.txt and /llms-full.txt, which list its documentation, free tools, use cases and these glossary pages. The free generator below drafts one for your own site from its home page, sitemap and page titles; you edit it before you publish.
Tools and docs
- llms.txt generator Draft a file from a site's home page and sitemap, then edit and download it.
- Website data checker See whether a site already publishes an llms.txt.
- Sitemap checker Validate the sitemap a generated llms.txt draws its links from.
Related terms
-
robots.txt
The file where a site tells crawlers what they may fetch, and what it does not do.
-
Sitemap
The XML file that lists a site's URLs, and how to use it to find pages and spot changes.
-
Structured data
JSON-LD and schema.org markup: what sites publish for machines and where it falls short.
Build an API from a website
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.