CleanedWeb

Glossary

llms.txt

llms.txt is a proposed convention: a Markdown file at /llms.txt that gives language models and AI agents a short, curated guide to a website, with links to its most useful pages. It was proposed in 2024 and is not an internet standard.

Format

# Example Store

> Desk lamps and lighting, shipped from Rotterdam.

## Products

- [Desk lamps](https://www.example.com/lamps.md): every lamp model with specs

## Help

- [Shipping and returns](https://www.example.com/shipping.md): delivery times and return policy

The proposal asks for a single H1 with the site name, a short quoted summary, then sections of links with a line of description each. Some sites also publish a larger llms-full.txt that includes the page content itself.

What it does and does not do

If you collect data, an llms.txt is an easy list of the pages a site considers its main ones, and its links often point to clean Markdown versions of pages. If you publish one, keep it short, keep it current, and link to pages that stand on their own.

In CleanedWeb

This site publishes /llms.txt and /llms-full.txt, which list its documentation, free tools, use cases and these glossary pages. The free generator below drafts one for your own site from its home page, sitemap and page titles; you edit it before you publish.

Tools and docs

Related terms

All glossary entries

Build an API from a website

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.