What Is llms.txt and Why Websites Are Adding It for AI Crawlers

llms.txt is a proposed file that websites place at their root, such as example.com/llms.txt, to give AI tools a short, clean guide to the site: what it is, and links to its most useful pages. It was proposed in September 2024 by Jeremy Howard of Answer.AI.

What goes in it

The file is written in Markdown, a simple text format. A typical one has:

• The site or project name as a heading.
• A one- or two-sentence summary.
• Sections listing key pages, each with a link and a short description.
• Optionally, a separate llms-full.txt with the full text of important pages.

The idea is that AI tools struggle with web pages full of navigation, ads and scripts, and a plain summary with links is easier to use.

How it differs from robots.txt

• robots.txt tells crawlers which parts of a site they may visit. It's widely recognized. See whether AI bots respect robots.txt.
• llms.txt doesn't control access. It's a guide to content, more like a curated sitemap written for AI.

Do AI systems use it?

As of 2026, there's no clear evidence that the major AI search systems rely on it. Google's John Mueller has said that AI services don't use it, comparing it to the old keywords meta tag. Some developer tools and coding assistants do read llms.txt files, especially for software documentation, which is where it's most common.

Who's adding it

Documentation sites for software tools and APIs adopted it first, because developers often feed docs to AI coding assistants. Some other sites add it as a low-cost experiment.

Should you add one?

• It's cheap: a small text file that takes an hour to write.
• It won't hurt your search rankings.
• Don't expect traffic from it. Being cited in AI answers depends far more on clear, useful, well-structured content and a site that's easy to crawl.

If you do add one, keep it current, link to your best pages, and describe each in plain language.

What matters more for AI visibility

• Pages that answer questions clearly, with the answer near the top.
• Structured data describing your organization and content.
• Pages crawlers can load without heavy scripts.

See how knowledge graphs power AI answers and controlling AI crawlers.