What Is llms.txt, and Does Your Site Need One?

An llms.txt file is a short Markdown file at a website's root that tells AI agents what the company is and which pages are worth reading. Proposed in 2024, it is useful and cheap to make. It is not a ranking factor, though, and Google says its AI features need no special files.

What is llms.txt?

llms.txt is a proposed standard for a Markdown file, placed at a website's root, that gives language models and AI agents a concise description of a site and links to its most important pages. The idea is to save agents from wading through navigation, scripts and adverts to work out what a company does.

The proposal is short. According to llmstxt.org, it was put forward by Jeremy Howard on 3 September 2024 to "standardise on using an /llms.txt file to provide information to help agents use a website". It is a community proposal, not a rule set by any search engine or AI company.

Think of it as a cover letter for machines. A sitemap lists every page. An llms.txt file picks the pages that matter and says, in plain words, what each one is for.

What goes in an llms.txt file?

The format is simple. It has a heading with the company name, a one-paragraph summary in a blockquote, optional detail, and sections of links with a short note on each. A good file for a B2B company holds a canonical description and somewhere between 20 and 60 of its most useful pages.

A minimal file for the fictional Halcyra has four parts:

Every claim in the summary should be one the company would publish anywhere else. No invented numbers, no price figures that will date, and nothing the product does not yet do. An AI agent may quote the file directly.

Does llms.txt help SEO or AI visibility?

Not directly, and it is worth being honest about that. It is not a known ranking factor for Google, and Google's guidance says its AI features need no special files. Some AI tools and agent frameworks do read llms.txt, so it can help agents describe you accurately. Expect a modest benefit, not a traffic lift.

Google is explicit. According to its Search Central guidance, "You don't need to create new machine readable files, AI text files, or markup to appear in these features." The same page says "The best practices for SEO remain relevant." Clear, well-structured pages still do the heavy lifting.

So why bother? Because it is cheap and forces the company to write one accurate description of itself. Agents browsing on a buyer's behalf can use it too. The guide to generative engine optimisation covers the work that moves visibility more.

What should you check before writing one?

Check robots.txt first. If it blocks the crawlers AI assistants use, they cannot read your site or your llms.txt file, so the file is theatre. Then check that the facts are approved, current and consistent with your website. Finally, decide who will keep the file up to date.

Crawler names matter. According to OpenAI's crawler documentation, OAI-SearchBot surfaces websites in ChatGPT's search features, and OpenAI recommends allowing it in robots.txt. GPTBot, by contrast, collects content that may be used for training. Some companies block it while allowing search crawlers.

A short pre-flight list:

1. robots.txt allows the AI search crawlers you want to reach you.
2. Key pages load without scripts and state what they cover in the first paragraph.
3. The description uses only claims the company already publishes.
4. An owner is named to update the file when pages or products change.

Should your site have an llms.txt file?

For most B2B companies, yes, as long as it is sized correctly. It takes an hour or two, costs nothing to host and gives agents an accurate summary. Just do not expect it to fix poor visibility on its own, and do not let it drift out of date, because a wrong description is worse than none.

Maintenance is the part teams forget. Treat the file as a living description of the company, revised when the product, pages or positioning change. A dated file listing retired pages tells agents the wrong story.

The deployment step should sit with whoever runs the website. An outside writer can draft and validate the file, but someone at the company puts it live and owns it. The guide to measuring AI visibility shows how to see whether anything changed.

Skein writes llms.txt files from a client's approved product facts and checks robots.txt first, with someone at the client deploying it; the AI engineering page shows where it fits. That suits a B2B company with a site of more than a handful of pages. A one-page website gains little from one.

Frequently asked questions

What is an llms.txt file?

An llms.txt file is a short Markdown file at a website's root that describes the company and lists its most useful pages for AI agents and language models. It was proposed in September 2024 as a way to help agents understand and use a website without parsing every page.

Does Google use llms.txt?

Google says you do not need new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode, and that standard SEO best practices still apply. It is not a known Google ranking factor. Some other AI tools and agent frameworks do read it.

What should be in an llms.txt file?

Include a heading with the company name, a one-paragraph summary, optional detail and sections of links to your most useful pages, each with a short note. Use only claims the company already publishes, avoid prices that will date, and keep it to roughly 20 to 60 pages.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers which parts of a site they may access, and it is respected by most major crawlers. llms.txt describes the company and points agents to the pages worth reading. If robots.txt blocks AI crawlers, an llms.txt file cannot help, so check robots.txt first.

Will llms.txt get my company mentioned by ChatGPT?

Not on its own. It may help agents describe you accurately, but mentions depend far more on clear, answer-first pages and on third-party coverage such as reviews and press. Make sure robots.txt allows AI search crawlers, then measure visibility over time to see what changes.

Related reading