SEO · 9 min read
What Is llms.txt? A Plain Guide With an Example
What is llms.txt? A plain guide to the proposed file for AI tools: the format, llms-full.txt, how it differs from robots.txt, and a ready example.
If you have been reading about AI search, you have probably seen llms.txt mentioned as the next thing every website needs. It is simpler than it sounds: a short text file that gives AI tools a map of your site in plain language. This guide explains what it is, where it came from, what it can and cannot do, and how to write one, with an example you can copy.
In short: llms.txt is a proposed Markdown file at the root of a website (yoursite.com/llms.txt) that summarises the site and links to its most useful pages for large language models. Jeremy Howard proposed it in 2024. It is optional, and no major AI search provider has confirmed using it.
Where did llms.txt come from?
Jeremy Howard, co-founder of Answer.AI and fast.ai, published the idea in September 2024, with the specification at llmstxt.org. The reasoning is practical. Web pages carry menus, scripts, cookie banners and layout code. A language model with a limited context window has to wade through all of that to find the useful part. A clean Markdown summary saves it the trouble.
He first aimed it at documentation sites, so that coding assistants could find the right reference pages quickly. Since then, plenty of businesses, software projects and blogs have added one.
What does an llms.txt file contain?
The format is ordinary Markdown with a few conventions. In order:
- An H1 heading with the name of the site or project. This is the only required part.
- A blockquote with a short summary: who you are and what the site covers.
- Optional paragraphs with any extra context that helps a reader understand the rest.
- H2 sections containing lists of links, each written as
[Page name](URL): a short note. - An optional section titled "Optional" for links a tool can skip when space runs short.
That is all. No special syntax, no code, no registration.
An example llms.txt for a small business
Here is what a file might look like for a fictional bakery:
# Riverside Bakery
> Riverside Bakery is an independent bakery in Leeds making sourdough, pastries and celebration cakes to order. Open Tuesday to Sunday. Contact: [email protected].
## Products
- [Celebration cakes](https://riversidebakery.example/cakes): made to order, prices and lead times.
- [Bread and pastries](https://riversidebakery.example/bread): the daily range and allergens.
## Ordering
- [How to order](https://riversidebakery.example/order): online orders, collection and delivery area.
- [FAQ](https://riversidebakery.example/faq): opening hours, allergens, payment and refunds.
## Company
- [About](https://riversidebakery.example/about): who runs the bakery and where it is.
- [Contact](https://riversidebakery.example/contact)
## Optional
- [Blog](https://riversidebakery.example/blog): recipes and seasonal news.
Save the text at the root of your site so it loads at /llms.txt. Keep the summary factual. This is not the place for slogans. It is the place for the facts an assistant would need to describe you accurately.
What is llms-full.txt?
Some sites also publish llms-full.txt, which holds the full text of their key pages in one Markdown file rather than a list of links. An AI tool can then read everything in a single request. It became common on documentation sites, where build tools generate it from the source content.
If you publish one, generate it from your real pages as part of your website build. A hand-copied file drifts out of date the first time someone edits a page, and an outdated summary does more harm than none.
How is llms.txt different from robots.txt and sitemap.xml?
These three files often appear together, but they do different jobs.
| File | What it does | Who reads it | Does anything enforce it? |
|---|---|---|---|
| robots.txt | Says which crawlers may visit which paths | Search and AI crawlers | Well-behaved crawlers respect it |
| sitemap.xml | Lists every page you want indexed, with dates | Search engines | A hint, not a command |
| llms.txt | Summarises the site and points to the clearest pages | AI tools and assistants, if they choose to | No, it works only as a guide |
The most important difference: this file does not control access. Crawler rules such as OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot or Google-Extended live in robots.txt, which follows the Robots Exclusion Protocol (RFC 9309). Google explains how to write those rules in its robots.txt introduction. A perfect guide file does nothing if robots.txt blocks the crawler that would read your pages. For the sitemap side, see the sitemaps.org protocol.
Does llms.txt actually help with AI search?
Honestly, nobody can say for sure yet. At the time of writing, OpenAI, Google, Perplexity, Microsoft and Anthropic have not publicly confirmed that their search products read the file when they choose what to cite. Google's page on AI features in Search says those features rely on the normal search index and normal SEO practices.
So treat it as a small, cheap extra rather than a strategy. The work that matters comes first, and the vendors document it:
- crawlers that robots.txt allows,
- pages that search engines index and load fast,
- clear answers near the top of each page,
- consistent facts about who you are,
- structured data that matches what the page shows.
We cover all of that in our guide to generative engine optimization, and the citation side in how to get cited by ChatGPT.
Who should add an llms.txt file?
Some sites gain more than others. The file suits you best when:
- You run documentation or a help centre. Developers and coding assistants look for exact reference pages, which is the case the proposal was written for.
- Your site is large or hard to navigate. A short curated list can show an AI tool which dozen pages matter out of thousands.
- Your content changes often. If you can generate the file automatically, it stays current without effort.
- You publish products, prices or policies. A plain summary helps tools describe you correctly.
A five-page brochure site gains little. Fix the basics first, then decide.
What are the best practices for this file?
- Lead with facts. Say what you do, where, and for whom in the opening summary.
- Choose pages, not everything. Link to the pages that answer real questions. Leave out thin, duplicate or outdated ones.
- Write one clear note per link. A few words on what each page answers beats a bare list of URLs.
- Use full URLs that return a normal page. Avoid redirects and pages behind a login.
- Generate it from your content. A build step keeps the links and summary in step with the site.
- Keep it short. The point is a quick map, so move long material into llms-full.txt if you need it.
How do you add llms.txt to your website?
- List your key pages. Usually the home page, main services or products, about, contact, pricing if public, and your most useful guides.
- Write the summary. One or two sentences: name, what you do, where, how to contact you.
- Write a short note for each link saying what the page answers.
- Publish it at the root so it loads at yoursite.com/llms.txt as plain text.
- Automate it if you can. Many site builders and frameworks can generate the file from your pages so it never goes stale.
- Review it when your site changes, the same way you would review your sitemap.
On this site, the build generates both llms.txt and llms-full.txt from the same content as the pages themselves, so they cannot drift apart.
How can you check the file works?
No official validator exists, so a few manual checks do the job:
- Open yoursite.com/llms.txt in a browser. It should load as plain text, not as a download or an error.
- Check the response is a normal 200 with no redirect chain.
- Click every link. Each one should load a live page that search engines can index.
- Compare the summary with your home page. Names, locations and contact details should match exactly.
- Make sure robots.txt does not block the file or the pages it lists.
Repeat after a redesign, a domain change or a large content update.
Common mistakes to avoid
- Stuffing it with keywords. The file exists for clarity. A wall of search phrases helps nobody.
- Listing pages that block indexing or carry a noindex tag. Point only to pages you want people and tools to read.
- Claims that your site does not back up. If the summary says you are "award-winning", the pages had better show the award.
- Treating it as a replacement for SEO. It is a signpost, not a foundation. If you are new to the foundations, start with what SEO is.
Where Tarzi fits
AI search readiness, including robots.txt rules for AI crawlers, structured data and optional files like this one, forms part of our SEO service. For the wider view of how agencies use AI in marketing, see what an AI marketing agency does. If you would like us to look at your setup, get in touch.
Frequently asked questions
What is llms.txt?
llms.txt is a proposed Markdown file that sits at the root of a website, such as example.com/llms.txt. It gives AI tools a short summary of the site and a list of links to its most useful pages. Nothing requires a site to publish it, and it does not grant or block access.
Who created llms.txt?
Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed it in September 2024, and the specification lives at llmstxt.org. It is a community proposal, not an official web standard, so each AI company decides for itself whether its tools read the file at all.
Do ChatGPT or Google use llms.txt?
Nobody has confirmed it publicly. At the time of writing, OpenAI, Google, Perplexity, Microsoft and Anthropic have not said their search products read the file when they choose sources. Google says its AI features rely on normal search indexing, so treat the file as an optional extra, not a ranking factor.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which paths they may visit, and well-behaved crawlers follow it. The newer file controls nothing and only points AI tools toward your clearest pages. To allow or block an AI crawler you still edit robots.txt, so a good guide file cannot rescue a page that robots.txt hides.
What is llms-full.txt?
llms-full.txt is a companion file that holds the full text of a site's key pages in one Markdown document. An AI tool can read everything in a single request instead of following links. It appears most often on documentation sites, where a build step usually generates it from the source content.
Should a small business add an llms.txt file?
It is a reasonable extra if it takes an hour or less, especially when your site builds it automatically so it stays accurate. Do the proven work first: crawlable pages, clear answers near the top and correct robots.txt rules. Then add the file as a low-cost signpost.
How do I create an llms.txt file?
Write a Markdown file with an H1 for the site name, a short blockquote summary and H2 sections that list links with one-line notes. Save it as plain text at the root so it loads at yoursite.com/llms.txt. Generate it from your real pages so it never drifts out of date.
How do I check that my llms.txt file works?
Open yoursite.com/llms.txt in a browser and confirm it loads as plain text with a normal 200 status, not a redirect or an error page. Then click every link, confirm each target can be indexed, and compare the summary with your live pages. Repeat after any major site change.
