What Is llms.txt? How to Make One, What to Expect
Yuuuk94·Published
Summary
llms.txt is a Markdown file at your site root (/llms.txt) that tells AI models what the site is and where its important pages are. It was proposed in September 2024 by Jeremy Howard of Answer.AI.
It takes ten minutes to write and costs nothing. But it isn't an official standard yet, and few major AI search providers say they use it for ranking or citation. Make one with accurate expectations.
01Why the file exists
Web pages carry a lot of material unrelated to the content: navigation, ads, scripts. A model has a limited context window when it answers, so reading a whole site as raw HTML is wasteful.
The idea behind llms.txt is simple: alongside the HTML meant for people, offer a short summary and a list of key links in Markdown that is easy for AI to read. It is especially useful for well-structured sites like developer documentation.
02Format: a fixed order
The format proposed at llmstxt.org follows the order below. Only the H1 on the first line is required, but the rest is what makes the file useful.
# Site name(H1, required)> One- or two-sentence summary(blockquote)- Optional paragraphs with more detail about the site
## Section nameheadings, each followed by a- [Link title](URL): descriptionlist- An
## Optionalsection for links that can be skipped when context is short
# META_SCAN.BIZ
> Free SEO/AEO/GEO checker. Enter one URL to check meta tags,
> robots.txt, sitemap, structured data, AI signals and Lighthouse.
## Tool
- [META_SCAN.BIZ](https://meta-scan.biz/en): checker home
## Blog
- [How to check robots.txt](https://meta-scan.biz/en/blog/robots-txt): syntax and common mistakes
## Optional
- [Terms](https://meta-scan.biz/en/terms)03Common mistakes
| Mistake | Problem | Fix |
|---|---|---|
/llms.txt returns the home page HTML | Looks like the file exists but it doesn't (soft 404) | Make sure it responds with real text/plain content |
| Listing every page | The summary stops being a summary | Keep it to the 10–30 key pages |
| Links without descriptions | The model can't tell which page to read | Add a one-line : description to each |
| Stale content | Points to pages that no longer exist | Generate it at build time, like a sitemap |
04How to add one in common setups
The simplest option is a static file. If your site keeps adding pages, though, a hand-maintained file goes stale quickly, so generating it at build time like a sitemap is better.
| Setup | How |
|---|---|
| Static site or plain hosting | Put llms.txt in your static folder (often public/) so it is served from the root. |
| Next.js (App Router) | Create a GET handler in app/llms.txt/route.ts that responds with text/plain, building the link list from your content. |
| WordPress | Use a plugin that generates llms.txt, or upload the file to the site root. |
Whatever you choose, open the URL after deploying, confirm the content-type header is text/plain, and check that the Markdown shows as-is.
05What to expect from it
To be candid, there is little evidence yet that llms.txt improves AI search visibility. Google says no special AI text files or markup are needed to appear in AI features such as AI Overviews.
There are still reasons to make one. Developer tools and agents that read site documentation increasingly use the file, and it costs almost nothing. Writing it also forces you to decide what your site's key pages are, which helps SEO directly.
What matters more for AI search: not blocking AI crawlers in robots.txt, having the main text in the HTML without JavaScript, answer-first paragraphs and FAQs, and structured data. META_SCAN's AI signals card checks these together.
FAQ
- Q. Without llms.txt, will I be left out of AI search?
- No. Most AI search products crawl ordinary web pages to build answers. llms.txt is an optional extra signal, and there is no evidence of a penalty for not having one.
- Q. How is it different from robots.txt?
- robots.txt is a set of rules that allow or block crawlers. llms.txt is a document that summarizes your site for AI. llms.txt can't block anything.
- Q. What is llms-full.txt?
- A variant that inlines the full text of key documents into one file instead of linking to them. It is mostly used by documentation-heavy developer sites. For a typical service site, llms.txt alone is enough.
Related reading
References
How does your site do?
Enter one URL to run the free SEO/AEO/GEO checklist, including everything covered in this post.
Check for free