What is llms.txt, and how is it different from robots.txt?
llms.txt is a Markdown file published at the root of a website that tells tools built on large language models what the site is about and which of its pages matter most. The format was proposed by Jeremy Howard on llmstxt.org in September 2024. It is not an official web standard; it is an open proposal adopted by the community.
The problem it addresses is simple: when an AI assistant reads your site, it has to dig the real information out of menus, scripts and repeated template text, and the amount of text it can read at once is limited. llms.txt gathers your most valuable pages, with short descriptions, in one place.
robots.txt, on the other hand, is an access file: it tells crawlers which paths they may or may not visit. sitemap.xml gives search engines the full list of URLs to crawl. llms.txt replaces neither of them; it does not grant or block access, and it does not try to list every URL. Think of it as a curated, annotated table of contents.
What should the llms.txt format look like?
The proposal asks for the file to be written in a specific order. This tool builds the draft in the same order, and if your site already has a file, it checks that file against these rules:
- A single H1 heading: # Site name. This is the only required part of the format.
- A short summary starting with >: one or two sentences on what the site offers and to whom.
- Optional paragraphs of notes; no headings are used in this part.
- Sections starting with ##, each with link lines in the form - [Page name](url): description.
- A section named exactly Optional: secondary links that can be skipped when a shorter context is needed.
How does this tool build the draft?
The draft is generated from signals your site already publishes; no content is made up. Link titles and descriptions come from your pages' title and meta description tags, and you can edit every line.
- It first looks at the Sitemap lines in robots.txt, or at /sitemap.xml if there are none, and follows up to five child sitemaps in a sitemap index. Internal links on the homepage are also taken into account.
- It collects up to 500 same-site URLs from the sitemap and selects up to 60 pages. Short paths and pages linked from the homepage come first.
- On multilingual sites, it uses the pages in the homepage's language. Image, feed, pagination, tag, search and cart URLs, paths that robots.txt closes to all crawlers, and pages marked noindex are skipped.
- Pages are grouped into sections by the first part of their path; pages such as privacy, cookie and terms of use go into the Optional section.
Does llms.txt increase visibility?
The honest answer: its effect is limited today and hard to measure. Google has not announced that it uses llms.txt for search rankings or AI summaries, and it is not clear how the crawlers of the large AI companies use the file. It is most useful when a user or developer gives your site to an AI tool as context.
Even so, it takes little effort to prepare and carries low risk. On sites where services, products or documentation are spread across many pages, it can help assistants reach the right page. Real visibility still depends on server-rendered, crawlable HTML, clear headings and correct structured data; keep in mind that AI crawlers generally do not execute JavaScript.