llms.txt: What It Is, the Format and Who Actually Serves One
llms.txt is a plain markdown file a website serves at /llms.txt to hand AI agents a short summary of the site and a list of links to the pages worth reading. Jeremy Howard put the proposal on llmstxt.org on 3 September 2024; its version 2 is dated 10 August 2026. When we requested the file on 17 November 2026, 38 of 49 well-known software, developer tool and docs sites served one, and so did 92 of the 192 directories and launch platforms we submit to that answered.
Searched as llm txt or llms-txt, it is the same file. This guide covers what an llms.txt file is for, the llms.txt format as the spec lays it out, this site's own two files as a worked example, the census of who serves one, what Google and the AI labs say about reading it, how to create and check one and how it sits next to robots.txt and sitemap.xml. For the crawler side (which AI bots a site lets in), see our LLM SEO guide.
llms.txt is a map for agents, not a gate for crawlers: robots.txt still decides who may crawl, and Google says its Search ignores llms.txt. Writing one takes an H1, a one-line summary and lists of links. On 17 Nov 2026, 38 of 49 well-known software sites served one, as did 92 of 192 directories we submit to, and 20 of those directory files linked individual listing pages, so a listing there can be one of the pages the directory's own file points agents to. HUNT0 sells one featured launch for $16.90; for $29 Submitator files 40 directories you pick out of the 202 we submit to.
What is llms.txt? What an llms.txt file is for
The proposal at llmstxt.org opens with its own definition: “A proposal to standardise on using an /llms.txt file to provide information to help agents use a website.” And then, plainly: “We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content.” The file is read when it is needed: “Agents are expected to view or search llms.txt to find the information they need, then follow the relevant links.” The detail stays behind the links, so an llms.txt works as a table of contents for a site, not a copy of it.
That makes it a different kind of file from robots.txt, which a crawler reads before it fetches anything. In the spec's words, “llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user.” On what the file feeds: “Our expectation was that llms.txt would mainly be useful for inference rather than training”, it says, and it adds that this is how the file has been used. Where it is used most: “llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials.”
Strictly, there is no llms.txt standard yet. The page calls itself a proposal to standardise on the file and closes with “The llms.txt specification is open for community input.” Version 2 records that “thousands of sites publish an llms.txt file”, that documentation platforms generate one and that Chrome's Lighthouse audits sites for one. It also settles where a file may live: “The file can be placed at the site root, or at any path within it, covering the pages under that path.” Its Changes page adds that “a file covers the pages under its path, and the most specific file applies.”
The llms.txt format, part by part
The spec defines a file by its sections, in a fixed order, so that a model can read it as markdown and a script can parse it with a regex. Six parts, of which the H1 is the one that must be there:
llms.txt format: the parts of the file, in order
| Part | Required | The spec's words |
|---|---|---|
| Byte-order mark | No | “An optional byte-order mark (BOM)” |
| H1: the name | Yes | “An H1 with the name of the project or site”, the one part the spec requires |
| Blockquote: the summary | No | “A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file” |
| Details | No | Paragraphs or lists “of any type except headings” |
| H2 file lists | No | Sections “delimited by H2 headers”, each a markdown list “containing a required markdown hyperlink [name](url), then optionally a : and notes about the file.” |
| An H2 named Optional | No | “The “Optional” section is used, by convention, for secondary information: links an agent can skip when a shorter context is needed.” |
From the Format section of "The /llms.txt file, v2" on llmstxt.org, by Jeremy Howard (the page's dates: September 3, 2024 and August 10, 2026), and its Changes page, read on 17 November 2026 and quoted character-exact.
Two conventions sit around the file. Pages an agent may need should exist as markdown: the spec asks for “a clean markdown version of those pages at the same URL as the original page”, either with .md appended (page.html.md) or with the extension swapped for .md (page.md). And the Optional section has changed status: since version 2 it carries no mechanical meaning, and the Changes page says “Optional sections are still allowed, and remain a useful convention for secondary links, but they no longer carry mechanical semantics.”
An llms.txt template to write by hand
Here is the whole format in one file, for a product we made up. The H1 names it, the blockquote says what it is and how it is priced, the plain lines carry the facts an agent should not have to follow a link for, and each H2 is a list of links with a note after the colon. Copy it, swap in your own pages and keep the Optional section for what an agent can skip.
llms.txt template (a made-up product; example.com is a reserved domain)
# Example Analytics
> Example Analytics is privacy-first web analytics for small SaaS teams. A free plan, then paid plans billed monthly.
Example Analytics runs as one script tag on your site. Data stays in the EU. The API is REST and returns JSON.
## Docs
- [Quickstart](https://example.com/docs/quickstart.md): install the script and see the first visit
- [API reference](https://example.com/docs/api.md): every endpoint, with request and response samples
## Product
- [Pricing](https://example.com/pricing.md): the plans, their limits and what each includes
- [Security](https://example.com/security.md): where data is stored and who can read it
## Optional
- [Changelog](https://example.com/changelog.md): what shipped, newest firstllms.txt example: the file on this site
This site serves both files, and we read both live on 17 November 2026. That day its llms.txt ran to 20,947 bytes and 138 lines: 8 H2 sections and 95 links, 36 of them to .md pages. Its llms-full.txt was 361,961 bytes over 9,214 lines. The opening lines and two of the sections, as served:
llms.txt file example: the opening of submitator.com/llms.txt ([...] marks lines left out)
# Submitator
> Submitator submits your startup to the directories you pick, for a one-time payment from $29. You add your product once, choose from a catalogue of 170+ directories and follow every submission in a dashboard. Agencies resell it under their own brand.
Plans, each a one-time payment:
- Starter: $29 for 40 directories
- Growth: $49 for 70 directories
- Scale: $79 for 110 directories
[...]
## Pages
- [Home and pricing](https://submitator.com/): what Submitator does, the plans with checkout and the FAQ
[...]
## Get started
[...]
- [Quickstart](https://submitator.com/docs/quickstart.md): Get a free test key, launch a client's project in test mode and receive a signed webhook when one of its listings goes live.- The H1.
# Submitatoris the name, and the one line every llms.txt needs. - The blockquote. What the product does and where its price starts, so an agent that reads nothing else still has the answer.
- The details. The plan lines sit in plain markdown under no heading: facts an agent should find without following a link.
- A file list.
## Pagesholds markdown links, each followed by a colon and a note on what the page holds. - Markdown twins. The API docs sections link .md addresses, the markdown version of each docs page, as the spec's .md convention asks; llms-full.txt carries the same docs pages in full.
Nobody types those plan lines. A script writes both files from the same data as the pricing cards and the docs, and a test fails when the files and the data disagree. A file that states prices goes stale the day the prices move, so build it the way the pages are built. The opening of the excerpt is built from that data as well, so it matches the file; the sizes above are from the read, and the live file grows as pages are added.
Who actually serves an llms.txt: 252 sites read on 17 Nov 2026
Among well-known software sites the file is the norm: 38 of the 49 that answered served one (78%), and 33 of those 38 had every part the spec describes. Among the 202 directories and launch platforms we submit to it is close to a coin toss: 92 of 192 (48%), with 72 complete. The directories' files are also shorter, a median 5.2 KB and 28 links against 14.9 KB and 106.
Who serves an llms.txt: 50 software sites against the 202 directories we submit to, 17 Nov 2026
| Measure | 50 well-known software and docs sites | 202 directories we submit to |
|---|---|---|
| Answered | 49 of 50 | 192 of 202 |
| Serve an llms.txt | 38 of 49 (78%) | 92 of 192 (48%) |
| Every part the spec describes | 33 of 38 | 72 of 92 |
| A blockquote summary | 34 of 38 | 84 of 92 |
| Links to .md pages | 24 of 38 | 11 of 92 |
| Links a pricing page | 32 of 38 | 42 of 92 |
| Median llms.txt | 14.9 KB, 160 lines, 106 links | 5.2 KB, 69 lines, 28 links |
| A fuller llms-full.txt too | 13 of 49 | 31 of 192 |
| An HTML page at /llms.txt | 0 of 49 | 11 of 192 |
GET /llms.txt and /llms-full.txt on each host, 17 November 2026, 01:36 to 01:42 UTC (the directories our catalog added that day, and PeerPush at its new address, at 16:14 UTC), a desktop browser user agent, redirects followed, then a curl retry of every miss. A site serves an llms.txt when the address answers 200 with a non-empty body that is not HTML and whose first non-blank line is an H1 ("# " and a name). Every part: one H1, a blockquote between it and the first H2, at least one H2 and a markdown list link under it, read outside code blocks. A fuller llms-full.txt answers 200 with text larger than the site's llms.txt. Answered leaves out hosts that sent a bot check, a server error, a paused deployment, a TLS failure or nothing on both passes: 1 software site (Screaming Frog) and 10 directories. Medians are over the served files, lines and links rounded; 1 KB is 1,000 bytes. Directories: every host of the catalog export of 17 November 2026 that a category hub lists, intercepts left out.
The AI labs keep their files on their developer docs, not on their home pages. OpenAI's developer docs, Anthropic's (which answer from platform.claude.com) and Perplexity's docs serve one; openai.com, anthropic.com and perplexity.ai answer 404. The spec itself links Gemini's file under ai.google.dev/gemini-api/docs/, a subpath the format allows, while the root of ai.google.dev answers 404.
SEO tools split: Semrush, Yoast and AIOSEO serve one, Ahrefs and Moz answer 404. Stripe serves one on stripe.com and another on docs.stripe.com; Paddle and Lemon Squeezy serve none. 7 of the 8 CMS and website builders do, Ghost being the exception. Sizes run across three orders of magnitude: the smallest file in the set is the spec's own, 637 bytes on llmstxt.org, and the largest is AIOSEO's at 787.2 KB with 3,468 links.
32 of the 38 files on well-known sites link a pricing page. If you have one, link it too; how SaaS companies structure that page is the subject of our post on SaaS pricing models. For more llms.txt examples than the 50 below, the spec lists three directories that index such files: llmstxt.site, directory.llmstxt.cloud and llmstxthub.com.
llms.txt examples: 50 software, developer tool and docs sites, read on 17 Nov 2026
| Site | Kind | llms.txt | Every part | .md links | Pricing link | llms-full.txt |
|---|---|---|---|---|---|---|
| OpenAIopenai.com | AI labs and model APIs | None (404) | None | |||
| OpenAI developer docsdevelopers.openai.com | AI labs and model APIs | 6.3 KB, 54 lines | Yes | 20 | No | 9.6 MB |
| Anthropicanthropic.com | AI labs and model APIs | None (404) | None | |||
| Anthropic developer docsdocs.anthropic.com | AI labs and model APIs | 102.1 KB, 994 lines | No | 916 | Yes | 31.5 MB or more |
| Perplexityperplexity.ai | AI labs and model APIs | None (404) | None | |||
| Perplexity docsdocs.perplexity.ai | AI labs and model APIs | 45.8 KB, 217 lines | Yes | 200 | Yes | 4.0 MB |
| Google AI for Developersai.google.dev | AI labs and model APIs | None (404) | None | |||
| Mistral AImistral.ai | AI labs and model APIs | 5.5 KB, 59 lines | Yes | 0 | Yes | None |
| Hugging Facehuggingface.co | AI labs and model APIs | None (404) | None | |||
| Stripestripe.com | Payments and billing | 69.8 KB, 401 lines | Yes | 24 | Yes | None |
| Stripe docsdocs.stripe.com | Payments and billing | 99.2 KB, 749 lines | No | 522 | Yes | None |
| Paddlepaddle.com | Payments and billing | None (404) | None | |||
| Lemon Squeezylemonsqueezy.com | Payments and billing | None (404) | None | |||
| Polarpolar.sh | Payments and billing | 16.7 KB, 238 lines | Yes | 122 | Yes | 792.3 KB |
| Chargebeechargebee.com | Payments and billing | 21.5 KB, 160 lines | Yes | 0 | Yes | None |
| Vercelvercel.com | Hosting and infrastructure | 14.9 KB, 151 lines | Yes | 104 | Yes | None |
| Netlifynetlify.com | Hosting and infrastructure | 3.0 KB, 46 lines | Yes | 28 | Yes | None |
| Cloudflarecloudflare.com | Hosting and infrastructure | 17.2 KB, 185 lines | Yes | 82 | Yes | 169.5 KB |
| Cloudflare developer docsdevelopers.cloudflare.com | Hosting and infrastructure | 16.6 KB, 143 lines | Yes | 0 | No | 31.5 MB or more |
| Supabasesupabase.com | Hosting and infrastructure | 3.5 KB, 59 lines | No | 32 | Yes | 6.8 MB |
| Renderrender.com | Hosting and infrastructure | 53.3 KB, 662 lines | No | 408 | Yes | 1.4 MB |
| Fly.iofly.io | Hosting and infrastructure | 5.9 KB, 77 lines | Yes | 14 | Yes | None |
| Neonneon.com | Hosting and infrastructure | 29.7 KB, 351 lines | Yes | 169 | Yes | 7.7 MB |
| WordPress.orgwordpress.org | CMS and website builders | 5.8 KB, 76 lines | Yes | 0 | No | None |
| Webflowwebflow.com | CMS and website builders | 13.3 KB, 143 lines | Yes | 0 | Yes | None |
| Framerframer.com | CMS and website builders | 9.8 KB, 95 lines | Yes | 0 | Yes | None |
| Wixwix.com | CMS and website builders | 38.7 KB, 263 lines | Yes | 1 | Yes | None |
| Squarespacesquarespace.com | CMS and website builders | 2.4 KB, 30 lines | Yes | 0 | Yes | None |
| Shopifyshopify.com | CMS and website builders | 12.6 KB, 112 lines | Yes | 2 | Yes | 19.0 KB |
| Ghostghost.org | CMS and website builders | None (404) | None | |||
| Sanitysanity.io | CMS and website builders | 29.9 KB, 179 lines | Yes | 0 | Yes | None |
| Ahrefsahrefs.com | SEO tools | None (404) | None | |||
| Semrushsemrush.com | SEO tools | 15.0 KB, 96 lines | Yes | 0 | Yes | None |
| Mozmoz.com | SEO tools | None (404) | None | |||
| Yoastyoast.com | SEO tools | 7.7 KB, 115 lines | Yes | 0 | No | None |
| AIOSEOaioseo.com | SEO tools | 787.2 KB, 3,504 lines | Yes | 0 | Yes | 18.7 MB |
| Screaming Frogscreamingfrog.co.uk | SEO tools | Bot check | Bot check | |||
| Mintlifymintlify.com | Docs platforms and frameworks | 6.7 KB, 79 lines | Yes | 36 | Yes | 141.6 KB |
| GitBookgitbook.com | Docs platforms and frameworks | 15.1 KB, 160 lines | Yes | 45 | Yes | None |
| ReadMereadme.com | Docs platforms and frameworks | 61.6 KB, 214 lines | Yes | 0 | Yes | None |
| Docusaurusdocusaurus.io | Docs platforms and frameworks | None (404) | None | |||
| Next.jsnextjs.org | Docs platforms and frameworks | 14.6 KB, 136 lines | Yes | 1 | No | None |
| llmstxt.orgllmstxt.org | Docs platforms and frameworks | 0.6 KB, 10 lines | Yes | 3 | No | None |
| GitHubgithub.com | Developer and SaaS tools | 28.7 KB, 178 lines | Yes | 0 | Yes | Not larger |
| Linearlinear.app | Developer and SaaS tools | 11.7 KB, 248 lines | Yes | 177 | Yes | HTML page |
| Notionnotion.com | Developer and SaaS tools | 6.8 KB, 74 lines | Yes | 0 | Yes | None |
| Zapierzapier.com | Developer and SaaS tools | 14.0 KB, 193 lines | Yes | 39 | Yes | 18.4 KB |
| PostHogposthog.com | Developer and SaaS tools | 514.9 KB, 5,125 lines | Yes | 4,883 | Yes | None |
| Sentrysentry.io | Developer and SaaS tools | 93.6 KB, 581 lines | No | 29 | Yes | Not larger |
| Resendresend.com | Developer and SaaS tools | 10.4 KB, 185 lines | Yes | 33 | Yes | Not larger |
Read on 17 November 2026 as in the table above, on the host shown. Served from another address after redirects: docs.anthropic.com from platform.claude.com/llms.txt, polar.sh from polar.sh/docs/llms.txt and squarespace.com from www.squarespace.com/assets/llms.txt. The spec allows a file below the root, and it links Gemini's at ai.google.dev/gemini-api/docs/llms.txt; this census asked each host for /llms.txt at its root. The Anthropic developer docs and Cloudflare developer docs llms-full.txt files were read up to 31,457,280 bytes and are at least that large. Not larger: /llms-full.txt answered with text no larger than the llms.txt (GitHub, Sentry and Resend). Bot check: a challenge page on both passes. Sizes are bytes as served, 1 KB = 1,000 bytes.
HUNT0: $16.90 for one featured launch. Submitator: $29 for 40 directories.
We file HUNT0 on its free and badge routes, one of 81 launch sites in our catalog. Add 39 more picks; we write a description for each site and keep every badge live. 70 for $49, 110 for $79, one payment.
Who reads llms.txt, and does it work?
Google and llms.txt
Google Search does not use it, and says so. Its guide to generative AI features, last updated 10 July 2026, puts llms.txt files among the things you can ignore for Search: you do not need them to appear there, it says, “as Google Search itself doesn't use them.” Then: “It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files.” “Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them.” The same passage notes that Google may crawl such a file like any other, and “this doesn't mean that the file is treated in a special way.”
Google's Chrome team does check for one. Lighthouse runs an llms.txt audit among its agentic browsing audits and describes the file this way: “The llms.txt file is an emerging convention used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents.” The audit is lenient: “Lighthouse flags the pages if a server error occurs when attempting to retrieve the llms.txt file.” “If the file is not provided by the server (resulting in a 404), the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment.” The page describes no check of the format itself.
OpenAI, Anthropic and Perplexity
The three labs document their crawlers in terms of robots.txt. OpenAI's crawler page: “OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.” Perplexity's crawler page: “Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity.” The Claude Help Center article on Anthropic's crawler says the bots respect do-not-crawl signals by “honoring industry standard directives in robots.txt”. None of the three pages says its crawlers read llms.txt, and Anthropic's does not mention the file.
Their docs use it, though. Anthropic's llms.txt, on its developer docs, is 102.1 KB with 917 links. OpenAI's crawler page opens with “For the complete documentation index, see llms.txt”, then “Markdown versions of documentation pages are available by appending .md to the page URL.” Perplexity's opens with “Fetch the complete documentation index at: /llms.txt” and “Use this file to discover all available pages before exploring further.” The labs write llms.txt for agents that use their own docs; none of their pages promises to read yours.
So does llms.txt work? For rankings in Google, no, by Google's own account. For agents that go looking for the file, its author says it does: the v2 change log reports that “coding agents use them reliably”. None of the four companies' pages above says its crawler ranks or cites a site because of the file.
LLM txt for SEO: what the file changes in search
Most questions about llm.txt for SEO come down to Google, and Google's answer is above: the file neither harms nor helps. For llm txt SEO in the newer sense, being read and cited by AI answers, access is still decided where it always was, in robots.txt rules for each crawler and in pages a crawler can fetch and read. An llms.txt adds a map for the agents that get in. Which AI crawlers the directories we submit to let in, robots.txt by robots.txt, is in the LLM SEO census; the groundwork a product site needs first is in our guide to SEO for a SaaS.
How to create an llms.txt file
Three routes: write it by hand, run a generator over your site, or let your CMS or docs platform write it. Whichever you take, the result has to answer at /llms.txt as plain text, not as a page of HTML.
By hand, in seven steps
- Put the product's name in an H1 on the first line.
- Write one blockquote line: what it is, who it is for, how it is priced.
- Add a few plain lines with the facts agents get wrong: limits, regions, what it does not do.
- Group links under H2s (docs, product, pricing, policies), each with a short note after a colon.
- Point the links at markdown versions of the pages where you have them.
- Move the nice-to-have links into an H2 named Optional.
- Save it as
llms.txtat the site root, served as text/plain or text/markdown.
llms.txt generators, free to start
An llms txt file generator crawls your site or reads your sitemap and drafts the file for you to edit. Three, each described from its own page on 17 Nov 2026. Firecrawl's llms.txt generator now opens with a deprecation notice: “This API endpoint will still remain active but we will no longer be maintaining it after June 30, 2025.” It points to an example repository instead. Writesonic's free llms.txt generator reads your sitemap, sets the terms as “No login. No email. 3 free files a day.” and says “Every file is checked against the llmstxt.org spec and scored, with anything that would fail listed out.” LLMrefs' generator is marked “Free, No card required”: “The tool will automatically crawl your website and create a standardized llms.txt file following the official specification.”
Whatever drafts it, the summary line and the notes are the parts worth writing yourself, since they are the lines that say what the product is. A list of every URL in a sitemap is not what the spec asks for: “The file itself stays small enough to fit in context.”
WordPress, Shopify, Wix and Webflow
On WordPress, Yoast SEO writes one: “Generate an llms.txt file automatically with a click.” “Available in one click, no upgrade needed.” Its page lists the feature in Yoast SEO for Shopify as well, which covers a Shopify store. The plugin directory has more WordPress llms.txt plugins; the largest it listed, Website LLMs.txt, shows “50,000+ active installations”. On Wix it is automatic: “Wix automatically generates and maintains your llms.txt file, but you can edit it at any time in your site's dashboard.” “Once you edit the file, it will stop updating automatically so your changes are preserved.” Webflow's own learning site describes the manual route: “In Webflow, you can create an llms.txt file by uploading it as an asset or by creating a page with the appropriate path.”
Mintlify, GitBook and Cloudflare's docs
Docs platforms generate the file on their own. Mintlify's llms.txt docs: “Mintlify automatically hosts an llms.txt file at the root of your project that lists all available pages in your documentation.” It caps what it generates: “Automatically generated llms.txt index files cannot exceed 100,000 characters.” A custom file has one rule: “Your custom llms.txt or llms-full.txt file must have a site title as an H1 heading.” GitBook: “GitBook produces llms.txt and llms-full.txt for published docs.” Cloudflare's developer docs show the pattern at scale. Their root llms.txt, in Cloudflare's words: “A directory of all Cloudflare documentation products, grouped by category.” Each product has its own llms.txt below it.
Next.js: a static file or a route handler
In Next.js the file can be static: “Next.js can serve static files, like images, under a folder called public in the root directory.” A public/llms.txt is served at /llms.txt. If it should be built from data, a route handler at app/llms.txt/route.ts does it: “You can use Route Handlers to return non-UI content.” This site serves both of its files from public/, written there by the script described above.
llms-full.txt: the whole site in one file
llms-full.txt is not in the spec; version 2 does not mention it. The ecosystem defines it. Mintlify: “The llms-full.txt file combines your entire documentation site into a single file as context for AI tools and LLM indexing.” Cloudflare's docs describe theirs as “The full contents of all Cloudflare documentation in a single file”. On 17 Nov 2026, 13 of the 49 well-known sites served a fuller llms-full.txt beside the llms.txt, and 31 of the 192 directories did. The big ones are big: OpenAI's developer docs file is 9.6 MB, and Anthropic's is at least 31.5 MB. Link it from your llms.txt so agents can find it; 11 of the 38 files on well-known sites do.
How to check an llms.txt: a validator checklist
The checks are short enough to run by hand: use the list below as your llms txt checker. Open the file, or fetch it, and go down it:
- It answers 200 at /llms.txt. After redirects, with a body. A 404 means there is no file.
- It is text, not a page. The content type is text/plain or text/markdown. On 17 Nov 2026, 11 of the directories we submit to answered /llms.txt with an HTML page, a soft 404 that a browser makes look fine.
- Line one is an H1. A single
#, a space and the name of the site or project. - A blockquote follows. One
>line with the summary. - Sections are H2s of links. Each item a markdown link,
[name](url), with an optional colon and note. - The links resolve, and point at pages an agent can read, markdown where you have it.
- It stays small. Mintlify splits a generated index past 100,000 characters, a fair ceiling for a hand-written one.
- Secondary links sit under Optional, so a short read can skip them.
The first two checks take one curl line each. The first prints the status and the content type; the second prints the opening lines, which should start with # and your product's name, then a > line:
llms txt check: two curl lines (swap in your own domain)
curl -sL -o /dev/null -w "%{http_code} %{content_type}\n" https://example.com/llms.txt
curl -sL https://example.com/llms.txt | head -n 5Run the same against /llms-full.txt if you serve one. As a validator, Lighthouse's llms.txt audit flags a server error and marks a 404 not applicable; Writesonic's generator scores a file against the spec.
llms.txt best practices
The spec's own four guidelines, in its order:
- “Use concise, clear language.”
- “When linking to resources, include brief, informative descriptions.”
- “Avoid ambiguous terms or unexplained jargon.”
- “Test your file by asking an agent questions about your content”, with nothing but your llms.txt to start from.
The census adds a few habits of the files on well-known sites. Lead with the blockquote: 34 of those 38 files carry one. Link the pricing page (32 of 38 do) and markdown where you have it (24 do). If a site has thousands of template pages, the kind programmatic SEO produces, link the pages that list them rather than every page.
llms.txt vs robots.txt vs sitemap.xml
Three files at the root, three different readers. The short answer to llms txt vs robots txt: robots.txt controls crawling, llms.txt does not. The spec places itself next to the other two: “It can complement robots.txt by providing context for allowed content.”
llms.txt vs robots.txt vs sitemap.xml: who reads each and what it says
| File | Who reads it | What it is for | What it controls | Format |
|---|---|---|---|---|
| robots.txt | Crawlers: Googlebot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot | “robots.txt lets automated tools know what access to a site is considered acceptable, such as for search indexing bots.” | The control each AI lab's crawler page names | Plain text: User-agent groups of Allow and Disallow lines |
| sitemap.xml | Search engines | “sitemap.xml is a list of all the indexable human-readable information available on a site.” | A list to crawl from, not a rule | XML, one entry per URL |
| llms.txt | Agents and AI tools, while they work on a task | “llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user.” | Nothing: it grants and blocks no access | Markdown: an H1, a blockquote, H2 lists of links |
| llms-full.txt | Tools that want a whole site in one read | Not in the spec. Mintlify: “The llms-full.txt file combines your entire documentation site into a single file as context for AI tools and LLM indexing.” | Nothing | Markdown, every page in full |
The quotes are from llmstxt.org's section "Existing standards" and Mintlify's llms.txt docs page, read on 17 November 2026. The crawlers named are the ones OpenAI, Anthropic and Perplexity document on their crawler pages and Google's Googlebot; each of the three labs' pages names robots.txt as the way to manage its crawlers.
The two can carry the same lines without trading jobs. HUNT0's llms.txt, read on 17 Nov 2026, has three sections, “Policy”, “Crawl Rules” and “Public Entry Points”, and the crawl rules section restates the groups of its robots.txt, so an agent reading the llms.txt sees the rules a crawler reads at /robots.txt. The rules bind there, in robots.txt; the llms.txt tells the agent about them.
Where directories come in: llms.txt files that point agents at listings
A directory's llms.txt is a map of its catalog, and some maps name the entries. Of the 92 directories and launch platforms we submit to that served an llms.txt on 17 Nov 2026, 20 linked individual product or listing pages in it, five or more each; counting llms-full.txt as well, 30 did. Some of those files link the newest entries, others far longer lists, so a listing on one of them can be a page the directory's own file hands to an agent. The longest of these files carries 9,145 links. Among the directories whose llms.txt links its listings are Infrabase.ai, DealMyApp and Launchboard, each with a guide on what it takes to get listed.
Directories we submit to with an llms.txt, by kind, 17 Nov 2026
| Kind | Answered | Serve llms.txt | Every part | Fuller llms-full.txt | llms.txt links listing pages | Either file does |
|---|---|---|---|---|---|---|
| Launch platforms and startup directories81 hosts | 77 | 37 of 77 (48%) | 27 | 11 | 6 | 9 |
| SaaS and software directories62 hosts | 56 | 31 of 56 (55%) | 26 | 11 | 8 | 12 |
| AI tool directories59 hosts | 59 | 24 of 59 (41%) | 19 | 9 | 6 | 9 |
| All202 hosts | 192 | 92 of 192 (48%) | 72 | 31 | 20 | 30 |
Every host of the catalog export of 17 November 2026 that a category hub lists (202), intercepts left out, read on 17 November 2026 as in the first census table; 10 hosts sent a bot check, a server error, a paused deployment, a TLS failure or nothing on both passes and are left out of Answered. Links listing pages: the file links at least five distinct pages of the directory's own site under one path, each page showing one product, tool or listing, found by grouping the file's same-site links on their first path segment and checked by hand; the second column counts that in llms.txt or llms-full.txt. No customer listing is named here.
All 202, with their Domain Ratings, are in the catalog table; the SaaS and AI ones also have hubs of their own, SaaS directories and AI tool directories.
Submitator puts your product on the directories you choose from that catalog. You make the picks yourself, from a dashboard that prints the Domain Rating beside every name. A site enters the catalog once a submission of ours has run its whole form through to the end. Once you pay, filling in the forms is our side of the job: every site receives its own description, sized to what its form allows, and we handle the CAPTCHAs along the way. Badge directories cost you no commits: one component on your site renders all the badges they ask for, and we keep them live. Every listing page we file is pushed through IndexMachine, the free indexing ($25 value) that comes with Starter, Growth and Scale, and there is nothing to set up on your side. Google decides which pages it indexes, as our indexing guide explains.
Each order is paid once. For 40 directories it is $29 (Starter), for 70 it is $49 (Growth) and for 110 it is $79 (Scale); All Directories costs $149 and covers the full catalog, including directories added after you buy. Of your directories, most are delivered within 24 to 48 hours and all within 21 days at most. Should a directory reject the submission, you get to pick a different one by default; All Directories has no swap, as your selection there is the whole catalog. A refund can be requested within 30 days of purchase, on the conditions in our terms. We have submitted 580+ startups to date; our submission statistics show what became of the orders.
We run no ads and no outreach. HUNT0, a launch platform on our list, sells its Premium Launch at $16.90, one-time, marked down from $29.90: a Featured listing in a launch week you reserve. We file customers there on its free and badge routes, as one of the 40 picks in a $29 order.
Every source was read on 17 November 2026, by GET request, and every quote is character-exact. The spec: llmstxt.org's "The /llms.txt file, v2" (dated September 3, 2024 and August 10, 2026 on the page) with its Changes page. Google: the Search Central guide "Optimizing your website for generative AI features on Google Search", updated 10 July 2026 and read in full, and the Lighthouse llms.txt audit page on Chrome for Developers, updated 5 May 2026. The labs: OpenAI's crawler overview, Perplexity's crawler page and the Claude Help Center article on Anthropic's crawlers. Tools and platforms, each from its own page: Mintlify's llms.txt docs, Cloudflare's AI consumability page in its style guide (updated 20 August 2026), GitBook's llms.txt blog post, Yoast's llms.txt feature page, Wix's help article, Webflow University, the Firecrawl, Writesonic and LLMrefs generators, the WordPress plugin directory and the Next.js docs on the public folder and route handlers; descriptions and terms are the makers' own, and naming a tool endorses none. Our own llms.txt and llms-full.txt, read live, matched byte for byte the files in this site's repository that day. Census: 252 hosts between 01:36 and 01:42 UTC under the rules in the table notes, directories from the catalog export of 17 November 2026. HUNT0's pricing page read "Premium Launch", "$16.90" next to a struck-through "$29.90", "One-time payment" and "Reserve your launch week"; its llms.txt and robots.txt were read the same morning, and its guide records that we file there on the free and badge routes. Statements about our own service have three sources. The founder: per-directory descriptions within each site's length rules, the forms and their CAPTCHAs handled on our side, one badge component for all badges, kept live by us, the IndexMachine push for every listing, 24 to 48 hours for most directories and the rule that a site enters the catalog after one complete submission. The pricing cards and the home page: plan prices, the free indexing and the 580+ count. The terms: delivery within 21 days at most, the swap when a directory rejects a submission (not on All Directories) and the 30-day refund window. Nothing here claims an agent will read, cite or recommend a listing, and nothing promises a ranking, traffic or indexing.
FAQ
What is an llms.txt file?
A markdown file served at /llms.txt that tells AI agents what a site is and where its useful pages are: an H1 with the site's name, a one-line summary in a blockquote, then H2 sections of links, each link optionally followed by a short note. Jeremy Howard proposed it in September 2024 and revised it as version 2 in August 2026. It grants and blocks nothing; robots.txt still decides what crawlers may fetch.
What is the llms.txt format?
In this order: an optional byte-order mark; an H1 with the name of the project or site, the one part the spec requires; a blockquote summary; any paragraphs or lists without headings; then H2 sections, each a markdown list of [name](url) links with an optional colon and note after each link. By convention an H2 named Optional holds the links an agent can skip.
How do I write an llms.txt file?
Put your product's name in an H1 and one sentence on what it does, for whom and how it is priced in a blockquote. Add a few plain lines an agent should not get wrong, then H2 sections that link your docs, pricing and policy pages with a note on each, pointing to markdown versions where you have them. Save it as llms.txt at the site root and serve it as plain text. Yoast on WordPress, Wix and docs platforms such as Mintlify write one for you.
How do I check the llms.txt of a website?
Open the site's /llms.txt in a browser or fetch it with curl -sL. It should answer 200 with a text content type rather than an HTML page, and its first line should be a # and the site's name, followed by a > summary and H2 sections of links. On 17 Nov 2026, 11 of the directories we submit to answered /llms.txt with an HTML page, which is a soft 404 and not a file.
Do llms.txt files actually work?
Not for Google Search: Google says the files will neither harm nor help a site's visibility or rankings there, as Search ignores them. For agents that look for one, the proposal's v2 change log says coding agents use them reliably, and OpenAI's and Perplexity's crawler pages open by pointing agents to their own docs' llms.txt. None of the AI labs' crawler pages says the file changes what their crawlers fetch, rank or cite.
Is llms.txt still relevant?
Among software companies, more serve one than not: 38 of 49 well-known software and docs sites did on 17 Nov 2026, against 92 of 192 directories we submit to. The proposal was revised in August 2026, and Chrome's Lighthouse runs an llms.txt audit among its agentic browsing checks. Google Search still ignores the file.
Should I add llms.txt to my website?
Yes if agents use your docs, your API or your pricing: it is one small text file, and it points them at pages you chose rather than whatever they find first. Not if the goal is Google rankings, since Google Search ignores it. In both cases robots.txt rules and pages a crawler can read come first.
Keep reading
Their $16.90: one featured launch. Our $29: 40 directories.
HUNT0 charges $16.90, marked down from $29.90, for one Premium Launch in a week you reserve. For $29 Submitator files 40 directories of your choosing, or 110 for $79, paid once: a fresh description for every site, and the badges kept live.
