SEO

llms.txt: What It Is, the Format and Who Actually Serves One

/17 min read

llms.txt is a plain markdown file a website serves at /llms.txt to hand AI agents a short summary of the site and a list of links to the pages worth reading. Jeremy Howard put the proposal on llmstxt.org on 3 September 2024; its version 2 is dated 10 August 2026. When we requested the file on 17 November 2026, 38 of 49 well-known software, developer tool and docs sites served one, and so did 92 of the 192 directories and launch platforms we submit to that answered.

Searched as llm txt or llms-txt, it is the same file. This guide covers what an llms.txt file is for, the llms.txt format as the spec lays it out, this site's own two files as a worked example, the census of who serves one, what Google and the AI labs say about reading it, how to create and check one and how it sits next to robots.txt and sitemap.xml. For the crawler side (which AI bots a site lets in), see our LLM SEO guide.

Short version

llms.txt is a map for agents, not a gate for crawlers: robots.txt still decides who may crawl, and Google says its Search ignores llms.txt. Writing one takes an H1, a one-line summary and lists of links. On 17 Nov 2026, 38 of 49 well-known software sites served one, as did 92 of 192 directories we submit to, and 20 of those directory files linked individual listing pages, so a listing there can be one of the pages the directory's own file points agents to. HUNT0 sells one featured launch for $16.90; for $29 Submitator files 40 directories you pick out of the 202 we submit to.

What is llms.txt? What an llms.txt file is for

The proposal at llmstxt.org opens with its own definition: “A proposal to standardise on using an /llms.txt file to provide information to help agents use a website.” And then, plainly: “We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content.” The file is read when it is needed: “Agents are expected to view or search llms.txt to find the information they need, then follow the relevant links.” The detail stays behind the links, so an llms.txt works as a table of contents for a site, not a copy of it.

That makes it a different kind of file from robots.txt, which a crawler reads before it fetches anything. In the spec's words, “llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user.” On what the file feeds: “Our expectation was that llms.txt would mainly be useful for inference rather than training”, it says, and it adds that this is how the file has been used. Where it is used most: “llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials.”

Strictly, there is no llms.txt standard yet. The page calls itself a proposal to standardise on the file and closes with “The llms.txt specification is open for community input.” Version 2 records that “thousands of sites publish an llms.txt file”, that documentation platforms generate one and that Chrome's Lighthouse audits sites for one. It also settles where a file may live: “The file can be placed at the site root, or at any path within it, covering the pages under that path.” Its Changes page adds that “a file covers the pages under its path, and the most specific file applies.”

The llms.txt format, part by part

The spec defines a file by its sections, in a fixed order, so that a model can read it as markdown and a script can parse it with a regex. Six parts, of which the H1 is the one that must be there:

llms.txt format: the parts of the file, in order

llms.txt format: the parts of the file, in order
PartRequiredThe spec's words
Byte-order markNo“An optional byte-order mark (BOM)”
H1: the nameYes“An H1 with the name of the project or site”, the one part the spec requires
Blockquote: the summaryNo“A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file”
DetailsNoParagraphs or lists “of any type except headings”
H2 file listsNoSections “delimited by H2 headers”, each a markdown list “containing a required markdown hyperlink [name](url), then optionally a : and notes about the file.”
An H2 named OptionalNo“The “Optional” section is used, by convention, for secondary information: links an agent can skip when a shorter context is needed.”

From the Format section of "The /llms.txt file, v2" on llmstxt.org, by Jeremy Howard (the page's dates: September 3, 2024 and August 10, 2026), and its Changes page, read on 17 November 2026 and quoted character-exact.

Two conventions sit around the file. Pages an agent may need should exist as markdown: the spec asks for “a clean markdown version of those pages at the same URL as the original page”, either with .md appended (page.html.md) or with the extension swapped for .md (page.md). And the Optional section has changed status: since version 2 it carries no mechanical meaning, and the Changes page says “Optional sections are still allowed, and remain a useful convention for secondary links, but they no longer carry mechanical semantics.”

An llms.txt template to write by hand

Here is the whole format in one file, for a product we made up. The H1 names it, the blockquote says what it is and how it is priced, the plain lines carry the facts an agent should not have to follow a link for, and each H2 is a list of links with a note after the colon. Copy it, swap in your own pages and keep the Optional section for what an agent can skip.

llms.txt template (a made-up product; example.com is a reserved domain)

# Example Analytics

> Example Analytics is privacy-first web analytics for small SaaS teams. A free plan, then paid plans billed monthly.

Example Analytics runs as one script tag on your site. Data stays in the EU. The API is REST and returns JSON.

## Docs

- [Quickstart](https://example.com/docs/quickstart.md): install the script and see the first visit
- [API reference](https://example.com/docs/api.md): every endpoint, with request and response samples

## Product

- [Pricing](https://example.com/pricing.md): the plans, their limits and what each includes
- [Security](https://example.com/security.md): where data is stored and who can read it

## Optional

- [Changelog](https://example.com/changelog.md): what shipped, newest first

llms.txt example: the file on this site

This site serves both files, and we read both live on 17 November 2026. That day its llms.txt ran to 20,947 bytes and 138 lines: 8 H2 sections and 95 links, 36 of them to .md pages. Its llms-full.txt was 361,961 bytes over 9,214 lines. The opening lines and two of the sections, as served:

llms.txt file example: the opening of submitator.com/llms.txt ([...] marks lines left out)

# Submitator

> Submitator submits your startup to the directories you pick, for a one-time payment from $29. You add your product once, choose from a catalogue of 170+ directories and follow every submission in a dashboard. Agencies resell it under their own brand.

Plans, each a one-time payment:

- Starter: $29 for 40 directories
- Growth: $49 for 70 directories
- Scale: $79 for 110 directories
[...]
## Pages

- [Home and pricing](https://submitator.com/): what Submitator does, the plans with checkout and the FAQ
[...]
## Get started

[...]
- [Quickstart](https://submitator.com/docs/quickstart.md): Get a free test key, launch a client's project in test mode and receive a signed webhook when one of its listings goes live.
  1. The H1. # Submitator is the name, and the one line every llms.txt needs.
  2. The blockquote. What the product does and where its price starts, so an agent that reads nothing else still has the answer.
  3. The details. The plan lines sit in plain markdown under no heading: facts an agent should find without following a link.
  4. A file list. ## Pages holds markdown links, each followed by a colon and a note on what the page holds.
  5. Markdown twins. The API docs sections link .md addresses, the markdown version of each docs page, as the spec's .md convention asks; llms-full.txt carries the same docs pages in full.

Nobody types those plan lines. A script writes both files from the same data as the pricing cards and the docs, and a test fails when the files and the data disagree. A file that states prices goes stale the day the prices move, so build it the way the pages are built. The opening of the excerpt is built from that data as well, so it matches the file; the sizes above are from the read, and the live file grows as pages are added.

Who actually serves an llms.txt: 252 sites read on 17 Nov 2026

Among well-known software sites the file is the norm: 38 of the 49 that answered served one (78%), and 33 of those 38 had every part the spec describes. Among the 202 directories and launch platforms we submit to it is close to a coin toss: 92 of 192 (48%), with 72 complete. The directories' files are also shorter, a median 5.2 KB and 28 links against 14.9 KB and 106.

Who serves an llms.txt: 50 software sites against the 202 directories we submit to, 17 Nov 2026

Who serves an llms.txt: 50 software sites against the 202 directories we submit to, 17 Nov 2026
Measure50 well-known software and docs sites202 directories we submit to
Answered49 of 50192 of 202
Serve an llms.txt38 of 49 (78%)92 of 192 (48%)
Every part the spec describes33 of 3872 of 92
A blockquote summary34 of 3884 of 92
Links to .md pages24 of 3811 of 92
Links a pricing page32 of 3842 of 92
Median llms.txt14.9 KB, 160 lines, 106 links5.2 KB, 69 lines, 28 links
A fuller llms-full.txt too13 of 4931 of 192
An HTML page at /llms.txt0 of 4911 of 192

GET /llms.txt and /llms-full.txt on each host, 17 November 2026, 01:36 to 01:42 UTC (the directories our catalog added that day, and PeerPush at its new address, at 16:14 UTC), a desktop browser user agent, redirects followed, then a curl retry of every miss. A site serves an llms.txt when the address answers 200 with a non-empty body that is not HTML and whose first non-blank line is an H1 ("# " and a name). Every part: one H1, a blockquote between it and the first H2, at least one H2 and a markdown list link under it, read outside code blocks. A fuller llms-full.txt answers 200 with text larger than the site's llms.txt. Answered leaves out hosts that sent a bot check, a server error, a paused deployment, a TLS failure or nothing on both passes: 1 software site (Screaming Frog) and 10 directories. Medians are over the served files, lines and links rounded; 1 KB is 1,000 bytes. Directories: every host of the catalog export of 17 November 2026 that a category hub lists, intercepts left out.

The AI labs keep their files on their developer docs, not on their home pages. OpenAI's developer docs, Anthropic's (which answer from platform.claude.com) and Perplexity's docs serve one; openai.com, anthropic.com and perplexity.ai answer 404. The spec itself links Gemini's file under ai.google.dev/gemini-api/docs/, a subpath the format allows, while the root of ai.google.dev answers 404.

SEO tools split: Semrush, Yoast and AIOSEO serve one, Ahrefs and Moz answer 404. Stripe serves one on stripe.com and another on docs.stripe.com; Paddle and Lemon Squeezy serve none. 7 of the 8 CMS and website builders do, Ghost being the exception. Sizes run across three orders of magnitude: the smallest file in the set is the spec's own, 637 bytes on llmstxt.org, and the largest is AIOSEO's at 787.2 KB with 3,468 links.

32 of the 38 files on well-known sites link a pricing page. If you have one, link it too; how SaaS companies structure that page is the subject of our post on SaaS pricing models. For more llms.txt examples than the 50 below, the spec lists three directories that index such files: llmstxt.site, directory.llmstxt.cloud and llmstxthub.com.

llms.txt examples: 50 software, developer tool and docs sites, read on 17 Nov 2026

llms.txt examples: 50 software, developer tool and docs sites, read on 17 Nov 2026
SiteKindllms.txtEvery part.md linksPricing linkllms-full.txt
OpenAIopenai.comAI labs and model APIsNone (404)None
OpenAI developer docsdevelopers.openai.comAI labs and model APIs6.3 KB, 54 linesYes20No9.6 MB
Anthropicanthropic.comAI labs and model APIsNone (404)None
Anthropic developer docsdocs.anthropic.comAI labs and model APIs102.1 KB, 994 linesNo916Yes31.5 MB or more
Perplexityperplexity.aiAI labs and model APIsNone (404)None
Perplexity docsdocs.perplexity.aiAI labs and model APIs45.8 KB, 217 linesYes200Yes4.0 MB
Google AI for Developersai.google.devAI labs and model APIsNone (404)None
Mistral AImistral.aiAI labs and model APIs5.5 KB, 59 linesYes0YesNone
Hugging Facehuggingface.coAI labs and model APIsNone (404)None
Stripestripe.comPayments and billing69.8 KB, 401 linesYes24YesNone
Stripe docsdocs.stripe.comPayments and billing99.2 KB, 749 linesNo522YesNone
Paddlepaddle.comPayments and billingNone (404)None
Lemon Squeezylemonsqueezy.comPayments and billingNone (404)None
Polarpolar.shPayments and billing16.7 KB, 238 linesYes122Yes792.3 KB
Chargebeechargebee.comPayments and billing21.5 KB, 160 linesYes0YesNone
Vercelvercel.comHosting and infrastructure14.9 KB, 151 linesYes104YesNone
Netlifynetlify.comHosting and infrastructure3.0 KB, 46 linesYes28YesNone
Cloudflarecloudflare.comHosting and infrastructure17.2 KB, 185 linesYes82Yes169.5 KB
Cloudflare developer docsdevelopers.cloudflare.comHosting and infrastructure16.6 KB, 143 linesYes0No31.5 MB or more
Supabasesupabase.comHosting and infrastructure3.5 KB, 59 linesNo32Yes6.8 MB
Renderrender.comHosting and infrastructure53.3 KB, 662 linesNo408Yes1.4 MB
Fly.iofly.ioHosting and infrastructure5.9 KB, 77 linesYes14YesNone
Neonneon.comHosting and infrastructure29.7 KB, 351 linesYes169Yes7.7 MB
WordPress.orgwordpress.orgCMS and website builders5.8 KB, 76 linesYes0NoNone
Webflowwebflow.comCMS and website builders13.3 KB, 143 linesYes0YesNone
Framerframer.comCMS and website builders9.8 KB, 95 linesYes0YesNone
Wixwix.comCMS and website builders38.7 KB, 263 linesYes1YesNone
Squarespacesquarespace.comCMS and website builders2.4 KB, 30 linesYes0YesNone
Shopifyshopify.comCMS and website builders12.6 KB, 112 linesYes2Yes19.0 KB
Ghostghost.orgCMS and website buildersNone (404)None
Sanitysanity.ioCMS and website builders29.9 KB, 179 linesYes0YesNone
Ahrefsahrefs.comSEO toolsNone (404)None
Semrushsemrush.comSEO tools15.0 KB, 96 linesYes0YesNone
Mozmoz.comSEO toolsNone (404)None
Yoastyoast.comSEO tools7.7 KB, 115 linesYes0NoNone
AIOSEOaioseo.comSEO tools787.2 KB, 3,504 linesYes0Yes18.7 MB
Screaming Frogscreamingfrog.co.ukSEO toolsBot checkBot check
Mintlifymintlify.comDocs platforms and frameworks6.7 KB, 79 linesYes36Yes141.6 KB
GitBookgitbook.comDocs platforms and frameworks15.1 KB, 160 linesYes45YesNone
ReadMereadme.comDocs platforms and frameworks61.6 KB, 214 linesYes0YesNone
Docusaurusdocusaurus.ioDocs platforms and frameworksNone (404)None
Next.jsnextjs.orgDocs platforms and frameworks14.6 KB, 136 linesYes1NoNone
llmstxt.orgllmstxt.orgDocs platforms and frameworks0.6 KB, 10 linesYes3NoNone
GitHubgithub.comDeveloper and SaaS tools28.7 KB, 178 linesYes0YesNot larger
Linearlinear.appDeveloper and SaaS tools11.7 KB, 248 linesYes177YesHTML page
Notionnotion.comDeveloper and SaaS tools6.8 KB, 74 linesYes0YesNone
Zapierzapier.comDeveloper and SaaS tools14.0 KB, 193 linesYes39Yes18.4 KB
PostHogposthog.comDeveloper and SaaS tools514.9 KB, 5,125 linesYes4,883YesNone
Sentrysentry.ioDeveloper and SaaS tools93.6 KB, 581 linesNo29YesNot larger
Resendresend.comDeveloper and SaaS tools10.4 KB, 185 linesYes33YesNot larger

Read on 17 November 2026 as in the table above, on the host shown. Served from another address after redirects: docs.anthropic.com from platform.claude.com/llms.txt, polar.sh from polar.sh/docs/llms.txt and squarespace.com from www.squarespace.com/assets/llms.txt. The spec allows a file below the root, and it links Gemini's at ai.google.dev/gemini-api/docs/llms.txt; this census asked each host for /llms.txt at its root. The Anthropic developer docs and Cloudflare developer docs llms-full.txt files were read up to 31,457,280 bytes and are at least that large. Not larger: /llms-full.txt answered with text no larger than the llms.txt (GitHub, Sentry and Resend). Bot check: a challenge page on both passes. Sizes are bytes as served, 1 KB = 1,000 bytes.

HUNT0: $16.90 for one featured launch. Submitator: $29 for 40 directories.

We file HUNT0 on its free and badge routes, one of 81 launch sites in our catalog. Add 39 more picks; we write a description for each site and keep every badge live. 70 for $49, 110 for $79, one payment.

Start: from $29

Who reads llms.txt, and does it work?

Google and llms.txt

Google Search does not use it, and says so. Its guide to generative AI features, last updated 10 July 2026, puts llms.txt files among the things you can ignore for Search: you do not need them to appear there, it says, “as Google Search itself doesn't use them.” Then: “It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files.” “Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them.” The same passage notes that Google may crawl such a file like any other, and “this doesn't mean that the file is treated in a special way.”

Google's Chrome team does check for one. Lighthouse runs an llms.txt audit among its agentic browsing audits and describes the file this way: “The llms.txt file is an emerging convention used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents.” The audit is lenient: “Lighthouse flags the pages if a server error occurs when attempting to retrieve the llms.txt file.” “If the file is not provided by the server (resulting in a 404), the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment.” The page describes no check of the format itself.

OpenAI, Anthropic and Perplexity

The three labs document their crawlers in terms of robots.txt. OpenAI's crawler page: “OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.” Perplexity's crawler page: “Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity.” The Claude Help Center article on Anthropic's crawler says the bots respect do-not-crawl signals by “honoring industry standard directives in robots.txt”. None of the three pages says its crawlers read llms.txt, and Anthropic's does not mention the file.

Their docs use it, though. Anthropic's llms.txt, on its developer docs, is 102.1 KB with 917 links. OpenAI's crawler page opens with “For the complete documentation index, see llms.txt”, then “Markdown versions of documentation pages are available by appending .md to the page URL.” Perplexity's opens with “Fetch the complete documentation index at: /llms.txt” and “Use this file to discover all available pages before exploring further.” The labs write llms.txt for agents that use their own docs; none of their pages promises to read yours.

So does llms.txt work? For rankings in Google, no, by Google's own account. For agents that go looking for the file, its author says it does: the v2 change log reports that “coding agents use them reliably”. None of the four companies' pages above says its crawler ranks or cites a site because of the file.

LLM txt for SEO: what the file changes in search

Most questions about llm.txt for SEO come down to Google, and Google's answer is above: the file neither harms nor helps. For llm txt SEO in the newer sense, being read and cited by AI answers, access is still decided where it always was, in robots.txt rules for each crawler and in pages a crawler can fetch and read. An llms.txt adds a map for the agents that get in. Which AI crawlers the directories we submit to let in, robots.txt by robots.txt, is in the LLM SEO census; the groundwork a product site needs first is in our guide to SEO for a SaaS.

How to create an llms.txt file

Three routes: write it by hand, run a generator over your site, or let your CMS or docs platform write it. Whichever you take, the result has to answer at /llms.txt as plain text, not as a page of HTML.

By hand, in seven steps

  1. Put the product's name in an H1 on the first line.
  2. Write one blockquote line: what it is, who it is for, how it is priced.
  3. Add a few plain lines with the facts agents get wrong: limits, regions, what it does not do.
  4. Group links under H2s (docs, product, pricing, policies), each with a short note after a colon.
  5. Point the links at markdown versions of the pages where you have them.
  6. Move the nice-to-have links into an H2 named Optional.
  7. Save it as llms.txt at the site root, served as text/plain or text/markdown.

llms.txt generators, free to start

An llms txt file generator crawls your site or reads your sitemap and drafts the file for you to edit. Three, each described from its own page on 17 Nov 2026. Firecrawl's llms.txt generator now opens with a deprecation notice: “This API endpoint will still remain active but we will no longer be maintaining it after June 30, 2025.” It points to an example repository instead. Writesonic's free llms.txt generator reads your sitemap, sets the terms as “No login. No email. 3 free files a day.” and says “Every file is checked against the llmstxt.org spec and scored, with anything that would fail listed out.” LLMrefs' generator is marked “Free, No card required”: “The tool will automatically crawl your website and create a standardized llms.txt file following the official specification.”

Whatever drafts it, the summary line and the notes are the parts worth writing yourself, since they are the lines that say what the product is. A list of every URL in a sitemap is not what the spec asks for: “The file itself stays small enough to fit in context.”

WordPress, Shopify, Wix and Webflow

On WordPress, Yoast SEO writes one: “Generate an llms.txt file automatically with a click.” “Available in one click, no upgrade needed.” Its page lists the feature in Yoast SEO for Shopify as well, which covers a Shopify store. The plugin directory has more WordPress llms.txt plugins; the largest it listed, Website LLMs.txt, shows “50,000+ active installations”. On Wix it is automatic: “Wix automatically generates and maintains your llms.txt file, but you can edit it at any time in your site's dashboard.” “Once you edit the file, it will stop updating automatically so your changes are preserved.” Webflow's own learning site describes the manual route: “In Webflow, you can create an llms.txt file by uploading it as an asset or by creating a page with the appropriate path.”

Mintlify, GitBook and Cloudflare's docs

Docs platforms generate the file on their own. Mintlify's llms.txt docs: “Mintlify automatically hosts an llms.txt file at the root of your project that lists all available pages in your documentation.” It caps what it generates: “Automatically generated llms.txt index files cannot exceed 100,000 characters.” A custom file has one rule: “Your custom llms.txt or llms-full.txt file must have a site title as an H1 heading.” GitBook: “GitBook produces llms.txt and llms-full.txt for published docs.” Cloudflare's developer docs show the pattern at scale. Their root llms.txt, in Cloudflare's words: “A directory of all Cloudflare documentation products, grouped by category.” Each product has its own llms.txt below it.

Next.js: a static file or a route handler

In Next.js the file can be static: “Next.js can serve static files, like images, under a folder called public in the root directory.” A public/llms.txt is served at /llms.txt. If it should be built from data, a route handler at app/llms.txt/route.ts does it: “You can use Route Handlers to return non-UI content.” This site serves both of its files from public/, written there by the script described above.

llms-full.txt: the whole site in one file

llms-full.txt is not in the spec; version 2 does not mention it. The ecosystem defines it. Mintlify: “The llms-full.txt file combines your entire documentation site into a single file as context for AI tools and LLM indexing.” Cloudflare's docs describe theirs as “The full contents of all Cloudflare documentation in a single file”. On 17 Nov 2026, 13 of the 49 well-known sites served a fuller llms-full.txt beside the llms.txt, and 31 of the 192 directories did. The big ones are big: OpenAI's developer docs file is 9.6 MB, and Anthropic's is at least 31.5 MB. Link it from your llms.txt so agents can find it; 11 of the 38 files on well-known sites do.

How to check an llms.txt: a validator checklist

The checks are short enough to run by hand: use the list below as your llms txt checker. Open the file, or fetch it, and go down it:

  1. It answers 200 at /llms.txt. After redirects, with a body. A 404 means there is no file.
  2. It is text, not a page. The content type is text/plain or text/markdown. On 17 Nov 2026, 11 of the directories we submit to answered /llms.txt with an HTML page, a soft 404 that a browser makes look fine.
  3. Line one is an H1. A single #, a space and the name of the site or project.
  4. A blockquote follows. One > line with the summary.
  5. Sections are H2s of links. Each item a markdown link, [name](url), with an optional colon and note.
  6. The links resolve, and point at pages an agent can read, markdown where you have it.
  7. It stays small. Mintlify splits a generated index past 100,000 characters, a fair ceiling for a hand-written one.
  8. Secondary links sit under Optional, so a short read can skip them.

The first two checks take one curl line each. The first prints the status and the content type; the second prints the opening lines, which should start with # and your product's name, then a > line:

llms txt check: two curl lines (swap in your own domain)

curl -sL -o /dev/null -w "%{http_code} %{content_type}\n" https://example.com/llms.txt
curl -sL https://example.com/llms.txt | head -n 5

Run the same against /llms-full.txt if you serve one. As a validator, Lighthouse's llms.txt audit flags a server error and marks a 404 not applicable; Writesonic's generator scores a file against the spec.

llms.txt best practices

The spec's own four guidelines, in its order:

  • “Use concise, clear language.”
  • “When linking to resources, include brief, informative descriptions.”
  • “Avoid ambiguous terms or unexplained jargon.”
  • “Test your file by asking an agent questions about your content”, with nothing but your llms.txt to start from.

The census adds a few habits of the files on well-known sites. Lead with the blockquote: 34 of those 38 files carry one. Link the pricing page (32 of 38 do) and markdown where you have it (24 do). If a site has thousands of template pages, the kind programmatic SEO produces, link the pages that list them rather than every page.

llms.txt vs robots.txt vs sitemap.xml

Three files at the root, three different readers. The short answer to llms txt vs robots txt: robots.txt controls crawling, llms.txt does not. The spec places itself next to the other two: “It can complement robots.txt by providing context for allowed content.”

llms.txt vs robots.txt vs sitemap.xml: who reads each and what it says

llms.txt vs robots.txt vs sitemap.xml: who reads each and what it says
FileWho reads itWhat it is forWhat it controlsFormat
robots.txtCrawlers: Googlebot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot“robots.txt lets automated tools know what access to a site is considered acceptable, such as for search indexing bots.”The control each AI lab's crawler page namesPlain text: User-agent groups of Allow and Disallow lines
sitemap.xmlSearch engines“sitemap.xml is a list of all the indexable human-readable information available on a site.”A list to crawl from, not a ruleXML, one entry per URL
llms.txtAgents and AI tools, while they work on a task“llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user.”Nothing: it grants and blocks no accessMarkdown: an H1, a blockquote, H2 lists of links
llms-full.txtTools that want a whole site in one readNot in the spec. Mintlify: “The llms-full.txt file combines your entire documentation site into a single file as context for AI tools and LLM indexing.”NothingMarkdown, every page in full

The quotes are from llmstxt.org's section "Existing standards" and Mintlify's llms.txt docs page, read on 17 November 2026. The crawlers named are the ones OpenAI, Anthropic and Perplexity document on their crawler pages and Google's Googlebot; each of the three labs' pages names robots.txt as the way to manage its crawlers.

The two can carry the same lines without trading jobs. HUNT0's llms.txt, read on 17 Nov 2026, has three sections, “Policy”, “Crawl Rules” and “Public Entry Points”, and the crawl rules section restates the groups of its robots.txt, so an agent reading the llms.txt sees the rules a crawler reads at /robots.txt. The rules bind there, in robots.txt; the llms.txt tells the agent about them.

Where directories come in: llms.txt files that point agents at listings

A directory's llms.txt is a map of its catalog, and some maps name the entries. Of the 92 directories and launch platforms we submit to that served an llms.txt on 17 Nov 2026, 20 linked individual product or listing pages in it, five or more each; counting llms-full.txt as well, 30 did. Some of those files link the newest entries, others far longer lists, so a listing on one of them can be a page the directory's own file hands to an agent. The longest of these files carries 9,145 links. Among the directories whose llms.txt links its listings are Infrabase.ai, DealMyApp and Launchboard, each with a guide on what it takes to get listed.

Directories we submit to with an llms.txt, by kind, 17 Nov 2026

Directories we submit to with an llms.txt, by kind, 17 Nov 2026
KindAnsweredServe llms.txtEvery partFuller llms-full.txtllms.txt links listing pagesEither file does
Launch platforms and startup directories81 hosts7737 of 77 (48%)271169
SaaS and software directories62 hosts5631 of 56 (55%)2611812
AI tool directories59 hosts5924 of 59 (41%)19969
All202 hosts19292 of 192 (48%)72312030

Every host of the catalog export of 17 November 2026 that a category hub lists (202), intercepts left out, read on 17 November 2026 as in the first census table; 10 hosts sent a bot check, a server error, a paused deployment, a TLS failure or nothing on both passes and are left out of Answered. Links listing pages: the file links at least five distinct pages of the directory's own site under one path, each page showing one product, tool or listing, found by grouping the file's same-site links on their first path segment and checked by hand; the second column counts that in llms.txt or llms-full.txt. No customer listing is named here.

All 202, with their Domain Ratings, are in the catalog table; the SaaS and AI ones also have hubs of their own, SaaS directories and AI tool directories.

Submitator puts your product on the directories you choose from that catalog. You make the picks yourself, from a dashboard that prints the Domain Rating beside every name. A site enters the catalog once a submission of ours has run its whole form through to the end. Once you pay, filling in the forms is our side of the job: every site receives its own description, sized to what its form allows, and we handle the CAPTCHAs along the way. Badge directories cost you no commits: one component on your site renders all the badges they ask for, and we keep them live. Every listing page we file is pushed through IndexMachine, the free indexing ($25 value) that comes with Starter, Growth and Scale, and there is nothing to set up on your side. Google decides which pages it indexes, as our indexing guide explains.

Each order is paid once. For 40 directories it is $29 (Starter), for 70 it is $49 (Growth) and for 110 it is $79 (Scale); All Directories costs $149 and covers the full catalog, including directories added after you buy. Of your directories, most are delivered within 24 to 48 hours and all within 21 days at most. Should a directory reject the submission, you get to pick a different one by default; All Directories has no swap, as your selection there is the whole catalog. A refund can be requested within 30 days of purchase, on the conditions in our terms. We have submitted 580+ startups to date; our submission statistics show what became of the orders.

We run no ads and no outreach. HUNT0, a launch platform on our list, sells its Premium Launch at $16.90, one-time, marked down from $29.90: a Featured listing in a launch week you reserve. We file customers there on its free and badge routes, as one of the 40 picks in a $29 order.

Every source was read on 17 November 2026, by GET request, and every quote is character-exact. The spec: llmstxt.org's "The /llms.txt file, v2" (dated September 3, 2024 and August 10, 2026 on the page) with its Changes page. Google: the Search Central guide "Optimizing your website for generative AI features on Google Search", updated 10 July 2026 and read in full, and the Lighthouse llms.txt audit page on Chrome for Developers, updated 5 May 2026. The labs: OpenAI's crawler overview, Perplexity's crawler page and the Claude Help Center article on Anthropic's crawlers. Tools and platforms, each from its own page: Mintlify's llms.txt docs, Cloudflare's AI consumability page in its style guide (updated 20 August 2026), GitBook's llms.txt blog post, Yoast's llms.txt feature page, Wix's help article, Webflow University, the Firecrawl, Writesonic and LLMrefs generators, the WordPress plugin directory and the Next.js docs on the public folder and route handlers; descriptions and terms are the makers' own, and naming a tool endorses none. Our own llms.txt and llms-full.txt, read live, matched byte for byte the files in this site's repository that day. Census: 252 hosts between 01:36 and 01:42 UTC under the rules in the table notes, directories from the catalog export of 17 November 2026. HUNT0's pricing page read "Premium Launch", "$16.90" next to a struck-through "$29.90", "One-time payment" and "Reserve your launch week"; its llms.txt and robots.txt were read the same morning, and its guide records that we file there on the free and badge routes. Statements about our own service have three sources. The founder: per-directory descriptions within each site's length rules, the forms and their CAPTCHAs handled on our side, one badge component for all badges, kept live by us, the IndexMachine push for every listing, 24 to 48 hours for most directories and the rule that a site enters the catalog after one complete submission. The pricing cards and the home page: plan prices, the free indexing and the 580+ count. The terms: delivery within 21 days at most, the swap when a directory rejects a submission (not on All Directories) and the 30-day refund window. Nothing here claims an agent will read, cite or recommend a listing, and nothing promises a ranking, traffic or indexing.

FAQ

What is an llms.txt file?

A markdown file served at /llms.txt that tells AI agents what a site is and where its useful pages are: an H1 with the site's name, a one-line summary in a blockquote, then H2 sections of links, each link optionally followed by a short note. Jeremy Howard proposed it in September 2024 and revised it as version 2 in August 2026. It grants and blocks nothing; robots.txt still decides what crawlers may fetch.

What is the llms.txt format?

In this order: an optional byte-order mark; an H1 with the name of the project or site, the one part the spec requires; a blockquote summary; any paragraphs or lists without headings; then H2 sections, each a markdown list of [name](url) links with an optional colon and note after each link. By convention an H2 named Optional holds the links an agent can skip.

How do I write an llms.txt file?

Put your product's name in an H1 and one sentence on what it does, for whom and how it is priced in a blockquote. Add a few plain lines an agent should not get wrong, then H2 sections that link your docs, pricing and policy pages with a note on each, pointing to markdown versions where you have them. Save it as llms.txt at the site root and serve it as plain text. Yoast on WordPress, Wix and docs platforms such as Mintlify write one for you.

How do I check the llms.txt of a website?

Open the site's /llms.txt in a browser or fetch it with curl -sL. It should answer 200 with a text content type rather than an HTML page, and its first line should be a # and the site's name, followed by a > summary and H2 sections of links. On 17 Nov 2026, 11 of the directories we submit to answered /llms.txt with an HTML page, which is a soft 404 and not a file.

Do llms.txt files actually work?

Not for Google Search: Google says the files will neither harm nor help a site's visibility or rankings there, as Search ignores them. For agents that look for one, the proposal's v2 change log says coding agents use them reliably, and OpenAI's and Perplexity's crawler pages open by pointing agents to their own docs' llms.txt. None of the AI labs' crawler pages says the file changes what their crawlers fetch, rank or cite.

Is llms.txt still relevant?

Among software companies, more serve one than not: 38 of 49 well-known software and docs sites did on 17 Nov 2026, against 92 of 192 directories we submit to. The proposal was revised in August 2026, and Chrome's Lighthouse runs an llms.txt audit among its agentic browsing checks. Google Search still ignores the file.

Should I add llms.txt to my website?

Yes if agents use your docs, your API or your pricing: it is one small text file, and it points them at pages you chose rather than whatever they find first. Not if the goal is Google rankings, since Google Search ignores it. In both cases robots.txt rules and pages a crawler can read come first.

Keep reading

Their $16.90: one featured launch. Our $29: 40 directories.

HUNT0 charges $16.90, marked down from $29.90, for one Premium Launch in a week you reserve. For $29 Submitator files 40 directories of your choosing, or 110 for $79, paid once: a fresh description for every site, and the badges kept live.