Skip to content

Does llms.txt Actually Work? Most Sites Shipped Half the Spec

Your llms.txt is doing about half the job you think it is, and the missing half is not in the file. I checked ours before I wrote this. Same problem.

The direct answer: llms.txt works, but not for the thing most people bought it for. It does nothing for your Google ranking, because Google Search Central states that you do not need to create new machine readable files, AI text files or markup to appear in AI Overviews or AI Mode. Where it does work is with agents that fetch it on demand while answering somebody, and there it only works properly if you shipped the second half of the specification, which almost nobody did.

I have published one of these files for imisofts, it is part of why an AI sent me a client, and I have built answer engine optimization for clients on top of it. That is exactly why the gap in mine annoyed me enough to write this.

Everybody printed the menu. Almost nobody opened the kitchen the menu points at.

The mistake: filing it under SEO

The file gets sold as a ranking asset. Generators, validators, checkers, a WordPress plugin, an audit line item. Then somebody publishes one, watches Search Console for a month, sees nothing, and concludes the whole idea was hype.

Both halves of that are wrong. Google Search Central says plainly that no new machine readable file is required for its AI features, so there was never a ranking to win. And the author of the specification is just as plain in the other direction: the expectation was that llms.txt would be useful for inference rather than training, and that is how it has been used. Inference means an agent reading it live, mid answer, on behalf of somebody who asked a question about you. Different instrument, different scoreboard.

How it gets measured

"We published llms.txt. Did rankings move?"

Google says no file is needed for AI Overviews or AI Mode. There was no ranking on offer.

How it should be measured

An agent fetches it while answering a buyer, follows a link, and needs clean text at the other end.

"Does a link in it lead anywhere an agent can read?"

The spec is two proposals. You shipped one.

Read llmstxt.org properly, in the version 2 revision published on August 10, 2026. It proposes the llms.txt file. It then proposes something separate: that pages an agent might need also publish a clean markdown version at the same URL, either with .md appended or with the extension replaced. The document calls these two proposals, and it says the links inside your llms.txt should point at that LLM friendly content.

That is the part everyone skipped. Open your own file and look at where the links go. If they resolve to ordinary HTML, an agent that follows one lands in navigation, a cookie banner, a footer and a pile of JavaScript, and then spends context stripping all of it back out to find your pricing. Which is the exact cost the file exists to avoid. The menu is clean. The kitchen is still locked.

I ran the count on ours before publishing this. Fifty seven links. Zero pointing at a markdown version of anything. We built the useful half and shipped the half that looks finished, and we are fixing it this week.

2
proposals in the llms.txt specification, the file and the markdown pages it should point at
57
links in our own llms.txt when I checked, none of which resolve to a markdown version
0
new machine readable files Google Search Central says you need to appear in its AI features

Lighthouse cannot fail you for not having one

Chrome shipped an llms.txt audit inside Lighthouse, under agentic browsing. Half the industry read that as Google now checking for the file, and started selling the check.

Read what the audit actually does. Chrome documents it flagging a page when a server error occurs while retrieving llms.txt. If the file is simply not there and the server returns a 404, the audit is marked Not Applicable, because providing the file is optional. So the audit cannot fail you for not having one. It can only fail you if your server breaks while trying to serve it.

Which leaves two Google teams saying compatible things that get quoted as a contradiction. Search says the file is not an input. Chrome says the file is optional and only checks that it is not erroring. Neither one is asking you for it. The people asking for it are agents, and they are not scored by either.

Version 2 also quietly moved two things nobody has implemented yet. The file no longer has to sit at your domain root: it can live at any path, it covers everything under that path, and where more than one applies an agent should use the most specific. And it defines two link relations, one pointing at the markdown version of a page and one pointing at the llms.txt that describes it, both deliverable as an HTTP Link response header from your CDN without editing a single page.

Agent reads llms.txt
Follows a link
Gets clean markdown
Names you in the answer

Four moves on your llms.txt this week

01
Ship the markdown half
Publish a plain markdown version of every page your llms.txt links to, at the same URL with .md appended or the extension replaced, then repoint the links at those. On a static site this is a build step, not a rewrite. It is the single change that makes the file do what it was designed to do, and it is ordinary web development work.
02
Add the two link relations at the CDN
Version 2 lets you advertise the markdown alternate and the describing llms.txt as an HTTP Link header, set in your CDN or web server config, with no page edits at all. If you are on Cloudflare or similar this is one rules change covering the whole site. Cheapest item on this list by a distance.
03
Split the file by path
One root file trying to describe a services site and a four hundred post blog serves neither well. Put a narrow file at the path it covers, keep the root one for what you sell, and let agents pick the most specific. Same instinct as scoping a workflow automation to one job rather than one giant flow.
04
Move it off the SEO scorecard
Stop looking for it in Search Console, because Google has told you it is not an input. Measure it where it is read: assistant referrals in analytics, and a how did you hear field on your booking form. That is how I caught our first AI referred client, and it is what AI search optimization is actually accountable to.

The bottom line

Does llms.txt work? For agents, yes, and only as well as what sits behind its links. For Google rankings, no, and the people who told you otherwise were selling a generator.

The tell is who publishes one properly. OpenAI, Anthropic and Google all serve an llms.txt for their own developer documentation, and documentation platforms generate the markdown pages alongside it as a matter of course. The AI labs did not ship the file and stop. They shipped the thing the file points at.

None of this is expensive. A markdown build step, a CDN header, and a file split by path. What it costs is admitting that the version you already published, and quietly counted as done, was the easy half. Mine was.

Book a 30 minute call: cal.com/zeeshanwaheed/30min or email [email protected]. I answer the first one myself, and you can see who you are talking to before you book.

Frequently Asked Questions

No. Google Search Central's documentation on AI features states that there are no additional requirements to appear in AI Overviews or AI Mode, and that you do not need to create new machine readable files, AI text files or markup. Standard SEO fundamentals are the stated path. llms.txt is aimed at a different consumer: an agent fetching it on demand while answering a question, which is what its own specification says it was expected to be used for.
Clean markdown versions of your pages. llmstxt.org proposes the llms.txt file and, separately, that pages an agent might need publish a markdown version at the same URL, with .md appended or the extension replaced. It describes these as two proposals and says the links inside llms.txt should point at that LLM friendly content. A file whose links all resolve to ordinary HTML hands the agent back the parsing problem the file was created to remove.
No. Chrome's documentation for the llms.txt audit says it flags a page when a server error occurs while retrieving the file, and that a 404 marks the audit Not Applicable because providing the file is optional at the moment. The audit therefore cannot fail for a missing file. It can only fail if your server errors while serving one, which makes a broken llms.txt worse than none at all.
Not in version 2 of the specification, published August 10, 2026. A file can sit at any path and covers the URLs beneath it, so a documentation set or a blog can carry its own, and where more than one applies agents are told to use the most specific. The specification explains that it deliberately avoided the well known URI convention for this reason, since anyone who controls only a path on a shared host can still publish one.

Want your llms.txt read the way an agent reads it?

Send me your file. I will tell you where its links actually land, which of them an agent can parse cleanly, and what the markdown half would take to ship on your stack.

Book a 30-Minute Call