What Is llms.txt And Does It Actually Work?
Every major shift in search creates the same pattern: a new signal emerges, the industry moves fast, and the tools multiply before the evidence does. llms.txt is the latest example.
Since late 2024, it's been pitched as the key to visibility in AI-generated answers. Agencies are selling it. Developers are implementing it. And the data, so far, tells a different story.
Where llms.txt Comes From
The idea was proposed in September 2024 by Jeremy Howard, co-founder of Fast.ai. The timing wasn't a coincidence.
By mid-2024, AI assistants had gone mainstream. Developers were building LLM-powered tools that needed to parse documentation. Companies were realizing their carefully crafted content was being consumed by AI models with no way to guide that consumption.
Howard identified a real problem: LLM context windows are too small to process most websites in their entirety. Navigation menus, scripts, ads, boilerplate - it all creates noise. His proposed fix was simple: a plain Markdown file at the root of your domain, telling AI crawlers which pages deserve their attention.
The idea is nice. And it attracted serious names early on: Anthropic, Cloudflare, Vercel, and LangChain all published their own llms.txt files. By mid-2025, nearly 800 sites had implemented the standard.
But publishing a file and reading one are two very different things.
Track your brand visibility in LLMs
Rank42 shows SEO managers where and how their brand appears across ChatGPT, Perplexity, and other AI tools.
What the Data Actually Shows
But adoption doesn't equal effectiveness. Here's what the research actually found. Multiple studies show llms.txt's lack of impact:
SERanking analyzed 300,000 domains and found zero correlation between having an llms.txt file and being cited by AI models.
Search Engine Land tracked 10 websites for 90 days before and after implementation. Eight sites saw no measurable change. Two showed a slight uptick — but deeper analysis traced the growth to content campaigns running in parallel, not the file itself. The telling detail: server logs confirmed that AI crawlers weren't even accessing the files.
Cyrus Shepard ran 54 experiments across 23 ranking factors for AI citations. llms.txt finished dead last — 23rd out of 23. His conclusion was blunt: "We're unable to find any credible evidence showing llms.txt files influence AI citations in any way."
The verdict is consistent across every study that has looked at it: llms.txt hasn't proven its value for AI visibility and the research pile keeps growing.
What Google Says About llms.txt
Google's position has been consistent and it's never been supportive.
John Mueller, Google's Search Advocate, has addressed the topic repeatedly. In June 2025: "No AI system currently uses llms.txt." When llms.txt files appeared on some Google properties, the Search team removed them within 24 hours.
His most recent statement, from June 2026, is even more direct: "It's purely speculative for now. The file has existed for years, yet none of the AI systems use it - what does it mean?"
He also pointed out something worth sitting with: most people are using an LLM to generate their llms.txt — so that another LLM doesn't have to parse their site. His response: if an LLM can generate the file, couldn't it just extract the same information directly from the HTML?
The answer, apparently, is yes.
So Why Is Everyone Talking About It?
Because the problem llms.txt is trying to solve is real.
Brands have genuinely lost visibility into how LLMs represent them. Showing up in AI-generated answers has become a competitive issue. SEO teams are looking for leverage on a channel they don't yet know how to measure.
In that context, llms.txt checks every box of a compelling story: a recognized problem, a simple solution, a reassuring analogy with robots.txt. That's exactly the kind of standard that spreads fast regardless of whether it actually works.
That's not bad faith. It's the normal mechanics of tech adoption: propose, experiment, measure, adjust. The issue is when the "experiment" phase gets skipped and everyone jumps straight to "sell."
The "What If It Becomes a Standard?" Argument
This is the most common pushback, and it deserves an honest answer.
It's true that many web standards started as informal proposals before becoming widely adopted. robots.txt followed that exact path. It's theoretically possible llms.txt does the same.
But the web is also full of proposed standards that never went anywhere. And even in the optimistic scenario where llms.txt eventually gets adopted at scale, an hour of work will be enough to implement it when the time comes. That's not the kind of competitive advantage you build today to cash in three years from now.
If your developer has better things to do - and they almost certainly do - this is a conversation worth cutting short.
What Actually Works for AI Visibility
AI visibility in generative responses runs on fundamentals that research is starting to confirm. None of them are called llms.txt.
Topical authority. LLMs cite sources they perceive as the most complete and reliable on a given subject. Not the most technically optimized. The most substantive. A piece that genuinely answers a question - with depth, nuance, and real data - has a far better shot at being cited than a well-tagged page sitting behind a clean llms.txt file.
Content structure. Direct answers at the top of each section, clear headings, well-built FAQ blocks, sourced and dated data points. These are the signals AI models extract naturally, with or without a governance file.
Being cited by authoritative sources. LLMs draw on what they saw during training and what crawlers are picking up today. Getting mentioned, linked, and referenced by sources that already carry authority remains one of the most reliable levers in the AEO playbook.
Measurement. This one is probably the most underrated. You can't optimize what you don't measure. Knowing whether your brand shows up in AI-generated answers, on which topics, in what context, and how you stack up against competitors - that's the foundation of any serious AEO strategy.
Track your brand visibility in LLMs
Rank42 shows SEO managers where and how their brand appears across ChatGPT, Perplexity, and other AI tools.
The Honest Verdict
llms.txt is a serious idea from serious people solving a real problem. But as of today, no data confirms it works, and the major players who would need to adopt it have confirmed they don't.
That's not a reason to dismiss it forever - the AEO landscape moves fast. But it is a reason not to make it a priority.
Spend that time where results are actually measurable. Understand how your brand is genuinely perceived by LLMs today. And if llms.txt ever becomes a real standard, you'll have an afternoon to implement it.
Until then, work on what moves the needle.
See other articles