I put an llms.txt file on bekee.com back in June, on the theory that AI search engines would eventually want one. Last week I got around to checking whether anything was reading it, and the answer was more useful to me than a clean result would have been.
What llms.txt is
It is a plain text file at the root of a site, at /llms.txt, in the same spot and the same spirit as robots.txt. You write the summary of your own site: a title, a one-line description of the business, then a list of your real pages with one honest sentence about each. It has been around since September 2024, and I will come back to what its author meant it for, because that is the part I had wrong.
Mine looks like this, trimmed:
# bekee gibson
> WordPress consultant in Madison, WI
## Pages
- [Services](https://bekee.com/services/): WordPress help for small
businesses and nonprofits in Madison: site triage, accessibility
audits, builds, migrations, and support that gets out of your way.
- [Portfolio](https://bekee.com/portfolio/): Selected WordPress projects
for small businesses and nonprofits in Madison: restaurants,
foundations, newsrooms, and food and sustainability groups.That is the whole format. There is no schema to validate, no plugin required, and no way to get it syntactically wrong in a way that breaks anything.
The test is in your access logs
Google has been consistent about this for over a year now. John Mueller of Google Search Relations wrote on Bluesky in June 2025, “FWIW no AI system currently uses llms.txt,” and he was still saying it a year later, calling the file “purely speculative for now” and noting that “the file has existed for years, yet none of the AI systems use it.”
The useful part of what he said was the test, not the verdict. A crawler that reads your llms.txt has to request it, and that request lands in your access log with a name attached. You do not have to take anyone’s word for it.
So I pulled bekee.com’s logs for the 41 days from June 30 to August 9. There were 315,426 requests. Of those, 86,092 came from AI crawlers, and I can name them: ClaudeBot 46,169, GPTBot 18,550, Meta’s crawler 17,605, then ChatGPT-User, OAI-SearchBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, and a few smaller ones.
Requests for /llms.txt from any of them: zero. Not one, out of eighty-six thousand fetches, while those same crawlers were reading my actual pages thousands of times a day.
Eleven requests for the file did turn up, and the list is worth reading. Five were ordinary browsers. Two were SemRush’s site audit bot. One was a data aggregator called Dataprovider, one was something called PipericBot, and one was my own Claude Code session on August 1, while I was doing this research. The only thing resembling an AI that has ever asked for my llms.txt was me, using an AI to go look at it.
The obvious objection is that bekee.com sits behind Cloudflare, so maybe the crawlers were blocked, or their requests were absorbed at the edge before reaching my log. Three facts rule that out. All 86,092 of those AI crawler requests are in the origin log, so nothing is blocking them. My robots.txt disallows exactly two paths, /wp-admin/ and one JSON file, so every crawler is welcome to fetch /llms.txt. And /llms.txt returns cf-cache-status: DYNAMIC, meaning Cloudflare passes each request through to my server rather than answering from its own cache, so a request for it cannot disappear before it reaches the log. The crawlers were allowed, invited, and counted. They still did not ask.
It also matters which crawlers those were, because Google is not the only company here and Mueller only speaks for Google. The agents that serve AI chat when it is being used as a search engine are OpenAI’s OAI-SearchBot and ChatGPT-User, Anthropic’s Claude-SearchBot, and Perplexity’s two. Together they hit bekee.com 3,056 times in those 41 days. Their requests for /llms.txt: zero. This is not a Google finding.
What the engines read instead
The logs say nothing reads the file. A separate test told me what they read in its place.
On August 1 I ran 31 queries across ChatGPT, Perplexity, Google’s AI Overviews, and Google’s AI Mode, the questions a Madison business owner would type, like “who can help me with my WordPress site in Madison” and “WCAG audit Wisconsin.” Across 29 discovery queries and four engines, which is 115 answer slots, bekee.com was named in 7. ChatGPT accounted for 5. Perplexity and AI Mode named me zero times.
The citations that did land clustered hard:
| Page cited | Times |
|---|---|
| the home page | 7 |
/services/ | 6 |
/case-studies/ | 2 |
/portfolio/ | 1 |
/about/ | 1 |
Two of those results are the reason I am writing this.
Every blog post I have published was cited zero times. Not once, by any engine, including three posts about ADA Title II that all sit in my llms.txt with their own description.
And my case studies were cited twice while being completely absent from my llms.txt. The generator I wrote walked pages and blog posts only, and the case studies are a different post type, so the file had never mentioned them. The engines found them anyway, by crawling, the ordinary way.
So if someone tells you llms.txt is how you show up in ChatGPT, they are ahead of the evidence. I would not sell it to you as a ranking factor, and I would be careful with anyone who does.
Why mine is still there
Three reasons, and the third is the real one.
It costs almost nothing. Mine is a small function in the theme that builds the file on request out of descriptions I had already written for search, so it updates itself when I publish and I never regenerate anything. Yoast will write one for you as a static file if you would rather not touch code, and the fact that Yoast ships the switch at all is worth a second look. Their own feature page says the file will “offer to guide LLMs like ChatGPT to your most important content,” and that it “can also reduce the chances of incorrect or misleading responses.” Offer. Can. Read the verbs. Yoast is not claiming anything reads it either.
If the engines do start reading it, the file is already correct and already current, which is a better position than scrambling later.
And building it made me inventory my own site, which is what found the problem. Sitting down to write one honest sentence about every page is a slower and more revealing exercise than it sounds. It is how I noticed the case studies were missing from the file, and then that my three Title II posts were listed and getting no traction, and then that the page I sell accessibility audits from was cited zero times in the entire run while /services/ was cited six. That last one is a real problem with how my site is built, and no amount of llms.txt fixes it. I only saw it because I went looking for something else.
So what is the file for
I went back and read the original proposal, which I should have done first. Jeremy Howard published it on September 3, 2024, and it is not a search document. The stated purpose is “to provide information to help LLMs use a website at inference time,” and it says the file “will mainly be useful for inference, i.e. at the time a user is seeking assistance.” The problem it solves is that context windows are too small to swallow a whole website, so if you are going to hand a model your site, hand it a clean index instead of your HTML.
The distinction that clears everything up is who does the pointing. llms.txt works when a person or a tool deliberately aims a model at your site. It does nothing when a crawler wanders by, because the crawlers do not look for it.
Which is why the places using it well are documentation sites. I checked nine, and every developer-docs site I tried serves one at real size: vercel.com at 207 KB, Stripe at 93 KB, Anthropic at 57 KB, Cloudflare’s developer docs at 15 KB. When someone asks a coding assistant how Stripe’s API works, that assistant can pull a clean index of the docs instead of scraping a rendered page. That is a real problem, and llms.txt solves it. Perplexity publishes one for its own documentation, incidentally, while PerplexityBot has never once fetched mine.
So the file was built to feed documentation to tools, and the SEO industry repurposed it as a visibility tactic. If you run a restaurant or a nonprofit or a clinic, you do not have API docs, and nobody is aiming a coding agent at your site. The job llms.txt does well is a job your site does not have.
What does move AI visibility
The part worth your attention is that those crawlers read my pages 86,092 times in 41 days. They are paying close attention. They are just reading the site, not a summary of it.
So the work is in the pages they already fetch. On my own site the citations went to the home page, /services/, /case-studies/, /portfolio/, and /about/, and nowhere else. Not one blog post. And the page I sell audits from was cited zero times while /services/ was cited six, which tells me the engines are reading the overview page and never reaching the page with the offer on it.
That is a structural problem, and it is the kind of thing I can fix. It comes down to whether the pages being crawled state plainly who you are, where you work, and what you sell, in extractable sentences near the top rather than buried in good prose; whether your schema names one unambiguous business instead of a person and a company sharing an identity; and whether the page that answers a question is a page a reader can reach. I rebuilt my own schema on August 1 for exactly this reason, and /about/ and /services/ both started getting cited after.
If you want to know what AI search currently says about your business, that is the kind of work I take on. It starts with measuring rather than guessing, because the whole point of this post is that I would have been wrong about my own site if I had not looked.
If you want to check this yourself first
Write the file by hand first, even if you plan to automate it later. One line per page, and describe what the page is for rather than what it is called. If you cannot write an honest sentence about a page, that is the finding, not a writing problem.
Then check your own logs rather than trusting mine. On cPanel hosting the raw access log is usually a click away under Metrics, and you are looking for two facts: whether any AI crawler has ever requested /llms.txt, and how often those same crawlers fetch your real pages. If your answer differs from mine, that is news and I would like to hear it.
And go look at what AI search says about you today, before you change anything. Ask ChatGPT and Perplexity the question your customers would ask, the one that does not include your name, and write down which pages get cited and which do not. The gap between the pages you think are doing the work and the pages that get cited is where the real work is, and you will not see it from inside your own site map.