Running llms.txt for LLM crawlers
llms.txt is an index that helps machines understand the core documents of a site quickly. Here is how we run it.
Key summary · TL;DR
llms.txt contains a one-line site summary, a list of key documents, and markdown links for each. Pairing it with llms-full.txt, which carries full text, increases the chance of citation.
What llms.txt does
llms.txt is a lightweight index that helps generative engines understand the shape of a site quickly. It is not a standard and it is not a ranking factor; it is a convention that makes a site easier to parse.
A useful file answers three questions in order:
- What is this site?
- Which documents matter most?
- Where is the machine-readable source for each document?
Operating rules
- Keep a title and a one-line summary for every entry. A bare list of URLs is not useful.
- Never include drafts, private pages or staging URLs.
- Link to the markdown source when one exists.
- Regenerate the file when content changes, not on a fixed schedule only.
- Keep the format stable and boring.
The last rule matters more than it sounds. A generated file that changes shape every week is hard to trust and hard to test.
Pair it with full text
llms.txt is an index; llms-full.txt is the corpus. Serving both lets a crawler choose between a cheap map and a full download. Keep the full-text file sorted by publication date so diffs are readable.
Measurement
Citation is hard to measure directly. A realistic approach combines:
- crawler statistics from search consoles
- referrer logs for AI domains
- a manual sample of target questions every month
- server logs for requests to llms.txt and raw markdown
Record the date of each sample. Answer engines change their sources without notice, and a single check is not a trend.
Common failure modes
- A list of 500 URLs with no summaries.
- Links that redirect or 404.
- Private documents included by accident.
- A file generated once and never refreshed.
Conclusion
Treat llms.txt as documentation for machines: short, accurate, stable and linked to the source. That is enough to be useful.
Was this article helpful?
Your rating tells other readers how reliable this article is.
Senior data editor · InsightLog
Builds data pipelines and measurement systems, and writes about the context behind every number.
Markdown source: /en/posts/llms-txt-operations/raw
Cite this article: https://insightlog.site/en/posts/llms-txt-operations
Continue reading
Read next
Designing content for the GEO and AEO era: from search engines to answer engines
Owning search results pages is no longer enough. Here is how to structure content so generative engines can cite it.
Stablecoin settlement is fast; refunds are where systems break
Finality is easy to measure and hard to operationalize. Refunds, reconciliation and compliance decide whether stablecoin payments survive production.