Skip to content
InsightLogEditorial Notes
Menu

Running llms.txt for LLM crawlers

llms.txt is an index that helps machines understand the core documents of a site quickly. Here is how we run it.

Daniel OkaforUpdated November 20, 20261 min read182 views
Share on X
97% positive

Key summary · TL;DR

llms.txt contains a one-line site summary, a list of key documents, and markdown links for each. Pairing it with llms-full.txt, which carries full text, increases the chance of citation.

What llms.txt does

llms.txt is a lightweight index that helps generative engines understand the shape of a site quickly. It is not a standard and it is not a ranking factor; it is a convention that makes a site easier to parse.

A useful file answers three questions in order:

  1. What is this site?
  2. Which documents matter most?
  3. Where is the machine-readable source for each document?

Operating rules

  1. Keep a title and a one-line summary for every entry. A bare list of URLs is not useful.
  2. Never include drafts, private pages or staging URLs.
  3. Link to the markdown source when one exists.
  4. Regenerate the file when content changes, not on a fixed schedule only.
  5. Keep the format stable and boring.

The last rule matters more than it sounds. A generated file that changes shape every week is hard to trust and hard to test.

Pair it with full text

llms.txt is an index; llms-full.txt is the corpus. Serving both lets a crawler choose between a cheap map and a full download. Keep the full-text file sorted by publication date so diffs are readable.

Measurement

Citation is hard to measure directly. A realistic approach combines:

  • crawler statistics from search consoles
  • referrer logs for AI domains
  • a manual sample of target questions every month
  • server logs for requests to llms.txt and raw markdown

Record the date of each sample. Answer engines change their sources without notice, and a single check is not a trend.

Common failure modes

  • A list of 500 URLs with no summaries.
  • Links that redirect or 404.
  • Private documents included by accident.
  • A file generated once and never refreshed.

Conclusion

Treat llms.txt as documentation for machines: short, accurate, stable and linked to the source. That is enough to be useful.

Was this article helpful?

Your rating tells other readers how reliable this article is.

97% positive

Daniel Okafor
Daniel Okafor

Senior data editor · InsightLog

Builds data pipelines and measurement systems, and writes about the context behind every number.

Markdown source: /en/posts/llms-txt-operations/raw

Cite this article: https://insightlog.site/en/posts/llms-txt-operations

Continue reading

Read next