llm.json and llms.txt: Why v8.1 Brought Them Back

Every site in the federation now serves four machine-readable summaries of itself.

/llm.txt is the plain-text summary. It never left.

/llm.json and its mirror /ai/llm.json are the structured version. v8.0 retired them. v8.1 put them back.

/llms.txt is the plural, markdown convention from llmstxt.org. It was added to the v8.1 build on 2026-09-22.

None of them is written by hand.

One builder derives all of them from the site's own /ai/manifest.json, /ai/catalog.json and /ai/health.json, right after those files are regenerated. They cannot disagree with the manifest, because they are made from it.

The v8.0 reasoning was clean.

The manifest already carries the site's identity, endpoints and relationships. A second document carrying the same fields is a second thing that can drift. So retire it.

That is technically true.

It also assumed crawlers read the spec.

They do not. They request the files they have learned to request, and /llm.json is one of them. On 2026-09-09 we found portfolio sites serving an llm.json that had not been regenerated in 51 days, while AI crawlers were still fetching it. Retiring a file does not stop anyone from asking for it. It just means the answer goes stale.

These are GET requests recorded in the Digital Karma Data Warehouse server logs across the portfolio, 2026-08-18 through 2026-09-22 (the 35 days of raw logs the warehouse retains).

That was enough.

When the thing you retired gets requested more than the thing you told everyone to use instead, the retirement was the mistake.

A request proves that a crawler asked for a URL and what the server answered.

It does not prove the crawler used the file, cited it, trained on it, or ranked anything because of it.

Crawler identity comes from user agents and published vendor IP ranges, grouped by purpose in the warehouse's crawler configuration. The llms.txt count is the weakest of the set: most of those 404s came from tools and unclassified agents, and the named AI crawlers mostly requested llms.txt on the few sites that already had one.

The decision does not depend on proving use. Serving a small, accurate, generated file costs almost nothing. Serving a 404 or a stale file to something that keeps asking costs the chance to be read at all.

About twenty different generator implementations run across the federation. Teaching each one to write llm.json its own way is how the original drift happened.

So v8.1 has one writer.

1. Each site's own federation generator rebuilds /ai/manifest.json, /ai/catalog.json, /ai/health.json, /ai/karma.json, /ai/federation.json and /llm.txt.
2. The portfolio builder, build-llm-json.php, runs immediately after and derives /llm.json, /ai/llm.json and /llms.txt from those fresh files.
3. A nightly conformance audit checks every site: lifecycle fields at 8.1, llm.json no older than the manifest, llms.txt present and well formed, every constellation peer listed, every peer URL answering 200 without a redirect.

Three rules keep it honest.

Generators never write llm.json themselves.

Generators never delete it. Several v8.0 generators actively removed llm.json on every run and reported it as retired. That is fixed.

A site that publishes its own hand-built llms.txt keeps it. The builder only overwrites an llms.txt it wrote, which it recognizes by its own marker line.

The full contract is in the federation specification. In short:

• The four lifecycle fields: protocol_version, supported_since, deprecated_after, removed_after.
• site: name, URL, tagline, role and summary.
• content_summary: collection counts, total items, and every dataset in the catalog with its item count.
• endpoints: the manifest's endpoint map, plus llm_json, llm_json_ai and llms_txt.
• related_sites and constellation, copied from the manifest.
• derived_from: the exact files it was built from, so a reader can check it.

It is a schema.org Dataset, like the other federation files.

llms.txt follows the llmstxt.org format: a markdown H1 with the site name, a one-paragraph summary in a blockquote, then sections of links.

• Machine-readable endpoints: manifest, llm.json, catalog, llm.txt and federation map.
• Datasets: every dataset the catalog lists, with its item count.
• Pages: the site's public pages and articles from the catalog's content index, with their descriptions.
• Optional: the site's constellation peers.

Everything in it comes from files the site already publishes. It adds a door, not a claim.

If you run a federation member outside this portfolio, the download kit includes build-llm-json.php. Point it at your site root after your federation files are generated:

php build-llm-json.php /path/to/your/site/root

Run it wherever your generator runs, a deploy step or a nightly cron, so llm.json is never older than your manifest. Then add llm_json and llm_json_ai to your manifest's endpoints and set federation_version to 8.1.

A v8.0 site is still a valid, compatible member. It simply has not taken the upgrade, and the audit reports it that way.