What we can and can't see

Every census has a denominator. Ours: on 2026-09-14 we read the policy of 646 of 1020 tracked domains (63.3%). The other 374 are listed below with the reason — never guessed, never quietly dropped. Snapshot: 2026-09-14

Every outcome, counted

Why nothing answered

Until 2026-09-14 this page had a single no HTTPS response bucket, which hid the most important distinction in the whole ledger. It is now split on recorded evidence, not on guesswork about domain names: 184 domains have no DNS record at all, and 79 resolve to a real host that then gave us nothing.

The first group is not censorship — it's the panel. Tranco ranks domains by how often the world's resolvers look them up, so its top list is full of names that were never websites: CDN edges (akam.net, tiktokcdn.com), connectivity and DNS endpoints (msftconnecttest.com, dns-parking.com), cloud plumbing (ax-msedge.net, azurewebsites.net). They have no robots.txt because they have no site, and a name that does not resolve today will almost certainly not resolve tomorrow. We leave them in the index — greyed out and marked, never deleted — so the panel stays reproducible from a public ranking instead of from our taste. The second group is the genuinely interesting one: those are live hosts, and a timeout today can be a readable policy next week, which is why coverage moves a little every day.

How the panel changes

The other half of the denominator is which domains are in it at all. The panel was filled once from a Tranco ranking and would quietly go stale — the ranking drifts about 2% a week — so it is re-examined weekly, additions only, and capped at 25 per run so one bad list day cannot reshape the census unattended. Every run is appended to panel-history.json, and this section is that file, read at build time: 1 refresh so far, 20 domains added, 20 kept after falling out.

A domain that falls out of the ranking stays in the index. Dropping it would delete history: its daily snapshots are append-only facts about what that site's robots.txt said on those mornings, and /site/<domain>/ is a live URL somebody may have linked. So a departure is reported and kept — it keeps its page, its history and its place in the counts — and the panel grows rather than churns. Same principle as leaving the 184 no-DNS domains listed and marked: the panel is reproducible from a public ranking plus a written ledger, not from our taste about which sites deserve to be in it.

The unreadable, by name

Listed for auditability: if you think one of these should be readable, fetch its robots.txt yourself and tell us. We never retry a refusal with a disguised user agent, and we never infer a policy from an HTML page.

Receipts for every llms.txt we report

We say 112 tracked sites publish an llms.txt. Each of those claims should be checkable by someone who is not us, so we archive the file we read. Bodies above the 256KB archive cap are kept as a SHA-256 prefix and a byte count instead — enough to prove the body existed, and enough to notice if it is silently swapped — because a claim with no evidence behind it is the failure mode this index has corrected three times.

How many sites actually answered the llms.txt question

An llms.txt probe has three outcomes, and only two of them are answers. A served file is a yes; a 404 is a no; a refusal, a rate-limit, a 5xx or a timeout is no answer at all, and we record it as such rather than counting it as a no. On 2026-09-14 we got a definitive answer from 622 of 1020 tracked domains.

Until 2026-09-14 the index drew "no answer" and "none found" the same way — a dash — which published an unknown as a fact. The clearest case is a site we hold a receipt for: shein.com's 64KB llms.txt is archived in this repository, and on the mornings its probe is refused the old rendering said the file did not exist. Those cells now read ?, and the site page says we got no answer instead of asserting a negative. Nothing about the adoption figure changed: 11% is a floor, counted only from sites that gave us a real yes, and the 135 live sites in the last row are the room above it.

How this affects the numbers

Headline rates — for example "31.9% block at least one AI crawler" — divide by 646, the readable count, never by 1020. Per-bot and per-category rates do the same. Change detection ignores any domain that was unreadable on either side of a diff, so a site going briefly unreachable never produces a fake policy flip. The raw counts behind this page are in latest.json: every domain carries its own fetch outcome.

Coverage moves day to day — timeouts and refusals are not permanent verdicts. This page is regenerated from the latest snapshot every morning, and the daily snapshots keep the history.