What we can and can't see

Every census has a denominator. Ours: on 2026-08-25 we read the policy of 650 of 1000 tracked domains (65%). The other 350 are listed below with the reason — never guessed, never quietly dropped. Snapshot: 2026-08-25

Every outcome, counted

Why nothing answered

The biggest bucket is no HTTPS response, and it is mostly not censorship — it's the panel. Tranco ranks domains by how often the world's resolvers look them up, so its top list is full of names that were never websites: CDN edges (akam.net, tiktokcdn.com), connectivity and DNS endpoints (msftconnecttest.com, dns-parking.com), cloud plumbing (ax-msedge.net, azurewebsites.net). They have no robots.txt because they have no site. We leave them in the index rather than curating them away, so the panel stays reproducible from a public ranking instead of from our taste.

The unreadable, by name

Listed for auditability: if you think one of these should be readable, fetch its robots.txt yourself and tell us. We never retry a refusal with a disguised user agent, and we never infer a policy from an HTML page.

How this affects the numbers

Headline rates — for example "31.2% block at least one AI crawler" — divide by 650, the readable count, never by 1000. Per-bot and per-category rates do the same. Change detection ignores any domain that was unreadable on either side of a diff, so a site going briefly unreachable never produces a fake policy flip. The raw counts behind this page are in latest.json: every domain carries its own fetch outcome.

Coverage moves day to day — timeouts and refusals are not permanent verdicts. This page is regenerated from the latest snapshot every morning, and the daily snapshots keep the history.