The state of the walls
Headline numbers from the latest crawl of 1020 tracked sites (644 with readable policy). Snapshot: 2026-10-04
Block rate by bot
Block rate by site category
| Category | Sites | Blocking ≥1 bot | Rate |
|---|---|---|---|
| news | 33 | 31 | 93.9% |
| food | 6 | 5 | 83.3% |
| jobs | 4 | 3 | 75% |
| social | 15 | 11 | 73.3% |
| entertainment | 12 | 8 | 66.7% |
| realestate | 3 | 2 | 66.7% |
| sports | 9 | 6 | 66.7% |
| health | 5 | 3 | 60% |
| reference | 15 | 9 | 60% |
| auto | 5 | 2 | 40% |
| finance | 17 | 5 | 29.4% |
| education | 7 | 2 | 28.6% |
| ecommerce | 18 | 5 | 27.8% |
| travel | 12 | 3 | 25% |
| top1k | 441 | 107 | 24.3% |
| tech | 34 | 4 | 11.8% |
| classifieds | 1 | 0 | 0% |
| government | 7 | 0 | 0% |
Blocklists nobody wrote
Not every block is a decision. Some arrive as infrastructure: a CDN inserts a managed section, or a site copies a community roster of AI user agents. Either can be switched on or off in one move by someone who never reads the names inside — on 2026-09-01 roblox.com un-blocked eight AI crawlers at once by deleting a Cloudflare managed block, and on 2026-09-02 semafor.com un-blocked 31 by deleting a copied list. Each cohort is small enough to name in full, so we do, and we re-count them from the archived files every day.
| Boilerplate | Sites | Which ones | What it is |
|---|---|---|---|
| Cloudflare Managed Content | 3 | gamespot.com (last read 2026-09-15) · nexusmods.com (last read 2026-09-01) · snopes.com | A block Cloudflare generates and maintains, dropped into the file between marker comments. Removing it is a dashboard toggle, not an edit. |
| Content Signals Policy | 24 | amplitude.com · avast.com (last read 2026-09-24) · bitdefender.net · cloudflare.com · corriere.it · forter.com · gamespot.com (last read 2026-09-15) · hostinger.com · intercom.io · launchpad.net · linktr.ee · netlify.com · nexusmods.com (last read 2026-09-01) · nvidia.com · oxylabs.io · patreon.com · segment.io · sentry.io · snopes.com · theatlantic.com · trendmicro.com · vercel.com · visma.com · weibo.com | Cloudflare's machine-readable usage preamble (search / ai-input / ai-train). It carries no Disallow of its own — it states intent alongside the rules. |
| robotstxt.com/ai list | 1 | launchpad.net | A community-maintained roster of AI user agents, copied in whole and credited in a comment. It grows when that list grows, without the site revisiting the question. |
Counted by matching the archived robots.txt of every readable domain against the marker each cohort leaves behind, on 2026-10-04. A site can appear in more than one row. Membership is a fact about the file, not a judgement about the site: the Content Signals preamble states a preference and blocks nothing by itself. A member we could not read on this crawl is counted from its last archived file, dated beside its name.
Citing these numbers
Data is CC BY 4.0. Cite as "Canicrawl, AI crawler access census, 2026-10-04" with a link. Raw data: latest.json. Methodology: about. Every figure is reproducible from the committed daily snapshots.