The state of the walls
Headline numbers from the latest crawl of 1020 tracked sites (649 with readable policy). Snapshot: 2026-09-13
Block rate by bot
Block rate by site category
| Category | Sites | Blocking ≥1 bot | Rate |
|---|---|---|---|
| news | 33 | 31 | 93.9% |
| food | 6 | 5 | 83.3% |
| jobs | 4 | 3 | 75% |
| social | 15 | 11 | 73.3% |
| entertainment | 13 | 9 | 69.2% |
| realestate | 3 | 2 | 66.7% |
| sports | 9 | 6 | 66.7% |
| health | 5 | 3 | 60% |
| reference | 15 | 9 | 60% |
| auto | 5 | 2 | 40% |
| ecommerce | 17 | 5 | 29.4% |
| finance | 17 | 5 | 29.4% |
| education | 7 | 2 | 28.6% |
| travel | 12 | 3 | 25% |
| top1k | 445 | 107 | 24% |
| tech | 34 | 4 | 11.8% |
| classifieds | 1 | 0 | 0% |
| government | 8 | 0 | 0% |
Blocklists nobody wrote
Not every block is a decision. Some arrive as infrastructure: a CDN inserts a managed section, or a site copies a community roster of AI user agents. Either can be switched on or off in one move by someone who never reads the names inside — on 2026-09-01 roblox.com un-blocked eight AI crawlers at once by deleting a Cloudflare managed block, and on 2026-09-02 semafor.com un-blocked 31 by deleting a copied list. Each cohort is small enough to name in full, so we do, and we re-count them from the archived files every day.
| Boilerplate | Sites | Which ones | What it is |
|---|---|---|---|
| Cloudflare Managed Content | 5 | gamespot.com · kick.com · nexusmods.com · patreon.com · snopes.com | A block Cloudflare generates and maintains, dropped into the file between marker comments. Removing it is a dashboard toggle, not an edit. |
| Content Signals Policy | 25 | amplitude.com · avast.com · bitdefender.net · cloudflare.com · corriere.it · forter.com · gamespot.com · hostinger.com · intercom.io · kick.com · launchpad.net · linktr.ee · netlify.com · nexusmods.com · nvidia.com · oxylabs.io · patreon.com · segment.io · sentry.io · snopes.com · theatlantic.com · trendmicro.com · vercel.com · visma.com · weibo.com | Cloudflare's machine-readable usage preamble (search / ai-input / ai-train). It carries no Disallow of its own — it states intent alongside the rules. |
| robotstxt.com/ai list | 1 | launchpad.net | A community-maintained roster of AI user agents, copied in whole and credited in a comment. It grows when that list grows, without the site revisiting the question. |
Counted by matching the archived robots.txt of every readable domain against the marker each cohort leaves behind, on 2026-09-13. A site can appear in more than one row. Membership is a fact about the file, not a judgement about the site: the Content Signals preamble states a preference and blocks nothing by itself.
Citing these numbers
Data is CC BY 4.0. Cite as "Canicrawl, AI crawler access census, 2026-09-13" with a link. Raw data: latest.json. Methodology: about. Every figure is reproducible from the committed daily snapshots.