← all issues

The web is blocking crawlers that barely exist

State of the Agent Web #2 · 2026-08-30

Second issue. Everything below is drawn from our daily census of AI-crawler access across 1,000 top websites, and every number is reproducible from the snapshots in the public repository. This issue covers 2026-08-25 to 2026-08-30.

The headline: sites are blocking crawlers that barely exist yet

Alongside the 32 AI crawlers we track, we scan the raw text of every robots.txt we read for user-agent tokens we don’t track. On 2026-08-30 that scan covered 597 archived files containing 2,075 distinct tokens, and it keeps surfacing the same surprise: a real slice of the web is already writing rules for AI agents that have almost no public footprint.

The clearest example is FirecrawlAgent, the crawler behind an AI scraping API, named in the robots.txt of 23 sites run by 22 independent operators — among them the BBC, the New York Times, Le Monde, Die Welt, The Verge, USA Today and Investopedia. Crawl4AI, an open-source agentic scraper, appears on 20 sites. Neither is a household name. Both are already on major publishers’ blocklists.

Two more deserve a mention for what they say about how these lists get written. Ai2Bot-Dolma — named not after a crawler but after the dataset it collects for — appears across 26 independent operators. And MyCentralAIScraperBot, a name with essentially no public presence at all, is named by 8 distinct operators including LinkedIn, CNN and the New York Times. Nobody writes that token from first principles. Blocklists are being copied between sites faster than the crawlers themselves are being documented.

A counting caveat we would rather state than hide

Raw site counts overstate how widely a token has spread, because one company’s robots.txt often sits on a dozen domains. Grok-DeepSearch and xAI-Grok each appear on 15 sites — but 13 of those are Amazon’s regional storefronts plus Prime Video, so the honest figure is 2 independent operators, not 15. We report both numbers from here on. It is the difference between “the web is turning against xAI” and “Amazon updated one file.”

The newest bots have the lowest walls — and half their blocks are accidents

Across the 638 sites we could read on 2026-08-30, 32% block at least one tracked AI crawler. The per-bot spread tracks age almost perfectly. CCBot is still the most-blocked at 25.7%, followed by Bytespider (25.2%), ClaudeBot (23.0%) and GPTBot (22.1%). At the other end sit the newcomers: Gemini-Deep-Research (11.3%) and GrokBot (10.3%).

But the interesting number is how they are blocked. Of the 66 sites blocking GrokBot, only 16 name it — the other 50 catch it with a blanket rule written before GrokBot existed. For CCBot the split runs the other way: 122 of its 164 blocks are explicit. So the picture is not that the web has made up its mind about xAI or Google’s research agent. It is that 52 sites run default-deny robots.txt files that silently swallow every crawler born after they were written, while explicit decisions still cluster on the bots people have been arguing about for two years.

The week’s real movements

Five days of daily diffs produced four genuine policy changes across 1,000 sites — a reminder that robots.txt is a slow-moving document, which is exactly why the day-level record is worth keeping.

The number of sites publishing an llms.txt stands at 108 of 1,000. Two of this week’s four changes were doors opening rather than closing — the welcome-mat trend from issue #1 is still running alongside the walls.

A correction, and what we changed

While fact-checking this issue against the raw archive, we found that our own changelog had been publishing a flip that never happened. A 404 on a bare domain caused our fetcher to stop before trying the www host; on days when un.org’s apex answered 404 instead of its usual redirect, we recorded “no robots.txt exists, therefore everything is allowed” — while www.un.org was serving a restrictive file the entire time. Diffed against a real reading the next day, that manufactured a phantom “United Nations reopens to all AI crawlers” story. It accounted for 66 of the 136 entries we had published.

Three fixes followed: the fetcher now continues to www after a 404; we no longer emit a diff across a transition between a successful read and a failed one; and an llms.txt probe that fails now records an explicit unknown rather than false — which had been making shein’s real 64KB file appear to vanish and return on alternating days. The changelog was then recomputed from the append-only snapshots rather than hand-edited: 136 entries became 68, with 68 dropped and zero gained. All four genuine stories above survived untouched.

The tradeoff is deliberate and worth stating plainly: a site that genuinely deletes its robots.txt looks identical, on the day, to a server hiccup. We now stay quiet in that case. We lose same-day news on a rare true positive in order to stop crying wolf on a common false one. For a record whose only value is being citable, that is the right side on which to err.

What’s next

Newly-discovered tokens that clear a meaningful independent-operator threshold graduate into tracked bots with their own pages, as 16 did last week. The coverage ledger continues to publish the exact denominator behind every percentage here — 638 readable of 1,000, with every unreadable domain listed by cause. The change feed lives on the changelog and its RSS.

Every number in this issue is reproducible from the committed daily snapshots. Cite "Canicrawl" with a link — data CC BY 4.0.