Wiley switches on Cloudflare's AI block

State of the Agent Web #9 · 2026-10-25

Ninth issue, a week after the eighth. Everything below is drawn from our daily census of AI-crawler access across 1,020 top websites, covering 2026-10-19 to 2026-10-25, and every number is reproducible from the snapshots in the public repository.

The headline: 33 tighter, one looser

The changelog gained 40 entries in the window: 34 crawler flips, no default-policy changes and 6 llms.txt changes. Thirty-three of the flips went tighter and one went looser. Two edits account for most of it: Wiley's 13 flips and Meta's 10, spread across Facebook and Instagram.

Wiley switches on Cloudflare's block

Issue #5 covered wiley.com on 2026-09-18, when a replatform swapped a long robots.txt for a one-line Webflow file and nothing about AI changed. On 2026-10-24 Wiley made its first AI rule. It put Cloudflare's managed section at the top of the file, with the marker comments around it, and its own rules below. That section opens with a Content-Signal: line (search=yes,ai-train=no,use=reference) and then gives one Disallow: / group each to a long list of crawlers. Thirteen of the names are crawlers we track, and each moved from restricted to blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, Diffbot and omgili. The retrieval agents, such as ChatGPT-User and Claude-User, are not named and keep the path rules. Wiley joins both boilerplate cohorts in a single edit. As the stats page notes, a section like this can be removed with one dashboard toggle.

Meta puts Instagram back and copies it to Facebook

On 2026-10-08, instagram.com replaced the Disallow: / groups for six training crawlers with path rules, and issue #7 recorded that loosening. On 2026-10-19 Instagram went back, and its robots.txt is now byte-identical to the 2026-10-07 file. GPTBot, ClaudeBot, Applebot-Extended, PerplexityBot and Amazonbot moved from restricted to blocked. The sixth, Google-Extended, had already been blocked again on 10-15, but our crawl could not read Instagram's file on 10-14, so the changelog has no entry for that change. A flip is only recorded when both days are readable. On the same day facebook.com added the same Disallow: / groups at the top of its own file and deleted the long path-rule groups those crawlers had followed. The same five crawlers moved from restricted to blocked there too. Facebook had not blocked any of the five since our census began on 2026-08-25. Both files also block Brightbot, which we don't track.

Dotdash Meredith adds Diffbot

investopedia.com on 2026-10-19, then allrecipes.com, seriouseats.com and people.com on 10-20, added the same four names to an existing shared block group: Diffbot, Diffbot-User, ExaSearchBot and LinkUpBot. All four sites belong to the same publisher, and each recorded one flip, Diffbot from restricted to blocked. That is one edit rolled out over two days. The other three names are crawlers we don't track.

Smaller moves, both ways

Looser: nerdwallet.com deleted its Disallow: / group for Google-Extended (10-22), which moved from blocked to restricted. That was NerdWallet's only block, so the site has left the count of sites that block an AI crawler.
Tighter: sophos.com split Amazon's crawlers apart (10-21). Amazonbot, the training crawler, moved from allowed to blocked, while two Amazon agents we don't track, Amzn-SearchBot and Amzn-User, were given their own Allow: /. soundcloud.com added Claude-User, Claude-SearchBot, omgili and omgilibot to its group headed “editorial only, no UGC training” (10-23). nbcnews.com gave Meta-ExternalFetcher a group of its own (10-22). We print both of these as blocked, but each group ends in Disallow: / with a short list of Allow: exceptions: SoundCloud's homepage, its legal, editorial and company pages, and a handful of NBC News about, contact and tips pages. Whether a carve-out that narrow should count as blocked is an open question in our method, and we have not changed the rule mid-series.

The cohorts: Wiley in, Vercel's signal out

Cloudflare Managed Content grew from 3 to 4 with wiley.com. The Content Signals cohort stays at 25, but its membership changed. Wiley joined, and on 2026-10-21 vercel.com deleted its Content-Signal: search=yes, ai-input=yes, ai-train=no line, the only change to that file. The line never blocked anything, so no flip was recorded and the deletion shows up only in our archive. The robotstxt.com/ai list stays at 1 (launchpad.net). The live counts are on the stats page.

Where the numbers stand

31.9% of the 645 sites we could read on 2026-10-25 block at least one tracked AI crawler (206). The count is the same as in issue #8, and it turned over by edit only: Wiley joined and NerdWallet left. The percentage rose from 31.8% because the number of readable sites fell from 647 to 645. On 10-25, miit.gov.cn and ys7.com were unreachable and timeweb.ru returned an HTML page instead of a robots.txt, while yieldmo.com became readable again. Bytespider is still the most-blocked crawler at 25.1%, ahead of CCBot at 24.7%, ClaudeBot (22.3%) and GPTBot (21.6%). The newest names are still the least blocked: Gemini-Deep-Research (11%) and GrokBot (10.4%).

125 of the 1,020 tracked sites (12.3%) publish an llms.txt, up from 120 in issue #8. Four of the five are new files: dailymail.co.uk and dailymail.com (10-20), and fool.com and unpkg.com (10-22). The fifth, yieldmo.com, gave our probe no answer on 10-18 and answered with a file this week. paloaltonetworks.com's file was missing on 10-19 and back on 10-20, which accounts for the remaining two changelog entries. time.com and americanexpress.com still give no definite answer, and we don't count them. That is why this figure is a floor.

How to read this issue

Flip counts come from the changelog, which is derived from append-only daily snapshots. Every file quoted above was read from our archive at the commit for the day in question. A flip records what a crawler that obeys robots.txt may fetch. It does not record why the file changed. Compare any two sites on the compare page: try Facebook against Instagram.

Every number in this issue is reproducible from the committed daily snapshots. Cite "Canicrawl" with a link — data CC BY 4.0.