Semafor puts its blocklist back, Delta opens up

State of the Agent Web #10 · 2026-11-01

Tenth issue, a week after the ninth. Everything below is drawn from our daily census of AI-crawler access across 1,020 top websites, covering 2026-10-26 to 2026-11-01, and every number is reproducible from the snapshots in the public repository.

The headline: 59 tighter, 18 looser

The changelog gained 79 entries in the window: 77 crawler flips, no default-policy changes and 2 llms.txt changes. Fifty-nine of the flips went tighter and 18 went looser. Three edits account for most of it: Semafor's 31 flips, Delta's 17 and Kick's 13, and the first two point in opposite directions. Three of the seven days, 10-26, 10-31 and 11-01, recorded no flips at all.

Semafor puts the list back

On 2026-09-02 semafor.com deleted a copied roster of AI user agents from its robots.txt and un-blocked 31 crawlers in one move, and the stats page has cited it since as the example of a blocklist nobody wrote. On 2026-10-27 the list came back. The new file opens with the comments “Block all known AI crawlers, AI-search indexers” and “Source: https://robotstxt.com/ai”, then names 83 user agents in a single group that ends in Disallow: /. It is the roster Semafor carried before September; the only lines missing are the experimental DisallowAITraining, Content-Usage and Content-Signal directives the old file carried alongside it. Thirty-one tracked crawlers moved from restricted to blocked, among them ClaudeBot, ChatGPT-User, OAI-SearchBot, Google-Extended, PerplexityBot, CCBot, Bytespider and Amazonbot. The thirty-second, GPTBot, did not flip because it was the one crawler Semafor kept blocking through September and October. Semafor rejoins the robotstxt.com/ai cohort, which had been down to a single member.

Delta names 17 crawlers and lets them all in

Until 2026-10-29, delta.com had a robots.txt with a single User-agent: * group: a long list of allowed and disallowed paths that applied to every crawler alike, so each AI crawler we track read as restricted. On 10-30 the file gained a new top section headed “LLM / AI Crawlers - Explicitly Permitted”. It names nineteen crawlers and gives each its own group with a single Allow: /. Seventeen are crawlers we track, and all seventeen moved from restricted to allowed: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, meta-externalagent, Meta-ExternalFetcher, Applebot-Extended, Amazonbot, PerplexityBot, CCBot, cohere-ai, Bytespider, DuckAssistBot, YouBot and Diffbot. The other two, FacebookBot and ai2-crawler, are names we don't track. Training crawlers such as CCBot, Bytespider and Google-Extended are on the list alongside the retrieval agents. In the same edit the * group gained Disallow: /en_US/ with two SkyMiles pages carved back out, so an unnamed crawler now has less of the site than a named AI crawler does. Only three single-day loosenings in the series have been larger, all of them in September.

Kick switches on Cloudflare's block

On 2026-10-29 kick.com did what Wiley did in issue #9. Its robots.txt now opens with Cloudflare's content-signals preamble, then the managed section after the BEGIN Cloudflare Managed content marker: a Content-Signal: search=yes,ai-train=no,use=reference line and one Disallow: / group per crawler on Cloudflare's training list. Thirteen of the names are crawlers we track, the same thirteen as at Wiley, and each moved from restricted to blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, Diffbot and omgili. Kick had blocked none of them before. It joins both Cloudflare cohorts in one edit, and, as with every managed section, the block can be removed with one dashboard toggle.

The Atlantic gives OpenAI's crawlers the house rules

theatlantic.com blocks twenty of the crawlers we track and had, until this week, given a handful of others a bare Allow: / group of their own, which reads as allowed. On 2026-10-29 it added to thirteen of its named groups the same six path disallows its * group carries, covering /ab/, /zephr/, video embeds and search, plus Crawl-delay: 1. Four of those groups belong to crawlers we track, GPTBot, ChatGPT-User, OAI-SearchBot and DuckAssistBot, and each moved from allowed to restricted. Nothing was blocked. The site kept naming these crawlers and gave them the ordinary rules instead of a blank cheque.

Smaller moves, both ways

Tighter: fool.com (10-28) added one Disallow: / group each for GPTBot, ClaudeBot and CCBot, each labelled with its operator and “Model Training”, and a Content-Signal: ai-train=no, search=yes, ai-input=yes line of its own, which puts it in the Content Signals cohort without a Cloudflare managed section. metacritic.com (10-28) blocked meta-externalagent and Meta-ExternalFetcher. ikea.com (10-28) blocked Amazonbot, next to a block for Amazon's product-discovery crawler that was already there. dailymail.co.uk and dailymail.com, which serve a byte-identical file, added cohere-training-data-crawler to their long block group on 10-29, along with three names we don't track: GPTBot-Search, Meta-WebIndexer and xAI-SearchBot. newsweek.com (10-30) blocked Perplexity-User, plus ShapBot and Shap-User, which we don't track.
Tighter, with a caveat: cnbc.com (10-29) added Claude-SearchBot and Perplexity-User to the merged group it built in September, which ends in Disallow: / with a single Allow: /select/. We print both as blocked, as we do the rest of that group. Whether a carve-out that narrow should count as blocked is an open question in our method, and we have not changed the rule mid-series.
Looser: the same Newsweek edit deleted the Disallow: / group for Amazonbot, which moved from blocked to restricted. It is the only loosening in the window outside Delta's.

Not a flip: X adopts the IETF training-preference token

On 2026-10-31 x.com and twitter.com, which serve a byte-identical file, added a group for User-agent: __usage_ai-training with Disallow: /. The comment above it cites the IETF draft draft-kelley-aipref-training-preference-signal, says Microsoft, Google, OpenAI and Anthropic honour it, and states plainly that it is a usage-policy token rather than a crawler: it does not change crawl access, and User-agent: * never matches it. No tracked crawler's verdict changed, so there is no changelog entry, and the change exists only in our archive. No other file in the panel carries the token. Our token radar, which scans every archived robots.txt for user-agent names we don't track, now counts 2,113 distinct tokens. Meta-WebIndexer appears in 42 of the panel's files, and GPTBot-Search and xAI-SearchBot have begun to appear as well.

The cohorts: three joiners, no leavers

Cloudflare Managed Content grew from 4 to 5 with kick.com. Content Signals grew from 25 to 27 with kick.com and fool.com; nobody left. The robotstxt.com/ai list grew from 1 to 2: semafor.com is back beside launchpad.net. We checked membership name by name against the files archived on 10-25, not just the counts. The live counts are on the stats page.

Where the numbers stand

32.4% of the 645 sites we could read on 2026-11-01 block at least one tracked AI crawler (209), up from 31.9% (206) in issue #9. The number of readable sites is the same, but it turned over: gamespot.com, which refused us on 10-25, and kwai.net and ys7.com, which did not answer, were readable on 11-01, while meraki.com and magnit.ru refused us and samsungosp.com did not answer. Two sites joined the blocking set by edit, kick.com and ikea.com; Semafor and Motley Fool were already in it. Two joined by becoming readable again, gamespot.com and kwai.net, a default-deny site that drops in and out of reach every few days, and meraki.com left the same way. Bytespider is still the most-blocked crawler at 25.6%, ahead of CCBot at 25.4%, ClaudeBot (22.9%) and GPTBot (22.0%), each up about half a point. The newest names are still the least blocked: Gemini-Deep-Research (11.2%) and GrokBot (10.5%).

128 of the 1,020 tracked sites (12.5%) publish an llms.txt, up from 125 in issue #9. robinhood.com and cloudns.net published files on 10-28 and account for the two changelog entries. The third, bluehost.com, also answered with a file from 10-28, but its robots.txt refuses our crawler and its earlier llms.txt probes gave no definite answer, so there is no earlier day to record a change against. As before, this figure is a floor.

How to read this issue

Flip counts come from the changelog, which is derived from append-only daily snapshots. Every file quoted above was read from our archive at the commit for the day in question. A flip records what a crawler that obeys robots.txt may fetch. It does not record why the file changed. Compare any two sites on the compare page: try Delta against Semafor.

Every number in this issue is reproducible from the committed daily snapshots. Cite "Canicrawl" with a link — data CC BY 4.0.