← all issues

Delta closes the door it opened

State of the Agent Web #11 · 2026-11-08

Eleventh issue, a week after the tenth. Everything below is drawn from our daily census of AI-crawler access across 1,020 top websites, covering 2026-11-02 to 2026-11-08, and every number is reproducible from the snapshots in the public repository.

The headline: 44 tighter, 8 looser

The changelog gained 55 entries in the window: 52 crawler flips, no default-policy changes and 3 llms.txt changes. Forty-four of the flips went tighter and 8 went looser. Three edits account for most of it: Delta's 17, Corriere della Sera's 17 and Seeking Alpha's 10. Two of the seven days recorded no flips: 11-03 had a single llms.txt change and 11-07 had nothing at all.

Delta closes the door it opened

Issue #10 led with delta.com. On 2026-10-30 its robots.txt gained a section headed “LLM / AI Crawlers - Explicitly Permitted”, nineteen named groups each carrying a single Allow: /, and 17 tracked crawlers moved from restricted to allowed, the fourth-largest single-day loosening in the series. On 2026-11-06 the section was gone. The day's edit is 58 deleted lines and nothing added: the heading and all nineteen groups, with the rest of the file untouched. The same 17 crawlers moved from allowed back to restricted, because each now inherits the long path rules of the User-agent: * group exactly as it did on 10-29: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, meta-externalagent, Meta-ExternalFetcher, Applebot-Extended, Amazonbot, PerplexityBot, CCBot, cohere-ai, Bytespider, DuckAssistBot, YouBot and Diffbot. The section lasted seven days. Delta edited the file twice more in the window without touching it: on 11-04 it reworked its flight-status paths, and on 11-07 it dropped a block of .jsp allows. The archive records what changed, not why. This is the second time in the series that a large opening has closed within a week: wiley.com opened to all 32 tracked crawlers on 09-18 and shut again on 09-24.

Corriere della Sera names 78 more crawlers

On 2026-11-04 corriere.it added 78 named groups to its robots.txt, each ending in Disallow: /. The roster runs from the familiar to the obscure: Claude-User, Claude-SearchBot, Perplexity-User, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, ImagesiftBot, AI2Bot, DeepSeekBot, GrokBot and PanguBot, fifteen crawlers we track, each moving from restricted to blocked, alongside names we don't track such as Claude-Code, ChatGPT Operator, NovaAct, Firecrawl, Crawl4AI, Turnitin, MoodleBot, xAI-Grok and Amazon Kendra. With the two flips below it is 17 flips in one edit, tied with Delta for the largest tightening of the window and the eighth-largest in the series, behind seven edits that blocked 31 or 32 crawlers at once. Corriere already carried a Content-Signal line and was already in that cohort; nothing about its membership changed.

With a caveat: in the same edit OAI-SearchBot and PerplexityBot went from a bare Allow: / to Disallow: / with a single Allow: /nativeadv/, the site's native-advertising section. That is the pattern ChatGPT-User and Claude-Web have had at Corriere since 09-11, and as there we print both as blocked. Whether a carve-out that narrow should count as blocked is an open question in our method, and we have not changed the rule mid-series.

Seeking Alpha swaps the fetchers for the crawlers

seekingalpha.com keeps a long alphabetical blocklist, one Disallow: / group per name. On 2026-11-05 it rewrote the list in both directions. Out went the groups for ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot, MistralAI-User, YouBot and Meta-ExternalFetcher, all seven moving from blocked to restricted because they now inherit the * group's path rules. Every one of them is a retrieval agent, the kind that fetches a page because a person asked a question. Alongside them it dropped Discordbot, Feedly, LinerBot, NewsNow, Operator, SeekrBot and TelegramBot, which we don't track. In went Amazonbot, restricted to blocked, in the slot where Andibot used to be, together with a run of SEO and archive crawlers: AhrefsBot, archive.org_bot, ia_archiver, Baiduspider, Barkrowler, rogerbot, SEOkicks, Yandex. Then on 11-08 it added GPTBot and Applebot-Extended, both restricted to blocked. Amazonbot and Applebot-Extended had been un-blocked by the same site on 09-27, so each spent six weeks restricted. The 11-05 edit also deleted the lower-case meta-externalagent group but kept Meta-ExternalAgent; robots.txt matches user agents without regard to case, so that crawler stays blocked and no flip was recorded. Seven of the window's eight loosenings are this one edit.

Hostinger says yes to training

On 2026-11-04 hostinger.com changed one word in both of its Content-Signal lines: ai-train=no became ai-train=yes, in the * group and in the named group it keeps for AI crawlers. In the same edit the standalone Disallow: / group for CCBot was deleted and CCBot's name was added to that AI group, which allows the site and disallows only tracking-parameter URLs, so CCBot moved from blocked to restricted, the window's one loosening outside Seeking Alpha. Bytespider, which Hostinger blocked on the same day as CCBot in September, is still blocked. The edit also removed about 340 Disallow lines for localised product pages, which has no bearing on any verdict. A Content-Signal line is a statement of preference and carries no access rule of its own, so the word change is not a flip and appears only in our archive. Of the 27 files in the Content Signals cohort, 11 now say ai-train=yes, up from 10; Hostinger is the only member whose answer changed this week.

Smaller moves

Not a flip: the training-preference token spreads to Meta

Issue #10 noted that x.com and twitter.com had added a group for User-agent: __usage_ai-training, the IETF draft token that states a training preference rather than naming a crawler, and that no other file in the panel carried it. A week later ten do. imdb.com added it on 11-03, wikihow.com on 11-05, and on 11-06 Meta added it across facebook.com, fb.com, instagram.com, whatsapp.com and threads.com, and linkedin.com followed on 11-07. Each is the same two lines, Disallow: / under the token. None changes what any crawler may fetch, so none is in the changelog. Also outside the changelog: github.com (11-07) added ChatGPT-User, Claude-SearchBot, Claude-User and Perplexity-User to the named AI group at the top of its file and dropped anthropic-ai from it. That group carries the same path rules as the * group plus a Crawl-delay: 1, so every verdict reads restricted before and after. And noaa.gov, which had refused our crawler on every day of the series, served its robots.txt for the first time on 11-03: ordinary path rules, no AI crawler named. Our token radar now counts 2,135 distinct user-agent tokens across the archive, up from 2,113, and 32 AI-looking names we don't track, up from 25. Meta-WebIndexer still appears in 42 files.

The cohorts: no joiners, no leavers

Cloudflare Managed Content stays at 5, Content Signals at 27 and the robotstxt.com/ai list at 2. We checked membership name by name against the files archived on 11-01, not just the counts: the same sites in every list. The one movement inside a cohort is Hostinger's change of answer, above. The live counts are on the stats page.

Where the numbers stand

32.6% of the 648 sites we could read on 2026-11-08 block at least one tracked AI crawler (211), up from 32.4% of 645 (209) in issue #10. Three sites became readable and none stopped: gosuslugi.ru, which had not answered, and noaa.gov and magnit.ru, which had refused us. Two sites joined the blocking set: researchgate.net by edit, and gosuslugi.ru by becoming readable, since its * group disallows everything and so blocks all 32 tracked crawlers at once. Delta, Corriere, Seeking Alpha, the BBC, AP and Le Monde were all already in the set, and Hostinger stayed in it through Bytespider. Bytespider is still the most-blocked crawler at 25.6%, ahead of CCBot at 25.3%, ClaudeBot (23.0%) and GPTBot (22.2%). The largest rise belongs to Claude-SearchBot, 14.9% to 15.4%, on the BBC's two domains, Corriere and gosuslugi.ru; GrokBot rose from 10.5% to 11.0% on Corriere, Le Monde and gosuslugi.ru. The newest names are still the least blocked: Gemini-Deep-Research (11.3%) and GrokBot (11.0%).

130 of the 1,020 tracked sites (12.7%) publish an llms.txt, up from 128 in issue #10. appsflyer.com (11-02), mixpanel.com (11-03) and digitalocean.com (11-05) published files and account for the three changelog entries. One site left the count without a changelog entry: comcast.com, whose file we read through 11-05, has given no definite answer to the probe since 11-06, so it shows as ? rather than as a removal; if the file comes back, nothing will have been recorded either way. As before, this figure is a floor.

How to read this issue

Flip counts come from the changelog, which is derived from append-only daily snapshots. Every file quoted above was read from our archive at the commit for the day in question. A flip records what a crawler that obeys robots.txt may fetch. It does not record why the file changed. Compare any two sites on the compare page: try Delta against Corriere.

Every number in this issue is reproducible from the committed daily snapshots. Cite "Canicrawl" with a link — data CC BY 4.0.