AliExpress names six crawlers and blocks them
State of the Agent Web #8 · 2026-10-18
Eighth issue, a week after the seventh. Everything below is drawn from our daily census of AI-crawler access across 1,020 top websites, covering 2026-10-12 to 2026-10-18, and every number is reproducible from the snapshots in the public repository.
The headline: the quietest week yet, and most of it tighter
The changelog gained 19 entries in the window: 16 crawler flips, no default-policy changes and 3 new llms.txt files. Eleven of the flips went tighter and five went looser. Six of the tighter ones came from a single site.
AliExpress names six crawlers and blocks them
On 2026-10-15, aliexpress.com appended seven one-line Disallow: / groups to the bottom of its robots.txt. Six of the names are crawlers we track, and each moved from restricted to blocked: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider and Amazonbot. The seventh is SemrushBot, an SEO crawler. Until then AliExpress named no AI crawler at all, and all of them fell under the * group's path rules. The same edit deleted a Googlebot-only group and moved its one rule, an API endpoint for item data, into the general groups. AliExpress is the only site in this window that started blocking because of an edit. The retrieval agents, such as ChatGPT-User and Perplexity-User, are still unnamed and keep the path rules.
Cisco deletes its welcome mat
On 2026-10-17, cisco.com deleted a group for GPTBot, CCBot, Google-Extended and ClaudeBot that held nothing but Allow: lines, pointing them to its US-English pages, its .md and .llms.txt files and its llms.txt. A companion group that kept Googlebot and Bingbot away from those same .md and .llms.txt files went with it. The four crawlers now follow the * group's path rules, so each moved from allowed to restricted. The change is small in practice: the * rules close off about forty paths, such as bug data, login pages and marketing asset folders, and Cisco still publishes an llms.txt. What went away is the only part of the file that addressed AI crawlers by name.
Meta's own fetcher, shut out by Pinterest
On 2026-10-16, pinterest.com deleted one line, User-agent: meta-externalfetcher, from a long list of named crawlers that share a group of path rules. Pinterest's * group is Disallow: /, so Meta-ExternalFetcher, the agent Meta uses to fetch a page when someone asks for it, moved from restricted to blocked. pinimg.com, Pinterest's image domain, serves a byte-identical file and recorded the same flip, so this is one edit counted twice. A day earlier, on 2026-10-15, Meta tightened its own file: facebook.com replaced a group of 41 path rules for Google-Extended with a one-line Disallow: / at the top, so Google's AI-training token moved from restricted to blocked on Meta's site.
Smaller moves, both ways
- Looser: cnn.com removed Google-Extended from its blocklist (10-15), so it moved from blocked to restricted. In the same edit CNN added two crawlers we don't track, ExaSearchBot and ExaUser. spiegel.de deleted its
Disallow: /group for Amazonbot (10-13), which moved to restricted. Spiegel's blocks on Applebot-Extended and dozens of others stay. - Tighter: pcgamer.com added Applebot and Applebot-Extended to its
Disallow: /group (10-15). Applebot-Extended moved from restricted to blocked. Applebot itself, the crawler behind Siri and Spotlight search, is not one we track.
The cohorts: one new Content Signals site
The Content Signals cohort from issue #3 grew from 24 to 25. On 2026-10-14, att.com rewrote its robots.txt. Its * group now opens with a Content-Signal: line, followed by Allow: lines for agent-discovery files such as /llms.txt and /.well-known/agent-card.json. The first version said ai-train=yes. Two days later, on 10-16, it said ai-train=no, and on 10-17 the same line was added to the Googlebot group. Search and ai-input stayed yes throughout. The line states a preference. It blocks nothing, so none of this produced a flip, and the only record that AT&T briefly said yes to training is our archive. Cloudflare Managed Content stays at 3 and the robotstxt.com/ai list at 1 (launchpad.net). The live counts are on the stats page.
Where the numbers stand
31.8% of the 647 sites we could read on 2026-10-18 block at least one tracked AI crawler (206). The count is unchanged from issue #7, but the percentage dipped from 31.9% because two more sites became readable. One site joined by an edit: aliexpress.com. One left because of reachability: gosuslugi.ru timed out, and it blocks. Bytespider is still the most-blocked crawler at 24.9%, ahead of CCBot at 24.4%, ClaudeBot (21.8%) and GPTBot (21%). The newest names are still the least blocked: Gemini-Deep-Research (11%) and GrokBot (10.4%).
120 of the 1,020 tracked sites (11.8%) publish an llms.txt, the same as in issue #7. Three new files appeared in the changelog: fontawesome.com and seedtag.com (10-16), and businessinsider.com (10-17). Business Insider's asks AI systems to cite articles by title, author, date and canonical URL. Three sites dropped out of the count without removing anything: on 2026-10-18 our probe got no definite answer from time.com, americanexpress.com or yieldmo.com, so we don't count them that day. That is why this figure is a floor.
How to read this issue
Flip counts come from the changelog, which is derived from append-only daily snapshots. Every file quoted above was read from our archive at the commit for the day in question. A flip records what a crawler that obeys robots.txt may fetch. It does not record why the file changed. Compare any two sites on the compare page: try AliExpress against Cisco.
Every number in this issue is reproducible from the committed daily snapshots. Cite "Canicrawl" with a link — data CC BY 4.0.