Changelog
Every AI-policy flip we observe lands here, newest first — detected by comparing daily snapshots. Subscribe via RSS.
2026-10-03
2026-10-01
- trustpilot.com stopped blocking Meta-ExternalFetcher (blocked → restricted)
- trustpilot.com stopped blocking meta-externalagent (blocked → restricted)
- trustpilot.com stopped blocking CCBot (blocked → restricted)
- checkpoint.com changed access for Claude-Web (restricted → allowed)
- checkpoint.com changed access for CCBot (restricted → allowed)
- checkpoint.com changed access for Perplexity-User (restricted → allowed)
- checkpoint.com changed access for PerplexityBot (restricted → allowed)
- checkpoint.com changed access for Google-Extended (restricted → allowed)
- checkpoint.com changed access for ClaudeBot (restricted → allowed)
- checkpoint.com changed access for OAI-SearchBot (restricted → allowed)
- checkpoint.com changed access for ChatGPT-User (restricted → allowed)
- checkpoint.com changed access for GPTBot (restricted → allowed)
- figma.com began blocking Bytespider (restricted → blocked)
2026-09-29
2026-09-27
- seekingalpha.com stopped blocking Amazonbot (blocked → restricted)
- seekingalpha.com stopped blocking PerplexityBot (blocked → restricted)
- seekingalpha.com stopped blocking Applebot-Extended (blocked → restricted)
2026-09-26
- airbnb.com began blocking AI2Bot (restricted → blocked)
- airbnb.com began blocking Applebot-Extended (restricted → blocked)
- airbnb.com began blocking ClaudeBot (restricted → blocked)
- airbnb.com began blocking GPTBot (restricted → blocked)
- etsy.com published an llms.txt — a welcome mat for AI readers
2026-09-25
- markmonitor.com changed access for Gemini-Deep-Research (allowed → restricted)
- markmonitor.com changed access for PanguBot (allowed → restricted)
- markmonitor.com changed access for GrokBot (allowed → restricted)
- markmonitor.com changed access for DeepSeekBot (allowed → restricted)
- markmonitor.com changed access for AI2Bot (allowed → restricted)
- markmonitor.com changed access for Meta-ExternalFetcher (allowed → restricted)
- markmonitor.com changed access for Timpibot (allowed → restricted)
- markmonitor.com changed access for ImagesiftBot (allowed → restricted)
- markmonitor.com changed access for YouBot (allowed → restricted)
- markmonitor.com changed access for omgilibot (allowed → restricted)
- markmonitor.com changed access for omgili (allowed → restricted)
- markmonitor.com changed access for Diffbot (allowed → restricted)
- markmonitor.com changed access for cohere-training-data-crawler (allowed → restricted)
- markmonitor.com changed access for cohere-ai (allowed → restricted)
- markmonitor.com changed access for Claude-Web (allowed → restricted)
- markmonitor.com changed access for anthropic-ai (allowed → restricted)
- markmonitor.com changed access for MistralAI-User (allowed → restricted)
- markmonitor.com changed access for DuckAssistBot (allowed → restricted)
- markmonitor.com changed access for Amazonbot (allowed → restricted)
- markmonitor.com changed access for meta-externalagent (allowed → restricted)
- markmonitor.com changed access for Bytespider (allowed → restricted)
- markmonitor.com changed access for CCBot (allowed → restricted)
- markmonitor.com changed access for Perplexity-User (allowed → restricted)
- markmonitor.com changed access for PerplexityBot (allowed → restricted)
- markmonitor.com changed access for Applebot-Extended (allowed → restricted)
- markmonitor.com changed access for Google-Extended (allowed → restricted)
- markmonitor.com changed access for Claude-SearchBot (allowed → restricted)
- markmonitor.com changed access for Claude-User (allowed → restricted)
- markmonitor.com changed access for ClaudeBot (allowed → restricted)
- markmonitor.com changed access for OAI-SearchBot (allowed → restricted)
- markmonitor.com changed access for ChatGPT-User (allowed → restricted)
- markmonitor.com changed access for GPTBot (allowed → restricted)
- markmonitor.com changed its default crawler policy (allowed → restricted)
- name.com began blocking meta-externalagent (restricted → blocked)
2026-09-24
- fwmrm.net published an llms.txt — a welcome mat for AI readers
- aol.com stopped blocking PerplexityBot (blocked → restricted)
- aol.com stopped blocking ChatGPT-User (blocked → restricted)
- wiley.com changed access for Gemini-Deep-Research (allowed → restricted)
- wiley.com changed access for PanguBot (allowed → restricted)
- wiley.com changed access for GrokBot (allowed → restricted)
- wiley.com changed access for DeepSeekBot (allowed → restricted)
- wiley.com changed access for AI2Bot (allowed → restricted)
- wiley.com changed access for Meta-ExternalFetcher (allowed → restricted)
- wiley.com changed access for Timpibot (allowed → restricted)
- wiley.com changed access for ImagesiftBot (allowed → restricted)
- wiley.com changed access for YouBot (allowed → restricted)
- wiley.com changed access for omgilibot (allowed → restricted)
- wiley.com changed access for omgili (allowed → restricted)
- wiley.com changed access for Diffbot (allowed → restricted)
- wiley.com changed access for cohere-training-data-crawler (allowed → restricted)
- wiley.com changed access for cohere-ai (allowed → restricted)
- wiley.com changed access for Claude-Web (allowed → restricted)
- wiley.com changed access for anthropic-ai (allowed → restricted)
- wiley.com changed access for MistralAI-User (allowed → restricted)
- wiley.com changed access for DuckAssistBot (allowed → restricted)
- wiley.com changed access for Amazonbot (allowed → restricted)
- wiley.com changed access for meta-externalagent (allowed → restricted)
- wiley.com changed access for Bytespider (allowed → restricted)
- wiley.com changed access for CCBot (allowed → restricted)
- wiley.com changed access for Perplexity-User (allowed → restricted)
- wiley.com changed access for PerplexityBot (allowed → restricted)
- wiley.com changed access for Applebot-Extended (allowed → restricted)
- wiley.com changed access for Google-Extended (allowed → restricted)
- wiley.com changed access for Claude-SearchBot (allowed → restricted)
- wiley.com changed access for Claude-User (allowed → restricted)
- wiley.com changed access for ClaudeBot (allowed → restricted)
- wiley.com changed access for OAI-SearchBot (allowed → restricted)
- wiley.com changed access for ChatGPT-User (allowed → restricted)
- wiley.com changed access for GPTBot (allowed → restricted)
- wiley.com changed its default crawler policy (allowed → restricted)
2026-09-18
- wiley.com changed access for Gemini-Deep-Research (restricted → allowed)
- wiley.com changed access for PanguBot (restricted → allowed)
- wiley.com changed access for GrokBot (restricted → allowed)
- wiley.com changed access for DeepSeekBot (restricted → allowed)
- wiley.com changed access for AI2Bot (restricted → allowed)
- wiley.com changed access for Meta-ExternalFetcher (restricted → allowed)
- wiley.com changed access for Timpibot (restricted → allowed)
- wiley.com changed access for ImagesiftBot (restricted → allowed)
- wiley.com changed access for YouBot (restricted → allowed)
- wiley.com changed access for omgilibot (restricted → allowed)
- wiley.com changed access for omgili (restricted → allowed)
- wiley.com changed access for Diffbot (restricted → allowed)
- wiley.com changed access for cohere-training-data-crawler (restricted → allowed)
- wiley.com changed access for cohere-ai (restricted → allowed)
- wiley.com changed access for Claude-Web (restricted → allowed)
- wiley.com changed access for anthropic-ai (restricted → allowed)
- wiley.com changed access for MistralAI-User (restricted → allowed)
- wiley.com changed access for DuckAssistBot (restricted → allowed)
- wiley.com changed access for Amazonbot (restricted → allowed)
- wiley.com changed access for meta-externalagent (restricted → allowed)
- wiley.com changed access for Bytespider (restricted → allowed)
- wiley.com changed access for CCBot (restricted → allowed)
- wiley.com changed access for Perplexity-User (restricted → allowed)
- wiley.com changed access for PerplexityBot (restricted → allowed)
- wiley.com changed access for Applebot-Extended (restricted → allowed)
- wiley.com changed access for Google-Extended (restricted → allowed)
- wiley.com changed access for Claude-SearchBot (restricted → allowed)
- wiley.com changed access for Claude-User (restricted → allowed)
- wiley.com changed access for ClaudeBot (restricted → allowed)
- wiley.com changed access for OAI-SearchBot (restricted → allowed)
- wiley.com changed access for ChatGPT-User (restricted → allowed)
- wiley.com changed access for GPTBot (restricted → allowed)
- wiley.com changed its default crawler policy (restricted → allowed)
2026-09-16
- onet.pl began blocking Timpibot (restricted → blocked)
- onet.pl began blocking YouBot (restricted → blocked)
- onet.pl began blocking omgilibot (restricted → blocked)
- onet.pl began blocking Diffbot (restricted → blocked)
- onet.pl began blocking cohere-ai (restricted → blocked)
- onet.pl began blocking Amazonbot (restricted → blocked)
- onet.pl began blocking meta-externalagent (restricted → blocked)
- onet.pl began blocking Bytespider (restricted → blocked)
- kick.com stopped blocking Amazonbot (blocked → restricted)
- kick.com stopped blocking meta-externalagent (blocked → restricted)
- kick.com stopped blocking Bytespider (blocked → restricted)
- kick.com stopped blocking CCBot (blocked → restricted)
- kick.com stopped blocking Applebot-Extended (blocked → restricted)
- kick.com stopped blocking Google-Extended (blocked → restricted)
- kick.com stopped blocking ClaudeBot (blocked → restricted)
- kick.com stopped blocking GPTBot (blocked → restricted)
- patreon.com stopped blocking Amazonbot (blocked → restricted)
- patreon.com stopped blocking meta-externalagent (blocked → restricted)
- patreon.com stopped blocking Bytespider (blocked → restricted)
- patreon.com stopped blocking CCBot (blocked → restricted)
- patreon.com stopped blocking Applebot-Extended (blocked → restricted)
- patreon.com stopped blocking Google-Extended (blocked → restricted)
- patreon.com stopped blocking ClaudeBot (blocked → restricted)
- patreon.com stopped blocking GPTBot (blocked → restricted)
- azure.com removed its llms.txt
- weather.com began blocking YouBot (restricted → blocked)
- weather.com stopped blocking cohere-ai (blocked → restricted)
- weather.com stopped blocking Claude-Web (blocked → restricted)
- weather.com stopped blocking anthropic-ai (blocked → restricted)
- weather.com stopped blocking meta-externalagent (blocked → restricted)
- weather.com stopped blocking PerplexityBot (blocked → restricted)
- weather.com stopped blocking Applebot-Extended (blocked → restricted)
- weather.com stopped blocking Google-Extended (blocked → restricted)
- weather.com stopped blocking ClaudeBot (blocked → restricted)
- weather.com stopped blocking GPTBot (blocked → restricted)
2026-09-14
- samsungcloud.com changed access for Gemini-Deep-Research (allowed → restricted)
- samsungcloud.com changed access for PanguBot (allowed → restricted)
- samsungcloud.com changed access for GrokBot (allowed → restricted)
- samsungcloud.com changed access for DeepSeekBot (allowed → restricted)
- samsungcloud.com changed access for AI2Bot (allowed → restricted)
- samsungcloud.com changed access for Meta-ExternalFetcher (allowed → restricted)
- samsungcloud.com changed access for Timpibot (allowed → restricted)
- samsungcloud.com changed access for ImagesiftBot (allowed → restricted)
- samsungcloud.com changed access for YouBot (allowed → restricted)
- samsungcloud.com changed access for omgilibot (allowed → restricted)
- samsungcloud.com changed access for omgili (allowed → restricted)
- samsungcloud.com changed access for Diffbot (allowed → restricted)
- samsungcloud.com changed access for cohere-training-data-crawler (allowed → restricted)
- samsungcloud.com changed access for cohere-ai (allowed → restricted)
- samsungcloud.com changed access for Claude-Web (allowed → restricted)
- samsungcloud.com changed access for anthropic-ai (allowed → restricted)
- samsungcloud.com changed access for MistralAI-User (allowed → restricted)
- samsungcloud.com changed access for DuckAssistBot (allowed → restricted)
- samsungcloud.com changed access for Amazonbot (allowed → restricted)
- samsungcloud.com changed access for meta-externalagent (allowed → restricted)
- samsungcloud.com changed access for Bytespider (allowed → restricted)
- samsungcloud.com changed access for CCBot (allowed → restricted)
- samsungcloud.com changed access for Perplexity-User (allowed → restricted)
- samsungcloud.com changed access for PerplexityBot (allowed → restricted)
- samsungcloud.com changed access for Applebot-Extended (allowed → restricted)
- samsungcloud.com changed access for Google-Extended (allowed → restricted)
- samsungcloud.com changed access for Claude-SearchBot (allowed → restricted)
- samsungcloud.com changed access for Claude-User (allowed → restricted)
- samsungcloud.com changed access for ClaudeBot (allowed → restricted)
- samsungcloud.com changed access for OAI-SearchBot (allowed → restricted)
- samsungcloud.com changed access for ChatGPT-User (allowed → restricted)
- samsungcloud.com changed access for GPTBot (allowed → restricted)
- samsungcloud.com changed its default crawler policy (allowed → restricted)
2026-09-12
2026-09-11
- corriere.it began blocking Claude-Web (restricted → blocked)
- corriere.it changed access for PerplexityBot (restricted → allowed)
- corriere.it changed access for OAI-SearchBot (restricted → allowed)
- corriere.it began blocking ChatGPT-User (restricted → blocked)
- ring.com published an llms.txt — a welcome mat for AI readers
- checkpoint.com published an llms.txt — a welcome mat for AI readers
2026-09-10
- hostinger.com began blocking Bytespider (restricted → blocked)
- hostinger.com began blocking CCBot (restricted → blocked)
2026-09-09
- welt.de began blocking Amazonbot (restricted → blocked)
- intel.com published an llms.txt — a welcome mat for AI readers
- t-online.de stopped blocking Google-Extended (blocked → restricted)
2026-09-08
- opera-api2.com added to the index
- xerox.com added to the index
- tbank.ru added to the index
- pccc.com added to the index
- immedia-semi.com added to the index
- dnse2.com added to the index
- bytefcdn.com added to the index
- zenecn.net added to the index
- epam.com added to the index
- ryanair.com added to the index
- dnse0.com added to the index
- usercontent.goog added to the index
- actify.nl added to the index
- usps.com added to the index
- pendo.io added to the index
- bitdefender.net added to the index
- buydomains.com added to the index
- hostinger.com added to the index
- h264.io added to the index
- statsigapi.net added to the index
- lg.com changed access for PerplexityBot (allowed → restricted)
- lg.com changed access for Google-Extended (allowed → restricted)
- lg.com changed access for ClaudeBot (allowed → restricted)
- lg.com changed access for OAI-SearchBot (allowed → restricted)
- lg.com changed access for ChatGPT-User (allowed → restricted)
- lg.com changed access for GPTBot (allowed → restricted)
- intel.com changed access for anthropic-ai (restricted → allowed)
- intel.com changed access for PerplexityBot (restricted → allowed)
- intel.com changed access for Google-Extended (restricted → allowed)
- intel.com changed access for ClaudeBot (restricted → allowed)
- intel.com changed access for ChatGPT-User (restricted → allowed)
- intel.com changed access for GPTBot (restricted → allowed)
- bonappetit.com began blocking YouBot (restricted → blocked)
- epicurious.com began blocking YouBot (restricted → blocked)
- wired.com began blocking YouBot (restricted → blocked)
- newyorker.com began blocking YouBot (restricted → blocked)
2026-09-05
2026-09-04
2026-09-03
2026-09-02
- worldbank.org changed access for Gemini-Deep-Research (restricted → allowed)
- worldbank.org changed access for PanguBot (restricted → allowed)
- worldbank.org changed access for GrokBot (restricted → allowed)
- worldbank.org changed access for DeepSeekBot (restricted → allowed)
- worldbank.org changed access for AI2Bot (restricted → allowed)
- worldbank.org changed access for Meta-ExternalFetcher (restricted → allowed)
- worldbank.org changed access for Timpibot (restricted → allowed)
- worldbank.org changed access for ImagesiftBot (restricted → allowed)
- worldbank.org changed access for YouBot (restricted → allowed)
- worldbank.org changed access for omgilibot (restricted → allowed)
- worldbank.org changed access for omgili (restricted → allowed)
- worldbank.org changed access for Diffbot (restricted → allowed)
- worldbank.org changed access for cohere-training-data-crawler (restricted → allowed)
- worldbank.org changed access for cohere-ai (restricted → allowed)
- worldbank.org changed access for Claude-Web (restricted → allowed)
- worldbank.org changed access for anthropic-ai (restricted → allowed)
- worldbank.org changed access for MistralAI-User (restricted → allowed)
- worldbank.org changed access for DuckAssistBot (restricted → allowed)
- worldbank.org changed access for Amazonbot (restricted → allowed)
- worldbank.org changed access for meta-externalagent (restricted → allowed)
- worldbank.org changed access for Bytespider (restricted → allowed)
- worldbank.org changed access for CCBot (restricted → allowed)
- worldbank.org changed access for Perplexity-User (restricted → allowed)
- worldbank.org changed access for PerplexityBot (restricted → allowed)
- worldbank.org changed access for Applebot-Extended (restricted → allowed)
- worldbank.org changed access for Google-Extended (restricted → allowed)
- worldbank.org changed access for Claude-SearchBot (restricted → allowed)
- worldbank.org changed access for Claude-User (restricted → allowed)
- worldbank.org changed access for ClaudeBot (restricted → allowed)
- worldbank.org changed access for OAI-SearchBot (restricted → allowed)
- worldbank.org changed access for ChatGPT-User (restricted → allowed)
- worldbank.org changed access for GPTBot (restricted → allowed)
- worldbank.org changed its default crawler policy (restricted → allowed)
- semafor.com stopped blocking Gemini-Deep-Research (blocked → restricted)
- semafor.com stopped blocking PanguBot (blocked → restricted)
- semafor.com stopped blocking GrokBot (blocked → restricted)
- semafor.com stopped blocking DeepSeekBot (blocked → restricted)
- semafor.com stopped blocking AI2Bot (blocked → restricted)
- semafor.com stopped blocking Meta-ExternalFetcher (blocked → restricted)
- semafor.com stopped blocking Timpibot (blocked → restricted)
- semafor.com stopped blocking ImagesiftBot (blocked → restricted)
- semafor.com stopped blocking YouBot (blocked → restricted)
- semafor.com stopped blocking omgilibot (blocked → restricted)
- semafor.com stopped blocking omgili (blocked → restricted)
- semafor.com stopped blocking Diffbot (blocked → restricted)
- semafor.com stopped blocking cohere-training-data-crawler (blocked → restricted)
- semafor.com stopped blocking cohere-ai (blocked → restricted)
- semafor.com stopped blocking Claude-Web (blocked → restricted)
- semafor.com stopped blocking anthropic-ai (blocked → restricted)
- semafor.com stopped blocking MistralAI-User (blocked → restricted)
- semafor.com stopped blocking DuckAssistBot (blocked → restricted)
- semafor.com stopped blocking Amazonbot (blocked → restricted)
- semafor.com stopped blocking meta-externalagent (blocked → restricted)
- semafor.com stopped blocking Bytespider (blocked → restricted)
- semafor.com stopped blocking CCBot (blocked → restricted)
- semafor.com stopped blocking Perplexity-User (blocked → restricted)
- semafor.com stopped blocking PerplexityBot (blocked → restricted)
- semafor.com stopped blocking Applebot-Extended (blocked → restricted)
- semafor.com stopped blocking Google-Extended (blocked → restricted)
- semafor.com stopped blocking Claude-SearchBot (blocked → restricted)
- semafor.com stopped blocking Claude-User (blocked → restricted)
- semafor.com stopped blocking ClaudeBot (blocked → restricted)
- semafor.com stopped blocking OAI-SearchBot (blocked → restricted)
- semafor.com stopped blocking ChatGPT-User (blocked → restricted)
2026-09-01
- roblox.com stopped blocking Amazonbot (blocked → restricted)
- roblox.com stopped blocking meta-externalagent (blocked → restricted)
- roblox.com stopped blocking Bytespider (blocked → restricted)
- roblox.com stopped blocking CCBot (blocked → restricted)
- roblox.com stopped blocking Applebot-Extended (blocked → restricted)
- roblox.com stopped blocking Google-Extended (blocked → restricted)
- roblox.com stopped blocking ClaudeBot (blocked → restricted)
- roblox.com stopped blocking GPTBot (blocked → restricted)
- webmd.com stopped blocking ChatGPT-User (blocked → restricted)
2026-08-28
2026-08-27
- stackadapt.com changed access for Gemini-Deep-Research (allowed → restricted)
- stackadapt.com changed access for PanguBot (allowed → restricted)
- stackadapt.com changed access for GrokBot (allowed → restricted)
- stackadapt.com changed access for DeepSeekBot (allowed → restricted)
- stackadapt.com changed access for AI2Bot (allowed → restricted)
- stackadapt.com changed access for Meta-ExternalFetcher (allowed → restricted)
- stackadapt.com changed access for Timpibot (allowed → restricted)
- stackadapt.com changed access for ImagesiftBot (allowed → restricted)
- stackadapt.com changed access for YouBot (allowed → restricted)
- stackadapt.com changed access for omgilibot (allowed → restricted)
- stackadapt.com changed access for omgili (allowed → restricted)
- stackadapt.com changed access for Diffbot (allowed → restricted)
- stackadapt.com changed access for cohere-training-data-crawler (allowed → restricted)
- stackadapt.com changed access for cohere-ai (allowed → restricted)
- stackadapt.com changed access for Claude-Web (allowed → restricted)
- stackadapt.com changed access for anthropic-ai (allowed → restricted)
- stackadapt.com changed access for MistralAI-User (allowed → restricted)
- stackadapt.com changed access for DuckAssistBot (allowed → restricted)
- stackadapt.com changed access for Amazonbot (allowed → restricted)
- stackadapt.com changed access for meta-externalagent (allowed → restricted)
- stackadapt.com changed access for Bytespider (allowed → restricted)
- stackadapt.com changed access for CCBot (allowed → restricted)
- stackadapt.com changed access for Perplexity-User (allowed → restricted)
- stackadapt.com changed access for PerplexityBot (allowed → restricted)
- stackadapt.com changed access for Applebot-Extended (allowed → restricted)
- stackadapt.com changed access for Google-Extended (allowed → restricted)
- stackadapt.com changed access for Claude-SearchBot (allowed → restricted)
- stackadapt.com changed access for Claude-User (allowed → restricted)
- stackadapt.com changed access for ClaudeBot (allowed → restricted)
- stackadapt.com changed access for OAI-SearchBot (allowed → restricted)
- stackadapt.com changed access for ChatGPT-User (allowed → restricted)
- stackadapt.com changed access for GPTBot (allowed → restricted)
- stackadapt.com changed its default crawler policy (allowed → restricted)
- shein.com published an llms.txt — a welcome mat for AI readers
2026-08-26
- semafor.com began blocking Gemini-Deep-Research (restricted → blocked)
- semafor.com began blocking PanguBot (restricted → blocked)
- semafor.com began blocking GrokBot (restricted → blocked)
- semafor.com began blocking DeepSeekBot (restricted → blocked)
- semafor.com began blocking AI2Bot (restricted → blocked)
- semafor.com began blocking Meta-ExternalFetcher (restricted → blocked)
- semafor.com began blocking Timpibot (restricted → blocked)
- semafor.com began blocking ImagesiftBot (restricted → blocked)
- semafor.com began blocking YouBot (restricted → blocked)
- semafor.com began blocking omgilibot (restricted → blocked)
- semafor.com began blocking omgili (restricted → blocked)
- semafor.com began blocking Diffbot (restricted → blocked)
- semafor.com began blocking cohere-training-data-crawler (restricted → blocked)
- semafor.com began blocking cohere-ai (restricted → blocked)
- semafor.com began blocking Claude-Web (restricted → blocked)
- semafor.com began blocking anthropic-ai (restricted → blocked)
- semafor.com began blocking MistralAI-User (restricted → blocked)
- semafor.com began blocking DuckAssistBot (restricted → blocked)
- semafor.com began blocking Amazonbot (restricted → blocked)
- semafor.com began blocking meta-externalagent (restricted → blocked)
- semafor.com began blocking Bytespider (restricted → blocked)
- semafor.com began blocking CCBot (restricted → blocked)
- semafor.com began blocking Perplexity-User (restricted → blocked)
- semafor.com began blocking PerplexityBot (restricted → blocked)
- semafor.com began blocking Applebot-Extended (restricted → blocked)
- semafor.com began blocking Google-Extended (restricted → blocked)
- semafor.com began blocking Claude-SearchBot (restricted → blocked)
- semafor.com began blocking Claude-User (restricted → blocked)
- semafor.com began blocking ClaudeBot (restricted → blocked)
- semafor.com began blocking OAI-SearchBot (restricted → blocked)
- semafor.com began blocking ChatGPT-User (restricted → blocked)
2026-08-25
- Index expanded by 768 domains (Tranco top sites; infrastructure and adult domains excluded)
- Index founded. First crawl: 1020 domains, 32 AI user agents. 32% of readable sites block at least one AI crawler; llms.txt adoption 11.4%; most-blocked bot Bytespider (25%). Daily change tracking starts here.