Canicrawl — AI crawler policy changes https://canicrawl.com/changelog/ Daily-detected changes in which websites allow or block AI crawlers. Index expanded by 768 domains (Tranco top sites; infrastructure and adult domains excluded) https://canicrawl.com/stats/ 2026-08-25|index|panel| Tue, 25 Aug 2026 06:30:00 GMT Index founded. First crawl: 1000 domains, 16 AI user agents. 31.4% of readable sites block at least one AI crawler; llms.txt adoption 10.8%; most-blocked bot CCBot (25.5%). Daily change tracking starts here. https://canicrawl.com/stats/ 2026-08-25|index|founding| Tue, 25 Aug 2026 06:30:00 GMT