← all sites
Social media sites vs AI crawlers
16 tracked social media sites, 16 with readable policy. Snapshot: 2026-08-25
68.8%
block ≥1 AI crawler (11/16)
ClaudeBot
most-blocked bot here (10 sites)
0
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| instagram.com | — | blocked | blocked | blocked | blocked | blocked* | blocked* |
| reddit.com | — | blocked* | blocked* | blocked* | blocked* | blocked* | blocked* |
| threads.com | — | blocked | blocked | blocked | blocked | blocked* | blocked* |
| pinterest.com | — | blocked* | blocked* | blocked* | blocked* | blocked* | blocked* |
| tiktok.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| linkedin.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| x.com | — | blocked* | blocked* | restricted | blocked* | blocked* | blocked* |
| facebook.com | — | restricted | restricted | restricted | restricted | blocked* | blocked* |
| snapchat.com | — | blocked | blocked | blocked | restricted | blocked | blocked |
| medium.com | — | blocked | blocked | restricted* | restricted* | restricted* | blocked |
| tumblr.com | — | restricted* | blocked | blocked | restricted* | blocked | restricted* |
| bsky.app | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| discord.com | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| substack.com | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| twitch.tv | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| youtube.com | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |