← all sites
Social media sites vs AI crawlers
16 tracked social media sites, 15 with readable policy. Snapshot: 2026-10-04
73.3%
block ≥1 AI crawler (11/15)
ClaudeBot
most-blocked bot here (10 sites)
0
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| instagram.com | ? | blocked | blocked | blocked | blocked | blocked | blocked |
| reddit.com | ? | blocked | blocked | blocked | blocked | blocked | blocked |
| threads.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| linkedin.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| pinterest.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| x.com | — | blocked | blocked | restricted | blocked | blocked | blocked |
| facebook.com | — | restricted | restricted | restricted | restricted | blocked | blocked |
| snapchat.com | — | blocked | blocked | blocked | restricted | blocked | blocked |
| tiktok.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| tumblr.com | — | restricted | blocked | blocked | restricted | blocked | restricted |
| medium.com | ? | blocked | blocked | restricted | restricted | restricted | blocked |
| bsky.app | — | restricted | restricted | restricted | restricted | restricted | restricted |
| discord.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| substack.com | ? | unknown | unknown | unknown | unknown | unknown | unknown |
| twitch.tv | — | restricted | restricted | restricted | restricted | restricted | restricted |
| youtube.com | — | restricted | restricted | restricted | restricted | restricted | restricted |