← all sites
Job-board sites vs AI crawlers
4 tracked job-board sites, 4 with readable policy. Snapshot: 2026-08-25
50%
block ≥1 AI crawler (2/4)
GPTBot
most-blocked bot here (2 sites)
2
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| ziprecruiter.com | yes | blocked | restricted | restricted | restricted | blocked | blocked* |
| glassdoor.com | — | blocked | blocked | blocked | restricted* | blocked | blocked |
| indeed.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| monster.com | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |