← all sites
Job-board sites vs AI crawlers
4 tracked job-board sites, 4 with readable policy. Snapshot: 2026-09-13
75%
block ≥1 AI crawler (3/4)
omgili
most-blocked bot here (3 sites)
2
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| ziprecruiter.com | yes | blocked | restricted | restricted | restricted | blocked | blocked |
| glassdoor.com | ? | blocked | blocked | blocked | restricted | blocked | blocked |
| indeed.com | ? | restricted | restricted | restricted | restricted | restricted | restricted |
| monster.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |