← all sites
Real estate sites vs AI crawlers
4 tracked real estate sites, 3 with readable policy. Snapshot: 2026-10-04
66.7%
block ≥1 AI crawler (2/3)
meta-externalagent
most-blocked bot here (2 sites)
0
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| zillow.com | ? | restricted | restricted | restricted | restricted | restricted | restricted |
| realtor.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| apartments.com | ? | unknown | unknown | unknown | unknown | unknown | unknown |
| redfin.com | — | restricted | restricted | restricted | restricted | restricted | restricted |