← all sites
Finance sites vs AI crawlers
18 tracked finance sites, 17 with readable policy. Snapshot: 2026-10-04
29.4%
block ≥1 AI crawler (5/17)
Bytespider
most-blocked bot here (4 sites)
5
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| marketwatch.com | ? | restricted | blocked | blocked | blocked | blocked | blocked |
| seekingalpha.com | ? | restricted | blocked | restricted | restricted | blocked | blocked |
| cnbc.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| fool.com | — | restricted | restricted | restricted | restricted | restricted | blocked |
| nerdwallet.com | — | restricted | restricted | blocked | restricted | restricted | restricted |
| americanexpress.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| bankofamerica.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| bankrate.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| capitalone.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| chase.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| coinbase.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| fidelity.com | ? | restricted | restricted | restricted | restricted | restricted | restricted |
| paypal.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| robinhood.com | ? | restricted | restricted | restricted | restricted | restricted | restricted |
| schwab.com | — | unknown | unknown | unknown | unknown | unknown | unknown |
| stripe.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| vanguard.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| wellsfargo.com | — | restricted | restricted | restricted | restricted | restricted | restricted |