← all sites
Finance sites vs AI crawlers
18 tracked finance sites, 16 with readable policy. Snapshot: 2026-11-15
25%
block ≥1 AI crawler (4/16)
ClaudeBot
most-blocked bot here (4 sites)
7
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| marketwatch.com | ? | restricted | blocked | blocked | blocked | blocked | blocked |
| cnbc.com | — | blocked | blocked | blocked | blocked | blocked | blocked |
| seekingalpha.com | yes | blocked | blocked | restricted | restricted | blocked | blocked |
| fool.com | yes | blocked | blocked | restricted | restricted | blocked | blocked |
| americanexpress.com | ? | unknown | unknown | unknown | unknown | unknown | unknown |
| bankofamerica.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| bankrate.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| capitalone.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| chase.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| coinbase.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| fidelity.com | ? | restricted | restricted | restricted | restricted | restricted | restricted |
| nerdwallet.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| paypal.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| robinhood.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| schwab.com | — | unknown | unknown | unknown | unknown | unknown | unknown |
| stripe.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| vanguard.com | — | allowed | allowed | allowed | allowed | allowed | allowed |
| wellsfargo.com | — | restricted | restricted | restricted | restricted | restricted | restricted |