Skip to content

Finance sites vs AI crawlers

18 tracked finance sites, 17 with readable policy. Snapshot: 2026-08-25

29.4%

block ≥1 AI crawler (5/17)

Bytespider

most-blocked bot here (4 sites)

5

publish llms.txt

Site	llms.txt	GPTBot	ClaudeBot	Google-Extended	PerplexityBot	CCBot	Bytespider
marketwatch.com	—	restricted	blocked^*	blocked^*	blocked^*	blocked^*	blocked^*
cnbc.com	—	blocked	blocked	blocked	blocked	blocked	blocked
seekingalpha.com	—	restricted^*	blocked	restricted^*	blocked	blocked	blocked
fool.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	blocked
nerdwallet.com	—	restricted^*	restricted^*	blocked	restricted^*	restricted^*	restricted^*
americanexpress.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
bankofamerica.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
bankrate.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
capitalone.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
chase.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
coinbase.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
fidelity.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
paypal.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
robinhood.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
schwab.com	—	unknown	unknown	unknown	unknown	unknown	unknown
stripe.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
vanguard.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
wellsfargo.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*