← all sites

Government sites vs AI crawlers

8 tracked government sites, 7 with readable policy. Snapshot: 2026-09-14

0%
block ≥1 AI crawler (0/7)
—
most-blocked bot here
0
publish llms.txt
Sitellms.txtGPTBotClaudeBotGoogle-ExtendedPerplexityBotCCBotBytespider
europa.eu — restrictedrestrictedrestrictedrestrictedrestrictedrestricted
gov.uk — restrictedrestrictedrestrictedrestrictedrestrictedrestricted
irs.gov — restrictedrestrictedrestrictedrestrictedrestrictedrestricted
nasa.gov ? unknownunknownunknownunknownunknownunknown
sec.gov — restrictedrestrictedrestrictedrestrictedrestrictedrestricted
usa.gov — restrictedrestrictedrestrictedrestrictedrestrictedrestricted
uspto.gov — restrictedrestrictedrestrictedrestrictedrestrictedrestricted
weather.gov — allowedallowedallowedallowedallowedallowed