Skip to content

Government sites vs AI crawlers

8 tracked government sites, 8 with readable policy. Snapshot: 2026-08-25

0%

block ≥1 AI crawler (0/8)

—

most-blocked bot here

0

publish llms.txt

Site	llms.txt	GPTBot	ClaudeBot	Google-Extended	PerplexityBot	CCBot	Bytespider
europa.eu	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
gov.uk	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
irs.gov	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
nasa.gov	—	allowed	allowed	allowed	allowed	allowed	allowed
sec.gov	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
usa.gov	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
uspto.gov	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
weather.gov	—	allowed	allowed	allowed	allowed	allowed	allowed