Of the 41 fully scanned U.S. state sites we tested, 0 (0%) have a robots.txt problem. Another 8 (20%) have no robots.txt file at all. No check fails for that, so they count as passing here. A robots.txt file tells search engines and AI agents which parts of a site they may crawl and where to find the sitemap. When a site has one, it has to follow the format and not block the whole site. A site fails if it fails any of these checks: Robots validand Robots allowed.
0 of 41
State sites that have a robots.txt problem
0%
Share of tested state sites that fail these checks
41
Tested state sites that pass these checks
42 of 51
State sites fully scanned
The 0 state sites that have a robots.txt problem
Sorted by name. Each site links to its botability report and shows its botability and overall grades from the same scan.
| Site | Botability | Overall |
|---|
About these checks
A robots.txt file tells search engines and AI agents which parts of a site they may crawl and where to find the sitemap. When a site has one, it has to follow the format and not block the whole site. These botability checks test each part of that. A site with no robots.txt file passes both checks, since nothing blocks crawling and there is no file to get wrong. Sites whose robots.txt could not be fetched are left out of the share.
- Robots valid. The site's robots.txt file is written correctly so search engines can follow it. Search engines ignore crawling rules they cannot read.
- Robots allowed. The robots.txt file allows search engines and AI agents to crawl the site. Your site is blocked from search results.
Guidance
- Robots valid standard, with how to fix it
- Robots allowed standard, with how to fix it
- The Web Robots Pages
- State botability rankings
About this data
ScanGov scans the homepage of each county site on our county list. Scores come from the checks in the ScanGov standards. Sites were scanned between 10/2/2026 and 11/6/2026. Each site's report shows its own scan date. A site counts as fully scanned when its homepage loaded and sent us its page content. Averages, grade shares, failure rates, and ranks use fully scanned sites only. We could not fully scan 9 sites. They did not respond, blocked the scanner, or sent no page content. They appear in the table without grades, with their scan status, and are left out of every figure. Grades follow the ScanGov grade scale. Rank follows the ScanGov ranking method. The full dataset, with every county's raw results, is in the ScanGov data repository.
1 fully scanned state site has no result for these checks and is left out of the share.