Of the 663 fully scanned U.S. federal sites we tested, 32 (5%) have a robots.txt problem. Another 142 (21%) have no robots.txt file at all. No check fails for that, so they count as passing here. A robots.txt file tells search engines and AI agents which parts of a site they may crawl and where to find the sitemap. When a site has one, it has to follow the format and not block the whole site. A site fails if it fails any of these checks: Robots validand Robots allowed.
32 of 663
Federal sites that have a robots.txt problem
5%
Share of tested federal sites that fail these checks
631
Tested federal sites that pass these checks
688 of 866
Federal sites fully scanned
The 32 federal sites that have a robots.txt problem
Sorted by name. Each site links to its botability report and shows its botability and overall grades from the same scan.
| Site | Botability | Overall |
|---|---|---|
| oversight.gov Council of Inspectors General on Integrity and Efficiency |
F | C |
| wwtg.gov Department of Commerce |
B | B |
| stopfakes.gov Department of Commerce |
B | B |
| cuidadodesalud.gov Department of Health and Human Services |
B | A |
| worklife4you.com Department of Health and Human Services |
F | C |
| healthcare.gov Department of Health and Human Services |
B | A |
| namrs.net Department of Health and Human Services |
C | F |
| dsac.gov Department of Justice |
F | B |
| neapmap.com Department of Labor |
B | B |
| jcmarketplace.com Department of Labor |
B | B |
| lacoast.gov Department of the Interior |
D | F |
| usdebitcard.gov Department of the Treasury |
F | D |
| usds.gov Executive Office of the President |
B | A |
| fedpartnership.gov Federal Reserve Board of Governors |
C | C |
| login.gov General Services Administration |
B | B |
| paymentaccuracy.gov General Services Administration |
D | C |
| fdlp.gov Government Publishing Office |
F | D |
| fhlbcin.com Interagency Collaboration |
F | D |
| globe.gov National Aeronautics and Space Administration |
C | B |
| nrc.gov Nuclear Regulatory Commission |
B | B |
| oakleyca.gov oakleyca.gov |
B | C |
| opm.gov Office of Personnel Management |
B | A |
| pclob.gov Privacy and Civil Liberties Oversight Board |
C | C |
| sahuaritaaz.gov sahuaritaaz.gov |
B | C |
| tvaoig.gov Tennessee Valley Authority |
D | D |
| ustaxcourt.gov U.S. Courts |
B | B |
| nutrition.gov U.S. Department of Agriculture |
B | B |
| lcacommons.gov U.S. Department of Agriculture |
D | C |
| usmc-mccs.org U.S. Marine Corps |
F | D |
| westminsterco.gov City of Westminster, CO |
B | C |
| lexingtonky.gov City of Lexington, KY |
B | B |
| cityofmesquite.com City of Mesquite, TX |
B | C |
About these checks
A robots.txt file tells search engines and AI agents which parts of a site they may crawl and where to find the sitemap. When a site has one, it has to follow the format and not block the whole site. These botability checks test each part of that. A site with no robots.txt file passes both checks, since nothing blocks crawling and there is no file to get wrong. Sites whose robots.txt could not be fetched are left out of the share.
- Robots valid. The site's robots.txt file is written correctly so search engines can follow it. Search engines ignore crawling rules they cannot read.
- Robots allowed. The robots.txt file allows search engines and AI agents to crawl the site. Your site is blocked from search results.
Guidance
- Robots valid standard, with how to fix it
- Robots allowed standard, with how to fix it
- The Web Robots Pages
- Federal botability rankings
About this data
ScanGov scans the homepage of each county site on our county list. Scores come from the checks in the ScanGov standards. Sites were scanned between 9/30/2026 and 11/8/2026. Each site's report shows its own scan date. A site counts as fully scanned when its homepage loaded and sent us its page content. Averages, grade shares, failure rates, and ranks use fully scanned sites only. We could not fully scan 178 sites. They did not respond, blocked the scanner, or sent no page content. They appear in the table without grades, with their scan status, and are left out of every figure. Grades follow the ScanGov grade scale. Rank follows the ScanGov ranking method. The full dataset, with every county's raw results, is in the ScanGov data repository.
25 fully scanned federal sites have no result for these checks and are left out of the share.