Of the 200 fully scanned U.S. city sites we tested, 71 (36%) have a robots.txt problem. Another 25 (13%) have no robots.txt file at all. No check fails for that, so they count as passing here. Among states with at least 10 tested sites, Texas has the highest share (51%). Florida has the lowest (18%). A robots.txt file tells search engines and AI agents which parts of a site they may crawl and where to find the sitemap. When a site has one, it has to follow the format and not block the whole site. A site fails if it fails any of these checks: Robots validand Robots allowed.
71 of 200
City sites that have a robots.txt problem
36%
Share of tested city sites that fail these checks
25 of 41
States with at least one failing city site
51%
Texas has the highest share: 25 of 49 tested city sites
By state
The share of tested city sites in each state that have a robots.txt problem.
| State | Tested | robots.txt problem | Share |
|---|---|---|---|
| Arkansas | 1 | 1 | 100% |
| Maine | 1 | 1 | 100% |
| Maryland | 1 | 1 | 100% |
| Georgia | 4 | 3 | 75% |
| Illinois | 4 | 3 | 75% |
| Montana | 3 | 2 | 67% |
| Texas | 49 | 25 | 51% |
| Michigan | 6 | 3 | 50% |
| Virginia | 6 | 3 | 50% |
| Louisiana | 4 | 2 | 50% |
| Tennessee | 4 | 2 | 50% |
| Alabama | 2 | 1 | 50% |
| New Hampshire | 2 | 1 | 50% |
| New York | 2 | 1 | 50% |
| North Dakota | 2 | 1 | 50% |
| Wisconsin | 2 | 1 | 50% |
| California | 30 | 11 | 37% |
| New Mexico | 3 | 1 | 33% |
| South Carolina | 3 | 1 | 33% |
| Missouri | 5 | 1 | 20% |
| Florida | 11 | 2 | 18% |
| Massachusetts | 6 | 1 | 17% |
| North Carolina | 6 | 1 | 17% |
| Ohio | 7 | 1 | 14% |
| Washington | 8 | 1 | 13% |
| Arizona | 5 | 0 | 0% |
| Colorado | 3 | 0 | 0% |
| Connecticut | 1 | 0 | 0% |
| Idaho | 1 | 0 | 0% |
| Indiana | 2 | 0 | 0% |
| Kansas | 0 | 0 | 0% |
| Minnesota | 3 | 0 | 0% |
| Nevada | 2 | 0 | 0% |
| New Jersey | 1 | 0 | 0% |
| Oklahoma | 2 | 0 | 0% |
| Oregon | 2 | 0 | 0% |
| Pennsylvania | 2 | 0 | 0% |
| Rhode Island | 1 | 0 | 0% |
| South Dakota | 1 | 0 | 0% |
| Utah | 1 | 0 | 0% |
| West Virginia | 1 | 0 | 0% |
The 71 city sites that have a robots.txt problem
Sorted by state, then name. Each site links to its botability report and shows its botability and overall grades from the same scan.
| Site | State | Botability | Overall |
|---|---|---|---|
| birminghamal.gov City of Birmingham, AL |
Alabama | C | B |
| fayetteville-ar.gov City of Fayetteville, AR |
Arkansas | B | C |
| elmonteca.gov City of El Monte, CA |
California | B | C |
| escondido.gov City of Escondido, CA |
California | B | C |
| fontanaca.gov City of Fontana, CA |
California | B | C |
| modestogov.com City of Modesto, CA |
California | B | C |
| murrietaca.gov City of Murrieta, CA |
California | B | C |
| cityofpalmdaleca.gov City of Palmdale, CA |
California | B | C |
| rialtoca.gov City of Rialto, CA |
California | B | C |
| richmondca.gov City of Richmond, CA |
California | B | C |
| sanbernardino.gov City of San Bernardino, CA |
California | C | C |
| cityofsanmateo.org City of San Mateo, CA |
California | B | C |
| temeculaca.gov City of Temecula, CA |
California | B | C |
| hialeahfl.gov City of Hialeah, FL |
Florida | B | C |
| miamigardens-fl.gov City of Miami Gardens, FL |
Florida | B | C |
| augustaga.gov City of Augusta, GA |
Georgia | B | C |
| savannahga.gov City of Savannah, GA |
Georgia | B | C |
| cityofsouthfultonga.gov City of South Fulton, GA |
Georgia | B | C |
| elginil.gov City of Elgin, IL |
Illinois | B | C |
| peoriagov.org City of Peoria, IL |
Illinois | B | C |
| rockfordil.gov City of Rockford, IL |
Illinois | C | C |
| brla.gov City of Baton Rouge, LA |
Louisiana | C | D |
| shreveportla.gov City of Shreveport, LA |
Louisiana | B | C |
| augustamaine.gov City of Augusta, ME |
Maine | B | C |
| annapolis.gov City of Annapolis, MD |
Maryland | C | C |
| lowellma.gov City of Lowell, MA |
Massachusetts | B | C |
| eastgrmi.gov City of Grand Rapids, MI |
Michigan | B | C |
| walkermi.gov City of Grand Rapids, MI |
Michigan | B | C |
| sterlingheights.gov City of Sterling Heights, MI |
Michigan | C | C |
| stlouis-mo.gov City of St. Louis, MO |
Missouri | D | A |
| billingsmtpublicworks.gov City of Billings, MT |
Montana | B | C |
| billingsmt.gov City of Billings, MT |
Montana | B | C |
| concordnh.gov City of Concord, NH |
New Hampshire | C | C |
| rrnm.gov City of Rio Rancho, NM |
New Mexico | B | C |
| albanyny.gov City of Albany, NY |
New York | B | B |
| highpointnc.gov City of High Point, NC |
North Carolina | C | C |
| bismarcknd.gov City of Bismarck, ND |
North Dakota | B | C |
| daytonohio.gov City of Dayton, OH |
Ohio | C | C |
| charleston-sc.gov City of Charleston, SC |
South Carolina | B | B |
| clarksvilletn.gov City of Clarksville, TN |
Tennessee | B | C |
| murfreesborotn.gov City of Murfreesboro, TN |
Tennessee | B | C |
| abilenetx.gov City of Abilene, TX |
Texas | B | C |
| baytown.org City of Baytown, TX |
Texas | B | C |
| bellairetx.gov City of Bellaire, TX |
Texas | C | C |
| deerparktx.gov City of Deer Park, TX |
Texas | B | C |
| ci.friendswood.tx.us City of Friendswood, TX |
Texas | B | C |
| friscotexas.gov City of Frisco, TX |
Texas | B | C |
| cityofgalenapark-tx.gov City of Galena Park, TX |
Texas | B | C |
| garlandtx.gov City of Garland, TX |
Texas | B | C |
| southsideplacetx.gov City of Houston, TX |
Texas | B | C |
| killeentexas.gov City of Killeen, TX |
Texas | B | C |
| laportetx.gov City of LaPorte, TX |
Texas | B | C |
| leaguecitytx.gov City of League City, TX |
Texas | B | C |
| midlandtexas.gov City of Midland, TX |
Texas | C | C |
| missouricitytx.gov City of Missouri, TX |
Texas | B | C |
| nassaubay.com City of Nassau Bay, TX |
Texas | B | C |
| newbraunfels.gov City of New Braunfels, TX |
Texas | B | C |
| odessa-tx.gov City of Odessa, TX |
Texas | C | C |
| pasadenatx.gov City of Pasadena, TX |
Texas | B | C |
| seabrooktx.gov City of Seabrook, TX |
Texas | B | C |
| southhoustontx.gov City of South Houston, TX |
Texas | B | C |
| springvalleytx.com City of Spring Valley Village, TX |
Texas | B | C |
| staffordtx.gov City of Stafford, TX |
Texas | B | C |
| sugarlandtx.gov City of Sugar Land, TX |
Texas | B | C |
| wallertexas.gov City of Waller, TX |
Texas | B | C |
| westutx.gov City of West University Place, TX |
Texas | B | C |
| cityofchesapeake.net City of Chesapeake, VA |
Virginia | B | C |
| nnva.gov City of Newport News, VA |
Virginia | B | C |
| norfolk.gov City of Norfolk, VA |
Virginia | C | C |
| spokanevalleywa.gov City of Spokane Valley, WA |
Washington | C | C |
| greenbaywi.gov City of Green Bay, WI |
Wisconsin | B | C |
About these checks
A robots.txt file tells search engines and AI agents which parts of a site they may crawl and where to find the sitemap. When a site has one, it has to follow the format and not block the whole site. These botability checks test each part of that. A site with no robots.txt file passes both checks, since nothing blocks crawling and there is no file to get wrong. Sites whose robots.txt could not be fetched are left out of the share.
- Robots valid. The site's robots.txt file is written correctly so search engines can follow it. Search engines ignore crawling rules they cannot read.
- Robots allowed. The robots.txt file allows search engines and AI agents to crawl the site. Your site is blocked from search results.
Guidance
- Robots valid standard, with how to fix it
- Robots allowed standard, with how to fix it
- The Web Robots Pages
- City botability rankings
About this data
ScanGov scans the homepage of each county site on our county list. Scores come from the checks in the ScanGov standards. Sites were scanned between 9/30/2026 and 11/5/2026. Each site's report shows its own scan date. A site counts as fully scanned when its homepage loaded and sent us its page content. Averages, grade shares, failure rates, and ranks use fully scanned sites only. We could not fully scan 54 sites. They did not respond, blocked the scanner, or sent no page content. They appear in the table without grades, with their scan status, and are left out of every figure. Grades follow the ScanGov grade scale. Rank follows the ScanGov ranking method. The full dataset, with every county's raw results, is in the ScanGov data repository.
12 fully scanned city sites have no result for these checks and are left out of the share.