Of the 2,629 fully scanned U.S. education sites we tested, 95 (4%) have a robots.txt problem. Another 191 (7%) have no robots.txt file at all. No check fails for that, so they count as passing here. A robots.txt file tells search engines and AI agents which parts of a site they may crawl and where to find the sitemap. When a site has one, it has to follow the format and not block the whole site. A site fails if it fails any of these checks: Robots validand Robots allowed.
95 of 2,629
Education sites that have a robots.txt problem
4%
Share of tested education sites that fail these checks
2,534
Tested education sites that pass these checks
2,687 of 2,991
Education sites fully scanned
The 95 education sites that have a robots.txt problem
Sorted by name. Each site links to its botability report and shows its botability and overall grades from the same scan.
| Site | Botability | Overall |
|---|---|---|
| alamo.edu alamo.edu |
C | B |
| alvincollege.edu Alvin Community College |
B | B |
| aspiracitycollege.edu ASPIRA City College |
B | B |
| aum.edu Auburn University at Montgomery |
B | B |
| bakersfieldcollege.edu Bakersfield College |
A | A |
| barry.edu Barry University |
B | B |
| heichaldovid.org Beis Medrash Heichal Dovid |
D | C |
| belhaven.edu Belhaven University |
A | B |
| broward.edu Broward College |
B | C |
| cameron.edu Cameron University |
A | B |
| cnm.edu Central New Mexico Community College |
C | B |
| cocc.edu Central Oregon Community College |
B | A |
| cerrocoso.edu Cerro Coso Community College |
A | B |
| chaffey.edu Chaffey College |
A | B |
| champlain.edu Champlain College |
B | A |
| chapman.edu Chapman University |
B | B |
| ccms.edu Cincinnati College of Mortuary Science |
A | B |
| crcds.edu Colgate Rochester Crozer Divinity School |
C | C |
| collegeofsanmateo.edu College of San Mateo |
B | C |
| cscc.edu Columbus State Community College |
A | A |
| cornellcollege.edu Cornell College |
A | D |
| cps.unh.edu cps.unh.edu |
B | B |
| cuesta.edu Cuesta College |
B | B |
| dakotacollege.edu Dakota College at Bottineau |
B | B |
| home.dartmouth.edu Dartmouth College |
B | B |
| dvc.edu Diablo Valley College |
B | C |
| dillard.edu Dillard University |
A | B |
| edisonohio.edu Edison State Community College |
B | B |
| endicott.edu Endicott College |
B | B |
| ftscc.org Faith Theological Seminary and Christian College |
B | B |
| flcc.edu Finger Lakes Community College |
A | A |
| fscj.edu Florida State College at Jacksonville |
D | C |
| fvtc.edu Fox Valley Technical College |
B | B |
| franklinpierce.edu Franklin Pierce University |
B | B |
| hmsom.edu Hackensack Meridian School of Medicine |
C | C |
| hpu.edu Hawaii Pacific University |
C | D |
| hillcollege.edu Hill College |
A | C |
| hiu.edu Hope International University |
B | D |
| hgtc.edu Horry-Georgetown Technical College |
B | B |
| jscc.edu Jackson State Community College |
D | B |
| jacksonvillecollege.edu Jacksonville College-Main Campus |
B | C |
| sunyjefferson.edu Jefferson Community College |
A | B |
| jcjc.edu Jones County Junior College |
D | C |
| lecom.edu Lake Erie College of Osteopathic Medicine |
B | A |
| lsc.edu Lake Superior College |
A | C |
| lyon.edu Lyon College |
C | C |
| marist.edu Marist University |
B | B |
| mhu.edu Mars Hill University |
C | C |
| maryville.edu Maryville University of Saint Louis |
D | B |
| msudenver.edu Metropolitan State University of Denver |
A | A |
| midmich.edu Mid Michigan College |
B | B |
| macu.edu Mid-America Christian University |
B | B |
| mvsu.edu Mississippi Valley State University |
F | C |
| moreheadstate.edu Morehead State University |
B | B |
| ncc.edu Nassau Community College |
C | C |
| nwic.edu Northwest Indian College |
B | B |
| occc.edu Oklahoma City Community College |
C | C |
| occ.edu Ozark Christian College |
B | A |
| piu.edu Pacific Islands University |
B | B |
| pacificoaks.edu Pacific Oaks College |
B | B |
| ptseminary.edu Pentecostal Theological Seminary |
B | D |
| plymouth.edu Plymouth State University |
D | C |
| providencecc.edu Providence Christian College |
D | D |
| rabbinicalcollegeboboveryeshiva.edu Rabbinical College Bobover Yeshiva Bnei Zion |
B | A |
| rabbinical.org Rabbinical Seminary of America |
B | B |
| rcc.mass.edu Roxbury Community College |
C | C |
| steu.edu Saint Elizabeth University |
B | C |
| saybrook.edu Saybrook University |
B | B |
| se-pr.edu Seminario Evangelico de Puerto Rico |
D | D |
| southeasttech.edu Southeast Technical College |
C | C |
| sco.edu Southern College of Optometry |
D | C |
| swcciowa.edu Southwestern Community College |
D | B |
| swosu.edu Southwestern Oklahoma State University |
B | C |
| sjf.edu St. John Fisher University |
A | B |
| newpaltz.edu State University of New York at New Paltz |
D | C |
| sunyopt.edu SUNY College of Optometry |
A | B |
| taylor.edu Taylor University |
B | B |
| tntech.edu Tennessee Technological University |
B | B |
| twu.edu Texas Woman's University |
B | B |
| thechicagoschool.edu The Chicago School |
B | B |
| collegesoflaw.edu The Colleges of Law |
C | A |
| uah.edu University of Alabama in Huntsville |
A | C |
| uaa.alaska.edu University of Alaska Anchorage |
C | D |
| manchester.unh.edu University of New Hampshire at Manchester |
C | B |
| law.unh.edu University of New Hampshire-Franklin Pierce School of Law |
C | B |
| uncw.edu University of North Carolina Wilmington |
D | B |
| sc.edu University of South Carolina-Columbia |
B | B |
| uopeople.edu University of the People |
C | C |
| usnh.edu University System of New Hampshire System Office |
B | B |
| valenciacollege.edu Valencia College |
B | B |
| vinu.edu Vincennes University |
A | B |
| ws.edu Walters State Community College |
A | B |
| wgu.edu Western Governors University |
A | A |
| mooretech.edu William R Moore College of Technology |
C | D |
| ysu.edu Youngstown State University |
D | C |
About these checks
A robots.txt file tells search engines and AI agents which parts of a site they may crawl and where to find the sitemap. When a site has one, it has to follow the format and not block the whole site. These botability checks test each part of that. A site with no robots.txt file passes both checks, since nothing blocks crawling and there is no file to get wrong. Sites whose robots.txt could not be fetched are left out of the share.
- Robots valid. The site's robots.txt file is written correctly so search engines can follow it. Search engines ignore crawling rules they cannot read.
- Robots allowed. The robots.txt file allows search engines and AI agents to crawl the site. Your site is blocked from search results.
Guidance
- Robots valid standard, with how to fix it
- Robots allowed standard, with how to fix it
- The Web Robots Pages
- Education botability rankings
About this data
ScanGov scans the homepage of each county site on our county list. Scores come from the checks in the ScanGov standards. Sites were scanned between 9/30/2026 and 11/8/2026. Each site's report shows its own scan date. A site counts as fully scanned when its homepage loaded and sent us its page content. Averages, grade shares, failure rates, and ranks use fully scanned sites only. We could not fully scan 304 sites. They did not respond, blocked the scanner, or sent no page content. They appear in the table without grades, with their scan status, and are left out of every figure. Grades follow the ScanGov grade scale. Rank follows the ScanGov ranking method. The full dataset, with every county's raw results, is in the ScanGov data repository.
58 fully scanned education sites have no result for these checks and are left out of the share.