A growing share of the visits a website gets are AI assistants that read the page to do something on someone’s behalf, such as compare prices or fill in a form. We built Agentic Web Census to find out how ready the web is for them, and we publish it every month with open data.
What Agentic Web Census is
It is a free website that scores the whole web on how ready it is for AI agents. We call the score AWCS, short for Agentic Web Census Score, and it goes from 0 to 100%. We publish it for the web as a whole and also by technology, by domain suffix and by AI agent. Each month is kept in a report you can cite, and the figures can be reused under a CC BY 4.0 licence.
At Omitsis we build websites, and more and more clients ask us what to do about AI agents: whether they should let them in and what they need to change. To answer with data, we had to measure it across the whole web.
Where the data comes from
Lighthouse, Google’s tool for auditing web pages, has a category called Agentic Browsing with audits designed for AI agents.
HTTP Archive runs Lighthouse on millions of websites every month and stores the results in a public database, but its reports do not show this category. We read it from that database for a representative sample of the web.
What the score measures
The score has five dimensions. Four come from Lighthouse audits, and we work out the fifth from the robots.txt file that HTTP Archive collects from each site. The percentages in the image are the share of the whole web that passes each dimension.
- Read (40% of the score). Whether an agent can tell how the page is organised by reading its HTML: the menu, the main content, the buttons and the form fields. It weighs the most because an agent that cannot read the page cannot do anything else with it.
- Stable (25%). Whether the page moves while it loads, measured with CLS (Cumulative Layout Shift). An agent that clicks where it saw a button ends up clicking something else if the page has shifted.
- Guide (20%). Whether the site serves a valid llms.txt, a file that tells AI models what the site is about and which pages matter.
- Act (15%). Whether the site offers WebMCP tools, actions such as searching for a product or asking for a quote that an agent can use without guessing which button to press.
- Reachable (multiplies the rest). How many of the 17 AI agents we track the robots.txt lets in. If it blocks all of them, the score is zero however good everything else is.
Here is how they combine, with the same site in different scenarios:
| Scenario | Score |
|---|---|
| An agent can read it, and nothing else | 40% |
| It can be read and does not move while loading | 65% |
| It can be read, does not move and has a valid llms.txt | 85% |
| It does everything and lets every agent in | 100% |
| It does everything, but robots.txt blocks every agent we track by name | 0% |
The score says nothing about a site’s design or business, and it is not an SEO score. The page what the score measures explains it in full.
How the score has changed
The score of the whole web has gone up every month since we started measuring it, from 30.2% to 34.3%, and the change is larger than the margin of error. What has grown the most is llms.txt. Every series is on the trends page.
The census also publishes the score of each technology, with its five dimensions and its month-by-month trend. On the Drupal page, for example, you can see how sites built with Drupal are doing and compare them with yours.
How to check your site
Besides the census, you can measure a specific site on the same five dimensions.
The score, free and with no account. Enter the address and we give you the score, each dimension compared with the whole web and with the platform you use, and what to fix first. For Reachable, besides reading robots.txt, we request the page the way an agent would, because some servers turn agents away even when robots.txt lets them in.
Once your site has been measured, you can add a badge to it with its score and the date, linking to the result.
The plan, by email. Leave us an address and we send you what to change in each dimension you fail, with an example for each step, and which agents your robots.txt turns away and on which line.
The advanced report, paid. A report written for your site from its own measurements, with the steps in order for a developer to follow and links to the documentation of your platform and your server. Or, if you prefer, one button copies it as plan.md so you can hand it to an AI agent to make the changes.
We do it for you. If you would rather hand it over, we start from what the report already found and apply it. We also make sure your site can handle AI agent traffic, which for many sites is already a load problem.
