AI Agents Read the Web Like Screen Readers Do

When an AI agent browses the web for you, it usually isn't looking at the page. It's reading the same structured representation of it that a screen reader uses — the accessibility tree. The web spent two decades building that hidden layer for blind users; agents inherited it, and now the two groups depend on the same fragile contract.

That single fact explains a lot of the agent experience you've had: why an agent can book a flight on one airline and fail on another, why a "simple" click sometimes lands nowhere, and why every serious browser is now racing to make the accessibility layer more explicit.

What the accessibility tree actually is

A web page is markup plus styling plus JavaScript. None of that is a description of what the page means. The accessibility tree is the browser's translation: an ordered structure of elements with a role ("button", "textbox", "heading"), an accessible name ("Search flights"), and state (checked, expanded, disabled).

Screen readers have driven that tree for years — NVDA, JAWS, and VoiceOver all navigate it. It exists only because accessibility standards require it, and it is only as complete as the developer who built the page made it. A <div> styled to look like a button is invisible to it. A <button> is not.

Why agents ended up there

Agents had three options for "seeing" a page: pixels, raw HTML, or the accessibility tree.

Raw HTML is enormous and mostly noise — a modern page is tens of thousands of lines of markup wrapped around forty interactive elements. Pixels are expensive to interpret and give no guarantee of what's clickable. The accessibility tree is the only one of the three that is already a list of the interactive things with names attached. For an agent trying to decide what to click, it is the obvious substrate, and this is now widely described as agents reading pages through the accessibility layer — the same layer that powered screen readers for two decades.

The standards work followed. Chrome's WebMCP preview gives sites a way to expose their functions as tools an agent can call directly, instead of guessing through the tree — which we covered when it first started reaching signed-in browsers.

The failure mode both groups share

Here is the uncomfortable part: the same sites break for both. A custom dropdown built from divs, an icon-only button with no accessible name, a modal that traps focus incorrectly — these are classic accessibility bugs that blind users have reported for a decade. To an agent, they look identical: an element that should be interactable has no name, no role, or no state.

Which reframes a common complaint. When your agent gets stuck on a site, "the agent is dumb" is often wrong. Frequently the site is inaccessible and the agent is the second party to notice. Agent blocking gets blamed on bot detection far more often than on this, but broken semantics is the quieter half of the problem.

Where the interests diverge

This is where it stops being a neat parallel. Researchers testing computer-use agents specifically with blind users have found that the agents fail on the same desktop applications those users struggle with — the study literally asks whether computer-use agents get blind users any further than a screen reader alone, and the answer is mixed rather than automatic. An agent that stumbles on an unlabeled control is not a solution for someone who already had to work around it.

Meanwhile the two groups are heading in opposite directions. Agents are getting more ways to see — screenshot interpretation, vision models, coordinate clicking — which lets them brute-force past poor semantics. Screen readers are not. A blind user cannot fall back to "just look at the pixels." So the accessibility tree may end up being something agents tolerate rather than need, while remaining the only path for the people it was built for.

That asymmetry is the thing to watch. If agents outgrow the accessibility tree, the market pressure to fix it weakens — and the layer blind users depend on gets maintained by regulation alone.

What to do with this

Three practical moves:

• Prefer semantically solid sites when you can choose. Booking portals, banks, and retailers with real accessible markup are the ones your agent will operate reliably. It's a decent proxy, and it costs nothing to notice.
• Test with the tree, not the screenshot. When an agent fails somewhere, check whether the target element has an accessible name and role. It tells you instantly whether you're looking at an agent limitation or a broken page.
• Judge agent demos by the sites they skip. A browser-agent vendor showing ten clean flows is showing you ten pages that had good semantics. The interesting question is what happens on the eleventh.

And if you build for the web: the argument for correct markup just doubled. Accessibility was already the right thing to do and a legal requirement in many jurisdictions. It's now also the difference between whether software agents can use your product at all — a constituency that is growing fast and does not complain in your support inbox.

Takeaway

Agents and blind users navigate the same invisible layer of the web, and it fails them in the same places. That's a useful diagnostic — an agent that breaks is often revealing a page that was already broken — and a warning: as agents gain pixel-level fallbacks, the pressure to fix accessibility may come only from the people who still need it. Accessibility stopped being a niche concern the moment software started reading the web on everyone's behalf.

The accessibility tree: what agents and screen readers both navigate