Can AI agents use your website?

A new kind of visitor

For thirty years a website had exactly one kind of visitor that mattered: a person, in a browser, looking at rendered pages. Everything about the modern web — the JavaScript bundles, the cookie banners, the visual bot challenges — was built for that visitor. AI agents break the assumption. An agent working for a user typically fetches raw HTML without executing JavaScript, identifies itself honestly with a user agent like GPTBot or ClaudeBot, reads the machine-facing files a site publishes (robots.txt, sitemaps, llms.txt, OpenAPI specifications), and decides within a few fetches whether this site can be understood and used at all. Sites that fail that test are not punished loudly; the agent simply goes elsewhere, and the visit never appears in any analytics dashboard you own.

This matters now, rather than eventually, because agents are becoming a real channel: assistants that research purchases, agents that book and buy, support agents that read your documentation to answer customers, crawlers that build the indexes those systems answer from. Every one of them grades your site on machine legibility, and most sites were never designed to be read that way. The gap is rarely malice or neglect — it is that the machine-facing surface (the crawl rules, the raw HTML, the API descriptions) grew by accident while the human-facing surface got all the design attention. Agent readiness is the discipline of closing that gap on purpose.

The four layers of agent readiness

The checker scores four layers, in the order an agent experiences them. Discovery (20 points) is whether the agent can find the site’s map: a parseable robots.txt and what it tells the major AI crawlers, a reachable sitemap, an llms.txt file, structured data, a canonical URL that points at itself. Access (30 points) is whether the agent can get in once it arrives: pages that answer a plain HTTP client, no refusal of AI user agents, no bot-challenge interstitial, real content in the raw HTML rather than an empty shell waiting for JavaScript, a clean redirect to HTTPS, and honest status codes — a true 404 for what does not exist, because an agent that cannot tell a missing page from a real one cannot trust anything else the site says.

Usability (40 points) is the largest layer, because arriving is not the same as transacting. It asks whether the site exposes machinery an agent can act through: an OpenAPI specification at the paths agents look, documentation a machine can find, an MCP endpoint that answers a protocol probe, authentication discovery metadata when there is an API to log into, and the humble web forms — signup, contact, lead — reachable without a challenge and labelled well enough to fill in correctly. Payments (10 points) asks whether an agent could pay: a discovery file for one of the emerging agent-commerce protocols (ACP, UCP, AP2, x402) and a checkout that does not slam a challenge in the agent’s face. Payments is scored only when a site actually sells something; on a site with no commerce signals the whole layer is not applicable and the score is rescaled without it.

How to read a score

The score runs 0 to 100, with letter grades A from 85, B from 70, C from 55, D from 40 and F below — and 70, the B threshold, is where the category convention puts the “agent-ready” line. Treat the number as a summary, not the product. The product is the findings list: every check that failed or warned, the evidence collected for it, what it means, the fix, and the points the fix recovers. The “fix first” ordering is just that list sorted by recoverable points, which is almost never the order a generic checklist would give you — a missing llms.txt might be worth more than three cosmetic warnings combined, and the sort is what tells you.

Two reading habits keep the number honest. First, respect the not-applicable mechanism: checks and whole layers drop out when their precondition is absent, and the score renormalises, so a documentation site is not being graded on its checkout. A score is only meaningful against the layers that applied. Second, remember what a verdict means: “not found” is always “not found at the paths an agent would try, from the scanner’s network, at scan time”. Agents try the conventional paths — that is the point of the exercise — but a scan is a sample of the machine-facing surface, not an audit of everything a determined agent could eventually discover with a human coaching it.

The failures that cost the most, and their fixes

The same handful of failures accounts for most lost points across real sites. AI crawlers blocked by default: a robots.txt copied from a template, or a CDN security preset, refuses GPTBot, ClaudeBot and friends outright — sometimes the opposite of what the site owner intends, since the same crawlers feed the answer engines the owner wants to appear in. The fix is a deliberate robots policy: decide per crawler, write it down, keep the sitemap allowed. The empty shell: the homepage returns a JavaScript bundle and a spinner, with the actual content rendered only client-side; humans never notice, agents see an empty page. Server-render or pre-render the content that matters. No machine map: no sitemap, no llms.txt, no structured data — each cheap to add, each a separate finding until you do.

The second handful is about acting rather than reading. Challenge-first security: a bot wall that challenges every unknown client also challenges every agent; the fix is to challenge on behaviour and risk signals rather than on unfamiliarity alone, and to keep the public, read-only surface reachable. Undocumented or undiscoverable APIs: the API exists, but there is no OpenAPI spec at a findable path, so every agent integration becomes a bespoke guessing game; publishing the spec is a one-time export in most frameworks. Forms an agent cannot fill: unlabelled fields, placeholder-only hints, and submit flows that require solving a visual puzzle for a routine contact form. None of these fixes requires redesigning the human site — the machine-facing layer is almost entirely additive, which is why the points are there for the taking.

Case study: this site, scanned

One disclosure before the number. sysosx.com itself cannot be scanned with the public checker above: like every sysosx check, the endpoint refuses sysosx.com as a blocked target, by design, so the tool cannot be pointed at its own infrastructure. The case study below was therefore produced differently — the same scanning engine, with the same checks and scoring, run locally against the live sysosx.com site. The score is real engine output, not an estimate; only the vantage point differs.

Where the points went: the top findings

Frequently asked questions