What a Site Crawler Can and Cannot Catch
Technical audits find a specific, finite class of problem. Knowing exactly where that class ends is what stops you from mistaking a clean report for a healthy site.
A site audit produces a list of issues, sorted by severity, with counts. It looks comprehensive. It is not comprehensive — it is exhaustive within a bounded category, which is a different and much more useful thing to understand.
We maintain a public test site, badseo.dev, where every page breaks exactly one rule on purpose. It exists as the end-to-end fixture for our own audit engine: every issue type the engine can detect is exercised by at least one page, and a harness fails the build if that stops being true. That constraint makes the boundary unusually easy to describe, because the fixture is the boundary.
What a crawler is actually doing
A crawler requests a URL, reads the response, and checks a set of assertions that can be answered from that response alone or from the graph of responses it has already collected. That is the whole mechanism, and it explains both the strengths and the limits.
The assertions fall into a handful of families:
Head tags and headings. Missing title, title too long or too short, missing meta description, meta too long, missing H1, multiple H1s, a skipped heading level. Cheap to check, and genuinely worth checking, because they break silently and nobody notices for months.
Content quality, mechanically defined. Thin content, images with no alt text, duplicate content, duplicate titles, duplicate meta descriptions. Note the qualifier. A crawler can tell you two pages are near-identical. It cannot tell you whether either is any good.
Indexability. noindex in a meta tag, and — the one people miss — noindex in an X-Robots-Tag HTTP header, which never appears in the HTML and so never appears in view-source. Canonical pointing at a different URL, and conflicting canonicals disagreeing with each other.
Status codes and links. 404s, 500s, 403s, and internal links pointing at any of them.
Redirects. Chains, loops, and the trailing-slash trap where two normalisations redirect to each other forever.
Response time. Slow time-to-first-byte, which is a server problem wearing an SEO costume.
Structure. Orphan pages nothing links to, and pages buried so many clicks deep that crawlers deprioritise them.
That list is worth reading twice, because everything on it shares a property: it is decidable. There is a right answer, the crawler can compute it, and you can fix it without a judgement call.
Where the boundary sits
Everything a crawler cannot see follows from the same mechanism.
It cannot tell you whether the page deserves to rank. Duplicate detection compares strings. It has no opinion on whether your unique page is worth reading, and "no issues found" on a thin, unhelpful page means only that the page is technically well-formed.
It cannot tell you whether you targeted the right thing. A page can be flawless and aimed at a search nobody makes, or at the wrong half of your audience entirely. That is a keyword and intent question, and it lives in your Search Console data and your SERP results, not in your HTML.
It cannot see what happens after the response. If your content renders client-side after a fetch, what a crawler reads and what a browser shows can differ substantially. Some crawlers execute JavaScript, some do not, and Google's own rendering is a queue with its own delays.
It cannot tell you what a real crawler experienced. This is the one that catches people. A page that returns 200 to you can return 429 or 403 to an actual bot, because of rate limiting, a firewall rule, or a CDN configuration that treats unfamiliar user agents as suspicious. Your audit crawls politely from one place. Google, and the AI crawlers, arrive differently and much more often. The only way to know is to read your own server logs, which no external audit can do for you.
It cannot prioritise. A hundred missing alt attributes and one redirect loop on your highest-converting page will both appear as counts in a list. Severity labels help, but they rank by class, not by what the page is worth to you.
Reading a report properly
Given all that, the useful order is not top-to-bottom by severity.
Start with anything that removes a page from the index — noindex where you did not intend it, a canonical pointing somewhere unexpected, a redirect loop. These take pages out of search entirely, and one of them on a page that matters outweighs everything else on the list.
Then the status codes, weighted by which pages link to them. A 404 nobody links to is tidy-up. A 404 linked from your homepage is a leak.
Then structure — orphans and deep paths — because those are usually a symptom of a navigation decision rather than a bug, and fixing the decision fixes a class of them at once.
Head tags last. They matter, they are the easiest thing to fix, and they are the reason audit reports look alarming when very little is wrong.
The part the audit was never going to do
The reason to be precise about this boundary is that a clean audit is genuinely reassuring, and the reassurance is narrow. It tells you the machine can read your site. It says nothing about whether the site says anything worth reading, to anyone who was looking for it.
Those are the questions that need your Search Console data, a look at who is actually ranking, and a decision about which audience you meant. The crawler clears the ground so those questions are worth asking. It does not answer them.