Key takeaways
- Run the audit in order: crawl, index, architecture, speed, rendering, markup. A page Google can't crawl doesn't need a faster LCP.
- INP replaced FID as a Core Web Vital in March 2024. "Good" means 200 ms or less at the 75th percentile of real visits.
- Googlebot now fetches only the first 2MB of an HTML file, so bloated pages can lose content from the index.
- Google's Page indexing report in Search Console is the single most useful screen in any site audit. Start there.
- Copy the template table below, sort by priority and fix the "critical" rows before you touch anything cosmetic.
A Melbourne homewares brand we spoke to last year had spent four months rewriting product descriptions and wondered why nothing moved. The answer took ten minutes to find: a theme update had added a noindex tag to every collection page. No copy, however good, ranks from behind a noindex. That is why every engagement we run starts with a technical SEO audit checklist, not a content plan.
This is the checklist we use, ordered the way problems compound, built on free tools where possible, and ending with a template you can copy. Every threshold comes from Google's documentation as of October 2026.
What a technical SEO audit checklist should cover
A technical SEO audit checklist is a structured list of checks that confirms search engines can crawl, render, index and understand your site. It covers robots.txt, XML sitemaps, status codes, canonicals, internal links, Core Web Vitals (LCP, INP, CLS), mobile parity, JavaScript rendering, structured data, HTTPS and hreflang, ranked by business impact.
What it leaves out matters just as much. Keywords, content quality and backlinks belong in separate audits; mixing them into one 200-row spreadsheet is why so many audits never get actioned. Keep this one about plumbing: can Google get in, see what's there, and know which version of each page to show?
Think of it in three layers: access (crawlability), understanding (rendering, canonicals, structured data) and experience (speed, mobile, security). If you pay for professional SEO services, ask which layer each recommendation fixes. A provider who can't answer is probably pasting a tool's default report.
The tools you need for a site audit
You don't need an expensive stack. For most sites under 10,000 URLs, four tools cover 90% of the work:
- Google Search Console. Free, and the only tool that shows what Google itself has crawled and indexed. The Page indexing, URL Inspection, Core Web Vitals and Sitemaps reports do most of the work.
- A crawler. Screaming Frog SEO Spider is the industry default, and its free version crawls up to 500 URLs. Sitebulb, Ahrefs and Semrush site audits are solid alternatives.
- PageSpeed Insights. Shows real-user field data from the Chrome UX Report alongside lab diagnostics from Lighthouse.
- Google's Rich Results Test. Validates structured data and shows the rendered HTML Googlebot sees.
Our opinion: crawler "health scores" are overrated. A site can score 94/100 with its best category canonicalised to the homepage. Use tools to collect data, then judge it yourself.
Phase 1: Crawlability checks
Crawlability is whether Googlebot can reach your URLs at all. Problems here silently cap everything else, so this phase gets checked first and fixed first.
Robots.txt
Open yourdomain.com/robots.txt in a browser. You're looking for three things: it returns a 200 status, it doesn't block anything you want ranked, and it isn't blocking the CSS and JavaScript files Google needs to render your pages.
A few rules from Google's robots.txt documentation catch people out. Google enforces a 500 KiB size limit and ignores anything after it. A 4xx response (other than 429) is treated as if no robots.txt exists. A 5xx response is worse: Google pauses crawling for 12 hours, then falls back to a cached copy for up to 30 days. A server returning 503 for robots.txt can stall crawling of an otherwise healthy site.
Using robots.txt to keep a page out of Google. It blocks crawling, not indexing, so a disallowed URL can still appear without a snippet. To remove a page, allow crawling and use noindex.
XML sitemaps
Your sitemap should list only canonical, indexable URLs that return 200. Not redirects, not 404s, not noindexed pages, not parameter variants. Per Google's sitemap guidelines, a single sitemap is limited to 50MB uncompressed or 50,000 URLs, and Google ignores <priority> and <changefreq> entirely. It does use <lastmod>, but only when the dates are consistently accurate. If your CMS stamps today's date on every URL every night, you've trained Google to ignore the field.
Crawl your sitemap in list mode and filter for anything that isn't a 200.
Status codes and server errors
Export every non-200 internal URL from your crawl. Google's status code documentation confirms 5xx and 429 errors make its crawlers slow down, recovering only gradually. Intermittent 5xx errors in Search Console are a conversation for your host, not your copywriter.
Crawl budget, by contrast, most sites can ignore. Google's crawl budget guide targets sites with over a million unique pages, or 10,000+ pages changing daily. A 60-page dental clinic site in Pune has no crawl budget problem. It might have a crawl access problem, which is different.
Phase 2: Indexing issues and canonicals
Once you know Google can reach your pages, check which ones it actually keeps. Go to Search Console > Indexing > Pages. The Page indexing report groups every URL Google knows about into indexed and not indexed, with a reason for each exclusion.
How to read the "Why pages aren't indexed" table
Not every exclusion is a problem. "Alternate page with proper canonical tag" and "Page with redirect" are usually Google doing exactly what you asked. The reasons that deserve your attention are these:
| Search Console reason | What it usually means | What to do |
|---|---|---|
| Crawled - currently not indexed | Google fetched the page and decided it wasn't worth keeping, often thin or near-duplicate content | Improve or consolidate the page; check it's linked from relevant pages |
| Discovered - currently not indexed | Google knows the URL but hasn't crawled it yet; on big sites this can signal crawl capacity issues | Strengthen internal links; check server response times |
| Duplicate without user-selected canonical | Google found duplicates and you haven't said which one to use | Add a self-referencing or consolidating rel=canonical |
| Duplicate, Google chose different canonical than user | Your canonical tag conflicts with other signals (links, sitemap, redirects) | Align internal links and sitemap with your chosen canonical |
| URL marked 'noindex' | A noindex tag or header is present | Confirm every listed URL is meant to be excluded |
| Soft 404 | The page returns 200 but looks empty or like an error | Return a real 404/410, or add genuine content |
The fastest win in most audits is "URL marked 'noindex'". Scan the list and ask of each URL: did someone mean to do this? Staging settings carried over to live sites cause more ranking crashes than any algorithm update we've seen.
Canonical tags
Every indexable page should carry a self-referencing canonical, and every duplicate (filtered category views, print versions, UTM-tagged URLs) should point at its main version. Google's canonicalisation documentation describes redirects and rel=canonical as strong signals and sitemap inclusion as a weak one, and notes that the signals stack. So if your canonical says one URL but your links and sitemap say another, Google may pick the one you didn't.
Canonicals are less universal than you'd expect: the 2025 Web Almanac found them on 67% of mobile pages, so a third of the web leaves the choice to Google.
Phase 3: Site architecture, internal links and redirects
Architecture is how authority and crawlers flow through your site. The test is simple: can a visitor (or Googlebot) reach every important page within three or four clicks of the homepage, through normal HTML links?
Internal linking and orphan pages
Compare your crawl against your sitemap and your Search Console data. Any URL that appears in the sitemap or gets impressions but isn't found by the crawler is an orphan page: nothing on your site links to it. Orphans are common after site migrations and on e-commerce sites where products are only reachable through on-site search. If you run an online store, our guide to advanced e-commerce SEO covers category linking in more depth.
Also check click depth; key pages should sit within three or four clicks of the homepage. Say you run a UK law firm and your highest-value "commercial lease disputes" page sits five clicks deep under a blog archive. Linking it from the practice area page is often worth more than a new backlink.
Redirect chains and loops
Google's crawlers follow up to 10 redirect hops by default, but every hop costs crawl time and adds latency for users. Our pass criteria: no internal link should point at a redirect, and no redirect should take more than one hop. Google treats a 301 as a strong canonical signal and a 302 as a weak one, so permanent moves should use 301 (or 308).
After a migration, crawl the old URL list and check each one returns a single 301 to its closest equivalent, not the homepage. Blanket homepage redirects tend to be treated as soft 404s.
Phase 4: Core Web Vitals (INP, LCP, CLS) and mobile
If you last audited speed before 2024, your checklist is out of date. On 12 March 2024, Interaction to Next Paint became a stable Core Web Vital, replacing First Input Delay. FID only measured the delay before the browser started handling your first tap. INP measures the responsiveness of interactions across the whole visit, which is much harder to fake.
Here are the current thresholds from web.dev, all assessed at the 75th percentile of real page loads, separately for mobile and desktop:
| Metric | Measures | Good | Needs improvement | Poor |
|---|---|---|---|---|
| Largest Contentful Paint (LCP) | Loading | ≤ 2.5 s | 2.5 s to 4.0 s | > 4.0 s |
| Interaction to Next Paint (INP) | Responsiveness | ≤ 200 ms | 200 ms to 500 ms | > 500 ms |
| Cumulative Layout Shift (CLS) | Visual stability | ≤ 0.1 | 0.1 to 0.25 | > 0.25 |
The web is slowly getting better. The 2025 Web Almanac reports that 48% of mobile websites passed all three metrics, up from 36% in 2023 and 44% in 2024. Desktop sat at 56%. So if your mobile site passes, you're ahead of roughly half the web.
How to check Core Web Vitals properly
Start with the Core Web Vitals report in Search Console. It groups similar URLs, so one slow template shows up as hundreds of pages. Open a representative URL from each group in PageSpeed Insights and read the field data at the top before the Lighthouse score. Lighthouse is a lab simulation; field data is what Google uses.
What we usually fix first: oversized hero images and render-blocking fonts for LCP; third-party scripts (chat widgets, review carousels, a dozen tracking pixels) for INP; images and ad slots without set dimensions for CLS. A Bengaluru SaaS company with a heavy React marketing site will almost always find its INP problems in hydration and tags, not the server.
Keep a sense of proportion. Google's Core Web Vitals documentation says the metrics align with what its ranking systems seek to reward. That doesn't mean a 180 ms INP beats a competitor's 250 ms. Move pages out of "poor" first; perfecting pages that already pass rarely pays back.
Mobile parity
Google indexes and ranks the mobile version of your site. Check that your mobile pages carry the same main content, headings, internal links, structured data and meta robots tags as desktop. Collapsed accordions are fine. Content that only exists on desktop effectively doesn't exist. Our mobile SEO team sees this most often on older sites with separate m-dot versions.
Phase 5: JavaScript rendering
Google processes JavaScript pages in three phases: crawling, rendering and indexing. Rendering can be delayed, and other crawlers (including many AI crawlers) may not run JavaScript at all.
"Google can only discover your links if they are <a> HTML elements with an href attribute."Google Search Central, Understand the JavaScript SEO basics
That line explains a lot of audits. Menus built from <div onclick> leave whole sections undiscoverable. The same page calls server-side rendering or pre-rendering "still a great idea" because not all bots can run JavaScript.
How to check: in URL Inspection, click "Test live URL", then "View tested page", and compare the rendered HTML with your browser's "View source". Body copy, prices or links missing from the raw HTML are findings.
Page weight and the 2MB limit
This one is new enough that many audit templates miss it. Google's Googlebot documentation now states that Googlebot crawls the first 2MB of a supported file type (PDFs get 64MB), and anything beyond the cutoff isn't sent for indexing. Most pages are nowhere near that, but page builders that inline SVGs, base64 images and JSON blobs can get close. Flag any HTML over about 1MB in your crawl.
Phase 6: Structured data, HTTPS and international signals
Structured data
Key templates should carry valid markup: Organization or LocalBusiness on the homepage, Product with offers on product pages, Article on posts, BreadcrumbList sitewide. Test each in the Rich Results Test.
Keep your expectations current. Google's documentation changelog shows a steady pruning of rich result types: several were retired in 2025, and in 2026 Google stopped showing FAQ rich results entirely, removing that documentation in June. Markup still helps machines understand your page, but don't sell a client on FAQ snippets in 2027.
HTTPS and security
Every HTTP variant should 301 to one HTTPS hostname in a single hop, with no mixed content. The 2025 Web Almanac puts HTTPS adoption at about 91.5% of mobile pages, so this is now table stakes.
Hreflang for multi-country sites
If you serve India, the UK and Australia from one domain, check that hreflang annotations are reciprocal, point at 200 canonical URLs and use valid codes (en-GB, not en-UK). Broken hreflang doesn't cause penalties, but it does let the wrong country's prices rank.
Phase 7: AI search and crawler access
Heading into 2027, an audit has to ask a question it didn't three years ago: which AI crawlers can reach your site, and is that deliberate? Check robots.txt, your CDN's bot settings and firewall rules, because some security tools block AI user agents by default.
The distinctions matter. OpenAI documents that OAI-SearchBot surfaces sites in ChatGPT's search features, while GPTBot collects content that may be used for model training. You can block one and allow the other. Decide per bot and make sure your CDN agrees with your robots.txt.
llms.txt is a proposed plain-text file that summarises a site for large language models. In June 2026 Google added a note to its documentation that llms.txt files aren't needed for visibility or rankings in Google Search. It's harmless, but it isn't a technical SEO priority.
Crawler access is only the entry ticket for AI visibility. For what happens after that, read our sister post on how to rank in ChatGPT, Perplexity and Google AI Overviews, or see how our AI search optimisation (GEO) service approaches it.
Free SEO audit template (copy this table)
Select the table, paste it into Google Sheets or Excel, and add "Status" and "Owner" columns. Priority reflects typical impact; adjust for your site.
| Check | Tool | Pass criteria | Priority |
|---|---|---|---|
| Robots.txt returns 200 and blocks nothing important | Browser, Search Console robots.txt report | 200 status; no Disallow on key sections, CSS or JS | Critical |
| No accidental noindex | Search Console Page indexing; crawler | Every noindexed URL is intentionally excluded | Critical |
| Server errors | Search Console; crawler; host logs | No recurring 5xx or 429 responses | Critical |
| Key pages indexed | URL Inspection | Top 20 revenue pages show "URL is on Google" | Critical |
| XML sitemap clean | Search Console Sitemaps; crawler list mode | Only 200, canonical, indexable URLs; under 50,000 URLs/50MB per file | High |
| Canonical tags | Crawler | Self-referencing on indexable pages; duplicates point to the main URL | High |
| Canonical conflicts | Page indexing report | Few or no "Google chose different canonical" URLs | High |
| Internal links to redirects or 404s | Crawler | Zero internal links to 3xx or 4xx URLs | High |
| Redirect chains | Crawler | Every redirect resolves in one hop with 301/308 | High |
| Orphan pages | Crawler + sitemap + Search Console | Every important URL has at least one internal link | High |
| LCP | Search Console CWV report; PageSpeed Insights | ≤ 2.5 s at p75 (field data, mobile) | High |
| INP | Search Console CWV report; PageSpeed Insights | ≤ 200 ms at p75 (field data, mobile) | High |
| CLS | Search Console CWV report; PageSpeed Insights | ≤ 0.1 at p75 | Medium |
| Mobile parity | URL Inspection (smartphone); manual review | Same content, links, schema and meta robots as desktop | High |
| JavaScript rendering | URL Inspection "View tested page" | Main content and links present in rendered HTML; links use <a href> | High |
| HTML size | Crawler | Well under Googlebot's 2MB fetch limit | Medium |
| Structured data | Rich Results Test; Search Console Enhancements | No errors on key templates | Medium |
| HTTPS and hostname | Crawler; browser | All variants 301 to one HTTPS hostname; no mixed content | High |
| Hreflang (if used) | Crawler | Reciprocal, valid codes, 200 canonical targets | Medium |
| Title tags and meta descriptions | Crawler | Unique, present on every indexable page | Medium |
| AI crawler access | Robots.txt; CDN/firewall bot settings | OAI-SearchBot and similar allowed or blocked by deliberate choice | Medium |
A first pass takes about a day on a small site and closer to a week on a 50,000-URL store. For what professional audits cost, see how much SEO costs in 2027.
How to prioritise what you find
An audit with 140 findings and no order is a to-do list nobody finishes. We score each issue on two questions: how many important URLs does it affect, and does it stop crawling or indexing?
- Fix access blockersRobots.txt mistakes, accidental noindex, server errors and broken canonicals.
- Fix template-level issuesOne fix to a product or category template can repair thousands of URLs.
- Clean up link equityInternal links to redirects, redirect chains and orphan pages. Tedious, but compounding.
- Improve experience metricsMove "poor" Core Web Vitals groups up, starting with your highest-traffic templates.
- PolishAlt text, meta descriptions, minor hreflang tidying. Never before steps one to three.
Platform matters too: some five-minute WordPress fixes need an app or workaround on Shopify. Our comparison of Shopify vs WordPress vs Webflow for SEO covers where each platform limits your options, and our web development team handles the fixes that need code.
What to do this week
You don't need to finish the whole template to make progress. Here's a realistic five-day plan for one person:
- Monday: Read every "not indexed" reason in the Page indexing report and export the noindex list.
- Tuesday: Check robots.txt, then crawl your sitemap and remove anything that isn't a 200 canonical URL.
- Wednesday: Run a full crawl and export internal links to redirects, 404s and redirect chains.
- Thursday: Run the worst mobile Core Web Vitals group through PageSpeed Insights and note the top three fixes.
- Friday: Inspect the rendered HTML of your five most valuable pages, then mark progress in the template.
If that week turns up more than you can fix, that's normal. Our small business SEO service exists for exactly that, and our case studies show how we work.