Technical SEO Basics: Crawling, Indexing, and Site Health
You can publish brilliant, well-researched articles and still watch them get zero traffic — not because the writing is weak, but because search engines struggle to reach, understand, or trust the pages. Technical SEO is the layer that fixes that. It's the plumbing beneath your content: how easily a search engine can discover your URLs, read them, decide which version to keep, and serve them fast on a phone.
The good news for beginners is that most technical SEO is a handful of concepts you can genuinely understand in an afternoon, plus one free tool (Google Search Console) that tells you exactly what's wrong. This guide walks through crawling and indexing, the controls you use to steer them, and a plain-English site-health checklist you can run yourself.
- Technical SEO is about access, not words. It helps engines crawl, render, index and rank pages your content team already wrote.
- Blocking crawling and noindexing are different tools. One says "don't fetch this," the other says "fetch it but keep it out of results."
- Sitemaps and canonicals guide, they don't force. They help engines find pages and pick the primary version of duplicates.
- Site architecture is crawlability. If a page has no internal links pointing to it, it's practically invisible.
- Search Console is your source of truth. The Pages report and URL Inspection show what Google actually did with each URL.
What technical SEO is (and isn't)
SEO is usually split into three overlapping jobs. Content SEO is about what you say — choosing the right topics and answering intent, which starts with solid keyword research. On-page SEO is how you present that content on a single page — titles, headings, internal links and image alt text, covered in our on-page SEO checklist. Technical SEO is everything that determines whether an engine can even reach and process the page in the first place.
Think of it as the difference between writing a great book, designing its cover, and making sure the library actually shelves it where people can find it. If crawling and indexing fail, the quality of the other two layers is irrelevant — nobody sees the page. That's why technical SEO is worth understanding early: it removes the silent, avoidable reasons good pages never rank.
Content earns the ranking; technical SEO makes sure the page is even eligible to compete for it.
How search engines crawl and index
Before a page can rank, it passes through four stages. Understanding them makes every later decision obvious.
Crawling is discovery. Automated programs — Googlebot and its relatives, described in Google's crawler overview — follow links and read sitemaps to find URLs, then fetch them. Rendering runs the page like a browser would, executing JavaScript so the crawler sees the same content a person does. Indexing is where Google stores and interprets the page: what it's about, its language, whether it duplicates another URL. Only pages that make it into the index are eligible for the final stage, ranking, where the engine picks the best results for a specific query. A page can be crawled but not indexed, so "Google found it" and "Google will show it" are two separate wins.
Robots.txt vs. noindex: two different controls
Beginners constantly confuse these, and the mistake can wipe a page out of Google. They act at different stages of the pipeline.
robots.txt is a file at the root of your site that tells crawlers which paths they may or may not fetch. It controls crawling, and it's mainly for managing crawler load — not for hiding pages. Google's robots.txt introduction is explicit: a disallowed URL can still be indexed if other pages link to it, because Google never fetched the page to see a "keep this out" instruction. Blocking is not hiding.
A meta robots tag (or an X-Robots-Tag HTTP header) with the value noindex is how you keep a page out of results. But there's a catch that trips everyone up: for Google to see noindex, it must be allowed to crawl the page. If you both block it in robots.txt and add a noindex tag, the crawler never fetches the page, never sees the tag, and the noindex is ignored.
robots.txt Disallow stops the crawler from fetching a path. Use it to keep bots out of admin areas, faceted search URLs or infinite calendars — places where crawling wastes budget. It does not reliably remove a page from Google, so never rely on it to hide something.
meta robots noindex keeps a page out of the index while still letting it be crawled. Use it for thank-you pages, thin tag archives or staging content. Leave the page crawlable so Google can actually read the noindex instruction.
| Goal | Use | Watch out for |
|---|---|---|
| Save crawl budget on junk URLs | robots.txt Disallow | Page can still be indexed from links |
| Keep a page out of search results | noindex meta / header | Page must stay crawlable to work |
| Consolidate duplicate URLs | rel="canonical" | It's a hint, not a command |
| Move a page permanently | 301 redirect | Update internal links to the new URL |
Sitemaps and canonical tags
An XML sitemap is a machine-readable list of the URLs you want indexed, with optional signals like when each page last changed. It doesn't guarantee indexing — it's a discovery aid. Google's sitemaps documentation notes they help most on large sites, sites with pages that aren't well linked internally, and new sites with few external links pointing in. For a tidy ten-page site with clean internal links, a sitemap adds little; for a 5,000-URL store, it's essential. Most content platforms generate one automatically at /sitemap.xml.
Canonical tags solve duplicate content. The same page is often reachable through several URLs — with and without a trailing slash, with tracking parameters, printer-friendly versions. A <link rel="canonical"> tag tells Google which URL is the primary one so ranking signals consolidate onto a single version instead of splitting across duplicates. Point every duplicate's canonical at the preferred URL, and make a page's canonical point to itself when it's the original. Remember it's a strong hint, not an order — Google can choose a different canonical if your signals conflict, which is exactly the kind of thing Search Console will tell you.
Site architecture and internal linking
Crawlers discover pages by following links, so your site's link structure is its crawlability. A flat, logical architecture — where any important page is reachable within a few clicks of the homepage — helps engines find and prioritise your content. Deeply buried pages with no internal links pointing to them are "orphans," and orphans rarely get crawled or ranked.
The practical fixes are simple. Group related content into clear sections. Link from strong, established pages to newer ones using descriptive anchor text. Keep your primary navigation pointing to the pages that matter. If you run a location-based business, a well-structured hierarchy also underpins your local SEO, where consistent internal links to service and area pages help both users and crawlers understand your coverage. Internal linking is the cheapest technical-SEO lever there is, and most beginners barely use it.
Core Web Vitals, mobile and HTTPS
Once a page is crawlable and indexable, page experience becomes a tiebreaker. Core Web Vitals are Google's three field metrics for real-world experience, explained on web.dev. You don't need to be an engineer to improve them — start with the biggest offender on each page.
LCP (Largest Contentful Paint) is usually slowed by a heavy hero image or render-blocking scripts — compress images and defer non-essential code. INP (Interaction to Next Paint) measures responsiveness to taps and clicks; keep JavaScript lean. CLS (Cumulative Layout Shift) tracks content jumping around as things load — always set width and height on images and reserve space for ads.
Two more baselines matter. Google uses mobile-first indexing, meaning it predominantly crawls and indexes the mobile version of your site — so if content or links exist only on desktop, they may as well not exist. And HTTPS (a valid SSL certificate) is a basic trust and security signal that's now expected as standard. Serve every page over HTTPS and make sure the mobile experience is complete, not a stripped-down copy.
Using Google Search Console to audit
Google Search Console is the free tool that turns technical SEO from guesswork into diagnosis. Verify your site once, then three features do most of the work.
The Pages report (formerly Coverage) shows how many URLs are indexed versus not, and groups the "not indexed" ones by reason — "Crawled – currently not indexed," "Blocked by robots.txt," "Excluded by noindex tag," "Duplicate without user-selected canonical" and so on. These reasons map directly to the concepts above, so the report is effectively a to-do list. URL Inspection lets you check any single URL: whether it's indexed, which canonical Google chose, when it was last crawled, and whether it's mobile-friendly — and you can request indexing for a new or updated page. Finally, the Sitemaps section is where you submit your sitemap URL and confirm Google is reading it without errors.
Run through this checklist when you audit a site. Your ticks save in this browser, so you can pick up a draft audit later:
- Site verified in Search Console and sitemap submitted successfully
- Pages report reviewed — understand why any URLs are "not indexed"
- robots.txt reviewed; no important pages accidentally disallowed
- No page is both blocked in robots.txt and relying on a noindex tag
- Every page serves over HTTPS with a valid certificate
- Mobile version contains the full content and internal links
- Canonical tags point to the preferred URL for duplicates
- Important pages are reachable within a few internal clicks (no orphans)
- Core Web Vitals checked in Search Console; worst pages prioritised
Common questions
Do I need technical SEO for a small site?
Some of it, yes — but far less than a large site. A small, well-linked site on a modern platform gets crawling, HTTPS and a basic sitemap almost for free. Focus your effort on verifying Search Console, checking the Pages report, and not accidentally blocking or noindexing pages. Skip the enterprise-scale crawl-budget worries.
How long until a new page gets indexed?
It varies from hours to weeks and is never guaranteed. You can nudge it by linking to the new page from existing pages, including it in your sitemap, and using URL Inspection to request indexing. But indexing is Google's decision — thin or duplicate pages may simply stay out of the index.
What's the difference between robots.txt and a noindex tag?
robots.txt controls whether a page is crawled; a noindex tag controls whether a crawled page is kept out of results. Crucially, a page must be crawlable for Google to see its noindex tag, so never block a page in robots.txt and expect a noindex on it to work.
Is technical SEO a one-time job?
No. Sites change — new sections, redesigns, migrations and plugins can all introduce crawling or indexing problems. Treat it as a periodic health check: skim the Pages report monthly, and audit thoroughly after any big structural change.
Where to go next
Technical SEO clears the path; content and on-page work fill it. With crawling and indexing sorted, point that healthy foundation at the right topics using our keyword research guide, tighten each individual page with the on-page SEO checklist, and if you serve a specific area, apply the local SEO guide so nearby searchers can find you. Do the technical groundwork once, keep an eye on Search Console, and the rest of your SEO work finally gets a fair chance to compound.