How Search Engines Work
SEO without understanding retrieval systems is superstition. Modern search engines run a three-stage pipeline — crawling (discovery), indexing (storage and analysis), ranking (retrieval and ordering) — refined by machine-learned relevance systems that evaluate hundreds of signals per query.
Key Concepts
- Crawling: bots follow links and sitemaps to discover URLs; crawl budget limits how much of large sites gets visited
- Indexing: parsed content enters the inverted index — the word-to-document map enabling instant lookup
- Rendering matters: JavaScript-dependent content may be crawled but indexed late or incompletely
- Ranking: query understanding (spelling, synonyms, intent) → candidate retrieval → ML-scored ordering
- Core signal families: relevance (content-query match), authority (links, brand), user context (location, device, language)
- robots.txt requests non-crawling; noindex directives request non-indexing — different tools, commonly confused
- Search features beyond blue links: featured snippets, People Also Ask, local packs, image/video verticals
In Practice
The mental model to keep: you are not 'pleasing Google' — you are making documents easy to discover, unambiguous to interpret and demonstrably worthy of citation. Everything in the next nine lessons operationalises those three verbs.
Try It Yourself
Run the pipeline on your own (or any) site: fetch robots.txt, locate the sitemap, run site:domain.com to see indexed pages, and inspect one page's source for its title and meta description. Write one line on each stage's health.