How search works now, and what AI changed

You can use an AI assistant well on SEO only if you know what SEO is trying to do. This chapter is the short version: how a page gets into search results, what AI answers changed, and where an assistant helps and where it makes things up.

Crawl, index, rank

Crawl. A search engine runs a program, called a crawler, that fetches pages by following links and reading sitemaps. If a page is blocked, broken or only reachable through a script, the crawler never sees it.

Index. The engine reads each page it fetched and stores what it found: the text, the title, the images, the links, the structured data. A page can be crawled and still not be indexed, for example because it is a duplicate of another page or marked noindex.

Rank. When someone searches, the engine picks pages from the index and orders them. Hundreds of signals feed that order. Nobody outside the search engine knows the weights, so be wary of anyone who gives you an exact formula.

A problem at one stage hides everything after it. A page that cannot be crawled cannot be indexed, and a page that is not indexed cannot rank. That is why audits start with crawl and index problems, and why a perfect title on a blocked page does nothing.

Google's own overview of this, how Search works, is worth reading once. It is short.

What AI changed

Two things now sit next to the list of links. Google shows AI Overviews and AI Mode, which write an answer and link to pages. Chat assistants such as ChatGPT, Claude and Gemini answer in a conversation, and some of them search the web and cite sources. A client's customers use all of these.

What changed is where the reader meets your client's content. Sometimes it is a link in a list, and sometimes it is a sentence in an answer. What did not change is the entry requirement. Google says a page has to be indexed and eligible to appear in Search with a snippet to be used in its AI features, and that there are no extra requirements beyond the normal ones. Chapter 8 covers that in detail.

The checks that follow in this guide are the same ones that mattered before assistants existed: clear titles, one heading outline, working links, one version of each URL, a sitemap, fast pages. Most of them are things Google's own documentation covers.

Where an assistant helps, and the three things it is bad at

An assistant is good at drafting and reshaping text to rules you give it: titles, descriptions, alt text, an outline, JSON-LD, a redirect rule, a client email. It is good at turning a vague finding into a concrete change for a stack you name. It is fast at doing that fifty times in a row.

It is bad at three things, and every chapter works around them:

1. It invents facts. Prices, opening hours, search volumes, awards, years in business. It will write them confidently. Tell it to use only facts you supply, and read what it returns.
2. It cannot see your live site. Unless it has a browsing feature turned on, it only knows what you paste. It does not know your redirects, your plugins or your server. Paste the real finding and name the stack.
3. It cannot check its own work. It will say a title is 58 characters when it is 64. Count with a tool, and re-run the audit. The audit is the judge, not the assistant.

A crawler up close

A crawler is easier to understand when you have the rules of a real one. Bluprent's audit crawler, BluprentSEOBot, reads robots.txt first, makes at most two requests at a time, waits half a second between request starts, and gives up on a page after 10 seconds. It audits 50 pages by default and never more than 200 per run, so on a large site it audits a sample, not everything. Search engines run on the same ideas at a far larger scale: obey robots.txt, be polite, give up on slow pages.

Don't bother with

• The meta keywords tag. Google has said for years that it ignores it. Leave it out.
• Submitting the site to hundreds of search engines. Sitemaps and Search Console cover the ones that matter, and the paid "submission" services add nothing.
• "Google penalises AI content." Not as such. Google's guidance, last updated 1 October 2026, says the problem is mass-producing pages that add no value, which it calls scaled content abuse, and that you should fact-check what a model writes. Read its page on generative AI content.

Next

• Chapter 2: Set up the audit, prompt, fix, re-audit loop
• All 44 checks and how the score works
• Google's SEO Starter Guide (Google Search Central)