What Is a Web Crawler? How Search Engines Find You

A web crawler is a program that follows links around the internet, downloading pages so a search engine can store and understand them. Google’s is called Googlebot. Bing has Bingbot. They are also called spiders, robots or bots, and all four words mean the same thing.

That is the short answer. The longer one matters if you run a website, because almost every “why isn’t my site on Google” problem turns out to be a crawling problem wearing a disguise.

How a crawler actually works

It is less clever than the name suggests. A crawler does four things on a loop.

It starts with a list of URLs it already knows about. It requests each one the way your browser would. It reads the HTML, extracts every link on the page, and adds any it has not seen to the list. Then it moves on and does it again, forever.

That is genuinely it. There is no crawler sitting somewhere deciding your business is worth visiting. It follows links, and if nothing links to a page and it is not in a sitemap, the crawler has no way to learn that the page exists.

This is why internal linking matters more than most people are told. A page nothing links to is a page a crawler has to be told about separately, and a page that links to nothing is a dead end that passes nothing on.

Crawling, indexing and ranking are three different things

These get used interchangeably and they are three separate stages. Almost every diagnosis depends on knowing which one has failed.

• Crawling is the crawler fetching the page. Either it visited or it did not.
• Indexing is the search engine deciding to store the page and what it is about. A page can be crawled and then not indexed, which is extremely common and catches people out.
• Ranking is where that stored page appears when somebody searches. A page can be indexed and rank at position 90, which functionally means invisible.

The practical value of splitting them up is that the fixes are completely different. If a page is never crawled, the problem is links, sitemaps or something blocking access. If it is crawled and not indexed, the problem is usually that the page is thin or duplicates another page. If it is indexed and ranking badly, the problem is competition, and that is a much longer job.

Google Search Console tells you which of the three you are dealing with, and it is free. The URL Inspection tool will say, for any page, whether Google has crawled it, whether it is indexed, and if not, why not.

Crawl budget, and who should ignore it

Crawl budget is the amount of crawling a search engine is willing to spend on your site. It gets discussed constantly and it is irrelevant to most of the people worrying about it.

If your site has a few hundred pages, you do not have a crawl budget problem. Google will happily crawl a small site thoroughly. Reading about crawl budget optimisation and applying it to a twelve-page brochure site is a way of avoiding the real problem.

Where it genuinely matters is large sites, and above all online shops. A store with filtering can generate tens of thousands of URLs from a few hundred products: every combination of colour, size, brand and price band becoming its own crawlable page. The crawler spends its time on near-identical filter pages and your actual product and category pages get visited rarely. New stock can take weeks to appear in search.

That is a real and expensive problem, and it is covered properly on our e-commerce SEO page.

Why crawlers skip pages on small business sites

After a lot of audits, the same causes come up again and again. None of them is exotic.

• Nothing links to the page. The commonest cause by a distance. A page built for a campaign, or added to the CMS but never put in the menu, is invisible unless something points at it.
• robots.txt is blocking it. Usually left over from a staging site. One line in one file can hide an entire section, and nothing warns you.
• A stray noindex tag. Same story: added during a rebuild and never removed. The page is crawled, read, and then explicitly told not to be stored.
• The content needs JavaScript to appear. Google can render JavaScript but it is slower and less reliable than reading HTML. If your main content only exists after a script runs, you are relying on a second pass that may not happen promptly.
• The site is too slow. Crawlers are polite. If your server is struggling, they back off to avoid making it worse, and your crawl rate drops.
• The page is a near-duplicate. Crawled, assessed, and judged not worth storing because three other pages on your site say the same thing. This is very common on sites with a page per town that differ only by the town name.

Sitemaps and robots.txt, briefly

Two files do most of the communicating with crawlers, and both are routinely misunderstood.

An XML sitemap is a list of the URLs you want crawled. It is a hint rather than an instruction, and its real value is on large sites, on new sites with few inbound links, and for pages that are not well linked internally. Including a page in a sitemap does not guarantee indexing, and a sitemap full of URLs that redirect or 404 actively wastes goodwill.

robots.txt tells crawlers where not to go. The critical misunderstanding: it prevents crawling, not indexing. A page blocked in robots.txt can still appear in results if other sites link to it, because Google knows the URL exists and simply cannot see what is on it. If you want a page kept out of search, use a noindex tag and let the crawler in to read it. Blocking it in robots.txt means the crawler never sees the tag telling it to stay away.

AI crawlers are a new category

The crawler landscape changed recently. Alongside the search engine bots, there are now crawlers gathering content for AI assistants: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others.

They behave similarly and they exist for a different purpose. Search crawlers gather pages to rank; AI crawlers gather content that may be summarised or cited when somebody asks a question. For a business that increasingly means being visible in AI answers depends on those crawlers being allowed in and finding content they can use.

You can block them in robots.txt, and some publishers do. For most small businesses that is the wrong call: being cited by an AI assistant is free distribution to somebody actively asking about what you sell. We wrote more on that in how to get your business recommended by ChatGPT, and it is the substance of AI search optimisation.

How to check whether your pages are being crawled

Three checks, all free, in the order worth doing them.

1. Search Console URL Inspection. Paste any page. It tells you whether Google has crawled it, when, whether it is indexed, and if not, the reason. Start here for a specific page.
2. The Pages report in Search Console. Groups every URL by status and reason, so patterns show up. “Discovered, currently not indexed” across a whole section usually means thin content. “Blocked by robots.txt” means exactly what it says.
3. Your server logs, for larger sites. They show what actually visited and when, which is the only way to see whether crawlers are wasting their time on filter URLs.

If those turn up something you cannot interpret, that is what technical SEO is, and the free audit includes it.

The honest summary

For most small business websites, crawling is not the problem. Google finds pages easily on a small, well-linked site.

The problem is nearly always the next stage: pages that get crawled and then not indexed because they duplicate something, or indexed and then ranked at position 70 because there is nothing on them worth ranking. Both feel like “Google can’t find my site” and neither is fixed by anything to do with crawlers.

If that sounds like your situation, why isn’t my website showing up on Google walks through the nine causes in order of how common they actually are.