Skip to content
MangoGrow

Marketing glossary

What is Crawl Budget?

Crawl Budget: How many pages a search engine is willing and able to crawl on your site in a given period.

Crawl budget mainly matters for large sites with thousands of URLs. You protect it by avoiding endless URL variations, keeping sitemaps clean, fixing broken links and redirect chains, and making important pages easy to reach through internal links.

Why it matters

For very large sites, crawl budget affects how quickly new and updated pages are discovered. For most small business sites it isn’t a concern, but wasted crawling on duplicate or low-value URLs can still slow discovery of important pages.

Example

An online store with thousands of filter combinations finds Google spending time on endless filtered URLs. Limiting crawlable filter links and keeping the sitemap focused on real product and category pages helps new products get indexed faster. (Illustrative.)

Common mistakes

  • Worrying about crawl budget on a small site instead of content quality
  • Allowing infinite URL variations from filters and parameters
  • Listing redirects, errors and noindexed pages in sitemaps
  • Slow servers that limit how much Google can crawl

← Back to the full glossary