Skip to content
MangoGrow

Marketing glossary

What is robots.txt?

robots.txt: A text file at the root of a website that tells crawlers which parts of the site they may or may not access.

robots.txt controls crawling, not indexing: a blocked page can still appear in results if other sites link to it. Businesses that want AI visibility generally allow AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot to access public content.

Why it matters

robots.txt controls which parts of your site crawlers can access. A mistake can block important pages or whole sites from search engines and AI crawlers, while sensible rules keep crawlers away from admin areas and endless filtered URLs.

Example

After launch, a business notices its site isn’t appearing in Google. The robots.txt file still contains a rule from development that blocks everything. Removing it lets Google crawl and index the site. (Illustrative.)

Common mistakes

  • Leaving development rules that block the whole site
  • Using robots.txt to hide pages that should use noindex instead
  • Blocking CSS or JavaScript that pages need to render
  • Blocking AI crawlers unintentionally

← Back to the full glossary