Marketing glossary
What is robots.txt?
robots.txt: A text file at the root of a website that tells crawlers which parts of the site they may or may not access.
robots.txt controls crawling, not indexing: a blocked page can still appear in results if other sites link to it. Businesses that want AI visibility generally allow AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot to access public content.
Why it matters
robots.txt controls which parts of your site crawlers can access. A mistake can block important pages or whole sites from search engines and AI crawlers, while sensible rules keep crawlers away from admin areas and endless filtered URLs.
Example
After launch, a business notices its site isn’t appearing in Google. The robots.txt file still contains a rule from development that blocks everything. Removing it lets Google crawl and index the site. (Illustrative.)
Common mistakes
- Leaving development rules that block the whole site
- Using robots.txt to hide pages that should use noindex instead
- Blocking CSS or JavaScript that pages need to render
- Blocking AI crawlers unintentionally
Related terms
- Crawl BudgetHow many pages a search engine is willing and able to crawl on your site in a given period.
- llms.txtA plain-text markdown file at the root of a website that summarizes the site and links to its key pages in a format that’s easy for large language models to read.
- Technical SEOOptimizing how search engines crawl, render, and index a website, covering speed, mobile usability, site structure, redirects, and structured data.