Marketing glossary
What is robots.txt?
robots.txt: A text file at the root of a website that tells crawlers which parts of the site they may or may not access.
robots.txt controls crawling, not indexing: a blocked page can still appear in results if other sites link to it. Businesses that want AI visibility generally allow AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot to access public content.
Related terms
- Crawl BudgetHow many pages a search engine is willing and able to crawl on your site in a given period.
- llms.txtA plain-text markdown file at the root of a website that summarizes the site and links to its key pages in a format that’s easy for large language models to read.
- Technical SEOOptimizing how search engines crawl, render, and index a website, covering speed, mobile usability, site structure, redirects, and structured data.