What Is Robots.txt?
A robots.txt file is a plain text file placed at the root of a website that tells search engine crawlers which parts of the site they’re allowed or not allowed to crawl.
Correct Syntax
A basic robots.txt uses simple directives: User-agent: specifies which crawler the rule applies to (or * for all), Disallow: blocks a specific path from being crawled, and Allow: can carve out exceptions within a disallowed section. A Sitemap: line pointing to your XML sitemap is also commonly included.
Common Mistakes
Frequent errors include accidentally disallowing the entire site with Disallow: /, blocking CSS or JavaScript files needed for Google to render pages properly, confusing “disallow crawling” with “prevent indexing” (a disallowed page can sometimes still appear in results without content if it’s linked to elsewhere — noindex is the correct tool for that), and forgetting that robots.txt rules are case-sensitive and applied literally.
