The Secrets of robots.txt - How Websites Ask Search Engines to Look Away

robots.txt - The "No Trespassing" Sign of the Web

In the root directory of most websites sits a text file called robots.txt. This file is a mechanism for telling search engine crawlers (bots) "please don't crawl this page." It grew out of the "Robots Exclusion Protocol" proposed by Martijn Koster in 1994 and was used for years as a convention with no written specification, until it became an IETF Standards Track document, RFC 9309, in September 2022. Koster, who made that original proposal, is among the authors of the RFC.

However, robots.txt harbors a surprising secret: it's a "request," not a "command."

robots.txt Has No Legal Binding Force

robots.txt is a "gentleman's agreement" with crawlers. Major search engines like Google, Bing, and Yahoo respect robots.txt, but technically any bot can ignore it and crawl anyway. Malicious scrapers and spam bots naturally disregard robots.txt.

In other words, robots.txt is a "polite request to well-behaved bots" and is not a security measure. The standard itself says so: the Security Considerations section of RFC 9309 states that the Robots Exclusion Protocol is not a substitute for valid content security measures, and that anyone who needs to control access to those paths should use a measure at the application layer serving the file, such as HTTP authentication. "Hiding" pages containing sensitive information via robots.txt is like putting a "No Burglars" sign on your front door. For actual protection, sites need proper HTTP security headers and access controls.

robots.txt Actually Leaks Information

Ironically, robots.txt publishes a list of "things you want to hide." The same section of RFC 9309 warns about exactly this: listing paths in the file exposes them publicly and makes them discoverable. Attackers check robots.txt first, using Disallow paths to infer the existence of admin panels, staging environments, internal APIs, backup files, and more. Knowing how to read URLs safely can help you spot suspicious paths that attackers might target.

This robots.txt is essentially telling attackers "paths /admin/, /staging/, /api/internal/, and /backup/ exist."

Peeking at Famous Sites' robots.txt

Since robots.txt is a public file, anyone can read it by visiting https://example.com/robots.txt for any site.

• YouTube: Surprisingly short, roughly 30 lines holding just over 20 Disallow rules. At the top sits a joke comment saying the file was "created in the distant future (the year 2000) after the robotic uprising of the mid 90's which wiped out all humans"
• Wikipedia: Blocks certain bots by name. The entry for MJ12bot comes with a comment explaining the reason: it kept fetching pages in bulk and ignored the 429 (too many requests) responses
• CIA (cia.gov): Names more than 30 bots and shuts most of them out with Disallow: /. It also asks every crawler for a Crawl-delay: 10 (a 10-second gap), showing that government sites run on the same mechanism
• Amazon: Allows crawling of product pages while excluding cart, account, and internal search results

robots.txt and the New Battle Against AI Crawlers

Since 2023, the collection of training data for generative AI has put robots.txt back in the spotlight. The turning point was 2023, when OpenAI published the name of its training crawler GPTBot along with instructions for blocking it, after which many news sites and publishers started adding AI-related rules to their robots.txt. The CCBot in the code sample below is the crawler of Common Crawl, which has been running since well before the generative AI boom, but because its public archive is used for training, it is increasingly listed alongside the newer bots.

One name that causes confusion is Google-Extended. It is usually mentioned in the same breath as the others, yet it is not a crawler of its own. It has no dedicated user agent; it exists purely as a name you can use in robots.txt to say whether already-collected content may be used for training and grounding in products such as Gemini, and it has no effect on your listing in Google Search.

Even after being standardized as RFC 9309, however, robots.txt remains a "request." There is no guarantee that every AI crawler will respect it, and as of August 2026 the role of robots.txt in the AI era, including its relationship with copyright law, is still being debated. Users concerned about their data being collected by AI services may want to explore privacy-focused search engines that take a different approach to data collection.

Summary

robots.txt has traveled from a 1994 proposal to standardization as RFC 9309 in 2022, and it is still in use as the "etiquette" of the internet. It has no legal binding force and doesn't serve as a security measure, but it remains an important tool for managing the relationship with search engines. The robots.txt on IP Check-san is configured both to have search engines crawl the site properly and to spell out, bot by bot, which AI-oriented crawlers are allowed. Anyone can read the file, so if you are curious, go take a look at the real thing.

Frequently Asked Questions

What is robots.txt?

robots.txt is a plain text file placed at the root of a website that tells web crawlers which pages they should or should not access. It follows the Robots Exclusion Protocol, which was standardized as RFC 9309 in September 2022, yet compliance is left to each crawler and the file carries no legal force.

Is robots.txt a security measure?

No. robots.txt is publicly accessible and only works as a polite request to well-behaved crawlers. Malicious bots ignore it entirely. Worse, listing sensitive paths in Disallow rules actually advertises their existence. For real access control, use authentication and authorization.

Can robots.txt block AI crawlers?

You can add Disallow rules for the user agents of known AI crawlers such as GPTBot and CCBot. Google-Extended is often listed alongside them, but it is not a crawler of its own: it is only a name used to say whether already-collected content may be used for training and grounding. Compliance is voluntary, new AI crawlers appear frequently, and as of August 2026 the legal standing of these rules is still being debated.