There’s a tiny text file sitting at the root of almost every website on the internet. Most visitors never see it, never think about it, and never need to. But search engine crawlers check it every single time they show up to your site before they touch anything else.
That file is robots.txt. And while it’s genuinely simple in concept, it has a reputation for causing serious SEO damage when people get it wrong. Entire sites have vanished from Google because of one misplaced line in this file. Not because of bad content. Not because of weak backlinks. Because of a text file roughly the size of a grocery list.
So let’s understand it properly.
What Is robots.txt?
Robots.txt is a plain text file that tells crawlers which parts of your website they’re allowed to access and which parts they should leave alone. It lives at the root of your domain, which means it’s always accessible at:
yourdomain.com/robots.txt
Go ahead and check your own site right now if you’re curious. Type that URL into your browser and see what comes up. If you see a structured list of instructions, you have one. If you get a 404, you don’t which is fine for most sites, as we’ll discuss shortly.
The file works on what’s called the Robots Exclusion Protocol, which sounds more impressive than it is. It’s basically an agreed-upon standard from the early 1990s that says: “Hey, crawlers, check this file before you do anything on our site.” Well-behaved crawlers like Googlebot, Bingbot, and most legitimate bots follow it consistently.
The keyword there is well-behaved. Robots.txt is not a security measure. It’s more like a polite request than a locked door. Malicious scrapers, spammers, and bad-faith bots frequently ignore it entirely. If you have genuinely sensitive content you need to protect, robots.txt alone won’t do that job.
What Does a robots.txt File Actually Look Like?
Here’s a basic example:
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Allow: /
Sitemap: https://www.yoursite.com/sitemap.xml
That’s it. A real, functional robots.txt file. Let’s break down what each part means.
User-agent identifies which crawler the following instructions apply to. The asterisk (*) is a wildcard that means “all crawlers.” You can also get specific:
User-agent: Googlebot
User-agent: Bingbot
User-agent: GPTBot
Each crawler identifies itself with a user-agent string when it visits your site, and robots.txt matches instructions to the right bot using these names.
Disallow tells a crawler not to access a specific path. The path is relative to your domain, so Disallow: /admin/ means don’t touch anything under yourdomain.com/admin/.
Allow overrides a disallow for specific paths. If you’ve blocked a whole folder but want one page within it to remain accessible, Allow lets you carve out that exception.
Sitemap points crawlers to your XML sitemap file. It’s a convenient place to declare this since any crawler that reads your robots.txt file will immediately know where your sitemap lives too.
User-Agents: Talking to Specific Crawlers
Here’s something worth understanding: you can write separate instructions for different bots. Google’s crawler, Bing’s crawler, and the newer AI crawlers all identify themselves differently.
| Crawler | User-agent Name |
| Googlebot | |
| Google Images | Googlebot-Image |
| Bing | Bingbot |
| DuckDuckGo | DuckDuckBot |
| Yandex | YandexBot |
| Baidu | Baiduspider |
| OpenAI | GPTBot |
| Anthropic | ClaudeBot |
| Apple | Applebot |
This matters in 2026 more than it did a few years ago. AI crawlers from companies like OpenAI and Anthropic are now making regular appearances in server logs across the web. If you want to control whether those specific bots can access your content for AI training or answer generation purposes, robots.txt is one of the tools available to you. Some site owners are choosing to block certain AI crawlers while keeping Google and Bing fully accessible.
For example:
User-agent: GPTBot
Disallow: /
User-agent: Googlebot
Allow: /
This blocks OpenAI’s crawler from the entire site while leaving Google full access.
One important technical note: each user-agent declaration in robots.txt is treated as its own separate block of rules. Instructions set for Googlebot don’t carry over to Bingbot. Each bot only follows the rules declared specifically for it or for the wildcard (*).
What Can You Actually Block?
This is where most of the practical use of robots.txt lives. Here are the types of content worth thinking about:
Admin and login areas. There’s no SEO benefit to having Google crawl your /admin/ or /wp-admin/ directories. Blocking them keeps crawl resources pointed at your actual content and away from backend tools visitors never see.
Internal search result pages. If your site has internal search functionality, the URLs it generates (like /search?q=shoes) are usually thin, duplicate-ish content that adds nothing to Google’s understanding of your site. Blocking them is generally good practice.
Staging or development sections. If you have a /staging/ or /dev/ folder that isn’t ready for the public, blocking it from crawlers keeps unfinished work out of the index.
Parameter-heavy URLs. Ecommerce sites often generate thousands of URLs through filter combinations (colour=blue&size=medium&sort=price). Many of these are near-duplicates of each other. Blocking crawlers from parameter-heavy paths can protect your crawl budget from being wasted on these.
Duplicate content paths. Things like print-friendly versions of pages, or multiple URL variations of the same content, can be managed partly through robots.txt, though canonical tags are usually the better tool for canonicalization specifically.
PDF files or media. If you don’t want certain file types appearing in search results, you can block specific file extensions.
What robots.txt Cannot Do
This is where a lot of confusion creeps in, and getting it wrong can actually damage your SEO rather than help it.
Blocking a page in robots.txt does not guarantee it stays out of Google’s index.
Read that again, because it’s one of the most important and most misunderstood things about this file.
If another website links to a page you’ve blocked in robots.txt, Google can learn that page exists from that external link and index it anyway even if it has never been able to crawl it. What Google typically does in this case is create a minimal listing: the URL appears in search results but with no title or description, just the bare URL. Not ideal.
If your actual goal is keeping a page out of Google’s search results, the right tool is a noindex meta tag placed on the page itself. Not robots.txt.
Here’s why this distinction matters in practice: if you block a page with robots.txt AND add a noindex tag to it, you’ve created a contradiction. The noindex tag on the page is telling Google not to index it. But the robots.txt block is preventing Google from crawling the page in the first place which means Google never even reads the noindex tag. The block wins, and Google may eventually index the page anyway if it discovers the URL through external links.
The correct approach when you genuinely want a page excluded from search results: allow crawling (remove any robots.txt block), and use a noindex meta tag. That way Google can crawl the page, read the noindex instruction, and respect it.
The Directives Worth Knowing
Disallow
The workhorse of robots.txt. Blocks the specified path.
Disallow: /private/
Allow
Overrides a disallow for a specific sub-path. Useful when you want to block a whole folder but make one exception.
User-agent: *
Disallow: /products/
Allow: /products/featured/
Wildcards
The asterisk (*) can match any string of characters within a path, making it possible to write efficient rules without listing every variation.
# Block all URLs with a question mark (parameterized URLs)
Disallow: /*?
# Block all PDF files sitewide
Disallow: /*.pdf$
The dollar sign ($) at the end of a pattern means “must end with this.” So /*.pdf$ matches any URL ending in .pdf, but not a URL like /document.pdf?download=true.
Crawl-delay
Some crawlers support this directive to throttle how fast they request pages from your server. Google does not support it. Bing and Yandex do. It’s worth knowing about for server load management on those other search engines, but it won’t change Googlebot’s behavior at all.
Noindex (in robots.txt)
You may see this in older robots.txt files. Google stopped supporting it in 2019. If you’re using it to try to exclude pages from Google’s index, it’s not working. Use a noindex meta tag on the page instead.
Common robots.txt Mistakes That Hurt SEO
These show up constantly, including on sites run by people who absolutely should know better.
Blocking the entire site. The most catastrophic version looks like this:
User-agent: *
Disallow: /
This tells every crawler to stay off your entire website. Every page, every file, everything. This is appropriate for a site under development that you genuinely don’t want indexed yet. It’s a disaster if it accidentally goes live on your production site, which happens more often than anyone wants to admit, usually right after a site migration when the development robots.txt gets pushed to the live server without anyone checking.
Blocking CSS and JavaScript files. Your site’s CSS and JavaScript files control how everything looks and behaves. If Google can’t access these files, it may render your pages poorly or incompletely during the indexing process, which can affect how Google understands and evaluates your content. There’s rarely a good reason to block these through robots.txt.
Blocking your sitemap. If you reference your sitemap location in robots.txt but your sitemap URL itself is blocked by a disallow rule, you’ve created a contradiction. The sitemap can’t be accessed even though you’ve pointed crawlers to it.
Forgetting that rules are case-sensitive. Paths in robots.txt are case-sensitive. Disallow: /Blog/ is different from Disallow: /blog/. If your URLs use a consistent case and your robots.txt uses a different one, the rules won’t match correctly.
Thinking specificity doesn’t matter. This one catches people off guard. If you write Disallow: /de, intending to block a German-language subdirectory you’re still building, you’ll accidentally block any URL that starts with /de including /delivery/, /demo/, /design/, and anything else that begins with those two letters. The fix is a trailing slash: Disallow: /de/ — which specifically targets only the /de/ directory.
Using robots.txt when canonical tags are the right tool. Robots.txt is for controlling access. Canonical tags are for telling Google which version of a page is the “real” one when duplicates exist. They’re different problems with different solutions. Trying to fix duplicate content issues purely through robots.txt blocks often creates more problems than it solves.
How to Check If a URL Is Blocked
Google Search Console has a built-in robots.txt tester that lets you paste any URL and see instantly whether your current robots.txt rules would block it. You can also see which specific rule is doing the blocking, which is useful when you have multiple overlapping rules and can’t immediately tell which one is firing.
The URL Inspection tool in Search Console is another useful check: paste any URL on your site and the results will tell you whether it’s currently blocked by robots.txt and what Google’s last known crawl status is for that page.
If you’re seeing pages in Search Console flagged as “Submitted URL blocked by robots.txt,” that means URLs from your sitemap are being blocked by your robots.txt rules a contradiction you want to fix immediately, since those are pages you’ve actively told Google to index, but also told it not to crawl.
Do Small Sites Actually Need robots.txt?
Honestly, for a small website with clean navigation, no admin areas exposed, no parameter URL problems, and no sensitive content robots.txt is not urgent. Google handles small, simple sites fine without one. The absence of a robots.txt file doesn’t hurt anything.
That said, having a basic one is quick to set up and gives you control if your needs change later. At minimum, pointing crawlers to your sitemap location through the Sitemap directive is worth doing even if you have no actual blocks to implement:
User-agent: *
Disallow:
Sitemap: https://www.yoursite.com/sitemap.xml
The empty Disallow here means “block nothing” . It’s a valid way of saying “all crawlers can access everything,” while still pointing them to your sitemap.
A robots.txt Checklist Before You Publish
Before your robots.txt goes live or if you haven’t reviewed yours in a while run through these:
- Check it in a browser first. Go to yourdomain.com/robots.txt and read what’s there. Make sure it looks intentional and nothing is blocked that shouldn’t be.
- Test specific URLs in Search Console. Use the robots.txt tester to confirm your rules do what you think they do.
- Make sure your sitemap isn’t blocked. If you’re referencing your sitemap, confirm the sitemap URL itself isn’t caught by any disallow rules.
- Verify CSS and JS aren’t blocked. Google needs these to render your pages properly.
- Check for leftover staging rules. The Disallow: / disaster usually comes from staging environments. If your site was recently migrated or redesigned, this is the first thing to verify.
- Cross-check with your noindex tags. Pages you want excluded from search should have noindex tags, not just robots.txt blocks. Pages blocked in robots.txt shouldn’t also have noindex tags you’re relying on Google can’t read those if it can’t crawl the page.
The Bottom Line
Robots.txt is one of those things that, when it works, nobody notices it. When it breaks, the consequences show up fast and painfully in the Search Console. The good news is that once you understand what it does control crawler access, not indexation and what it doesn’t do guarantee pages stay out of Google most of the confusion around it clears up quickly.
Keep your robots.txt honest, simple, and reviewed whenever you make major changes to your site structure. It’s not glamorous technical SEO work, but it’s the kind of thing that, done wrong, can undo months of other effort in one push to production.