How to Implement Noindex via Robots.txt (SEO Guide) |Using noindex in robots.txt is a common SEO misunderstanding. Google no longer recognizes or supports the noindex directive within the robots.txt file. This guide explains why that approach doesn’t work, compares noindex with robots.txt, outlines the correct methods, and shows how to properly prevent pages from appearing in search results.
The concept of using noindex in robots.txt is one of the most commonly misunderstood topics in technical SEO. In the past, many SEO professionals tried to prevent pages from appearing in search results by adding noindex directives directly to the robots.txt file.
However, this approach is no longer valid Google does not support the noindex directive within robots.txt.
In this guide, you’ll discover why this method no longer works, the correct alternatives to use, how noindex and robots.txt differ, and the safest ways to manage indexing without negatively impacting your SEO performance.
What Is the Noindex Directive in Robots.txt?
How to Implement Noindex via Robots.txt (SEO Guide) | Noindex is a directive used to instruct search engines not to display a page in search results.
In the past, some SEO practitioners tried to apply it within the robots.txt file using rules like:
User-agent: *
Noindex: /private-page/
This approach was commonly referred to as using a noindex directive in robots.txt.
However, Google has officially discontinued support for this method, meaning Googlebot now ignores any noindex instructions placed in robots.txt.
Why Google No Longer Supports Noindex in Robots.txt
Google stopped supporting noindex in robots.txt for several key reasons:
Robots.txt controls crawling, not indexing: It only tells search engines which pages they can or cannot access.How to Implement Noindex via Robots.txt (SEO Guide)
No visibility without crawling: If a page is blocked, Google cannot read a noindex directive on it
Limited signal verification: Blocking access prevents Google from evaluating important page signals
Confusing and inconsistent results: This method often led to unpredictable indexing behavior
👉 Google now recommends using meta robots tags or HTTP headers as the correct and reliable ways to apply noindex instructions.
Noindex in Robots.txt Explained (Reality Check)
Any guide suggesting that you can safely use a directive like:
Noindex: /page/
is outdated.
Even if you include it in your robots.txt file, Googlebot will simply ignore it.
So, when people ask “how to add noindex in robots.txt,” the accurate answer is:
👉 You don’t use proper and supported noindex methods instead.
Noindex Nofollow in Robots.txt Explained
The nofollow directive is also not supported in robots.txt.
Google ignores any nofollow instructions placed in the robots.txt file, meaning that using noindex, nofollow in robots.txt does not work.
The correct way to apply these directives is by using a meta robots tag, such as:
This tag should be added within the section of the webpage.
Robots.txt vs Meta Robots Tag: Noindex Explained
❌ Robots.txt (Incorrect)
User-agent: *
Disallow: /thank-you/
Meta Robots Tag: Best Practice
<meta name="robots" content="noindex">HTTP Header Method for Noindex
X-Robots-Tag: noindex
This directive is applied via HTTP headers and is fully supported by Google for controlling indexing behavior. How to Implement Noindex via Robots.txt (SEO Guide)
Using Meta Robots for Noindexing Pages
Best suited for HTML pages:
<meta name="robots" content="noindex">Use this method when:
- You want search engines to crawl the page but prevent it from appearing in search results
- The page contains internal links that you still want search engines to follow.
Using X-Robots-Tag in HTTP Headers
X-Robots-Tag: noindex
This method is effective for controlling indexing of:
- PDF files
- DOC and other document formats
- Images
- Video files
Blocking Search Engine Crawlers via Robots.txt
Here is a rewritten version of your content:
Use this method when:
- You want to conserve crawl budget
- The page should not be accessed by search engine bots
Example:
User-agent: *
Disallow: /admin/
Common Mistakes Using Noindex in Robots.txt
❌ Blocking pages with robots.txt while expecting them to be de-indexed
❌ Using noindex in robots.txt and assuming it will work
❌ Preventing crawling before implementing a proper noindex meta tag
❌ Removing the noindex directive before the page has been fully re-crawled
❌ Incorrectly combining noindex and disallow directives
When to Apply Noindex vs Robots.txt in SEO
- The page should not appear in search results
- The page remains accessible to users
- The page includes internal links that should still be followed by search engines
- How to Implement Noindex via Robots.txt (SEO Guide)