If you have a website, you’ve probably wondered at least once how to stop search engines from indexing a page.
There are numerous scenarios when such a move would make sense, and there are multiple ways to go about it.
In this guide, we’ll show you how to prevent search engines from crawling one or more pages on your site.
Why would you want to stop search engines from indexing a page?
The first question that may pop into your mind when reading the title of this article may very well be, “why would I want to prevent search engines from indexing my site?”
And you’d be totally right to be puzzled by the idea of blocking search engines. After all, the whole point of building a website is to rank in search engine searches, not be ignored by them.
However, there are multiple situations when preventing search engines from indexing a page makes sense.
Maybe you’re still developing your website on a domain name that’s publicly accessible, and you don’t want search engines to index your maintenance mode page.
Or perhaps you’re creating a private blog that you don’t want to be accessible to the general public.
Finally, many people use WordPress sites for intranet or project management and want to avoid making any internal documents accessible to users outside your organization.
What web pages don’t need to be indexed?
Besides the reasons listed above, you may also want to stop search engines from indexing various pages on a live, public website.
Many blog owners don’t know this, but not all pages on a website need to be indexed by search engines.
Some of the pages that you should de-index include the following:
Admin and login pages
You shouldn’t have your admin and login pages appearing in Google search results. To prevent the pages from being indexed, you can simply add a noindex tag to them. The only exceptions should be those login pages that serve a community.
Thank you pages
These are those pages you use to thank your newsletter subscribers or new customers/commenters. They are considered thin content pages, and since they are of no value for search engines, you shouldn’t have them indexed.
Author archives if you’re the only author on your blog
Google and other search engines may consider author pages duplicate content if there’s only one author on the entire website.
You can either disable the author archive to prevent that from happening, but if for any reason you decide to keep it, you should noindex your author archives.
Internal search results pages
An internal search results page is a page that results when a user performs a search on your site using the search function.
You shouldn’t have any of your internal search results pages indexed by search engines because they are likely to harm your user’s experience and your website in general.
Google’s Quality Guidelines specify that you should prevent crawling to search result pages using robots.txt because they don’t add any value for users who come to your website from a search engine.
How to stop search engines from indexing a page?
There are multiple ways to discourage search engines from indexing a page. The easiest of them is to include the noindex tag, but if you can’t do that for some reason, there are other ways to de-index a page.
Here’s a quick overview of the most common methods.
Add a “noindex” metatag
Adding a “noindex” metatag to a page sends a signal to search engines that it cannot add it to the search index, even though it can still crawl it.
In other words, if you add a noindex metatag to any page on your website, it won’t be shown in the search engine results.
The good news is that it’s quite easy to use metatag to prevent a specific page on your website from appearing in SERPs.
Here’s how to do it.
- Copy the following tag: <META NAME=”robots” CONTENT=”noindex”>
- Add it to the <head> section of your page’s HTML.
To do so, go to the beginning of your header by opening the source code of the page you want to de-index and paste the tag into a new line within the header.
Once you’ve added the tag, the search engine won’t index the page and it won’t appear in any search results.
It’s important to note that you need to do this for every page that you want to de-index.
If you also want search engines to not follow the links on the de-indexed page, you should use the “noindex,nofollow” metatag instead:
< META NAME =”robots” CONTENT =”noindex,nofollow”>.
Add a meta robots tag
The purpose of a robots.txt file is to tell search engines crawlers which pages they can access on a site.
According to Google, you can use a robots meta tag if you want to control how a specific page should be indexed.
This means that you can hide certain media files such as images and videos from appearing in search results.
Setting up your robots.txt file is pretty technical, so it’s a good idea to consult Google’s developer documentation.
Remove the URL from Google Search Console
This method only works for Google, but it’s a quick way to stop the most popular search engine of all from indexing a page on your site.
To do so, you can use Google Webmaster’s Remove URL Tool. According to Google, you should employ this method when you need to take off a specific URL off Google Search quickly.
The URL in question needs to be on a Search Console property that you own, and it’s important to keep in mind that this method will only block the URL temporary.
Here’s how to use the Google Remove URL tool to exclude pages from search engine results.
- Go to the website’s property in Search Console.
- Select “Temporary Hide”, then enter the URL of the page you want to exclude.
- Select “Clear URL from cache and remove from search”.
This action will clear the cached copy of the page from Google’s index and also hide the page from Google search results.
Use a plugin to noindex one or more pages in Google
Another easy way to noindex a single page or multiple ones in Google is with the help of a plugin.
This is the perfect solution for beginner users who don’t have a lot of coding experience and would like to be able to noindex pages in Google with just a couple of clicks.
There are multiple plugins that make it easy to noindex pages in Google, with Yoast SEO, Rank Math and AIOSEO being the most popular.
Let’s have a look at how you can noindex pages using each of these plugins.
Yoast SEO
If you’re running Yoast SEO, all you have to do to noindex a page is change a simple setting.
Go to the Advanced tab in the Yoast SEO meta box and select “No” for the question “Allow search engines to show this post in search results?
The default setting for this box should typically be “Yes,” and it’s the setting that you’ve probably set when installing Yoast.
To noindex posts, simply select “No” manually for every page or post that you don’t want to appear in Google searches.
If the post you’re deindexing has already been crawled by Google and appears in search results, it will take a while for them to disappear.
Yoast SEO assumes that you want to set your posts to “no index, follow,” which means that Google and other search engines will continue to follow the links on those posts.
If you don’t want that to happen, answer “No” to the next question, “Should search engines follow links to this post?”
Rank Math
If you use Rank Math instead of Yoast SEO, it’s still very easy to noindex one or more pages on your site.
Just like in the case of Yoast SEO, Rank Math indexes your posts by default. If you don’t want a particular post or page to be indexed, you need to set them to “No Index” manually.
Go to “Advanced Mode” and check the “No Index” check box that’s located in the Robots Meta section.
Alternatively, you can set a page to noindex from the admin menu bar. Go to Rank Math SEO > Mark this page > As NoIndex to do so with just one click.
AIOSEO
As you might expect by now, the default setting in AIOSEO is to index all pages and posts.
To exclude specific pages, you need to do so manually for each post or for entire categories.
To do so, scroll down to AIOSEO Settings and go to the “Advanced” tab. In the “Robots Setting” category, you’ll see the “Use Default Settings” setting turned on.
Change the toggle to off to reveal the Robots Meta heading. Check the No Index checkbox (and the No Follow one if needed), then click “Update” to apply the new settings to your post.
Password protect your entire site via cPanel
Using a password to protect your site via cPanel is another way to prevent search engines from indexing a site.
To access a password-protected site and see its contents, users will have to enter a username and password.
You can protect your entire site via cPanel by logging into your hosting account and navigating to “Password Protect Directories”. The location of the icon may differ from one host to another, but you can generally find it under the “Security” section.
Choose the public_html directory or the folder where you have installed WordPress if it’s a different one.
You’ll then have to set permissions for the folder and create a username and password.
This will add password protection to your entire site, which means that whenever a search engine or user attempts to visit the site, they’ll be prompted to enter the username and password.
Password protect your WordPress site with a plugin
If you don’t have access to cPanel via your hosting or you don’t want to go down that route, you can use a plugin to password protect your site.
Some popular plugins for password protection include:
Password Protected
This plugin is one of the easiest ways to password protect your site. You only need to check a box to enable protection, and there’s no need to create a username.
One of the best things about the Password Protected plugin is that it offers the option to whitelist IP addresses as well as various types of requests.
You can easily configure these options by going to Settings → Password Protected after you install and activate the plugin.
SeedProd
Another plugin you can use to password-protect your WordPress site, SeedProd is a popular coming soon plugin that also comes with access control and permissions features.
By adding a username and password in SeedProd, you can hide your website from users and search engines alike.
Passster
With Passster, you can password protect your entire site or specific pages or posts. Besides restricting content with a password, this plugin allows you to customize the headline and restrict content with a captcha.
Other options include multiple passwords to protect your content and unlocking protected areas of the website via a link.
Conclusion
As you can see, there are multiple options available if you want to stop search engines from indexing a page.
Whether you’re adding a “noindex” metatag or using a plugin to password protect your entire site, it’s quite easy to prevent search engines from indexing one or more of your pages.
Hopefully, our guide has made it easier to choose the best option for your particular website and type of content.
