Indexing as many pages on your website as possible can be very tempting for your digital marketing team who are trying to boost their search engine authority. However, sometimes there’s more value in keeping certain pages out of a search engine’s index.
This post will walk you through why you might want to remove certain webpages from the SERPs (search engine results pages), and exactly how to go about doing it.
Understanding the Temptation to Index Many Pages
For marketers aiming to improve search rankings, it can be tempting to index as many pages as possible. Publishing more pages relevant to particular keywords can boost your ranking, provided those pages are of high quality.
The Benefit of De-indexing Pages
In some cases, keeping specific pages out of the search engine index could be more beneficial. This strategic move can help you maintain better control over your content and improve user experience.
Reasons for De-indexing
There are a few occasions where you may want to exclude a webpage or a portion of a webpage from search engine crawling and indexing:
- Avoiding Duplicate Content: Prevent different versions of the same content from being indexed, such as printer-friendly pages.
- Securing Admin and Login Pages: Ensure internal-use pages are not indexed unless meant for community access.
- Protecting Thank-You Pages: Prevent thank-you pages from being found via search, ensuring visitors pass through lead-capture forms.
Impact on Lead Generation
If thank-you pages are indexed, visitors might find them through search engines and bypass your lead-capture forms. This not only leaves valuable leads on the table but also means high-ranking thank-you pages could divert traffic away from more important content.
Methods to De-Index Pages
There are three primary methods to prevent pages from being indexed:
- Using Robots.txt
- Htaccess No Index No Follow
- Meta No Index No Follow
Let’s delve into each method in detail.
Using Robots.txt
Use if: You want more control over what you de-index and have the necessary technical resources.
Adding a robots.txt file to your site gives you control over what content search engine bots can index. This method allows you to block bots from accessing specific pages, directories, or files. To create and customize a robots.txt file:
- Create a Robots.txt File: Open a text editor and create a new file named robots.txt.
- Add Directives: Specify which pages or directories to block by adding lines like:
- User-agent: *
- Disallow: /thank-you/
- Upload the File: Place the robots.txt file in the root directory of your website.
- Using Htaccess No Index No Follow
Use if: Your website runs on Apache and mod_headers is enabled.
For a quick solution, you can add a single line to your .htaccess file:
Header set X-Robots-Tag "noindex, nofollow"
This directive tells search engines not to index the site or follow any links.
Using Meta No Index No Follow
Use if: You prefer an easier, less technical solution for de-indexing pages or links.
Metatags like “noindex” and “nofollow” provide simple and effective ways to control indexing. Here’s the difference:
- “Noindex” Tag: Tells search engines to crawl but not index a page.
- “Nofollow” Tag: Instructs search engines not to follow or pass authority through the links on a page.
Choosing the Right Tags
- Use “noindex”: When you want pages to be crawled but not indexed, which can benefit the linked pages’ ranking.
- Use “nofollow”: When you want pages to be indexed but not to pass authority through their links.
- Use Both: When you don’t want pages indexed or their links followed, like thank-you pages.
Do the best you can until you know better. Then when you know better, do better.
Implementation: Adding Metatags to Your Website
Step-by-Step Guide:
Copy the Relevant Tag:
- For “noindex”:
<meta name=”robots” content=”noindex”>
- For “nofollow”:
<meta name=”robots” content=”nofollow”>
- For both “noindex” and “nofollow”:
Add the Tag to the Head Section:
- Open the source code of the target webpage.
- Paste the full tag into a new line within the <head> section of the page’s HTML.
Here’s an example for both “noindex” and “nofollow”:
Remember, every small step counts towards achieving your big goals. Keep pushing forward! By following these steps, search engines will be instructed to ignore the specified page in search results.
Conclusion
You’ve learned how to de-index pages from search results effectively. This strategy can help you capture more leads and ensure your content is indexed as you intend. While changes won’t take effect immediately, frequent content updates can speed up the process. Monitor your crawl stats using Google Webmaster Tools to see how search engines interact with your pages.
