Our job as search engine optimization experts is to make sure Google and other search engines can first find our content so that they can then understand it, index it, and rank it appropriately. When we have an excess of indexed pages, we are not being clear with how we want search engines to treat our pages. As a result, they take whatever action they deem best which sometimes translates to indexing more pages than needed.<\/p>\n
Before you know it, you\u2019re dealing with\u00a0index bloat<\/a>.<\/p>\n Put simply, index bloat is when you have too many low-quality pages on your site indexed in search engines. Similar to bloating in the human digestive system (disclaimer: I\u2019m not a \u00a0doctor), the result of processing this excess content can be seen in search engines indices when their information retrieval process becomes less efficient.<\/p>\n Index bloat can even make your life difficult\u00a0without you knowing it. In this puffy and uncomfortable situation, Google has to go through much more content than necessary (most of the times low-quality and\u00a0internal duplicate content<\/a>) before they can get to the pages you\u00a0want\u00a0them to index.<\/p>\n Think of it this way: Google visits your XML sitemap to find 5,000 pages, then crawls all your pages and finds even\u00a0more\u00a0of them via\u00a0internal linking<\/a>, and ultimately decides to index 30,000 URLs. This comes out to an indexation excess of approximately 500% or even more.<\/p>\n But don\u2019t worry, diagnosing your indexation rate to measure against index bloat can be a very simple and straight forward check. You simply need to cross-reference which pages you want to get indexed versus the ones that Google is indexing (more on this later).<\/p>\n The objective is to find that disparity and take the most appropriate action. We have two options:<\/p>\n You will find that most of the time, index bloat results in removing a relatively large number of pages from the index by adding a \u201cNOINDEX\u201d meta tag. However, through this indexation analysis, it is also possible to find pages that were missed during the creation of your XML sitemap(s), and they can then be added to your sitemap(s) for better indexing.<\/p>\n Index bloat can slow processing time, consume more resources, and open up avenues outside of your control in which search engines can get stuck. One of the objectives of SEO is to remove roadblocks that hinder great content from ranking in search engines, which are very often technical in nature. For example, slow load speeds, using\u00a0noindex<\/a>\u00a0or nofollow meta tags where you shouldn\u2019t, not having proper internal linking strategies in place, and other such implementations.<\/p>\n Ideally, you would have a 100% indexation rate. Meaning every quality page on your site would be indexed \u2013 no pollution, no unwanted material, no bloating. But for the sake of this analysis, let\u2019s consider anything above 100% bloat. Index bloat forces search engines to spend more resources (which are limited) than needed processing the pages they have in their database.<\/p>\n At\u00a0best, index bloat causes inefficient crawling and indexing, hindering your ranking capability. But index bloat at\u00a0worst\u00a0can lead to\u00a0keyword<\/a>\u00a0cannibalization across many pages on your site, limiting your ability to rank in top positions, and potentially impacting the user experience by sending searchers to low-quality pages.<\/p>\n To summarize, index bloat causes the following issues:<\/p>\n Unintentional duplicate content\u00a0is one of the most common sources of index bloat. This is because most sources of internal duplicate content revolve around technical errors that generate large numbers of URL combinations that end up indexed. For example, using URL parameters to control the content on your site without proper\u00a0canonicalization<\/a>.<\/p>\n Faceted navigation has also been one of the \u201cthorniest SEO challenges\u201d for large ecommerce sites,\u00a0as Portent describes<\/a>, and has the potential of generating billions of duplicate content pages by overlooking a simple feature.<\/p>\n It\u2019s important to mention an issue introduced by the\u00a0Yoast SEO plugin version 7.0<\/a>\u00a0around attachment pages. This WordPress plugin bug led to \u201cPanda-like problems\u201d in March of 2018 causing\u00a0heavy ranking drops for affected sites<\/a>\u00a0as Google deemed these sites to be lower in the overall quality they provided to searchers. In summary, there is a setting within the Yoast plugin to remove attachment pages in WordPress \u2013 a page created to include each image in your library with minimal content \u2013 the epitome of thin content for most sites. For some users, updating to the newest version (7.0 then) caused the plugin to overwrite the previous selection to remove these pages and defaulted to index all attachment pages.<\/p>\n This then meant that having five images per blog post would lead to 5x-ing the number of indexed pages with 16% of actual quality content per URL, causing a massive drop in domain value.<\/p>\n Pagination refers to the concept of splitting up content into a series of pages to make content more accessible and improve user experience. This means that if you have 30 blog posts on your site, you may have ten blog posts per page that go three pages deep. Like so:<\/p>\n You\u2019ll see this often on shopping pages, press releases, and news sites, among others.<\/p>\n Within the purview of\u00a0SEO<\/a>, the pages beyond the first in the series will very often contain the same page title and meta description, along with very similar (near duplicate) body content, introducing keyword cannibalization to the mix. Additionally, the purpose of these pages is for a better browsing user experience for users already on your site, it doesn\u2019t make sense to send search engine visitors to the third page of your blog.<\/p>\n If you have content on your site that is not generating traffic, has not resulted in any conversions, and does not have any\u00a0backlinks<\/a>, you may want to consider changing your strategy. Repurposing content is a great way to maximize any value that can be salvaged from under-performing pages to create stronger and more authoritative pages.<\/p>\n Remember, as SEO experts our job is to help increase the overall quality and value that a domain provides, and improving content is one of the best ways to do so. For this, you will need a content audit to evaluate your own individual situation and what the best course of action would be.<\/p>\n Even a 404 page that results in a 200 Live\u00a0HTTP status code<\/a>\u00a0is a thin and low-quality page that should not be indexed.<\/p>\n One of the first things I do when auditing a site is to pull up their XML sitemap. If they\u2019re on a WordPress site using a plugin like Yoast SEO or All in One SEO, you can very quickly find page types that do not need to be indexed. Check for the following:<\/p>\n To determine if the pages in your XML sitemap are low-quality and need to be removed from search really depends on the purpose they serve on your site. For instance, sites do not use author pages in their blog, but still, have the author pages live, and this is not necessary. \u201cThank you\u201d pages should not be indexed at all as it can cause conversion tracking anomalies. Test pages usually mean there\u2019s a duplicate somewhere else. Similarly, some plugins or developers build custom features on web builds and create lots of pages that do not need to be indexed. For example, if you find an XML sitemap like the one below, it probably doesn\u2019t need to be indexed:<\/p>\n Remember that our objective here is to find the greatest contributors of low-quality pages that are bloating the index with low-quality content. Most times it\u2019s very easy to find these pages on a large scale since a lot of thin content pages follow a pattern.<\/p>\n This is a quantitative analysis of your content, looking for volume discrepancies based on the number of pages you have, the number of pages you are linking to, and the number of pages Google is indexing. Any disparity between these numbers means there\u2019s room for technical\u00a0optimization<\/a>, which often results in an increase in organic rankings once solved. You want to make these sets of numbers as similar as possible.<\/p>\n As you go through the various methods to diagnose index bloat below, look out for patterns in URLs by reviewing the following:<\/p>\n Next, I will walk you through a few simple steps you can take on your own using some of the most basic\u00a0tools<\/a>\u00a0available for SEO. Here are the tools you will need:<\/p>\n As you start finding anomalies, start adding them to a spreadsheet so they can be manually reviewed for quality.<\/p>\n Under Configuration > Spider > Basics, configure Screaming Frog to crawl (check \u201ccrawl all subdomains\u201d, and \u201ccrawl outside of start folder\u201d, manually add your XML sitemap(s) if you have them) for your site in order to run a thorough scan of your site pages. Once the crawl has been completed, take note of all the indexable pages it has listed. You can find this in the \u201cSelf-Referencing\u201d report under the Canonicals tab.<\/p>\n Take a look at the number you see. Are you surprised? Do you have more or fewer pages than you thought? Make a note of the number. We\u2019ll come back to this.<\/p>\n Open up your Google Search Console (GSC) property and go to the Index >\u00a0Coverage report<\/a>. Take a look at the valid pages. On this report, Google is telling you how many total URLs they have found on your site. Review the other reports as well, GSC can be a great tool to evaluate what the Googlebot is finding when it visits your site.<\/p>\n How many pages does Google say it\u2019s indexing? Make a note of the number.<\/p>\n This one is a simple check. Visit your XML sitemap and count the number of URLs included. Is the number off? Are there unnecessary pages? Are there not enough pages?<\/p>\n Conduct a crawl with Screaming Frog, add your XML sitemap to the configuration and\u00a0run a crawl analysis<\/a>. Once it\u2019s done, you can visit the Sitemaps tab to see which specific pages are included in your XML sitemap and which ones aren\u2019t.<\/p>\n Make a note of the number of indexable pages.<\/p>\n This one is a simple check too, don\u2019t overthink it. How many pages on your site do you have? How many blog posts do you have? Add them up. We\u2019re looking for quality content that provides value, but more so in a quantitative fashion. It doesn\u2019t have to be exact as the actual quality a piece of content has can be measured via a content audit.<\/p>\n Make a note of the number you see.<\/p>\n At last, we come to the final check of our series. Sometimes Google throws a number at you and you have no idea where it comes from, but try to be as objective as possible. Do a \u201csite:domain.com\u201d search on Google and check how many results Google serves you from its index. Remember, this is purely a numeric value and does not truly determine the quality of your pages.<\/p>\n Make a note of the number you see and compare it to the other numbers you found. Any discrepancies you find indicates symptoms of an inefficient indexation. Completing a simple quantitative analysis will help direct you to areas that may not meet minimum qualitative criteria. In other words, comparing numeric values from multiple sources will help you find pages on your site that contain a low value.<\/p>\n The quality criteria we evaluate against can be found in\u00a0Google\u2019s Webmaster guidelines<\/a>.<\/p>\n Resolving index bloat is a slow and tedious process, but you have to trust the optimizations you\u2019re performing on the site and have patience during the process, as the results may be slow to become noticeable.<\/p>\n In an ideal scenario, low-quality pages would not exist on your site, and thus, not consume any limited resources from search engines. If you have a large number of outdated pages that you no longer use, cleaning them up (deleting) can often lead to other benefits like fewer redirects and 404s, fewer thin-content pages, less room for error and misinterpretation from search engines, to name a few.<\/p>\n The less control you give search engines by limiting their options on what action to take, the more control you will have on your site and your SEO.<\/p>\n Of course, this isn\u2019t always realistic. So here are a few alternatives.<\/p>\n When you use this method at the page level please don\u2019t add a site-wide noindex \u2013 happens more often than we\u2019d like), or within a set of pages, is probably the most efficient as it can be completed very quickly on most platforms.<\/p>\n All of the above can be noindexed and removed from your XML sitemap(s) with a few clicks on WordPress if you use Yoast SEO or All in One SEO.<\/p>\n Using the robots.txt file to disallow sections or pages of your site is not recommended for most websites unless it has been explicitly recommended by an SEO Expert\u00a0after auditing your website<\/a><\/strong><\/span>. It\u2019s incredibly important to look at the specific environment your site is in and how a disallow of certain pages would affect the indexation of the rest of the site. Making a careless change here may result in unintended consequences.<\/p>\n Now that we\u2019ve got that disclaimer out of the way, disallowing certain areas of your site means that you\u2019re blocking search engines from even reading those pages. This means that if you added a noindex, and also disallowed, Google won\u2019t even get to read the noindex tag on your page or follow your directive because you\u2019ve blocked them from access. Order of operations, in this case, is absolutely crucial in order for Google to follow your directives.<\/p>\nWhat Is The Index Bloat?<\/h2>\n
\n
Why Index Bloat is Detrimental for SEO<\/h2>\n
\n
Sources of Index Bloat<\/h2>\n
1. Internal Duplicate Content<\/h3>\n
2. Thin Content<\/h3>\n
3. Pagination<\/h3>\n
\n
4. Under-performing Content<\/h3>\n
Common Index Bloat Issues<\/h2>\n
\n
\n
Different Methods to Diagnose Index Bloat<\/h2>\n
\n
\n
1. Screaming Frog Crawl<\/h3>\n
![\"screenshot](\"https:\/\/www.searchenginewatch.com\/wp-content\/uploads\/2019\/07\/asg_JJyXkIVUfq0pGhHU2Fart_uPAmtJAKMUHu7Z3b2F1561682418591-1561682418591.png\")
<\/p>\n2. Google\u2019s Search Console<\/h3>\n
![\"screenshot](\"https:\/\/www.searchenginewatch.com\/wp-content\/uploads\/2019\/07\/asg_JJyXkIVUfq0pGhHU2Fart_uPAmtJAKMUHu7Z3b2F1561682416856-1561682416856.png\")
<\/p>\n3. Your\u00a0XML Sitemaps<\/a><\/h3>\n
![\"example](\"https:\/\/www.searchenginewatch.com\/wp-content\/uploads\/2019\/07\/asg_JJyXkIVUfq0pGhHU2Fart_uPAmtJAKMUHu7Z3b2F1561682417038-1561682417038.png\")
<\/h3>\n4. Your Own Content Management System (CMS)<\/h3>\n
5. Google<\/h3>\n
![\"screenshot](\"https:\/\/www.searchenginewatch.com\/wp-content\/uploads\/2019\/07\/asg_JJyXkIVUfq0pGhHU2Fart_uPAmtJAKMUHu7Z3b2F1561738910365-sitesearch.png\")
<\/p>\nHow to Resolve Index Bloat<\/h3>\n
1. Deleting Pages (Ideal)<\/span><\/h2>\n
2. Using Noindex (Alternative)<\/span><\/h2>\n
\n
3. Using Robots.txt (Alternative)<\/span><\/h2>\n
4. Using Google Search Console\u2019s Manual Removal Tool (Temporary)<\/span><\/h2>\n