Before content can be ranked by Google, which is the primary goal of any SEO campaign, we must first have it discovered, crawled, and indexed. These are three very distinct things, and we’ll take a look why they may not be happening for you here.
We often see complaints that Google isn’t indexing and ranking content. The first step is understanding WHY this is happening, & the logical question you should start with is “Why should they?”.
The hard truth: It’s not Google’s job to find your content & get it in front of people. Instead, it’s your job to make content Google finds valuable, and build the correct technical environment around it.
With the web growing at an epic pace, it would be unreasonable to assume that Google will quickly, if at all, identify every new piece of content, understand it, and then rank it for related searches. John Mueller of Google said as much in February of 2022 when he indicated if Google doesn’t deem the content “important”, it’s unlikely to be indexed.
So it’s an SEO’s job to “make it important”.
Table of Contents
The Three States of Content
In Google Search Console, there’s really only 3 ways that a piece of content that doesn’t have significant technical issues (accidentally redirected or no-indexed, etc) can exist. In this case, we’re talking about content we want to rank, is in the XML sitemap, and is high quality, human written, and useful to the end user. It should also be VERY unique.
Crawled and Indexed
This is step one to ranking a product page, category page, or a piece of blog content. Google has discovered the page through internal, external, or XML sitemap links. They’ve deemed it worthy of being seen, so they’ve included it in their index database.
You might not rank yet, or receive traffic… But we’re saying there’s a chance.
Crawled – Not Currently Indexed
Google knows about this content, has read it over, but hasn’t yet included it for ranking consideration. No mention made yet in Google Search Console if there’s a technical issue preventing it, so you’ll have to go determine that on your own.
If Google had detected issues that prevent indexing, these specific pages would fall into another category of “excluded” content, such as they have canonical links pointing to other pages, or you’ve noindexed them through meta or X tags.
Discovered – Not Currently Indexed
This is the Google equivalent of “Yeah, we’ll get to it…”.
Google knows the content exists, but its mobile or desktop crawlers have yet to even have a look. Maybe it’s in the queue, or maybe it’s not. There’s no way of knowing for sure.
You can manually submit these in Search Console, but if any of the issues we describe below exist, it might not help.
In the larger context, we ultimately have to ask WHY they weren’t crawled when discovered. It could be as simple as you were at your crawl budget and Google will eventually get to it.
There COULD be other reasons though. Some of these are conjectures or hypotheses, but if you think about how Google generally treats the web, it’s probably “your fault”. The rest of this article will attempt to explain why.
On-Page SEO and Indexing Content
You don’t have to be an SEO expert to know that search engines try to value high-quality, original content. To be sure, we can all show high-ranking examples of content that clearly isn’t original. It’s probably “good” though, even if pulled together from other places on the web.
That is to say, it follows all the standard best practices of web page building, and it does have value to the user.
We can’t control whether or not Google can identify the original source (or maybe more so even cares), so unless you’re going to run around filing DMCA complaints all day, we mostly have to live with this fact. Tony Wright takes a look at this issue in Why Do Aggregator Sites Outperform the Original Source in Google.
Now let’s look at what we CAN control.
Unique Content – How are you Providing User Value?
As I stated very early in this piece, before we ask ourselves why Google isn’t indexing and ranking our content, we need to answer a better question:
Why should they?
With an immeasurable number of pieces of content being published daily, you need to stand out. Don’t forget, Google even crawls Twitter… Can you imagine reading every Tweet from even just your feed? That’s just one web platform- consider how much content they must be exposed to across the web every day.
So to get indexed, you need to bring your content game strong.
You need to write pieces that wow a user. You can do this in a multitude of ways, but you can never go wrong with providing a unique perspective, or offering a new solution to a problem.
At its core, that is what makes good content.
If you consistently publish things that assist a reader in a better way than other articles (or product descriptions… whatever the case may be), you are going a long way towards possibly ranking.
Then, we need to solve for the below concerns.
Quality Backlinks – Do you Have Any, and are you Looking for Them?
Your super-impressive, well researched piece of content is written, edited, and ready to publish.
You press the button, and now you wait. And wait. And probably wait some more.
Where’s all the Google organic traffic? Unless you are one of the undisputed authorities in your space, it might be a while unless you do a LOT more to get indexed and ranked.
As Tony Wright mentioned in his piece on aggregator sites I referenced above, one of the key differentiators between sites that rank and sites that don’t is a robust backlink profile. Many websites we see don’t have this, and it will take some time to develop it.
In short, a link from your XML Sitemap isn’t good enough.
I highly suggest that you amplify the page through your social channels, perhaps do some press releases, and otherwise go out and get some links on external sites to all of your core new content. White hat links, of course… I don’t suggest outright buying them.
On a side note, the links from your social channels themselves probably don’t help that much. That would be far too easy to manipulate. However, you’re getting the piece read, and then some of the readers who have websites could potentially link to you. That’s where the value really lies.
Since those readers can’t find it in Google yet, you have to give them a way to discover it, so they can help you amplify it.
Orphaned Pages
Just as important as external links is your internal linking strategy. As John Mueller said, if Google doesn’t think it’s important, they won’t index it.
To that end, if you publish a piece and you’re not linking to it from anywhere else but your sitemap, you’re showing that YOU don’t think it’s important. So why should Google?
I’d go so far as to say anything you really want to rank for should be linked to from your home page. Especially if it’s core to your business, and has the chance of driving revenue.
You can also go through all your existing content and add links to the new piece where it makes sense. Pay particular attention to pages that do well in search currently– they can be a goldmine of a linking opportunity for new content.
Technical SEO and Indexing Content
For the rest of this post, I’m going to be talking about some technical issues that prevent web pages from being indexed. I should make clear here that most of these things should be itemized in Search Console for any page that Google has crawled. And again, we’re talking about pages in your XML sitemap that you want to be indexed.
For example, there’s an entire report on Mobile Usability. If Google has crawled a page on your site, and they feel as though a user’s experience will be bad on their mobile, they will certainly tell you.
I bring these issues to light because it would stand to reason that if Google runs into these issues frequently on your site, they might stop or throttle the crawling/indexing of new content until the old issues are resolved.
I can’t promise you that to be true, but if you’re struggling to get indexed, we need to look at every possibility. And fixing broken pages is never a bad idea.
Here’s what you should be looking at:
Mobile First Indexing – Your Pages are Broken
This can be one of the most frustrating things about Google.
You can sit at your desk, bring up your website on your phone, and everything looks great. Yet Google says the Text is Too Small, there’s Clickable Elements too Close Together, and other scary things.
There could be a technical reason here. Sometimes, a CSS file could be hidden from Google. Here’s some of the possible reasons:
- A CDN could be blocking it
- It could be slow to load, which causes other SEO issues
- There could be a deeper technical reason its blocked
These can be tough to diagnose, so you might need someone like us to take a look.
You’ve Told Google You Don’t Want This Content Indexed
I’m going to assume that you don’t have the site blocked from Google with either a basic Yoast setting or something in your core WordPress settings.
That said, there are other little things you can do to screw up your indexing.
You incorrectly tagged an individual page. This you definitely don’t want. If you’re covering a unique topic on a page, you want this page in the index. If for any reason there’s a noindex tag here, we want to remove it. This would also include having a canonical tag to another weaker page. Make sure your best content is what you present to Google to index.
Your Server is Sending Google Mixed Signals
This one is rare, but it happens.
If you have developers working in an Amazon AWS environment (or something similar), they can have server responses sent to Google that you don’t know about. These things would be in a Header response known as X-responses, and could send instructions to Google that you don’t want.
If your content isn’t getting indexed, and the simpler issues above don’t apply, have them take a look. There could be an issue here.
How our SEO Audit Roadmap will Help
If you feel some of the issues we’ve outlined here are the reasons you are struggling to get indexed, it may be a great time to have a complete SEO audit.
We can identify not only what issues are preventing the indexing, but many others that may be negatively impacting your overall SEO performance. See what else we include in our SEO audits.
Ready to get ranked? We’re ready to help.
