Isn’t it crazy when pages that should be blocked by robots.txt end up in search results anyway? During this week’s Wednesday Workshop, we break down just how and why this happens.
VIDEO TRANSCRIPT
Welcome back to another Wednesday Workshop from DealerOn.
Last month, during our video on Technical SEO, we discussed that Google will no longer support robots.txt files that contain noindex directives starting September 1st. Rather than using robots.txt to block pages from being crawled, implementing noindex in robots meta tags is a better option.
But when you’re using robots.txt to hide a page from search engines, have you ever noticed that sometimes it gets indexed and ranked anyway? Today, we are going to investigate why this happens.
Google has said that it cannot index content that is blocked by robots.txt, and will typically index more accessible pages first; but it does its best to determine how relevant the content is to a search query through a couple of methods.
First, Google will compare a page’s URL with other URLs to find similarities and consistencies that may be descriptive of page content. However, this can still be difficult when a page is set to noindex.
Then, Google will evaluate the sources and types of links coming to the robotted page. In doing this, they are trying to establish relevance as it pertains to the search query.
If Google determines through backlinks to the blocked page that the content could be relevant, it may then display the page in search results; and if people are linking to the robotted content, Google feels it may be worthwhile for other users.
So what does this mean for you?
You need to limit the type of content that you block with your robots.txt file, as Google may choose to guess what it is and rank it anyway. Guessing probably won’t work out to your benefit! If it is something you truly need to have omitted from search, implementing noindex in robots meta tags is one of the best ways to prevent indexation.
That’s all the time we have left for today’s workshop. We covered a very challenging topic today; so feel free to leave your questions and comments down below, and we’ll get back to you shortly. Thanks for watching. We’ll see you next week with another Wednesday Workshop from DealerOn.
Thanks Kelcey Drapp, Sharing a great article,
I have one issue regarding URL, I edited the old URL, but in Search Console, in the coverage section its showing Error due to 404, also showing URL in Index, I want to show my edited URL. I also deleted this URL in webmaster but still showing in the index, plz guide what can I do for removing forever.
Another Question is I have deleted my old page in Search console in Remove section, still, show in Index, how can I delete forever? Kindly Guide me.
Ravi, make sure you’re using 301 redirects to point old pages to new pages. You can also look into 410 status codes to makes those pages drop from the index for good; but if you’ve notated these changes in search console, you should see pretty quick results.
The information you’ve shared in blog is remarkable. Thanks for sharing such quality information.