Google’s John Mueller explains when it’s useful to fix 404 errors from inbound links and when it’s a waste of time
Google’s John Mueller responded to a thread in Reddit about finding and fixing inbound broken links, offering a nuanced insight that some broken links are worth finding and fixing and others are not.
Reddit Question About Inbound Broken Links
Someone asked on Reddit if there’s a way to find broken links for free.
This is the question:
“Is it possible to locate broken links in a similar manner to identifying expired domain names?”
The person asking the question clarified if this was a question about an inbound broken link from an external site.
John Mueller Explains How To Find 404 Errors To Fix
John Mueller responded:
“If you want to see which links to your website are broken & “relevant”, you can look at the analytics of your 404 page and check the referrers there, filtering out your domain.
This brings up those which actually get traffic, which is probably a good proxy.
If you have access to your server logs, you could get it in a bit more detail + see which ones search engine bots crawl.
It’s a bit of technical work, but no external tools needed, and likely a better estimation of what’s useful to fix/redirect.”
In his response, John Mueller answers the question on how to find 404 responses caused by broken inbound links and identify what’s “useful to fix” or to “redirect.”
Mueller Advises On When Not To “Fix” 404 Pages
John Mueller next offered advice on when it doesn’t make sense to not fix a 404 page.
“Keep in mind that you don’t have to fix 404 pages, having things go away is normal & fine.
The SEO ‘value’ of bringing a 404 back is probably less than the work you put into it.”
Some 404s Should Be Fixed And Some Don’t Need Fixing
John Mueller said that there are situations where a 404 error generated from an inbound link is easy to fix and suggested ways to find those errors and fix them.
Mueller also said that there are some cases where it’s basically a waste of time.
What wasn’t mentioned was what the difference was between the two and this may have caused some confusion.
Inbound Broken Links To Existing Webpages
There are times when another sites links into your site but uses the wrong URL. Traffic from the broken link on the outside site will generate a 404 response code on your site.
These kinds of links are easy to find and useful to fix.
There are other situations when an outside site will link to the correct webpage but the webpage URL changed and the 301 redirect is missing.
Those kinds of inbound broken links are also easy to find and useful to fix. If in doubt, read our guide on when to redirect URLs.
In both of those cases the inbound broken links to the existing webpages will generate a 404 response and this will show up in server logs, Google Search Console and in plugins like the Redirection WordPress plugin.
If the site is on WordPress and it’s using the Redirection plugin, identifying the problem is easy because the Redirection plugin offers a report of all 404 responses with all the necessary information for diagnosing and fixing the problem.
In the case where the Redirection plugin isn’t used one can also hand code an .htaccess rule for handling the redirect.
Lastly, one can contact the other website that’s generating the broken link and ask them to fix it. There’s always a small chance that the other site might decide to remove the link altogether. So it might be easier and faster to just fix it on your side.
Whichever approach is taken to fix the external inbound broken link, finding and fixing these issues is relatively simple.
Inbound Broken Links To Removed Pages
There are other situations where an old webpage was removed for a legitimate reason, like an event passed or a service is no longer offered.
In that case it makes sense to just show a 404 response code because that’s one of the reasons why a 404 response should be shown. It’s not a bad thing to show a 404 response.
Some people might want to get some value from the inbound link and create a new webpage to stand in for the missing page.
But that might not be useful because the link is for something that is irrelevant and of no use because the reason for the page no longer exists.
Even if you create a new reason, it’s possible that some of that link equity might flow to the page but it’s useless because the topic of that inbound link is totally irrelevant to anyting but the expired reason.
Redirecting the missing page to the home page is a strategy that some people use to benefit from the link to a page that no longer exists. But Google treats those links as Soft 404s, which then passes no benefit.
These are the cases that John Mueller was probably referring to when he said:
“…you don’t have to fix 404 pages, having things go away is normal & fine.
The SEO ‘value’ of bringing a 404 back is probably less than the work you put into it.”
Mueller is right, there are some pages that should be gone and totally removed from a website and the proper server response for those pages should be a 404 error response.
Google Removing Crawl Rate Limiter Tool From Search Console
The Googlebot Crawl Rate Limiter Tool will soon be removed from search console because it’s no longer necessary
Google announced that it is sunsetting the search console crawl rate limiter tool, scheduled to be removed on January 8, 2024, citing improvements to crawling that has essentially made it unnecessary.
Search Console Crawl Rate Limiter Tool
The crawl rate limiter tool was introduced to search console fifteen years ago in 2008. The purpose of the tool was to provide publishers a way to control Googlebot crawling so that it didn’t overwhelm the server.
There was a time when some publishers experienced too much crawling, which could result in the server being unable to server webpages to users.
Enough people complained that Google eventually released the tool within search console.
The impact of the tool was to provide Google with data. According to Google, requests to limit crawling typically took about a day to go into effect and remained in effect for 90 days.
Why Google Is Removing Rate Limiter Tool
The announcement stated that crawling algorithms have reached a state where Googlebot can automatically sense when a server is reaching capacity and take immediate action to slow down the crawl rate.
Furthermore, Google stated that the tool was rarely used and when it was used, the crawl rate was generally set to the lowest setting.
Moving forward, the minimum crawl rate will by default be set to a lower rate similar to what publishers tended to request.
According to the announcement:
“With the deprecation of the crawl limiter tool, we’re also setting the minimum crawling speed to a lower rate, comparable to the old crawl rate limits.
This means that we effectively continue honoring the settings that some site owners have set in the past if the Search interest is low, and our crawlers don’t waste the site’s bandwidth.”
Making Search Console Less Complex
Removing the tool makes search console easier to use because it’s less cluttered with tools that are rarely used.
This in turn should improve the user experience for search console.
Publishers who continue to have problems with Googlebot’s crawl rate can still use the Googlebot report form to send feedback to Google.
Read Google’s official announcement:
Google On Traffic Metric & SEO
Google’s John Mueller shared his thoughts on the traffic metric and how SEOs should use it
Google’s recent Search Off the Record podcast discussed the important details of SEO, including a thought provoking segment about how SEOs can benefit from realign their thoughts about traffic with other possibly more important goals.
Traffic As A Measure Of Success?
One of the most misinformed things that search marketers do is measure their success by citing traffic statistics. We’ve all seen the posts on Facebook or the articles on blogs, where a search marketer relates how they did X for a client and traffic exploded exponentially within months.
Link builders use traffic as a metric of success, content writers do it, SEOs do it.
The question that goes through my mind when I see that is, well, what effect did that have on sales or ad clicks?
Because if earnings remain flat then the traffic increase probably doesn’t matter and probably neither did the SEO work done for that traffic.
Ask any Pay Per Click expert or any affiliate marketer about the importance of conversions versus traffic and the response is clear: conversions are everything, not traffic.
Adam J Humphreys, CEO of search marketing and design consultancy Making 8 (LinkedIn profile) said this about success metrics:
“I focused on analytics early on and learned the language of business which is ROAS. This is all executives care about other than awareness.
Our job is to grow their business first as well as awareness second.
Many don’t know how to measure attribution from SEO.
Things like form fills and calls are almost always unmeasured and unreported. Clients get super excited when I talk about this because it’s almost like nobody ever cared about their success.”
What Google Says About SEO & Traffic
Martin Splitt asked the question about the business impact of traffic.
“…when you say traffic drops, what does that mean to you? Is it impressions?
Are you going to Search Console and click on the Performance report and you just look at the impressions going down or clicks?
Or do you actually measure a real business impact? Like you’re selling only half as many things as you used to sell last month?”
John Mueller added his thoughts on the topic. He tried to explain why SEOs don’t focus on return on investment (ROI) or the impact of SEO on earnings.
He offered his guess that maybe it’s the time between doing SEO versus the impact of it.
“Usually what I see from people is that they focus on the traffic on their site.
And they look at something like Analytics and they say, ‘Oh, I get so many visitors and so many visitors from search engines. And that number went down significantly.’
…And the aspect of ROI or kind of the value of that traffic, I see that as something that a lot of these SEOs tend not to focus on primarily, because my assumption is it’s just a very long lead time there.
Like you can turn your website off now, but you might still have people who are kind of like paying for something for a while.
And then, it’s like, wow…”
Mueller followed up is thoughts on why SEOs might not focus on earnings by discussing how website traffic can be a misleading metric because it doesn’t tell a lot about why something happened, it only shows what happened.
“But I do think it’s something where I sometimes feel it’s misleading to just purely focus on the traffic.
And I see that with our sites as well. A year or two ago, we would rank for the word “Google” in Canada or something like that, the search documentation, of course.
I hope the Google website ranks for the word Google. But like the Search documentation would rank for the word Google somewhere on the first page.
And we got tons of traffic there, but all of that traffic was basically irrelevant.
And then if you only look at the traffic and all of that irrelevant traffic goes away, then it might look like you lost a lot of traffic, but actually it’s like all of those things are people that weren’t relevant for your site anyway.
So you almost need to look at the bigger picture of all of the traffic that’s gone, but also keep in mind like, well, a lot of this was useless and I should maybe focus on the queries that people use, and then clicks and impressions for those, or individual like lower level pages of the site and kind of track those a little bit more.”
Traffic And SEO
SEOs promote themselves through case studies showing all the traffic their efforts brought. But those studies are hollow and maybe even deceptive if there’s absolutely no reference to how much sales lift resulted after their effort.
Link builders do a similar thing where they promote all the hundreds or thousands of links they acquired for a client and sometimes mention the lift in traffic. Just as consistently as SEOs, they always leave out the effect on sales or earnings.
Why do they do that? I suspect that many of those SEOs have never actually built a business around monetizing affiliate sales or ad clicks and thus don’t have first hand experience from that side of the SEO fence.
A lot of the old school SEOs like myself learned what we know from building and monetizing websites, maybe because client work for SEO wasn’t really as much of a thing then as it is now.
Using traffic as a metric is useful for measuring the impact of SEO but traffic should not be the goal of SEO. The reason is because there are different kinds of traffic.
Some traffic converts into sales. Some traffic has a lead time from the first visit to the sales. Some traffic is useful for building a brand name.
But some traffic is not relevant or useful.
When it comes to diagnosing traffic drops related to ranking changes, it may be helpful to understand if there’s any impact on sales and if not, to understand why the traffic drop had no monetary impact and if content and SEO efforts might be better directed in a different direction.
In a world ruled by algorithms, SEJ brings timely, relevant information for SEOs, marketers, and entrepreneurs to optimize and grow their businesses — and careers.