The Dreaded 404 Page (Not Found): What It Is, and How to Find & Fix It on Your Website

Websites are constantly changing and evolving, which often leads to the disappearance of some pages or changes in their URLs. As a result, site visitors or search engine crawlers end up on non-existent (or deleted) URLs. When the desired URL can’t be located, the server returns a 404 response code (Page not found). In this article, we'll take a look at what 404 pages are, how they affect your site, and we’ll tell you how to find and optimize them.

What Is a 404 Page (Not Found)?

A 404 page, or ‘not found’ URL, is a standard HTTP response from the server informing the user that the requested resource is unavailable. 404 page errors are displayed when a user tries to access a non-existent or deleted URLs of a site. They may read ‘404 page not found,’ ‘404 http response,’ or other similar messages. Here’s an example of such a URL on the Netpeak Software site:

How Does the Presence of Urls With a 404 Server Response Affect a Website?

404 pages can negatively affect the user experience and search engine optimization (SEO) of a website in a variety of ways:

• reduces user trust in the site
• increases the bounce rate
• reduces the time users spend on the site
• wastes the crawling budget (search engine resources for website crawling), resulting in less comprehensive crawling and indexing of real site URLs

404 pages do not have a direct negative impact on website promotion, but the above factors that arise from such URLs can cause a deterioration in website rankings and, as a result, a decrease in traffic.

How to Find 404 Pages

There are many different ways to identify 404 pages on a website. Let's take a look at some of them below.

Search for 404 pages using Google Search Console

Google Search Console (GSC, formerly known as Google Webmaster Tools) is a free service provided by Google that helps site owners, web developers, and SEO managers monitor, customize, and track their sites on the Google search engine. This tool provides important data and analytics that help to optimize sites in terms of search engine optimization (SEO) and solve possible problems related to the display of the site in search results.

Among the many GSC reports, there is a report with URLs that return a 404 server response (page not found). To view the data in this report, go to the ‘Pages’ section and select ‘Not found (404)’. Here’s an example:

The report will show a graph of the dynamics of the number of URLs with a 404 server response and examples of such URLs on the site:

This method of finding 404 pages is one of the most common, but it has several drawbacks:

1. To check the report, you need to have a Google account with access to the Google Search Console of the desired website. Therefore, you cannot check someone else's site in this way.
2. The report is limited to a list of only 1000 URLs. If your website has more than 1000 incidents of 404 pages, you must first fix them and send a request for the site to be rechecked, and only then will the next batch of URLs with a 404 response be loaded.
3. The GSC report identifies only those 404 pages of the website that the search robot has already found and tried to crawl. Accordingly, the URLs that the search engine crawler has not yet reached will not be shown in the report.

Despite the above disadvantages, the Google Search Console report is a fairly effective method of finding 404 pages on small websites. To find such URLs on large sites, Netpeak Spider is a great option, which we will discuss below.

Finding 404 pages with Netpeak Spider

Netpeak Spider is a desktop SEO audit and site analysis tool that helps to identify various issues related to website structure, navigation, and optimization. This helps site owners and SEO managers improve their websites’ search engine rankings and provide a better user experience.

To check the presence of 404 pages on your site with Netpeak Spider, enter the URL in the program field, start scanning the site and select the report ‘4xx Error Pages: Client Error’ report. The ‘URL’ column will show the URLs with a 404 server response:

By scanning your site with Netpeak Spider, you can find 404 pages in advance and fix them before they are discovered by a search robot.

How to Fix and Optimize 404 Pages

Here are some suggestions for how to fix a 404 page not found error on your website:

1. Remove or fix broken links. Check all the links that lead to 404 pages and remove them or replace them with appropriate ones.
2. Create a useful 404 page. Instead of a standard ‘not found’ page, create a useful and user-friendly URL that explains the reason for the error and offers options to continue on the site (for example, a search or navigation menu). See the example from the Netpeak Software above, or do a quick search to find other helpful 404 page examples.
3. Set up redirects. If a page has been deleted or moved, set up a 301 redirect from the old URL to the new one to ensure that users and search robots are correctly redirected.

Regularly check your site for new 404 pages to detect and fix them before they cause problems for users.

Conclusions

A 404 page is a standard server response to a missing resource. The presence of such URLs can negatively affect the user experience and the SEO of a website. To find 404 pages, you can use tools such as Netpeak Spider, which helps to analyze the site and identify problematic URLs. Learning how to fix 404 pages is critical to maintaining a user-friendly site. By following the tips and tricks provided in this article, you can reduce the negative impact of 404 pages on your site, and improve user experience and search engine optimization.