How To Configure 301 Redirects From Old URLs To New Ones When Upgrading Or Migrating a Website

A website is a dynamic system that’s constantly changing and improving as it adapts to new conditions. Sometimes, when updating an Internet resource (say, if you’re moving to another CMS), the URLs of all a site’s landing pages change. This leads to a problem: all the old URLs show a 404 server response (page not found). Example:

• old URL: https://website.com/category/subcategory/page-1.html
• new URL: https://website.com/page-635.html

Instead of pages not found, we can see a standard apology page from the website administration, similar to this one:

As a result, the resource quickly loses its positions in search results, and along with them, its traffic, which can be catastrophic. To avoid this, you must immediately redirect users from the old pages to the new versions.

If your website is small (several dozens of pages) you can compare old and new pages manually, but this won’t be feasible for larger resources with hundreds or thousands (or even tens of thousands) of pages.

In this article, we’ll analyze how you can use Netpeak Software to automatically compare pages before and after updating a website in order to set up page-by-page 301 redirects. This method is great for very large websites with hundreds of thousands or even millions of pages.

How 301 redirects work

301 is one of many possible HTTP status codes and works like a mail forwarder. Once you've moved a content piece away from a specific URL, anyone trying to access it will receive a 404 error message. To improve the user journey, you can send a request to a server to forward visits from the old URL to a new location exactly by using a 301 redirect to new domain.

When you try to visit the old URL, the server sends back the 301 — Permanently Moved status code and then brings you to the newly selected location.

This happens so quickly you're usually unaware of it, yet you might notice the URL differs from the one you clicked on or typed in.

The other key role of a 301 redirect is related to search engines. Having useful status codes that properly signal where you've moved a content piece allows search engines (e.g., Google and Bing) to keep their index up-to-date.

When should you use a 301 redirect to new domain

Here’s the list of cases when you should use a 301 permanent redirect instead of another similar type.

Permanently moving a page to a new URL

Sometimes, you need to change the URL of a page on your site due to changes in the product name or to better categorize your site's pages. The reasons are numerous.

Implementing a 301 redirect will make sure users and search engines get redirected to the new URL. This will also help search engines pass the SEO power of the old URL to the new URL.

Deleting a page

If you want to delete a page on your site for any reason, redirect it to another relevant page if possible.

It creates a better user experience and offers people an alternative. For instance, if you just delete the page, users who click on a link to it will land on a page with an error message and will likely bounce.

In most cases, using a 301 redirect is the best way to send users to a similar page on your site with the same search intent.

Migrating a site to a new domain

You may also want to migrate your site to a new domain entirely — for example, from https://www.website.com to https://www.newwebsite.com.

Imagine you’re migrating from a .net to a .com, or you’ve rebranded your business and need to move to a different domain name. Using a 301 redirect is the best way to forward URL to another URL, as well as notify Google using Google Search Console's Change of Address tool.

Changing the website's structure

Change your website's structure to organize your content more easily. Thus, you’ll help Google better understand how your pages are related, so forward webspge via 301 redirects.

Whether that’s a blog, an ecommerce category, or basically any other, a permanent redirect is the best option to implement the change.

Moving from non-WWW to WWW URLs or solving duplication issues

For example, you have to move from https://www.website.com to https://website.com.

In this case, make sure your website's main version is either non-www or www. This will make your site look more consistent to users and search engines and avoid duplication issues.

A 301 redirect will help indicate the main version so that users and search engines can properly redirect to a preferred version.

Switching from HTTP to HTTPS

If you want to reroute website from HTTP to HTTPS due to security concerns and make use of it as Google's ranking factor, use a 301 redirect.

Doing so will ensure Google properly indexes the new protocol and redirect users to the right pages.

Merging two or more domains

Let's take this example: from https://www.website.de to https://www.website.com/de/.

Sometimes, you'll need to merge a domain with another one—for instance, after business mergers and acquisitions. Merging using a 301 redirect is the way to save and maintain as much SEO potential as possible.

Resolving upper-case vs. lower-case issues

For example, you need a change from https://www.website.com/Page-Name/ to https://www.website.com/page-name/.

Upper- and lower-case URLs can be seen as different pages — even one capitalized letter in a URL can make a difference, and search engines may consider it a duplicate. Hence, using 301 redirects is usually the best way to resolve it.

Collect the pages of the old and new site using Netpeak Spider

First, you must collect all the important pages of the old and new sites. There are two ways to do this:

Get the list of URLs from XML map files

This method is suitable if the old or new website has an XML map listing the desired pages. To get this list, go to ‘List of URLs’ → select ‘Download from Sitemap’ → enter the URL of the XML sitemap → click ‘Start’ → click ‘Transfer’:

Collect the list of URLs using a website crawler

This method will take a little longer, since the program will need to scan the entire website, but it’s a great idea if the old or new website doesn’t have XML maps listing the pages it needs. To use the web crawler, enter the URL of the website and click the ‘Start’ button:

The crawling will start from the specified URL. It works until it finishes crawling all pages (unless the program settings limit the crawling). Use the hints to quickly add a URL — they'll appear below the line as you enter text. The hints are created based on your previous inquiries.

If you decide to enter them manually, an additional window will open. In this window, you can enter a list of URLs, each one on a new line. After you've entered all URLs, click Add to save them.

You can also upload target URLs and forwarding links by uploading a separate file or extracting them from a sitemap.

Once you start crawling, keep in mind the following:

• Netpeak Spider only crawls pages based on HTTP / HTTPS protocols.
• If the protocol is not specified, it will automatically add a safe HTTPS protocol to the beginning of the URL address.
• The main table can only contain unique URLs.
• URLs will be percent-decoded in a user-friendly form.
• Anchors in hash links will be removed. For instance, instead of https://example.com/test/#anchor, Netpeak Spider will add https://example.com/test.

Moreover, depending on the way you've added URLs, the crawling process will differ:

• If you've entered an initial URL but the main table remains empty, the app will start crawling the whole website, starting from the initial page and going on until it crawls all pages.
• If the Initial URL field and the table contain some URLs, Netpeak Spider will continue crawling, adding new pages to the previous results in the table.
• When there are no initial URLs, but the table contains a list of pages, Netpeak Spider will crawl this list, detecting outgoing links but not following them.

Netpeak Spider also allows you to change the initial URL and add a list of URLs in a single project. This is convenient if you need:

1. To crawl multiple websites in a single project, for instance, to check for duplicate content or their interlinking. Follow these steps to do it:

• When the crawling is over, replace the address in the Initial URL field with a new one or add a list of target URLs to the main table.
• Click Start.

1. To crawl specific website categories in a single project. Follow these steps to do it:

• Before crawling, set it up in a folder on the General tab in the settings.
• Enter the necessary URL and click Start.
• When the crawling is over, replace the initial URL and resume crawling.

Recrawling pages

Netpeak Spider helps you recrawl only part of the URLs you've added — for instance, after you made some changes on a website.

Recrawling specific pages

To recrawl one or several pages, select them in "All results," right-click to access the context menu, and choose "Recrawl URL." Netpeak Spider will update the parameters and links for all the selected pages.

5.2. Recrawling a table of results

You can also recrawl a table of results — for instance, after you've applied a particular filter. Here's what you need to do for that:

1. Save a project to have a chance to return to the initial condition later — plus, it'll save you some memory.
2. Choose a necessary issue report in "Issues" on the sidebar by which you want to filter the results (e.g., "Missing or Empty Title")
3. Right-click on any cell in "Filtered results" and select "Current tabl", then "Recrawl the table." You can also use the Ctrl+Shift+R hotkey.

5.3. Recrawling all results

You can also recrawl all the pages (e.g., after moving a website to HTTPS protocol). To do so, clear the Initial URL field and click Restart. After that, Netpeak Spider will only recrawl the pages that were added to the main table.

5.4. Recrawling broken pages

If you have removed the links to the broken pages on your website to make the app fully display the changes, you'll have to recrawl the website.

If you haven't removed the links and made the pages available using the same URLs, you can see the changes by recrawling the table "Broken pages."

Extract information for mapping

To match old and new URLs, you need to highlight the elements that will be the same for both versions of the pages. As a rule, these elements are:

• the page’s header H1
• unique product identifiers (SKU)

You can also match other blocks that are unique within the website, but the same for the pages being compared.

To add H1 headers for the list of pages, check the ‘H1 content’ option and click ‘Start’:

For SKU scraping or other unique page elements, you should use the Netpeak Spider parser: open ‘Settings’ → select ‘Scraping’ → enter the name of the parsing parameter in the ‘Name’ field → select the ‘XPath’ scraping type → fill in the field with the XPath value of the block where the site displays the SKU or other identifier → Select ‘Inner text’ as the type of information → Save the settings by clicking ‘OK’ → Mark the scraping name in the right column with parameters → Click ‘Start’:

To copy the XPath of the desired block, open the browser inspector panel (Ctrl + Shift + C) → select the block in the code → right-click → select the ‘Copy’ item → click ‘Copy XPath’:

Example XPath:

//*[@id="productOverview_feature_div"]/div/table/tbody/tr[2]/td[2]/span

To export scraped data, open the ‘Export’ item → select ‘XL (extra large) reports from database’ → click ‘Scraping data and all results in single file (XL)’:

Map the URL

To match the URLs, we need a Google Spreadsheet with a list of URLs and SKUs (or some other shared parameter) and a VLOOKUP function. Insert the data in the table as follows:

• column A: unique identifiers (SKUs) or H1 titles of old website pages
• column B: unique identifiers (SKUs) or H1 titles of new website pages
• column C: new website URLs
• column D: old website URLs

Enter this formula in column E:

=VLOOKUP(A2;B:C;2;0)

Example Spreadsheet:

Decoding the obtained results:

• The URLs in column D are the pages from which you need to set up a 301 redirect
• The URLs in column E are the pages to which you need to configure 301 redirects.

Important! All redirects from old URLs to new ones must be implemented with a 301 code.

Related issues and questions

Now that you know how to redirect a website and configure different types of redirects, let's look at some of the most widespread issues you may encounter while working on it.

Why is my website redirecting to another website?

Sometimes, hackers can insert malicious redirect codes into websites. This code can redirect a visitor to another site to generate promotional impressions or boost their SEO. Quite a few malicious redirections can have severe, damaging effects.

How can I resolve a 301 redirect conflict?

If you try to set up an automated redirect for a page you've already redirected manually, you'll receive a notification that it wasn't possible to create it. However, you can resolve this conflict in the URL Redirect Manager: either keep the old redirect or save the new one instead.

To resolve a conflict:

1. Click Fix Now on the pop-up or launch your SEO Dashboard and select Go to URL Redirect Manager below Tools and settings.
2. Click Retry.
3. Select how you want to resolve the conflict:

• Update Redirect: Update your manual website forward to a new URL slug.
• Keep Existing Redirect: Keep the manual redirect you've already created.

Check URLs for SEO parameters with Netpeak Checker

As we've established earlier, regular checkups are essential if you want to streamline your website's performance and make sure everything works perfectly, including the forwarding links. In that sense, Netpeak Checker comes in handy.

This powerful tool offers plenty of features and enables integrations with various SEO-related services. Here's what else you can do with Netpeak Checker:

Integration with 25 other services to analyze 450+ parameters

Netpeak Checker enables integrations with 25+ SEO services: Moz, SimilarWeb, Ahrefs, Serpstat, Google Analytics, and many more.

50+ on-page parameters

Netpeak Checker delivers research results in a real-time dashboard. The metrics you can monitor include redirects, titles, response time, status codes, mobile-friendliness, and many others. All you have to do is choose the required stats and hit "Start."

Website traffic estimation

Netpeak Checker shows the traffic volumes on a target page, potential link-building donors’ share ratios, and traffic by location. Plus, it shows what types of traffic are prevailing on a selected page (search, organic, direct, mail, social, etc.).

Batch Core Web Vitals checkup

You can also retrieve data from Google PageSpeed Insights to analyze and monitor your website's loading speed, responsiveness, and visual stability.

Integration with Google Drive & Sheets

Connect your Google Drive account to Netpeak Checker so you can promptly export any report to Google Sheets and easily share them at any time.

Conclusions

We don’t recommend changing the URLs of website pages unless absolutely necessary. However, if URLs have changed as a result of updating the resource, you must configure 301 redirects from the old URLs to the new ones. This should be done simultaneously with the release of a new version, or shortly after it, if there must be a delay. This is the only way to minimize the potential drop in positions and traffic. If you wait too long to set up the redirects, the resource’s audience may weaken substantially, and it will take a long time to restore it.