Netpeak Spider 3.6: Search Console Queries, Custom HTTP Headers

UpdatesVideo
11Like
Comments
Share

You probably haven't recovered from joy in the wake of Netpeak Spider 3.5 release, but we're ready to strike you with a new version of the crawler! Yes, we don't allow your ideas to get bottlenecked 😊 Normally, it happens like this:

This is how our Product Manager suggested cool feature, and Lead Software Engineer managed to implement it in just two weeks

We decided not to stiffen our development zeal. Here you go, mates, meet not only export of queries from Google Search Console but also custom HTTP headers.

So each in turn, below 👇

  • 1. Export of Search Queries from Google Search Console (Pro plan)
  • 2. Customization of HTTP Request Headers
  • Perks for You
  • In a Nutshell

1. Export of Search Queries from Google Search Console

We keep on scaling up our Pro plan, so go and catch a new PROfessional feature.

With new features in Netpeak Spider 3.6, you'll be like Aquaman among mortals :)

We've started to analyze queries and connect them with data on the crawled pages – so you better believe that a new portion of aha moments is waiting for you 😎

Here's a post which describes four optimization insights to using this feature: 'How to Use Export of Queries from Google Search Console (GSC) in Netpeak Spider'.

1.1. Why Do You Need This Feature?

Export of search queries from Google Search Console allows you to:

  1. Analyze data from GSC in the format ‘URL – search query – major metrics (impressions, clicks, CTR, average position)’ in Netpeak Spider. It significantly empowers your opportunities, since in GSC itself, you're confined only to two options: analyze main indicators by URL, or by search queries.

    Queries in Google Search Console are displayed in a separate ‘Queries’ tab. To see the URL, you need to open another tab
    Search queries analysis in GSC
    This is how the URLs that match the queries are displayed in Google Search Console
    URL analysis in GSC
    In the database table of Netpeak Spider you can compare this and other data from Google Search Console in one table at once
    This is how it looks in a single report in Netpeak Spider

  2. Quickly export absolutely all queries URL ranks for. While in GSC, you're restrained to 1000 results only 😢
  3. Search for the missed optimization opportunities and find the growth points in your project → for instance, export only those queries which are close to the top but not in the top-10 (from 10 to 30 position) and optimize pages to drive more traffic to them.
  4. Optimize meta tags: steer by relevant queries from GSC – it helps catch inspiration when all ideas for a unique title, description, or h1 left you.
  5. Assess your website performance, enriching GSC data by additional parameters: device (mobile, desktop) and country. Thus you can see which types of devices and countries respectively get a small number of impressions / clicks, low CTR / average position. Or where the things are too bright 👌
  6. Compare dynamics → you can find new, missed, increased, or decreased queries. The algorithm is simple:
    • Export data on one period to the file
    • Do the same for the previous period
    • Compare two sets of data from files in the pivot table in Google Sheets or Excel. View the example of table →

1.2. This Is What We Added

  • A new section with queries settings on the 'Google Analytics & Search Console' tab
  • New data enrichment parameters (device / country)
  • 'GSC: Queries' parameter
  • An inner table that allows approaching received data
  • XL report ‘Queries from Search Console' in the 'Export' menu
  • 'GSC: queries' item in the main table context menu

New featutes that we added in Netpeak Spider 3.6

Read other details in the User Guide: 'Export of Search Queries in Google Search Console.'

Grab the benefits of new functionality in the Pro plan. Users with an active license have the opportunity to upgrade their Standard plan to Pro.

Upgrade to Pro Plan

2. Customization of HTTP Request Headers

We've implemented the opportunity to configure HTTP request headers thanks to our users' feedback mentioning the range of some specific tasks:

  1. Crawling of test websites with HTTP headers protection. Sometimes, developers set up protection on the website that allows you to get to it only after entering a particular HTTP header (usually they deploy IP protection or login / password protection, but we've already covered these cases).
  2. Visiting the site logged in as a specific user. To do so, go to the website in a browser, log in, copy cookie, enter it in the program, and Netpeak Spider will crawl website on behalf of the user. Be cautious and attentive when performing this task to ensure that Spider crawls the website correctly and doesn't turn everything upside down. Potentially, it can follow the links an administrator has access to and delete pages, create new topics, etc.
  3. Testing the website in various situations. For example, if you need to check how a website determines with 'Accept-language' header what language of the page to show to the user → with 'Source code and HTTP headers analysis' tool, these checks are quick-fire. Likewise, you can check any other headers (take, for example, 'If-Modified-Since'). Our QA team regularly uses Netpeak Spider for testing – they save their time and feel upbeat about it.

You can choose any of the 19 draft headers, or write your custom one. Also, this header is passed to the inner browser for crawling in JavaScript rendering mode (to put it mildly, it wasn't that easy at all 😂).

On the 'HTTP headers' settings tab, set your own headers, open the 'Source Code and HTTP headers analysis' tool, enter the URL you need and see how the crawler will request the server with the given headers

We'll show you the ropes in the User Guide: 'Custom HTTP Headers.'

Perks for You

All our users get a free trial period until February 14, inclusive – pamper yourself with new functionality 💗 If you're not registered yet, simply hit the button and let it roll!

Sign Up and Start a Free Trial

Need assistance in harnessing new features? Our specialists will be delighted to help you get through all these hard knocks and bumps! Book personal video demonstration to learn all the cases and ways of using new features.

Book a Demo

In a Nutshell

In the updated Netpeak Spider 3.6, we've implemented:

  • Export of search queries from Google Search Console – professional features for comprehensive analysis of website performance. Export all queries URLs rank for from GSC, find missed optimization opportunities, and new growth points. Top is the limit 🏁
  • Customization of HTTP headers – an important feature for users with specific tasks, be it a versatile website testing or crawling of the websites with HTTP headers protection, or visiting the website under the guise of a specific user.

That's the way the cookie crumbles! Please leave comments, tell your friends and relatives, subscribe to channel, smash that thumbs up button... well, you got it 😬