Netpeak Spider – SEO Crawler Overview: Program Tour and the Main Advantages
How to
If you want to better understand our SEO crawler Netpeak Spider, this post is the right place to start learning more. The program will help you efficiently optimize websites, save your time, and:
- Perform a complete technical SEO audit of any website and even a competitor’s one.
- Spot critical optimization issues that are slowing you down on the way to the top of the SERP.
- Scrape any data from websites – prices, reviews, comments, and a whole lot more.
- Find traffic growth points and insights exploring your site from bird’s-eye view.
- Regularly create optimization reports, including white label ones – they can be used to set up tasks for developers. At the same time, a special white label feature lets you customize SEO audits with your branding for prospects and clients.
- Upgrade your SEO skills. Explore our internal knowledge base inside the program and plenty of other content on our website (in Help Center and blog).
In this post I’ll give you as much useful content about Netpeak Spider as I can in a brief way. To make sure you got all the information and earn several points for Gryffindor, don’t miss a short quiz after reading the tutorial.
- 1. Start Crawling
- 2. Dashboard, Tables, and Reports
- 3. Segmentation
- 4. Parameters and Settings
- 5. Reports Export
- 6. Internal Tools
- 7. Brief Summary
1. Start Crawling
Let’s dive into the program to find out how it works. To start the website analysis – enter its URL into the address bar and click ‘Start’. The crawler will get the necessary data to create all the reports you can instantly use at work.
2. Dashboard, Tables, and Reports
10 minutes later we have the following reports:
- Dashboard → after the crawling is over, you will find here the main diagrams that will help you understand the 'health score' of your project. If necessary, you can share it after exporting in 1 click.
- Tables → it’s the heart of the program where it brings together every piece of data after crawling. We have integrated tables with the highest performance on the market which allow you to work with huge data sets with maximum comfort. There are rumors going on that they cope with enormous amounts of information even faster than Excel. Here is the shortlist of main features:
- Grouping
- Sorting
- Quick search
- Custom filters
- Internal databases
- Context menu
- Issues → the program spots more than 100 SEO problems and marks them with different colors according to their severity (errors, warnings, and notices). Click on any issue to see a list of pages where it was detected and read a detailed description of this problem: how we dig it out, how it can harm your website, how to fix it, and even several useful resources to leave no stone unturned.
- Overview → based on the retrieved data, the crawler calculates some insights, for example the page status or the file type.
- Site structure → on the next tab, you can explore the site structure by URL segments in tree-view form. Click on any section to explore pages from the necessary folder. Thus, you can check if all pages and files are located in the right place. Also, using the 'Export' menu, you can get the report with complete site structure and crawling parameters like title, description, and other, which may come in handy when analyzing a competitor’s website.
- Scraping → one of the main Netpeak Spider features that allows you to collect any data from the website pages. Such as prices, reviews, SEO texts, the presence of specific tags on the page, publication date, and many more. In this report, you can filter results by each scraper separately (I’ll give more details about them a little bit later in this post).
We have prepared a detailed scraping guide with real examples so you can start gleaning more right now. Scraping is a useful feature for solving a variety of tasks and saving your time, don't underrate it 😉
3. Segmentation
If you want to dig further into your website, we have a unique segmentation feature for this task (pretty much like in Google Analytics). This feature will help you separate a specific part of a website and rebuild all reports to narrow down on a necessary segment. Try applying any filter as a segment to spot issues that have been found in that exact place of the website or get an overview information.
4. Parameters and Settings
We all have extraordinary tasks that require an out-of-the-box thinking – that’s the moment when you truly enjoy customization opportunities of the tool with ‘Parameters’ and ‘Settings’.
- Parameters
You can configure which parameters the program should collect and analyze. For example, turn some of them off to save your computer resources, increase crawling speed, and use available resources to check even more webpages. By the way, clicking on any parameter, you will move to the corresponding column of the table, and a detailed description of it will appear on the info panel.
- Settings
- General → here you can:
- Set up crawling speed by changing the number of simultaneous threads.
- Enable JavaScript rendering, if you need the program to execute these scripts and find content that has been generated by them.
- Add different file types to the analysis.
- Enable multi-domain crawling. It’s a Pro feature that allows you to simultaneously crawl plenty of websites and get all the results within one report.
- Advanced → by default we crawl as many pages as we can to find more data and insights. Specially for advanced users, we created the corresponding settings section for tricky tasks.
- Virtual robots.txt → here you can test your hypotheses with robots.txt file before uploading it on your website.
- Scraping → create your own scrapers to extract any website data. I recommend you to read our subject related articles first, but if you still need some help, don’t hesitate to write to our support team.
- User agent → if you want to check your website using different values of this header, you can use the default ones or set a custom header.
- HTTP headers → some tasks may require using custom server request HTTP headers, and this is the right place to set them before the crawling has started.
- Restrictions → you can tailor issue detection to a specific project. Note that by default we’ve already gauged precise settings and always keep them up-to-date.
- Rules → this useful configuration may help you crawl only the necessary part of a website, excluding other pages by part of a URL. For example, you can check only blog posts or product pages.
- Google Analytics, Search Console, and Yandex,Metrica → Netpeak Spider is integrated with Google Analytics, Search Console, and Yandex.Metrica to give SEOs an opportunity to enrich technical website scores with data about traffic analytics, user behavior, and search queries. Merge this data to find even more insights and growth points for your projects.
- Export → here you can set up preferences on how to export reports from the tool and even right to the Google Drive (to be more accurate, right to the Google Spreadsheets) for comfortable teamwork with your colleagues or clients.
- Authentication → if your website is under development and is restricted with basic authentication, enter your access credentials here. Thus, the program will use them when requesting data from a server.
- Proxy → proxies may come in handy when you need to crawl pages from different geolocations or when a website blocks many requests from the same IP address.
- White label → it’s a Pro feature that allows you to add your branding such as logo and contact details into an SEO audit in PDF format which the crawler automatically creates in just a few clicks. Using this feature may help your report to be more personalized for prospects or colleagues.
- General → here you can:
5. Reports Export
To export the audit after the crawl, open the ‘Export’ menu and choose the corresponding item. Let’s take a look at this amazing report created with Netpeak Spider!
- The report is based on a massive data set so we’ve added contents for navigation.
- Diagrams let you quickly determine current project status and find the right place to focus your efforts for achieving maximum results.
- At the end of the report, a recipient will get your contact details in order to never forget to discuss it with you.
Besides the PDF audit, the program lets you export any tables: get more complex reports or all data in one folder in just a few clicks.
6. Internal Tools
Netpeak Spider is a program with a variety of functions and use cases. We have deployed special built-in tools to solve some tasks without unnecessary distractions:
- Source code and HTTP headers analysis
- On the left, you will see all server request and response HTTP headers as well as GET-parameters if they appear in the URL.
- On the right, you will find the page's source code and the extracted text.
- For more effective use, don’t forget about the search option to quickly find something within all this data.
- Internal PageRank calculation → it’s a tool for link analysis that may help you improve internal linking structure.
- XML Sitemap Validator → if you want to check your sitemap, this feature will help you spot more than 30 issues according to Google, Bing, and Sitemap Protocol documentations. It’s like a little Spidey within Netpeak Spider focused on sitemaps. For example, let’s check the Toyota's sitemap file. If you still haven't got a sitemap file, the next tool is right up your alley.
- Sitemap Generator → it helps you generate not only XML sitemaps but also HTML ones for even better indexing.
7. Brief Summary
As a conclusion, I want to recap what makes Netpeak Spider an outstanding crawler compared to many other programs:
- Focus on fixing SEO issues → we’ve made a dedicated report with severity groups and quick export of all website problems in a few clicks to make a seamless workflow with your developers and content managers.
- SEO audit in PDF + white label → brand your reports and add contact details to quickly conduct express audits and sign new contracts for SEO services.
- Integration with Google Analytics, Search Console, and Yandex.Metrica → find new insights after enriching crawling results with analytics data and search queries. I bet it’s not that easy to find that much useful data on the market!
- Data segmentation → a truly unique feature for a new point of view on your website. Once, I crawled our website and spent a whole week researching its deepest parts using different segments and exploring new points of growth.
- Comprehensive customizations → make parameters and settings work best for you to get maximum profit!
- Advanced results tables → the most powerful tables on the market with impressive features: sorting, grouping, custom filters, quick search, and tweaking it for your preferences.
- Internal tools → source code and HTTP headers analysis, internal PageRank calculation, sitemap validator and generator – no need to leave the tool while solving the most interesting tasks anymore.
- And as a cherry on top, optimized consumption of computer resources → this toolset uses minimum RAM while an internal database allows you to work with a huge number of URLs.
P.S. Don't miss an overview of a tool for bulk URL analysis – Netpeak Checker. I invite you to watch a video, read a post, and, of course, have a crack at quiz in the end to be the brightest spark of Gryffindor.
