This is such a great feature, I'm glad I'll be able to generate sitemaps inside my favorite tool from now on. Netpeak Spider makes it so easy to analyze my own site, those of my clients and competitors. I'll keep my eyes open for more updates! Many thanks:-)
Thank you very much for the feedback! ;)
Thanks for the update guys, I've been waiting for this!
Thanks for the feedback, Terry!
Please, check it out and tell us, how can we improve this feature ;)
Good work guys! Can't wait to see what's gonna happen to Netpeak Checker;-)
Thanks for the feedback, Holly!
Right on time. This was on my wishlist because I've been doing a lot more Xpath scraping lately with that other crawler.
Devin, it's a good news for us! ;) I appreciate you sharing your thoughts. Now we going to make Netpeak Checker great again... and it's not a joke!)
Hi guys! Aleh Barysevich here, CTO and Founder of SEO PowerSuite. Congrats on the launch of the updated spider, it's always good to see the tools getting better with time :) Could you please share the methodology and data you used to run the experiment and compare other crawlers? I think this information will be useful for everyone reading your blog. Thanks in advance :)
Greetings from me and Netpeak Software team! ;) Sorry for a long response.
As I’ve mentioned in the post, we used identical settings in tools (as much as humanly possible according to their settings) and crawled the same websites to make comparison. I’ll explain in more details:
1. Same settings → crawling directory, number of threads, types of crawled content, indexation instructions, User Agent, IP address.
2. Same hardware → Windows Server 64-bit, i7 2.67 GHz processor, RAM 24 GB, usual hard disk (not SSD).
3. Same domain → we’ve run the experiment using opera.com website. It has quite a lot of webpages and fast server response time that allows to crawl up to 70 URLs per second.
4. Same full access to all tools → we’ve purchased Screaming Frog SEO Spider license, WebSite Auditor (‘Enterprise’) and Sitebulb.
5. We’ve checked 1K pages with Netpeak Spider before every crawling for consistency with such benchmarks as crawling speed and the quality of results (they almost never change during in-depth crawling). If everything was OK, we started crawling with the next tool. This is necessary to exclude the factor like ‘Of course, you’ve crawled 100K with your own tool and after that Opera blocked us, that’s why we’re slower.’
6. We’ve run the experiment several times (quite a few times with some tools because some of them couldn’t take even 100K). We’ve got similar results, I mean they’re quite representative.
Thank you for words about Netpeak Spider development, I’m really happy to hear it. We’re constantly working on it! You’re working great too and have a good tool :)
Aleh, it there anything else I can help you with?