How to Analyze Competitors’ Content with Netpeak Spider

4
2
4
2
How to Analyze Competitors’ Content with Netpeak Spider
Use Cases

Creating and improving your website content are some of the most complex parts of comprehensive website promotion. How to find proper post topic and how to make it more interesting and viral? If you focus only on your own website, it will be difficult to find answers to these questions. Our advice is to look closely at your competitors. That’s right, we propose you to use your competitors’ experience and automatically figure out which of their posts are most popular and why.

In this article, you will learn how to:

  • find the most popular posts on your competitors’ websites
  • scrape automatically their user engagement rates
  • figure out the length of the most popular posts
  • get new ideas from competitors for your publications

As an example, we used businessoffashion.com website.

To perform these tasks we’ll use Netpeak Spider. We’ll analyze all necessary parameters using general crawling settings (or crawling only the directory with analyzed content) and scraping with a few simultaneous searches.

1. Engagement Rates: Scraping Settings

To scan and scrape data from competitors’ websites, we’ll set two simultaneous scraping searches: by shares and by comments. Keep in mind that metrics to measure engagement on selected websites can differ from comments and shares to views, likes (upvotes), etc.

To set and launch scraping, perform the following actions:

  1. Open one of the content pages.
  2. Find shares (comments, views) counter and highlight it.
  3. Right-button click on it and select ‘Inspect’.
  4. In the opened window with page source code you’ll find an element responsible for showing number of shares (it will be highlighted when you hover the mouse over).
  5. Right-click on it and select ‘Copy’ → ‘Copy XPath’. Keep in mind that the type of data which is used for scraping (XPath, RegExp, CSS-selector) may differ according to the website structure. In most cases, this task is carried out through XPath.
  6. How to copy Xpath
  7. Launch Netpeak Spider and open ‘Settings’ → ‘Scraping’ and check ‘Use HTML scraping’.
  8. To make the process clear, set the name of the conditions according to the type of scraping data (‘Comments’, ‘Views’, ‘Shares’, etc.).
  9. Select type of data (XPath) and Data Extraction mode (Inner text).
  10. Insert the piece of resulting code into the search box.
  11. Scraping settings to analyze competitors content in Netpeak Spider
  12. Repeat points 2-9 for each metric you interested in and create additional scraping searches.
  13. Open ‘General’ tab in the Settings window.

2. Scanning Settings: Basic parameters

Before starting scanning and scraping process you need to set several key parameters:

  1. Find ‘Basic crawling settings’ in the ‘General’ tab . Turn off ‘check images’, ‘check PDF’, ‘check CSS’, ‘check JavaScript’ and ‘check other MIME types’.
  2. Choose scan mode. If you’re interested in content from the whole website, you need to choose ‘Crawl all subdomains’. If you’re interested in selected category or directory choose ‘Crawl only in directory’. In case of Business of Fashion we’ve chosen /articles/ directory and set a limit for crawled pages – 1000 URL.
  3. General settings to analyze competitors content
  4. Set custom crawling ‘Rules’, if you need some specific pages not to be limited by one directory. Details on how to perform this procedure are available here.
  5. In the Netpeak Spider main window open ‘All results’ tab and right-click on the line with rows names (painted in blue). Turn off all parameters excluding ‘Title’, ‘Description’, ‘Words in <p>’, ‘Characters in <p>’ and other potentially useful indicators.
  6. Start crawling.

3. Results Export and Analysis

Now Netpeak Spider final results table includes only the parameters you need for building your content strategy (paragraph 2, point 4). Based on this data, you can draw a conclusion about optimal post length and method of building winning Titles and Descriptions.

After the completion of scanning, you can export resulting data in a convenient table format. Push the ‘Export’ button placed above scanning results window.

Export results about analyzed competitors content

Received report will contain information about all parameters you have chosen in basic crawling settings and scraped data. If you need to download scraped data as a separate file, you can proceed as follows:

  • Open ‘Scanning’ tab in sidebar and choose ‘Scraping’.
  • To see scraping results table push the ‘All Results’ button.

    Scraping results in Netpeak Spider

    In the opened window you’ll see only the custom scraping data (in our case it’s a number of shares and comments). By sorting out results by one or the other parameter you can see posts with the biggest (or smallest) number of comments, shares, likes or views.

    On the basis of businessoffashion.com scanning we’ve sorted gained data and figured out which posts were the most popular. Among those, we’ve highlighted the following leaders with the biggest number of shares or comments:

    • Miroslava Duma and Ulyana Sergeenko Accused of Racism, Homophobia and Transphobia: 77 comments.
    • Gucci Bans Fur: ‘It’s Not Modern’: 44 888 shares.

    However, do not draw conclusions based only on the two (or more if you have more than two key metrics) most popular posts: we highly recommend to analyze more representative samples with a high number of shares or/and comments.

    Based on the received data, we can determine some parameters of highly demanded content: its topic, length (number of words or symbols) and way of writing title and description.

    Export all scraping data

    It is worth mentioning that each scanned URL gets its own number. It means that if you work with two separate reports, one URL in both of them will have the same number and it makes your work with data much easier.

    Unique URL sequence number

    Summary

    To analyze your competitors content and find their most popular posts (most viewed, commented, viral), you need to follow these easy steps:

    1. Decide which user engagement metrics you need to analyze first.
    2. Copy their XPath.
    3. Choose scanning mode and set corresponding Netpeak Spider settings.
    4. Launch crawling and scraping.
    5. Filter resulting data.
    6. Export crawling and scraping results and draw the appropriate conclusions.

    And how do you analyze competitors’ content? Do you use similar methods? Share your experience with us: maybe it will provide a basis for our next article ;)

    If you still have any questions, feel free to ask them: we’ll be glad to answer.

Read this post inRussian