Follow

How to Validate Web Scraping Protection

 

Description

  • This document explains the simplest procedure for validating Silverline Web Scraping Protection: How to validate via a browser
  • However, you may use other automated tools including wget, POSTMAN etc.
  • Requirement: To display the inline JS, you will need an environment that supports JS execution.

 

Environment

  • Silverline Web Scraping
  • Silverline Shape Defense In Flag mode
  • Web browser

 

Procedure

Prerequisite

Important: Silverline Portal must be in Flag mode before sending validation request.

 

Step 1: Validate the Initial Response Using Browser

The simplest way to validate Silverline Web Scraping Protection is by executing a GET request to the protected endpoint. This will return the inline SSD JS.

  1. Open an incognito browser window
  2. Enter the URL of the protected endpoint
  3. The initial GET request is made and will be served the initial SSD JS response.

Validating_Interstitial.png

Figure 1. SSD Web Scraping Protection workflow

 

Initial_SSD_response.png

Figure 2. Initial Response with SSD JS tag

 

Step 2: Validate Shape Headers in a Subsequent GET Response

Once the initial response is served and the browser executes the SSD JS, a subsequent GET XHR request is made to the same path. This JS will contain the Shape headers attached to the response.

  1. Open a second incognito window
  2. Open your Developer Tools from your browser
  3. Click the Network tab
  4. Type in the URL in the address bar
  5. Click the request (Request that was typed in the address bar that needs to be protected) from the list in the Request window (should appear first in the list)
  6. Locate the inline JS in the Response body:
    • In the request window you should see the same GET request being made for the second time. Click on this and look for request headers X-Vk2Vjf8v-*
  7. Go to the Silverline Portal dashboard in Flag mode to validate the request
  8. While Portal is in Flag mode, go to the Events Center to verify that your request was flagged as automated.

Request_Headers.png

Figure 3. Request Headers

 

Related Articles

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request