Description
- This document explains the simplest procedure for validating Silverline Web Scraping Protection: How to validate via a browser
- However, you may use other automated tools including wget, POSTMAN etc.
- Requirement: To display the inline JS, you will need an environment that supports JS execution.
Environment
- Silverline Web Scraping
- Silverline Shape Defense In Flag mode
- Web browser
Procedure
Prerequisite
Important: Silverline Portal must be in Flag mode before sending validation request.
Step 1: Validate the Initial Response Using Browser
The simplest way to validate Silverline Web Scraping Protection is by executing a GET request to the protected endpoint. This will return the inline SSD JS.
- Open an incognito browser window
- Enter the URL of the protected endpoint
- The initial GET request is made and will be served the initial SSD JS response.
Figure 1. SSD Web Scraping Protection workflow
Figure 2. Initial Response with SSD JS tag
Step 2: Validate Shape Headers in a Subsequent GET Response
Once the initial response is served and the browser executes the SSD JS, a subsequent GET XHR request is made to the same path. This JS will contain the Shape headers attached to the response.
- Open a second incognito window
- Open your Developer Tools from your browser
- Click the Network tab
- Type in the URL in the address bar
- Click the request (Request that was typed in the address bar that needs to be protected) from the list in the Request window (should appear first in the list)
- Locate the inline JS in the Response body:
- In the request window you should see the same GET request being made for the second time. Click on this and look for request headers X-Vk2Vjf8v-*
- Go to the Silverline Portal dashboard in Flag mode to validate the request
- While Portal is in Flag mode, go to the Events Center to verify that your request was flagged as automated.
Figure 3. Request Headers