Follow

How To Troubleshoot Silverline Web Scraping Protection

Description

This document explains the process for troubleshooting issues with Silverline Web Scraping Protection. If you are experiencing issues with this service, please follow the procedures in this document.

 

Environment

  • Silverline Web Scraping
  • Silverline Shape Defense in Flag mode
  • Web browser

 

Procedure

Prerequisite

Make sure your Web Scraping Protection (Shape Defense) is configured properly: How to Configure Silverline Web Scraping Protection (Shape Defense)

 

Step 1: Validate the Initial Response Using Browser

Validate Silverline Web Scraping Protection by executing a GET request to the protected endpoint. This will return the inline SSD JS (Silverline Shape Defense JavaScript).

Important: Silverline Portal must be in Flag mode before sending validation request.

  1. Open an incognito browser window
  2. Enter the URL of the protected endpoint
    • Fill in the Protected Endpoints with the endpoint (URI) that you want protected.
    • All traffic sent to this endpoint's address is sent to Shape, where it is categorized as a human or automated.
    • Total # of protected endpoints allowed is limited: see Q&A: What are the Silverline Portal Configuration Limits?
    • Protected endpoints support glob pattern match characters: * ? [ ]
    • Must start with /
    • May contain characters valid for an HTTP path [a-ZA-Z0-9 - _]
    • Any open bracket [ must have a close bracket ]
    • May contain glob characters in any position after the initial /
    • Does not allow /* as protected Endpoint
  3. The initial GET request is made and will be served the initial SSD JS response.blobid0.png

Figure 1. SSD Web Scraping Protection workflow

blobid1.png

Figure 2. Initial Response with SSD JS tag

 

Step 2: Validate Shape Headers in a Subsequent GET Response

Once the initial response is served and the browser executes the SSD JS, a subsequent GET XHR request is made to the same path. This JS will contain the Shape headers attached to the response.

  1. Open a second incognito window
  2. Open your Developer Tools from your browser
  3. Click the Network tab
  4. Type in the URL in the address bar
  5. Click the request (Request that was typed in the address bar that needs to be protected) from the list in the Request window (should appear first in the list)
  6. Locate the inline JS in the Response body
    1. In the request window you should see the same GET request being made for the second time. Click on this and look for request headers X-Vk2Vjf8v-*

 

blobid3.png

Figure 3. Request Headers

  1. Go to the Silverline Portal dashboard in Flag mode to validate the request
  2. While Portal is in Flag mode, go to the Events Center to verify that your request was flagged as automated.

Note: If the valid Request Headers are not present, refer to Portal. Verify:

  • Web Scraping Method is enabled

  • Proxy HTTPS port is set to 443/443

 

Related Content

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request