Follow

Q&A: How does Silverline Web Scraping Protection Work? (Shape Defense)

Question

How does Silverline Web Scraping Protection Work?

If you need more background info on web scraping and bots, see:

 

Environment 

  • Silverline Shape Defense
  • Silverline Web Scraping Protection
  • Bot defense
  • Bot protection
  • Anti-bot

 

Answer

How does Silverline Web Scraping Protection Work?

  1. The web scraping protection is initiated when an initial HTTP GET request is sent from the client (web browser) through the SSDE
  2. SSDE identifies whether the request is from a known or unknown visitor.
  3. If the request is from an unknown visitor, the initial response is sent back to the client with a JS page, which is rendered innocuously in the browser.
  4. If this JS can be processed by the client, it sends telemetry back to the SSDE via GET XHR.
  5. The SSDE then analyzes the telemetry and determines if the user is a legitimate or malicious visitor.
  6. If the request is from a legitimate user,
    1. the original HTTP GET is sent on to the Origin
    2. and a cookie is set on the response to the client. This cookie is then inspected on subsequent requests to ensure the visitor continues to be a valid human user.

Silverline_Shape_Defense_Interstitial.png

Figure 1. Silverline Shape Defense request flow

 

 

Workflows for Types of Visitor

Any returning good visitor (with a valid cookie) or allowlisted visitor is not served the JS client page. The JS client page is served only to traffic with no valid cookie and that's not allowlisted.

The following is the general workflow that determines whether to serve the JS client page or to trust a returning or allowlisted user:

 

Unknown visitor

An unknown visitor is any traffic on a protected page who doesn't have a valid cookie and isn't allowlisted. This can be:

  • A first-time visitor
  • A returning visitor who doesn't have a valid cookie
  • A returning visitor whose cookie is expired
  • Automation

The following is a typical workflow for an unknown visitor:

  1. An end-user initiates an HTTP GET request to a Silverline protected resource
  2. Silverline Shape Defense:
    • a) Receives the GET request
    • b) Responds with a JS page to collect telemetry from the client
    • c) If the client can process JavaScript, then the telemetry is returned to Silverline
    • d) Silverline inspects the telemetry and takes the appropriate action
      • If Human:
        • i.) Passes the request on to the Origin 
        • ii.) Sends the request to the origin server to retrieve the resource
        • iii.) Receives the response from the origin server with the resource
        • iv.) Inserts a web cookie in the response to identify future requests from this visitor 
        • v.) Passes the response downstream to display in the user’s browser
      •  If Malicious: Performs the configured action (Flag, Redirect, Block)

Returning good visitor

A returning good visitor is any traffic previously determined to be good, as indicated by having the defined cookie.

The following is a typical workflow for a visitor with a valid cookie:

  1. An end-user initiates an HTTP GET request to a Silverline protected resource
  2. Silverline Shape Defense:
  • a) Receives the GET request
  • b) Detects a valid cookie associated with the request
  • c) Sends the request to the origin server to retrieve the resource
  • d) Receives the response from the origin server with the resource and passes it downstream to display in the user's browser     

Allowlisted Visitor

An allowlisted visitor (How to Configure Shape Defense Allowlists) is any traffic that is on the trusted allowlist for the application. No allowlisted visitor is served a Silverline JS client page.

The following is a typical workflow for an allowlisted visitor:

  1. A request from end-user initiates an HTTP GET request for a Silverline protected web resource.
  2. Silverline Shape Defense:
    1. a) Receives the GET request
    2. b) Determines that the request is from an allowlisted requestor
    3. c) Sends the request to the origin server to retrieve the resource
    4. d) Receives the response from the origin server with the resource and passes it downstream to display in the user's browser      

 

Additional Details

An SSD JS page is served for any unknown visitor. Web scraping protection operates in two stages. The following is a typical workflow.

Stage I

The initial request and the subsequent response to it which contains telemetry data to use in the stage II request.

  1. An end-user initiates an HTTP GET request for a protected web resource.
  2. Silverline:
    • a) Determines that the request is not from an allowlisted requestor
    • b) Silverline sends the GET for the protected endpoint to the SSE always
  3. The SSE:
    • a) Receives the request
    • b) Detects no JS decorations associated with the request
    • c) Responds with web page that includes Silverline Shape Defense’s client JavaScript
      •  Silverline:
        • a) Forwards the response from the SSE to the client

At this point the client browser attempts to execute the JS in the response. If the client cannot process JavaScript, there is no further action. If the client can process JavaScript, the process continues with Stage II.

Stage II

The Shape JS that was returned in Stage I gathers telemetry about the client browser and initiates a GET XHR request that includes both the original request and the gathered telemetry.

  1. Silverline:
    • a) Determines that the request is not from an allowlisted requestor
    • b) Silverline sends the GET for the protected endpoint to the SSE always
  2. SSE:
    • a) Receives the initiated request.
    • b) Detects the JS decorations and evaluates the signals against inference rules to detect automation
    • c) Responds with an Inference (Human or Malicious)
    • d) If Human, responds with the cookie to be set on the response from the Origin
  3. Silverline: 
    • a) If the SSE Inference is Malicious, perform the appropriate mitigation: 
      • i.) Flag 
      • ii.) Redirect
      • iii.) Block
    • b) If the SSE Inference is Malicious, perform the appropriate mitigation:
      • i.) Store the received cookie 
      • ii.) Send the Request on to the Origin 
      • iii.) Insert the cookie on response from the origin 

 

Related Articles

Silverline Web Scraping Protection

 

Silverline Shape Defense

 

 

  •  
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request