Q&A: What Data Does Silverline Need for WAF Proxy Troubleshooting? – F5 Silverline Knowledge Base

Description

The intent of this article is to provide a guideline that can be used for troubleshooting general proxy availability and intermittent connectivity or backend application slowness related issues.

Environment

Silverline WAF
Proxy
WAF Policy

Procedure

Please collect and gather the following data so that we may quickly identify the issue or be able to provide better assistance.
Then open a ticket with the SOC.

Data Collection

Check if any changes to your DNS or application recently took place, and include that information. Check with application owner. This is key to effective troubleshooting.

List out the steps to reproduce the issue that you are experiencing/reporting

Fill in the following template with information. The more information provided, the better.

0) How is the impact observed? Support ID in case of WAF blocks, timeouts, resets, etc.

1) Proxy label/FQDN and request URI

2) Public source IP address of the impacted client/testers

3) Scope of impact (all users, some users, users of a specific carrier or region, etc.)

4) Steps to reproduce impact

5) Output of the failure condition from browser, curl, etc. 

6) If the issue can be reproduced using the browser, collect a HAR file. 
     For instructions refer How to Generate HAR Files

7) Traceroutes to proxy VIP from affected clients, if available

8) Packet capture on client, if available

9) Testing account/credentials (if necessary)

10) Start time of observed impact, end time if applicable, or ongoing

Check and verify the following features using this as a guideline (there may be other features that are not listed below)

1) Use the portal health check Proxy Health Monitoring, in order to check the status of pool members (any warning/signal?)

2) Check if the backend responds by connecting to the backend directly, bypassing Silverline proxy
   Adding an entry to the host file can help to bypass WAF, for instructions refer to How To Test The Silverline Proxy Using 'Hosts' File On Windows PC

3) For WAF proxies, check recent violation logs (Do you observe any blockings?)
   * How to View WAF Violation Logs in WAF Violation Summary

4) For Proxies with iRule(s) applied, check iRule stats for any blocks or other events that might be causing the issue
   * iRule Stats

5) For Proxies with Threat Intelligence Profile, check with Threat Intelligence stats for any blocks
   Threat Intelligence: Monitoring
   * This includes IP Denylist in the portal 
   * How To Denylist IP Addresses For Proxy Services (WAF And DDoS) 
          * Will show up as "Denylisted" under Category

6) For Proxies with L7 DDoS Profile(s), check with L7 DDoS profile stats for any issues
   * L7 DDoS Profiles have Support IDs searching

Tools and Commands

Resolve the domain name to IP address

For Windows,

nslookup <host_name>
# nslookup www.example.com

For *nix users can use dig

dig <host_name> -short
# dig www.example.com -short

On *nix systems one can use tools such as Wget or cURL to check the connectivity with the destination host using bash. For the use and syntax please refer to online documentation

Wget

time wget https://<domain>/
time wget https:/192.168.1.1/

cURL

#For a quick test measured in time:
curl -k -w "Testing Website Response Time for: %{url_effective}\n\nLookup Time:\t\t%{time_namelookup}\nConnect Time:\t\t%{time_connect}\nAppCon Time:\t\t%{time_appconnect}\nRedirect Time:\t\t%{time_redirect}\nPre-transfer Time:\t%{time_pretransfer}\nStart-transfer Time:\t%{time_starttransfer}\n\nTotal Time:\t\t%{time_total}\n" -Lvi https://<domain-name>/ --resolve <domain-name>:<port>:<ip>

Lookup Time: 0.000065
Connect Time: 0.000067
AppCon Time: 0.000067
Redirect Time: 1.219776
Pre-transfer Time: 0.000165
Start-transfer Time: 0.216281

Total Time: 1.438582

Modify "https://<domain-name>/  --resolve <domain-name>:<port>:<ip>
The "--resolve <domain>:<port>:<ip>" is useful for SNI enabled applications/servers

curl -vkLi https://www.example.com/  --resolve www.example.com:443:192.168.1.1

For SSL related issue 'openssl s_client -connect ' can be used
- ```
openssl s_client -connect <IP>:<Port> -servername <domain-name>
The "-servername <domain>" is useful for SNI enabled applications/servers

openssl s_client -connect 192.168.1.1:443 -servername www.example.com
```
- If there are no errors, it's a good indicator that the TLS communication layer works as expected
- If you encounter any errors, get a copy of the output for analysis
For latency, especially with congestion, take a traceroute and provide the source IP of the traceroute
- Traceroutes from impacted clients will be necessary to understand if the Tier 3 carrier has congestion
- For Windows
  - ```
  tracert <ip or domain>
  tracert 192.168.1.1
```
- For *nix systems
  - ```
  traceroute <ip or domain>
  traceroute 192.168.1.1
```

Engaging the SOC

Once data has been collected, reach out F5 Silverline team by opening a ticket with us: How to Contact SOC / How to Contact Silverline Support

Description

Environment

Procedure

Data Collection

Engaging the SOC

Related Content