Description
- The intent of this article is to provide a guideline that can be used for troubleshooting general proxy availability and intermittent connectivity or backend application slowness related issues.
Environment
- Silverline WAF
- Proxy
- WAF Policy
Procedure
- Please collect and gather the following data so that we may quickly identify the issue or be able to provide better assistance.
- Then open a ticket with the SOC.
Data Collection
Check if any changes to your DNS or application recently took place, and include that information. Check with application owner. This is key to effective troubleshooting. |
- List out the steps to reproduce the issue that you are experiencing/reporting
- Fill in the following template with information. The more information provided, the better.
-
0) How is the impact observed? Support ID in case of WAF blocks, timeouts, resets, etc.
1) Proxy label/FQDN and request URI
2) Public source IP address of the impacted client/testers
3) Scope of impact (all users, some users, users of a specific carrier or region, etc.)
4) Steps to reproduce impact
5) Output of the failure condition from browser, curl, etc.
6) If the issue can be reproduced using the browser, collect a HAR file.
For instructions refer How to Generate HAR Files
7) Traceroutes to proxy VIP from affected clients, if available
8) Packet capture on client, if available
9) Testing account/credentials (if necessary)
10) Start time of observed impact, end time if applicable, or ongoing
-
- Check and verify the following features using this as a guideline (there may be other features that are not listed below)
-
1) Use the portal health check Proxy Health Monitoring, in order to check the status of pool members (any warning/signal?)
2) Check if the backend responds by connecting to the backend directly, bypassing Silverline proxy
Adding an entry to the host file can help to bypass WAF, for instructions refer to How To Test The Silverline Proxy Using 'Hosts' File On Windows PC
3) For WAF proxies, check recent violation logs (Do you observe any blockings?)
* How to View WAF Violation Logs in WAF Violation Summary
4) For Proxies with iRule(s) applied, check iRule stats for any blocks or other events that might be causing the issue
* iRule Stats
5) For Proxies with Threat Intelligence Profile, check with Threat Intelligence stats for any blocks
Threat Intelligence: Monitoring
* This includes IP Denylist in the portal
* How To Denylist IP Addresses For Proxy Services (WAF And DDoS)
* Will show up as "Denylisted" under Category
6) For Proxies with L7 DDoS Profile(s), check with L7 DDoS profile stats for any issues
* L7 DDoS Profiles have Support IDs searching
-
- Tools and Commands
- Resolve the domain name to IP address
- For Windows,
-
nslookup <host_name>
# nslookup www.example.com
-
- For *nix users can use dig
-
dig <host_name> -short
# dig www.example.com -short
-
- For Windows,
- On *nix systems one can use tools such as Wget or cURL to check the connectivity with the destination host using bash. For the use and syntax please refer to online documentation
- Wget
-
time wget https://<domain>/
time wget https:/192.168.1.1/
-
- cURL
-
#For a quick test measured in time:
curl -k -w "Testing Website Response Time for: %{url_effective}\n\nLookup Time:\t\t%{time_namelookup}\nConnect Time:\t\t%{time_connect}\nAppCon Time:\t\t%{time_appconnect}\nRedirect Time:\t\t%{time_redirect}\nPre-transfer Time:\t%{time_pretransfer}\nStart-transfer Time:\t%{time_starttransfer}\n\nTotal Time:\t\t%{time_total}\n" -Lvi https://<domain-name>/ --resolve <domain-name>:<port>:<ip>
Lookup Time: 0.000065
Connect Time: 0.000067
AppCon Time: 0.000067
Redirect Time: 1.219776
Pre-transfer Time: 0.000165
Start-transfer Time: 0.216281Total Time: 1.438582
-
Modify "https://<domain-name>/ --resolve <domain-name>:<port>:<ip>
The "--resolve <domain>:<port>:<ip>" is useful for SNI enabled applications/servers
curl -vkLi https://www.example.com/ --resolve www.example.com:443:192.168.1.1
-
- For SSL related issue 'openssl s_client -connect ' can be used
-
openssl s_client -connect <IP>:<Port> -servername <domain-name>
The "-servername <domain>" is useful for SNI enabled applications/servers
openssl s_client -connect 192.168.1.1:443 -servername www.example.com - If there are no errors, it's a good indicator that the TLS communication layer works as expected
- If you encounter any errors, get a copy of the output for analysis
-
- For latency, especially with congestion, take a traceroute and provide the source IP of the traceroute
- Traceroutes from impacted clients will be necessary to understand if the Tier 3 carrier has congestion
- For Windows
-
tracert <ip or domain>
tracert 192.168.1.1
-
- For *nix systems
-
traceroute <ip or domain>
traceroute 192.168.1.1
-
- Wget
- Resolve the domain name to IP address
Engaging the SOC
- Once data has been collected, reach out F5 Silverline team by opening a ticket with us: How to Contact SOC / How to Contact Silverline Support
Related Content
- Reporting: Proxy Health Monitoring
- Threat Intelligence: Monitoring
- iRule Stats
- How to Generate HAR Files
- How To Test The Silverline Proxy Using 'Hosts' File On Windows PC
- How to View WAF Violation Logs in WAF Violation Summary
- How To Denylist IP Addresses For Proxy Services (WAF And DDoS)
- How to Contact SOC / How to Contact Silverline Support