Juniper's JTAC team investigated the SRX300 Gateway, where Pulse Secure VPN client suppose to connect, while the VPN connectivity was failing and found out that it was caused by an over-utilization of its Routing Engine.
Next, we will show the Juniper commands the JTAC engineer ran on the SRX in config mode
user-SRXGateway: run show chassis routing-engine
Routing Engine status:
Temperature 45 degrees C / 113 degrees F
CPU temperature 59 degrees C / 138 degrees F
Total memory 4096 MB Max 983 MB used ( 24 percent)
Control plane memory 2624 MB Max 656 MB used ( 25 percent)
Data plane memory 1472 MB Max 309 MB used ( 21 percent)
5 sec CPU utilization:
User 50 percent
Background 0 percent
Kernel 45 percent
Interrupt 0 percent
Idle 5 percent <---- idle was down to 0% when we initally executed the command at the time the VPN client was failing to connect
Last reboot reason 0x200:normal shutdown
Load averages: 1 minute 5 minute 15 minute
user-SRXGateway: run show system processes extensive | except 0.00
last pid: 49064; load averages: 2.33, 1.67, 1.38 up 3+07:22:08 10:40:48
160 processes: 16 running, 130 sleeping, 2 zombie, 12 waiting
Mem: 304M Active, 144M Inact, 1574M Wired, 385M Cache, 112M Buf, 1572M Free
Swap:
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
........................
.......................
49023 nobody 3 84 0 11600K 5032K ucondt 0 0:01 1.46% httpd <---- was in the low 90% when the VPN client could not connect
Finally he executed
run show security flow session interface st0.21
and that returned a list of pair of communicating IPs, their Session IDs and the Policy Names that allow them.
The overwhelming majority of the records were indicating 3 specific IP addresses trying to aggressively connect to the SRX itself. All IPs were assigned on Hardware in the internal network and there was no config to get accessed externally, via DDNS for example.
Two of those IPs were given to IP power switches [to power up hardware when power from the grid restores after an outage] and the 3rd one on a server.
The IP power switches by design ping 5 different external preset targets every few seconds; if those become unreacheable then they would power-cycle one or both of their AC power outlets according to the user's configuration.
Unexpectedly, the SRX300 logs showed the traffic from the IP power switches wasn't icmp, as expected, but plain http, so the fact it was directed to the SRX raised suspicions of having been hacked, so I removed them from the network.
The server among other SW had 2 very resources intensive network mngt SW [SolarWinds and PRTG Network Monitor] that were hitting the Gateway to extract descriptive network info and stats for reporting purposes.
Both pieces of SW were completely removed [down to the Windows registry level] from the server and the machine was deeply scanned with a couple of AVs to assess/ensure its virus/spyware clean status.
The Juniper engineer also disabled http mngt access on the SRX
[edit system services web-management] <--- config setting he changed
and we agreed not to restart the SRX again but monitor Pulse connectivity daily. So far it has been 6 days and we haven't experienced any Pulse secure VPN outage hoping it'll stay that way.
However, Juniper highly recommends NOT to use Pulse Secure as a VPN client accessing their gateways, especially from Win10 machines [albeit, from personal experience Pulse Secure still works from Win7 and it's pretty stable and reliable].
Instead they propose to use NCP. Below is a KB article about how to set it up on the SRX and a PDF with NCP set up instructions for the Client:
The URLs I'll try to attach on the thread by placing them in text file
I hope the above info is helpful providing enough insight to help others in their troubleshooting efforts.
Thanks
Stavros
