Connection crashing after random periods of time

[chl]

Hello.

I often experience problems with connections in Pulsesecure dropping. 

A disconnect does NOT occur, but the routes

Pulse Version: 9.0R3(923)

Ubuntu Version: 18.04.2 LTS

I have tried a few different builds of 9.0R2 and R3 and the problem persists.

Has anyone had similar problems and know how to fix it?

I have attached pulsesvc example logs of when the problem occurs:

....
20190604104631.164014 pulsesvc[p4680.t11925] pulseui.info bytes sent = 142349, recvd = 1270016 (pulseUiLib.cpp:72)
20190604104633.86372 pulsesvc[p4706.t4706] rmon.info got a netlink route update: type = 16 (routemon.cpp:351) # Connection drops
20190604104633.166284 pulsesvc[p4680.t11925] pulseui.info bytes sent = 144169, recvd = 1271299 (pulseUiLib.cpp:72)
20190604104641.175585 pulsesvc[p4680.t11925] pulseui.info bytes sent = 149660, recvd = 1278400 (pulseUiLib.cpp:72)
...
# Connectivity drops
20190604104813.273856 pulsesvc[p4706.t4706] ipsec.info Restarting keep-alives (engine.cpp:355)
20190604104813.274034 pulsesvc[p4680.t11925] pulseui.info bytes sent = 188352, recvd = 1623903 (pulseUiLib.cpp:72)
20190604104815.276361 pulsesvc[p4680.t11925] pulseui.info bytes sent = 188881, recvd = 1623903 (pulseUiLib.cpp:72)
20190604104817.277246 pulsesvc[p4680.t11925] pulseui.info bytes sent = 189311, recvd = 1623903 (pulseUiLib.cpp:72)
20190604104819.279430 pulsesvc[p4680.t11925] pulseui.info bytes sent = 189745, recvd = 1623903 (pulseUiLib.cpp:72)
20190604104821.281709 pulsesvc[p4680.t11925] pulseui.info bytes sent = 190163, recvd = 1623903 (pulseUiLib.cpp:72)
20190604104823.284239 pulsesvc[p4680.t11925] pulseui.info bytes sent = 190421, recvd = 1623903 (pulseUiLib.cpp:72)
20190604104825.286501 pulsesvc[p4680.t11925] pulseui.info bytes sent = 191100, recvd = 1623903 (pulseUiLib.cpp:72)
20190604104827.288890 pulsesvc[p4680.t11925] pulseui.info bytes sent = 191682, recvd = 1623903 (pulseUiLib.cpp:72)
# Connectivity restoring..20190604104828.828930 pulsesvc[p4706.t4706] ipsec.warn keep alive failed for ESP tunnel in:0x80797086, out:0x1B63B8D2 (engine.cpp:382)
20190604104828.828996 pulsesvc[p4706.t4706] ipsec.warn Cleaning up sa 0x80797086 (esp.cpp:76)
20190604104828.829008 pulsesvc[p4706.t4706] ipsec.info Switching to NCP mode (tunnel.cpp:666)
20190604104828.829015 pulsesvc[p4706.t4706] ipsec.info send kmp message 303 size 13 (tunnel.cpp:240) 
# Connectivity is restored
20190604104829.289176 pulsesvc[p4680.t11925] pulseui.info bytes sent = 192471, recvd = 1625119 (pulseUiLib.cpp:72)
20190604104831.291403 pulsesvc[p4680.t11925] pulseui.info bytes sent = 193287, recvd = 1626615 (pulseUiLib.cpp:72)
20190604104833.293452 pulsesvc[p4680.t11925] pulseui.info bytes sent = 193632, recvd = 1627184 (pulseUiLib.cpp:72)
20190604104835.295757 pulsesvc[p4680.t11925] pulseui.info bytes sent = 193632, recvd = 1627184 (pulseUiLib.cpp:72)
20190604104837.298143 pulsesvc[p4680.t11925] pulseui.info bytes sent = 193632, recvd = 1627184 (pulseUiLib.cpp:72)
20190604104839.76641 pulsesvc[p4706.t4706] rmon.info got a netlink route update: type = 16 (routemon.cpp:351)
20190604104839.300428 pulsesvc[p4680.t11925] pulseui.info bytes sent = 193632, recvd = 1627184 (pulseUiLib.cpp:72)

...

Best regards.

[[email protected]]

From the logs, I am able to see that the transport mode is being switched from ESP to SSL, which is expected if the client is not able to reach/doesn't get any reply for the ESP keep-alive packets from the VPN server. ESP (UDP/4500 - default)

>> Keep-alives got failed.

20190604104813.273856 pulsesvc[p4706.t4706] ipsec.info Restarting keep-alives (engine.cpp:355)

20190604104828.828930 pulsesvc[p4706.t4706] ipsec.warn keep alive failed for ESP tunnel in:0x80797086, out:0x1B63B8D2 (engine.cpp:382)

>> Switches over to SSL (NCP) mode.

20190604104828.829008 pulsesvc[p4706.t4706] ipsec.info Switching to NCP mode (tunnel.cpp:666)

By default, 15 seconds fallback time will be configured on the VPN server side which instructs the client to send ESP keep-alives upto 15 seconds and fallback to SSL if there is no reply from the server.

10:48:13 - Starts to send keep-alive messages.

10:48:28 (after 15 seconds) - Considers that the ESP mode is blocked, hence switches to SSL.

[[email protected]]

20190604104813.273856 pulsesvc[p4706.t4706] ipsec.info Restarting keep-alives (engine.cpp:355)

"Restarting keep-alives" will be triggered if the ESP mode tunnel was working for sometime and then the client fails to receive any ESP packets from the server i.e. after some key exchanges (16 minutes - default)

I remember once @zanyterp has referred this KB article to an other user: KB19900 - Connection drops after every ESP re-key using D-Link router

Please review and let me know if that helps.

[zanyterp]

in addition to what [email protected] commented, there is a possibility that the admin has disabled SSL transport as an option. this would prevent access if there is anything that drops the UDP 4500 traffic.
however, based on your log snippet, there is more likely an intermediate device that is closing the heartbeat connection (TCP 443) between the client and the VPN server.
can you work with your admin for a coordinated TCP dump on your client and the VPN server to make sure that the traffic is flowing properly?