cancel
Showing results for 
Search instead for 
Did you mean: 

Pulse secure connections randomly and temporarily stop working

New Contributor

Pulse secure connections randomly and temporarily stop working

Hi,

 

We are running a fairly old PCS appliance.

Model: SA-2000

System Version: 7.1R14 (build 23943)

Pulse Desktop Clients: 9.1.1 (607)


Users will connect and suddenly report that they cannot reach anything anymore. Upon testing I have been able to see this problem but not recreate it. So it happens at a random time and also dissapears randomly. When it happens the pulse desktop client does not show any sign of obstructed connection. and in the appliance I cannot see anything that points towards the problem.

 

i have already rebooted the device because it also seemed to have excessive memory in use and although this freed about 50% of the memory this did not solve the problem. To put our load in perspective as well: we have about 8-10 concurrent users maximum.

 

In the debuglog of my desktopclient I can see that the client loses connection but i cannot pinpoint the exact cause of it. I have 1 instance of normal debuggin and 1 instance of detailed. I will first paste the normal one because the detailed on is extensive.

 

Issues start at 12u11+- and resume again around 12u14+-

 

If anyone could help me out with pinpointing what is causing this I will be very happy. I did have to redact some personal/company information though but this should not influence the clarity of the log too much.

 

00142,09 2020/02/04 12:05:32.458 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t6FE0 dsDnsUtil.cpp:117 - 'dsDnsUtil' flush DNS successful
00140,09 2020/02/04 12:05:32.458 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t6FE0 dsDnsUtil.cpp:134 - 'dsDnsUtil' timeElapsed: 10000
00134,09 2020/02/04 12:05:32.471 3 <REDACTED> Pulse.exe Pulse p13600 t3524 JamTrayWindow.cpp:923 - 'JamUI' Tray update skipped due to lockscreen
00208,09 2020/02/04 12:05:33.250 3 SYSTEM PulseSecureService.exe 8021xAccessMethod p3508 t1C60 8021xAccessMethod_win.cpp:1524 - '8021xSuppression' NotifyThread: WAIT_OBJECT_0, onNewAdapter TestWirelessSuppression()
00217,09 2020/02/04 12:05:33.250 3 SYSTEM PulseSecureService.exe 8021xAccessMethod p3508 t1C60 8021xAccessMethod_win.cpp:2074 - '8021xSuppression' TestWirelessSuppression: m_bWirelessSuppression (Supression Setting Value): 0
00247,09 2020/02/04 12:05:33.250 3 SYSTEM PulseSecureService.exe 8021xAccessMethod p3508 t1C60 8021xAccessMethod_win.cpp:2084 - '8021xSuppression' TestWirelessSuppression: Wireless Suppression OFF. Call setWirelessSuppression(NOT_SUPPRESSING) and return.
00242,09 2020/02/04 12:05:33.250 3 SYSTEM PulseSecureService.exe 8021xAccessMethod p3508 t1C60 8021xAccessMethod_win.cpp:1920 - '8021xSuppression' setWirelessSuppression eSuppress: NOT_SUPPRESSING (Enable), suppressState: 2, Wireless AdapterExist: 1
00149,09 2020/02/04 12:05:33.270 3 <REDACTED> Pulse.exe Pulse p13600 t3524 JamTrayWindow.cpp:879 - 'JamUI' CJamTrayWindow:Smiley SurprisednSessionChange() WTS_SESSION_UNLOCK
00138,09 2020/02/04 12:05:33.270 3 <REDACTED> Pulse.exe Pulse p13600 t3524 JamTrayWindow.cpp:417 - 'JamUI' Tray state updated - State: Connected (2)
00187,09 2020/02/04 12:05:33.270 3 SYSTEM PulseSecureService.exe ConnectionManager p3508 tFD0 SessionChangeMonitor.cpp:98 - 'SessionMonitor' WTS_SESSION_UNLOCK --> SessionId=1 has been unlocked.
00205,09 2020/02/04 12:05:33.277 3 SYSTEM PulseSecureService.exe ConnectionManager p3508 t4EEC ConnectionManagerService.cpp:3256 - 'LocationManager' user [S-1-5-21-1578818540-1520087055-3346786075-49948] UNLOCKED
00138,09 2020/02/04 12:05:33.307 3 <REDACTED> Pulse.exe Pulse p13600 t3524 JamTrayWindow.cpp:417 - 'JamUI' Tray state updated - State: Connected (2)
00140,09 2020/02/04 12:05:42.459 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t6FE0 dsDnsUtil.cpp:115 - 'dsDnsUtil' About to flush DNS
00142,09 2020/02/04 12:05:42.469 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t6FE0 dsDnsUtil.cpp:117 - 'dsDnsUtil' flush DNS successful
00140,09 2020/02/04 12:05:42.469 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t6FE0 dsDnsUtil.cpp:134 - 'dsDnsUtil' timeElapsed: 20000
00183,09 2020/02/04 12:06:09.845 3 SYSTEM PulseSecureService.exe ConnectionManager p3508 tFD0 SessionChangeMonitor.cpp:90 - 'SessionMonitor' WTS_SESSION_LOCK --> SessionId=1 has been locked.
00203,09 2020/02/04 12:06:09.845 3 SYSTEM PulseSecureService.exe ConnectionManager p3508 t4EEC ConnectionManagerService.cpp:3256 - 'LocationManager' user [S-1-5-21-1578818540-1520087055-3346786075-49948] LOCKED
00147,09 2020/02/04 12:06:09.845 3 <REDACTED> Pulse.exe Pulse p13600 t3524 JamTrayWindow.cpp:885 - 'JamUI' CJamTrayWindow:Smiley SurprisednSessionChange() WTS_SESSION_LOCK
00170,09 2020/02/04 12:06:10.418 3 SYSTEM LogonUI.exe jamSSOCredProv p28364 tB38 jamSSOCredProv.cpp:42 - 'jamSSOCredProv' Inside DllGetClassObject(), dsLog has been initialized.
00187,09 2020/02/04 12:10:09.988 3 SYSTEM PulseSecureService.exe ConnectionManager p3508 tFD0 SessionChangeMonitor.cpp:98 - 'SessionMonitor' WTS_SESSION_UNLOCK --> SessionId=1 has been unlocked.
00149,09 2020/02/04 12:10:09.990 3 <REDACTED> Pulse.exe Pulse p13600 t3524 JamTrayWindow.cpp:879 - 'JamUI' CJamTrayWindow:Smiley SurprisednSessionChange() WTS_SESSION_UNLOCK
00138,09 2020/02/04 12:10:09.990 3 <REDACTED> Pulse.exe Pulse p13600 t3524 JamTrayWindow.cpp:417 - 'JamUI' Tray state updated - State: Connected (2)
00205,09 2020/02/04 12:10:10.001 3 SYSTEM PulseSecureService.exe ConnectionManager p3508 t6568 ConnectionManagerService.cpp:3256 - 'LocationManager' user [S-1-5-21-1578818540-1520087055-3346786075-49948] UNLOCKED
00138,09 2020/02/04 12:10:10.023 3 <REDACTED> Pulse.exe Pulse p13600 t3524 JamTrayWindow.cpp:417 - 'JamUI' Tray state updated - State: Connected (2)
00143,09 2020/02/04 12:13:30.913 1 SYSTEM PulseSecureService.exe iftProvider p3508 t3448 session.cpp:211 - 'IftTls' DSSSL_recv failed with error 10054
00139,09 2020/02/04 12:13:30.913 1 SYSTEM PulseSecureService.exe iftProvider p3508 t3448 IftTlsClient.cpp:349 - 'iftProvider' recv failure: 0x2746
00158,09 2020/02/04 12:13:30.913 3 SYSTEM PulseSecureService.exe iftProvider p3508 t3448 IftTlsClient.cpp:371 - 'iftProvider' IftTlsClient::run() exiting: err=0x2746
00166,09 2020/02/04 12:13:30.914 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t3448 connInstance.cpp:2030 - 'iveConnectionMethod' Setting state to retry-connect
00182,09 2020/02/04 12:13:30.914 1 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t3448 connInstance.cpp:1678 - 'iveConnectionMethod' Error Connecting channel. Stage 0 Error 9 0x9
00162,09 2020/02/04 12:13:30.914 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t3448 connInstance.cpp:2525 - 'iveConnectionMethod' Queuing schedule retry 9 0
00153,09 2020/02/04 12:13:30.914 3 SYSTEM PulseSecureService.exe iftProvider p3508 t3448 iftProvider.cpp:328 - 'iftProvider' iftProvider:Smiley TongueroviderThread exiting
00156,09 2020/02/04 12:13:30.914 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t3BFC connInstance.cpp:2543 - 'iveConnectionMethod' Scheduling retry 9 0
00183,09 2020/02/04 12:13:30.914 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t3BFC connInstance.cpp:1388 - 'iveConnectionMethod' Sending broadcast message sessionncAccessMethod
00151,09 2020/02/04 12:13:30.915 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t3BFC ncAccessMethod.cpp:785 - 'ncAccessMethod' Stopping ESP engine
00158,09 2020/02/04 12:13:30.915 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t3BFC connInstance.cpp:1390 - 'iveConnectionMethod' Sent broadcast message
00200,09 2020/02/04 12:13:30.915 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t3BFC ChannelRetryInfo.cpp:38 - 'iveConnectionMethod' onChannelDisconnected untilNow 1 vpn 1 error 9 time 1580814810
00228,09 2020/02/04 12:13:30.915 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t3BFC ChannelRetryInfo.cpp:63 - 'iveConnectionMethod' onRetry same direct uri <REDACTED> addr <REDACTED> now (just lost connection)
00225,09 2020/02/04 12:13:30.915 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t3BFC connInstance.cpp:2683 - 'iveConnectionMethod' Retry: error 0x9 rr 0, <REDACTED> via direct address <REDACTED> in 0 sec
00162,09 2020/02/04 12:13:30.915 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t3BFC connInstance.h:369 - 'iveConnectionMethod' Beginning connection retry...
00165,09 2020/02/04 12:13:30.916 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t3BFC connInstance.cpp:333 - 'iveConnectionMethod' No session type set, using IVE
00146,09 2020/02/04 12:13:30.916 1 SYSTEM PulseSecureService.exe UiModel p3508 t5B6C StatusListener.cpp:323 - 'UiModel' Inconsistent TNC info - ID: 65535
00175,09 2020/02/04 12:13:30.916 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t3BFC connInstance.cpp:397 - 'iveConnectionMethod' No transport type set, defaulting to IFT
00186,09 2020/02/04 12:13:30.917 3 <REDACTED> Pulse.exe Pulse p13600 t3524 JamConnectionModel.cpp:1068 - 'JamUI' Notification of connection status update: '' (ive:2bbcad1949fbd64bb86a02c1be90638a)
00246,09 2020/02/04 12:13:30.917 3 SYSTEM PulseSecureService.exe CloudAppVisibility p3508 t5B6C CloudAppVisibilityService.cpp:69 - 'CloudAppVisibility' onConnectionStatusChange connType: iveAccessMethod, connId: 2bbcad1949fbd64bb86a02c1be90638a state: 1
00139,09 2020/02/04 12:13:30.918 3 <REDACTED> Pulse.exe Pulse p13600 t3524 JamConnectionModel.cpp:1069 - 'JamUI' Connection Status: Connecting
00270,09 2020/02/04 12:13:30.918 3 SYSTEM PulseSecureService.exe ConnectionManager p3508 t5B6C ConnectionManagerService.cpp:2179 - 'ConnectionManager' status update (kStateConnecting) received from iveAccessMethod:2bbcad1949fbd64bb86a02c1be90638a with connection-identity: user
00139,09 2020/02/04 12:13:30.918 3 <REDACTED> Pulse.exe Pulse p13600 t3524 JamTrayWindow.cpp:417 - 'JamUI' Tray state updated - State: Connecting (4)
00228,09 2020/02/04 12:13:30.919 3 SYSTEM PulseSecureService.exe ConnectionManager p3508 t5B6C PolicyEvaluator.cpp:60 - 'ConnectionManager' PolicyEvaluator::evaluate policy [] auto kEvalResultInvalid connect 0 disconnect 1 transition 0
00138,09 2020/02/04 12:13:30.919 3 SYSTEM PulseSecureService.exe iftProvider p3508 t3BFC iftProvider.cpp:124 - 'iftProvider' Deleted IFT Provider
00541,09 2020/02/04 12:13:30.919 3 SYSTEM PulseSecureService.exe ConnectionManager p3508 t5B6C ConnectionEntry.cpp:989 - 'ConnectionManager' [shouldBlockForLockdown] Before blocking [ive:2bbcad1949fbd64bb86a02c1be90638a], settings are alwaysOnVPN 0, isOnlyBlockTrafficOnVPNDisconnectEnabled 0, connLevelLockdown 0, hasIpForL3 1, sConnect: 0, state: 1
00492,09 2020/02/04 12:13:30.919 3 SYSTEM PulseSecureService.exe ConnectionManager p3508 t5B6C ConnectionEntry.cpp:1010 - 'ConnectionManager' [shouldUnBlockForLockdown] Before unblocking [ive:2bbcad1949fbd64bb86a02c1be90638a], settings are alwaysOnVPN 0, isOnlyBlockTrafficOnVPNDisconnectEnabled 0, hasIpForL3 1, sConnect: 0, state: 1
00240,09 2020/02/04 12:13:30.920 3 SYSTEM PulseSecureService.exe ConnectionManager p3508 t5B6C ConnectionEntry.cpp:1811 - 'ConnectionManager' [startLockDownModeFirewallIfNeeded] Lockdown not enabled for connection: 2bbcad1949fbd64bb86a02c1be90638a
00331,09 2020/02/04 12:13:30.920 3 SYSTEM PulseSecureService.exe ConnectionManager p3508 t5B6C ConnectionEntry.cpp:1845 - 'ConnectionManager' onNetworkChange(ive:2bbcad1949fbd64bb86a02c1be90638a), manual: 1, hasIp: 1, connect: 0/1, disconnect: 1/0, machineSuspended: 0, connSuspended: 0, context: 1, identity: user, state kStateConnecting
00146,09 2020/02/04 12:13:30.920 1 SYSTEM PulseSecureService.exe UiModel p3508 t5B6C StatusListener.cpp:323 - 'UiModel' Inconsistent TNC info - ID: 65535
00172,09 2020/02/04 12:13:30.920 3 SYSTEM PulseSecureService.exe iftProvider p3508 t3BFC iftProvider.cpp:1362 - 'iftProvider' Using authentication network timeout of (120000) msec
00137,09 2020/02/04 12:13:30.922 3 SYSTEM PulseSecureService.exe iftProvider p3508 t6730 iftProvider.cpp:676 - 'iftProvider' Connecting directly
00178,09 2020/02/04 12:13:30.922 3 SYSTEM PulseSecureService.exe iftProvider p3508 t6730 iftProvider.cpp:647 - 'iftProvider' Connecting to host <REDACTED>:443 ip <REDACTED>
00165,09 2020/02/04 12:13:30.922 3 SYSTEM PulseSecureService.exe iveConnectionMethod p3508 t6730 connectionInstance.cpp:224 - 'iveConnectionMethod' No resolved hostname ips

 

3 REPLIES 3
Moderator

Re: Pulse secure connections randomly and temporarily stop working

the good news is i can tell you the problem; the bad news is i can't tell you the why of the problem. the messages:
DSSSL_recv failed with error & 10054recv failure: 0x2746
are showing that an intermediate device is terminating the connection (specifically the SSL control channel is being blocked/terminated)
New Contributor

Re: Pulse secure connections randomly and temporarily stop working

Thanks for the reply.

 

What makes this error even more weird is that we have a second PCS appliance a SA5000. It follows the exact same path of firewalls/routers. only the subnet at the end is different and this appliance is not having the same issues. So whatever is terminating the SSL control only terminates the connections to the old appliance and not the new. 

 

the path they follow is: WAN => Colt Cpe (not under our management => Main Firewall Palo Alto(static routing, application firewall, user firewall) => L3 switch (only static routing) => appliance

 

 

New Contributor

Re: Pulse secure connections randomly and temporarily stop working

Hi again,

 

so another update with something I have found in the logs which I cannot explain is the following:

ive - [src wan] username(realm)[roles] - username/realm logged out from (src wan) because user started new session from IP (src wan)

 

So the source wan is always the same public IP. Why would the SA delete a session because supposedly the user started a new connection from the same IP? I thought the user maybe just disconnected and reconnected because he was experiencing issues but I tried recreating this and the logs show something very differently when you do it by hand.

 

any idea why this might happen?