Re: Mac OS crashes

vkiryakov · ‎10-06-2020

Hi guys. After Pulse Secure client update I faced OS crashes.

Client version 9.1.8(3999)

Can anybody explain what is going on?

panic(cpu 4 caller 0xffffff800cec1eff): assertion failed: (MBUFQ_EMPTY(&cfq->q_mq) && cfq->q_start == cfq->q_end) || (!MBUFQ_EMPTY(&cfq->q_mq) && cfq->q_start != cfq->q_end), file: /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-6153.141.2/bsd/net/content_filter.c, line: 1067
Backtrace (CPU 4), Frame : Return Address
0xffffff923164b730 : 0xffffff800c71a65d
0xffffff923164b780 : 0xffffff800c854a75
0xffffff923164b7c0 : 0xffffff800c8465fe
0xffffff923164b810 : 0xffffff800c6c0a40
0xffffff923164b830 : 0xffffff800c719d27
0xffffff923164b930 : 0xffffff800c71a117
0xffffff923164b980 : 0xffffff800cec1a6c
0xffffff923164b9f0 : 0xffffff800cec1eff
0xffffff923164ba00 : 0xffffff800cad7bde
0xffffff923164ba40 : 0xffffff800cad7ef8
0xffffff923164bbd0 : 0xffffff800cad63e8
0xffffff923164bc20 : 0xffffff800cad320e
0xffffff923164bc80 : 0xffffff800cc5b5c9
0xffffff923164bcd0 : 0xffffff800ccd32eb
0xffffff923164bdf0 : 0xffffff800cca7eac
0xffffff923164be20 : 0xffffff800cca14a0
0xffffff923164bee0 : 0xffffff800cca1260
0xffffff923164bf40 : 0xffffff800cd84be7
0xffffff923164bfa0 : 0xffffff800c6c1206

BSD process name corresponding to current thread: net.pulsesecure.
Boot args: chunklist-security-epoch=0 -chunklist-no-rev2-dev chunklist-security-epoch=0 -chunklist-no-rev2-dev

Mac OS version:
19H2

Kernel version:
Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64
Kernel UUID: 05D51A3D-3A87-3FF0-98C3-9CF3827A3EDD
Kernel slide: 0x000000000c400000
Kernel text base: 0xffffff800c600000
__HIB text base: 0xffffff800c500000
System model name: MacBookPro15,1 (Mac-937A206F2EE63C01)
System shutdown begun: NO

System uptime in nanoseconds: 386284110690
last loaded kext at 20177425215: @kext.AMDFramebuffer 3.1.0 (addr 0xffffff7f91da3000, size 245760)
loaded kexts:
@kext.AMDFramebuffer 3.1.0
@kext.AMDRadeonX4000 3.1.0
@kext.AMDRadeonServiceManager 3.1.0
>!AGraphicsDevicePolicy 5.2.6
@fileutil 20.036.15
@AGDCPluginDisplayMetrics 5.2.6
>!AHV 1
|IOUserEthernet 1.0.1
|IO!BSerialManager 7.0.6f7
>!AUpstreamUserClient 3.6.8
>pmtelemetry 1
>AGPM 111.4.4
>!APlatformEnabler 2.7.0d0
>X86PlatformShim 1.0.0
>AGDCBacklightControl 5.2.6
>!A!IKBLGraphics 14.0.7
@Dont_Steal_Mac_OS_X 7.0.0
@kext.AMD9500!C 3.1.0
>!ABacklight 180.3
>!AThunderboltIP 3.1.4
>ACPI_SMC_PlatformPlugin 1.0.0
>BridgeAudioCommunication 6.70.7
>!ATopCaseHIDEventDriver 3430.1
>!AFIVRDriver 4.1.0
>!AMCCSControl 1.14
>!AMuxControl2 5.2.6
>!AHIDALSService 1
>!ABridgeAudio!C 6.70.7
>!AGFXHDA 100.1.429
>!A!IPCHPMC 2.0.1
>!A!ICFLGraphicsFramebuffer 14.0.7
>!A!ISlowAdaptiveClocking 4.0.0
>!AAVEBridge 6.1
@filesystems.autofs 3.0
>BCMWLANFirmware4355.Hashstore 1
>BCMWLANFirmware4364.Hashstore 1
>BCMWLANFirmware4377.Hashstore 1
>!AFileSystemDriver 3.0.1
@filesystems.hfs.kext 522.100.5
@BootCache 40
@!AFSCompression.!AFSCompressionTypeDataless 1.0.0d1
@!AFSCompression.!AFSCompressionTypeZlib 1.0.0
>!AVirtIO 1.0
>!ABCMWLANBusInterfacePCIe 1
@filesystems.apfs 1412.141.1
@private.KextAudit 1.0
>!ASmartBatteryManager 161.0.0
>!AACPIButtons 6.1
>!ASMBIOS 2.1
>!AACPIEC 6.1
>!AAPIC 1.7
$!AImage4 1
@nke.applicationfirewall 303
$TMSafetyNet 8
@!ASystemPolicy 2.0.0
|EndpointSecurity 1
@kext.AMDRadeonX4100HWLibs 1.0
@kext.AMDRadeonX4000HWServices 3.1.0
|IOAVB!F 850.1
>!ABacklightExpert 1.1.0
>!AHDA!C 283.15
|IOHDA!F 283.15
@!AGPUWrangler 5.2.6
>IOPlatformPluginLegacy 1.0.0
>!AHS!BDriver 3430.1
>IO!BHIDDriver 7.0.6f7
@kext.AMDSupport 3.1.0
>!ASMBus!C 1.0.18d1
>!AActuatorDriver 3440.1
>!AMultitouchDriver 3440.1
>!AInputDeviceSupport 3440.8
>!AGraphicsControl 5.2.6
>!AThunderboltEDMSink 4.2.3
>!AThunderboltDPOutAdapter 6.2.6
>!AHIDKeyboard 209
|IONDRVSupport 576.1
|IO!BHost!CUARTTransport 7.0.6f7
|IO!BHost!CTransport 7.0.6f7
>!A!ILpssUARTv1 3.0.60
>!A!ILpssUARTCommon 3.0.60
>!AOnboardSerial 1.0
>!ASMBusPCI 1.0.14d1
@!AGraphicsDeviceControl 5.2.6
|IOAccelerator!F2 438.7.3
|IOGraphics!F 576.1
|IOSlowAdaptiveClocking!F 1.0.0
>X86PlatformPlugin 1.0.0
>IOPlatformPlugin!F 6.0.0d8
@plugin.IOgPTPPlugin 840.3
|IOEthernetAVB!C 1.1.0
@kext.triggers 1.0
>usb.IOUSBHostHIDDevice 1.2
>usb.cdc.ecm 5.0.0
>usb.cdc.ncm 5.0.0
>usb.cdc 5.0.0
>usb.networking 5.0.0
>usb.!UHostCompositeDevice 1.2
|IOSurface 269.11
@filesystems.hfs.encodings.kext 1
|IOAudio!F 300.2
@vecLib.kext 1.2.0
>!ABCMWLANCore 1.0.0
>mDNSOffloadUserClient 1.0.1b8
>IOImageLoader 1.0.0
|IOSerial!F 11
|IO80211!FV2 1200.12.2b1
>corecapture 1.0.4
|IOSkywalk!F 1
>!AThunderboltPCIDownAdapter 2.5.4
>!AThunderboltDPInAdapter 6.2.6
>!AThunderboltDPAdapter!F 6.2.6
>!AHPM 3.4.4
>!A!ILpssI2C!C 3.0.60
>!A!ILpssDmac 3.0.60
>!A!ILpssI2C 3.0.60
>!AXsanScheme 3
>!AThunderboltNHI 5.8.6
|IOThunderbolt!F 7.6.1
>usb.!UVHCIBCE 1.2
>usb.!UVHCI 1.2
>usb.!UVHCICommonBCE 1.0
>usb.!UVHCICommon 1.0
>!AEffaceableNOR 1.0
|IOBufferCopy!C 1.1.0
|IOBufferCopyEngine!F 1
>usb.!UHostPacketFilter 1.0
|IONVMe!F 2.1.0
|IOUSB!F 900.4.2
>usb.!UXHCIPCI 1.2
>usb.!UXHCI 1.2
>!AEFINVRAM 2.1
>!AEFIRuntime 2.1
>!ASMCRTC 1.0
|IOSMBus!F 1.1
|IOHID!F 2.0.0
$quarantine 4
$sandbox 300.0
@kext.!AMatch 1.0.0d1
>!AKeyStore 2
>!UTDM 489.120.1
|IOSCSIBlockCommandsDevice 422.120.3
>!ACredentialManager 1.0
>!AFDEKeyStore 28.30
>!AEffaceable!S 1.0
>!AMobileFileIntegrity 1.0.5
@kext.CoreTrust 1
|CoreAnalytics!F 1
|IOTimeSync!F 840.3
|IONetworking!F 3.4
>DiskImages 493.0.0
|IO!B!F 7.0.6f7
|IO!BPacketLogger 7.0.6f7
>!ASSE 1.0
>KernelRelayHost 1
>!ASEPManager 1.0.1
>IOSlaveProcessor 1
|IOUSBMass!SDriver 157.140.1
|IOSCSIArchitectureModel!F 422.120.3
|IO!S!F 2.1
|IOUSBHost!F 1.2
>usb.!UCommon 1.0
>!UHostMergeProperties 1.2
>!ABusPower!C 1.0
|IOReport!F 47
>!AACPIPlatform 6.1
>!ASMC 3.1.9
>watchdog 1
|IOPCI!F 2.9
|IOACPI!F 1.4
@kec.pthread 1
@kec.corecrypto 1.0
@kec.Libm 1

[email protected] · ‎10-10-2020

BSD process name corresponding to current thread: net.pulsesecure.

Did the crash occur after installing the client or while connected to VPN?

Current thread name is not complete, can you please check if its net.pulsesecure.firewall component?

Pulse Connect Secure Certified Expert

vkiryakov · ‎10-12-2020

Yes, it's net.pulsesecure.firewall

[email protected] · ‎10-13-2020

@vkiryakov Thanks! 9.1R8.2 (3999) is not using KEXT (upgraded to use Security Extensions), so you can safely unload that using the following command and check the behavior:

sudo kextunload "/Applications/Pulse Secure.app/Contents/Plugins/Kext/PulseSecureFirewall.kext"

In the meanwhile, open a support ticket with Apple and see if they can provide us the root cause from the crash log.

Pulse Connect Secure Certified Expert

[email protected]gmail.com · ‎10-19-2020

Have the exact same issue, however the remedy does not seem to work:

$ sudo kextunload "/Applications/Pulse Secure.app/Contents/Plugins/Kext/PulseSecureFirewall.kext"
(kernel) Kext net.pulsesecure.PulseSecureFirewall not found for unload request.
Failed to unload net.pulsesecure.PulseSecureFirewall - (libkern/kext) not found.

Pulse Secure version: 9.1.8 (3999)

OS: macOS Catalina 10.15.7 (19H2)

panic(cpu 0 caller 0xffffff8003ec1eff): assertion failed: (MBUFQ_EMPTY(&cfq->q_mq) && cfq->q_start == cfq->q_end) || (!MBUFQ_EMPTY(&cfq->q_mq) && cfq->q_start != cfq->q_end), file: /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-6153.141.2/bsd/net/content_filter.c, line: 1067
Backtrace (CPU 0), Frame : Return Address
0xffffffa3eb293730 : 0xffffff800371a65d 
0xffffffa3eb293780 : 0xffffff8003854a75 
0xffffffa3eb2937c0 : 0xffffff80038465fe 
0xffffffa3eb293810 : 0xffffff80036c0a40 
0xffffffa3eb293830 : 0xffffff8003719d27 
0xffffffa3eb293930 : 0xffffff800371a117 
0xffffffa3eb293980 : 0xffffff8003ec1a6c 
0xffffffa3eb2939f0 : 0xffffff8003ec1eff 
0xffffffa3eb293a00 : 0xffffff8003ad7bde 
0xffffffa3eb293a40 : 0xffffff8003ad7ef8 
0xffffffa3eb293bd0 : 0xffffff8003ad63e8 
0xffffffa3eb293c20 : 0xffffff8003ad320e 
0xffffffa3eb293c80 : 0xffffff8003c5b5c9 
0xffffffa3eb293cd0 : 0xffffff8003cd32eb 
0xffffffa3eb293df0 : 0xffffff8003ca7eac 
0xffffffa3eb293e20 : 0xffffff8003ca14a0 
0xffffffa3eb293ee0 : 0xffffff8003ca1260 
0xffffffa3eb293f40 : 0xffffff8003d84be7 
0xffffffa3eb293fa0 : 0xffffff80036c1206 
BSD process name corresponding to current thread: net.pulsesecure.
Boot args: chunklist-security-epoch=0 -chunklist-no-rev2-dev chunklist-security-epoch=0 -chunklist-no-rev2-dev
Mac OS version:
19H2
Kernel version:
Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64
Kernel UUID: 05D51A3D-3A87-3FF0-98C3-9CF3827A3EDD
Kernel slide:     0x0000000003400000
Kernel text base: 0xffffff8003600000
__HIB  text base: 0xffffff8003500000
System model name: MacBookPro16,1 (Mac-E1008331FDC96864)
System shutdown begun: NO
System uptime in nanoseconds: 215329550858704
last loaded kext at 204558740613818: >usb.IOUSBHostHIDDevice	1.2 (addr 0xffffff7f879d4000, size 45056)
last unloaded kext at 204743585807967: >!UAudio	323.4 (addr 0xffffff7f8aabb000, size 434176)
loaded kexts:
@filesystems.smbfs	3.4.4
@kext.AMDRadeonX6000	3.1.0
@kext.AMDRadeonServiceManager	3.1.0
>!AGraphicsDevicePolicy	5.2.6
@fileutil	20.036.15
@AGDCPluginDisplayMetrics	5.2.6
>!AHV	1
|IOUserEthernet	1.0.1
|IO!BSerialManager	7.0.6f7
>!AUpstreamUserClient	3.6.8
>pmtelemetry	1
>AGPM	111.4.4
>X86PlatformShim	1.0.0
>!APlatformEnabler	2.7.0d0
>AGDCBacklightControl	5.2.6
>!A!IKBLGraphics	14.0.7
@Dont_Steal_Mac_OS_X	7.0.0
>BridgeAudioCommunication	6.70.7
>!AThunderboltIP	3.1.4
>!AHIDALSService	1
>!AAVEBridge	6.1
>!AMCCSControl	1.14
>!AMuxControl2	5.2.6
>!ATopCaseHIDEventDriver	3430.1
>!ABridgeAudio!C	6.70.7
>!AGFXHDA	100.1.429
>!A!IPCHPMC	2.0.1
>!A!ICFLGraphicsFramebuffer	14.0.7
>!A!ISlowAdaptiveClocking	4.0.0
@filesystems.autofs	3.0
>BCMWLANFirmware4355.Hashstore	1
>BCMWLANFirmware4364.Hashstore	1
>BCMWLANFirmware4377.Hashstore	1
>!ABCMWLANBusInterfacePCIe	1
@filesystems.hfs.kext	522.100.5
@BootCache	40
@!AFSCompression.!AFSCompressionTypeDataless	1.0.0d1
@!AFSCompression.!AFSCompressionTypeZlib	1.0.0
>!AVirtIO	1.0
@filesystems.apfs	1412.141.1
@private.KextAudit	1.0
>!ASmartBatteryManager	161.0.0
>!AACPIButtons	6.1
>!ASMBIOS	2.1
>!AACPIEC	6.1
>!AAPIC	1.7
$!AImage4	1
@nke.applicationfirewall	303
$TMSafetyNet	8
@!ASystemPolicy	2.0.0
|EndpointSecurity	1
>usb.!UHub	1.2
@kext.AMDRadeonX6100HWLibs	1.0
@kext.AMDRadeonX6000HWServices	3.1.0
|IOAVB!F	850.1
>!ABacklightExpert	1.1.0
@kext.AMDRadeonX6000Framebuffer	3.1.0
@!AGPUWrangler	5.2.6
>!AActuatorDriver	3440.1
>!AHIDKeyboard	209
|IO!BHost!CUARTTransport	7.0.6f7
|IO!BHost!CTransport	7.0.6f7
>!A!ILpssUARTv1	3.0.60
>!A!ILpssUARTCommon	3.0.60
>!AOnboardSerial	1.0
>!ASMBus!C	1.0.18d1
>!AGraphicsControl	5.2.6
>!AHS!BDriver	3430.1
>IO!BHIDDriver	7.0.6f7
>!AMultitouchDriver	3440.1
>!AInputDeviceSupport	3440.8
|IONDRVSupport	576.1
@kext.AMDSupport	3.1.0
@!AGraphicsDeviceControl	5.2.6
|IOAccelerator!F2	438.7.3
|IOGraphics!F	576.1
|IOSlowAdaptiveClocking!F	1.0.0
>X86PlatformPlugin	1.0.0
>IOPlatformPlugin!F	6.0.0d8
@plugin.IOgPTPPlugin	840.3
|IOEthernetAVB!C	1.1.0
@kext.triggers	1.0
>usb.cdc.ncm	5.0.0
>usb.cdc	5.0.0
>usb.networking	5.0.0
>usb.!UHostCompositeDevice	1.2
>!ABCMWLANCore	1.0.0
>mDNSOffloadUserClient	1.0.1b8
>IOImageLoader	1.0.0
|IO80211!FV2	1200.12.2b1
>corecapture	1.0.4
|IOSkywalk!F	1
|IOSurface	269.11
@filesystems.hfs.encodings.kext	1
|IOAudio!F	300.2
@vecLib.kext	1.2.0
|IOSerial!F	11
>usb.!UVHCIBCE	1.2
>usb.!UVHCI	1.2
>usb.!UVHCICommonBCE	1.0
>usb.!UVHCICommon	1.0
>!AEffaceableNOR	1.0
|IOBufferCopy!C	1.1.0
|IOBufferCopyEngine!F	1
|IONVMe!F	2.1.0
>!AThunderboltPCIDownAdapter	2.5.4
>!AThunderboltDPInAdapter	6.2.6
>!AThunderboltDPAdapter!F	6.2.6
>!AHPM	3.4.4
>!A!ILpssI2C!C	3.0.60
>!A!ILpssDmac	3.0.60
>!A!ILpssI2C	3.0.60
>!AThunderboltNHI	5.8.6
|IOThunderbolt!F	7.6.1
|IOUSB!F	900.4.2
>usb.!UXHCIPCI	1.2
>usb.!UXHCI	1.2
>!AEFINVRAM	2.1
>!AEFIRuntime	2.1
>!ASMCRTC	1.0
|IOSMBus!F	1.1
|IOHID!F	2.0.0
$quarantine	4
$sandbox	300.0
@kext.!AMatch	1.0.0d1
>!AKeyStore	2
>!UTDM	489.120.1
|IOSCSIBlockCommandsDevice	422.120.3
>!ACredentialManager	1.0
>!AFDEKeyStore	28.30
>!AEffaceable!S	1.0
>!AMobileFileIntegrity	1.0.5
@kext.CoreTrust	1
|CoreAnalytics!F	1
|IOTimeSync!F	840.3
|IONetworking!F	3.4
>DiskImages	493.0.0
|IO!B!F	7.0.6f7
|IO!BPacketLogger	7.0.6f7
>!ASSE	1.0
>KernelRelayHost	1
>!ASEPManager	1.0.1
>IOSlaveProcessor	1
|IOUSBMass!SDriver	157.140.1
|IOSCSIArchitectureModel!F	422.120.3
|IO!S!F	2.1
|IOUSBHost!F	1.2
>usb.!UCommon	1.0
>!UHostMergeProperties	1.2
>!ABusPower!C	1.0
|IOReport!F	47
>!AACPIPlatform	6.1
>!ASMC	3.1.9
>watchdog	1
|IOPCI!F	2.9
|IOACPI!F	1.4
@kec.pthread	1
@kec.corecrypto	1.0
@kec.Libm	1

Regards,

Oli

cbrauckmiller · ‎10-19-2020

As previously stated, please upgrade to the 9.11R8.2 Mac Client.

We removed the requirment for Kernel Extentions in this release and this should resolve your issue.

Thank you

Craig Brauckmiller

Eastern Region Escalation Manager

Pulse Secure

[email protected]gmail.com · ‎10-27-2020

We have now upgraded to a newer client (Version 9.1.9 (4983)), the issue persists.

Still cannot remove the extension:

$ sudo kextunload "/Applications/Pulse Secure.app/Contents/Plugins/Kext/PulseSecureFirewall.kext"
(kernel) Kext net.pulsesecure.PulseSecureFirewall not found for unload request.
Failed to unload net.pulsesecure.PulseSecureFirewall - (libkern/kext) not found.

Regards,

Oli

cbrauckmiller · ‎10-29-2020

All, it does appear that this is a slightly different issue than what was reported earlier with KEXT and Pulse. Do any of you have access to the PCS Appliance Admin UI? If so, I'd like ot know if the role you are assigned to has the Traffic Enforcement option enabled? This feature will install a network filter driver on the Mac and seems to be causing an issue. If you open the network settings page, do you have a Pulse Secure adapter listed?

I'd request that those who are seeing the crash reports, PLEASE, open a case with us so we can collect all of the appropriate configuration data and client logs.

Thank you

Craig Brauckmiller

Pulse Secure

ucsb_jbalkind · ‎11-05-2020

I've been hitting this same issue almost daily for a couple of weeks, with both 9.1.8 and 9.1.9. In the last 24 hours my machine rebooted without warning 3 times despite me quitting the application and never connecting the VPN. I uninstalled Pulse Secure yesterday and it still crashed again today. Looking now I see that the network adapter you mention was still there despite me uninstalling the application.

The logs sent by others are the same as what I was seeing. Every time the failure is in content_filter.c with the process being pulse. I can't really keep this installed if it's crashing my entire machine at random even when not in use, but I need to be able to use the VPN.

cbrauckmiller · ‎11-06-2020

Have you tried to disable the Pulse adapter? Does that resolve the crash? Uninstalling the app "should" remove the adapter. Did you remove Pulse via the standard method or did you just delete directories manuall?

Again, please have your IT group open a case with us so we can get more data to our developers. The more crash report data we get, the better.

Thanks

Craig Brauckmiller