Hi James,
Do you use SSL or ESP? with ESP you should have better performance. What if you enlarge the UDP4500 port for ESP traffic and check for max UDP Packets Per Second, might be an issue in that.
Cheers!