As you are stating that pure SSL is working fine, but using ESP gives problems this issue could relate to UDP.
I have seen a similar case where ESP was used.
When NAT is involved ESP uses NAT-T(encapsulation in UDP-4500).
What i saw was a large amount of UDP packets with sourceport 4500 being dropped by the firewall in front of the SSL VPN device. The firewall saw those packet as an attempt to initiate a UDP-session from the SSL VPN device.
To solve this problem you can either choose to run pure SSL org try to adjust the timeout values for the UDP sessions through the firewall.