For ESP you'll need both 443/TCP and 4500/UDP. 443/TCP is used for the control channel.
Is there a way to eliminate using port 443 at all?
Yes, I've been told that the control data for the session always flows over TCP 443. If this is true then I dont understand how your sessions could be working. I would check your firewall logs to see if you have both 443 and 4500 coming from a client.