My last post did not take... Trying again... Juniper says that we have to set Network Connect IP Address filters per device. Currently the cluster is set with each device having its own Network Connect Connection Profile and separate users pools are assigned to each device via the NC Connection Profile. Juniper says that because this is a cluster, one member can and does grab IP pools from another member and that is what is causing the changing of IP address and "moving" of users from one box to another. Implementation Steps: To add filters, perform the following steps on the primary cluster member: 1) Login 2) Click "Network" in left-navigation pane 3) Click "Network Connect" tab 4) Pick a cluster member from the drop-down "Settings for:" and click update. 5) Set the filter below you would like to apply for this particular cluster member. We just implemented this and will monitor. ... View more

Juniper believes that this is a configuration issue. We have the user IP pools separated via the Network Connect Connection Profiles for each device in the cluster. However, they say that because it is a cluster one device can and does use the user pool from the other device unless we set up Network Connect IP Address filters per device. That way the only IP Pool used for the member of the cluster will be what you set in the filter. Implementation Steps: To add filters, perform the following steps on the primary cluster member: 1) Login 2) Click "Network" in left-navigation pane 3) Click "Network Connect" tab 4) Pick a cluster member from the drop-down "Settings for:" and click update. 5) Set the filter you would like to apply for this particular cluster member. We just implemented this and will check going forward if it has the desired affect. ... View more

Thanks - our firewall team - who supports the F5s - worked with F5 support to be sure that we are configured properly and to review the logs in great detail. It is set for persistence and the inactivity is at 12 hours (43,200 seconds) as you suggest. It had been set to 30 minutes before. It didn't make the slightest difference. ... View more

Kita - The ticket opened with JTAC is 2011-1026-0834. There were several opened by our vendor to Juniper and I believe that this one has the latest and greatest detail. Getting very frustrating as after all of this time and all of the conf calls, testing with the load balancers, escalation to F5 to ensure that the load balancers are perfectly configured and working perfectly, and emails around this we received another email from the JTAC engineer working this blaming it on the load balancers again. See below as it has some detail that may be helpful to anyone tracking these posts. I removed names for privacy and security reasons. I know you are new to this case but we have beaten the persistence setting on the load balancerÓ topic to death already. It has been set to persistence for over a month now and JTAC requested that the inactivity time out for the persistence on the load balancer be set to 12 hours instead of the 30 minutes that we had it set to previously. That was implemented on Tuesday. Persistence is absolutely, positively, without a doubt, enabled. See screenshot below and note that currently there are 2,774 persistence records for ("URL") RIGHT nowÉ ($ grep "pool \"("URL")" ldns | wc -l >> 2774). Furthermore, as has been stated several times already, the (other region ive) & (other region ive)Singapore are experiencing the same exact issue and they are set to 100/0 with persistence and inactivity time out set to 12 hours and the failover is the (main region ive) (not the secondary device in those locations). The load balancer has absolutely no knowledge of the secondary boxes in the (other region ive) and (other region ive). Yet, somehow, someway, the users are bouncing back and forth between the primary and secondary IVEs in the (other region ive) and (other region ive) per the logs as well. This is NOT a load balancer persistence issue. I agree with xxxxxx Ð we need to have yet another call on this as we keep going around in circles. ... View more

Niol - Thanks for the response. JTAC has asked for tcpdumps, user logs, event logs, user traces, etc. from the IVEs and our vendor who manages the devices sent everything to JTAC for review. I'm not sure if they specifically looked at the tcpdumps to verify the swapping of users but JTAC acknowledges that they see the issue. I'm going to ask the vendor to send me the tcpdumps and see if I can track it. Note that JTAC also asked for and received many wireshark captures and client-side logs from several end users as well. The same load balancers resolve the URLs for users internally and externally and we have tested extensively and also escalated to F5 to ensure that this is not an F5 issue. In all testing the we absolutely resolve to the same IP we were originally resolved to both internally and externally. ... View more

Thanks for your response. Below are answers to your questions and some more detail. I am very eager to get this resolved and will provide any detail you require. Note that as this has been escalated to very high level JTAC engineers - each JTAC engineer has put a lot of time and effort into this but unfortunately the issue is still very prevalent. It is not due to lack of effort from Juniper. - Since when did this issue start? From the best of our knowledge it has ALWAYS been this way. We are only now getting a great deal of complaints as users who had been migrated from our old IPsec platform to this new platform were really not using this new platform until we disabled their old accounts. So now all users, no matter what, are using the Junipers and the complaints are growing daily. - Any upgrade on the IVE OS / Any changes on the network / Any changes on the load balancer? No changes to IVE - we are running 7.0R5.1 for many months now. We cannot upgrade any higher as it wreaks havoc with the "logoff on connect" feature that about 50% of our users use. When we upgrade the new version of NC does not keep the setting for "logoff on connect" so all users that have it enabled have to re-enable it. We can't push policy for "logoff on connect" as only 50% use it and we can't hit everyone... Note that I escalated this with JTAC and in the next release they hope to have this issue fixed as well. No changes at all to network. Only changes to Load Balancer was to set persistence - which per Juniper Admin Guide we should have always had on! - and then recently we upped the load balancer inactivity timer to 12 hours instead of 30 minutes - What load balancer are you using there and have you involved your load balancer team on to this and if yes, what do they say? We are using F5 load balancers and the F5 team is fully on top of this and they see absolutely no issue with the F5 deployment and we use these F5 devices for a ton of load balancing for many other applications with no issue. - Do you see any logs on the load balancer regarding this? We do not have F5 logs that show exactly what is happening and we would have to set up a sniffer trace to understand if there is truly an issue but it is very important to understand that we have no issues with load balancing any other servers/apps and more importantly our UK and Singapore solutions are set to 100/0 on the load balancer where if the primary goes down in UK or Singapore the traffic is sent to the United States - not the secondary in the UK or Singapore cluster. We see absolutely no evidence of users in UK or Singapore bouncing back and forth between UK/Singapore and the US. That speaks volumes here as it really points us to the Juniper devices/clusters. The secondary devices in UK and Singapore are not used in any way. shape, or form via our network, load balancer, etc. yet we see users boucning over to them in the Juniper logs. Also, the load balancer logs shows absolutely no instances of any of the monitored devices going down. So the LB is not moving anyone to another IP. - Are there any time pattern you see this issue? (Peak hours / Non-peak hours / Anytime / Always) Always - Is there any reason why JTAC feels it's your load balancer issue? (Any specific logs pointing to it?) I'm going to try to sum up what JTAC is saying and I hope I capture it 100% properly... They say that although state tables are shared within a cluster, Juniper has no mechanism for moving users back and forth between the clusters unless one of the clusters goes down. <<tnone of the bosxes are ever down even momentarily per montioring tools, event logs, etc>> So JTAC keeps thinking it is our load balancer even though we supply evidence that it cannot be the load balancer. <<For example - sorry to repeat - LB is set to 100/0 with persistence and inactivity at 12 hours in UK and Singapore and the LB does not even know about the IP addresses of the secondary devices out there and failover is to US. So LB in this situation really can't be involved.>> JTAC keeps thinking that the user - on initial sign-in - resolves the name to one member IP and then somehow starts resolving to the other member IP. They also said that when Host Checker performs its 60 minute interval scan to ensure AV is running and user is still aligned with security policy it could reach out to the user and when the response comes back it could be going to the other cluster member and therefore the member that the user is on thinks it does not get a response and blows the user away. Because they see the users bouncing back and forth between boxes they think this is the cause of the drops. However, the LB now keeps the user going to the same IP for 12 hours with persistence and the maximum session length set in the IVE is 12 hours so during the life of the session the user will resolve to the same IP. On top of that, when a user connects with Network Connect their hosts table is configured by Juniper to use the IP address of the IVE that they connected to. So really, once a user is connected, they should always resolve via their hosts file to the correct IP. PLEASE NOTE - We have SA 6500s in US and UK and SA 4500s in Singapore and in all cases, nearly all users are using Network Connect so we cannot turn off acceleration per your detail. <<<Please "DO NOT" disable the SSL hard acceleration if you have many NC (Network Connect) users. Since our NC (Network Connect) client uses 3DES/AES by default, by disabling acceleration, there could be a noticeable performance impact on the system. You may not see a process restart per-se, but packet loss will be greater (for NC) and the page response time will be slower (for core).>>>> HOWEVER:::: Either late last year or early this year we were getting complaints of slow response and JTAC suggested that we turn OFF hardware acceleration. So if your points in your post are accurate, could that be why we are still having slow response reports from our end users???? If all of our users are using Network Connect and we have hardware acceleration disabled are we adding to the slow response misery???? See JTAC response about this below: It has to do with SSL Acceleration option. This feature is enabled by default on all SA6500 devices. It is used to offload SSL operation from the main CPUÓ Due to some issues seen in the past with enabling the feature, the vendor recommends disabling it. Below is a statement we received from the vendor: We are recommending that it be disabled for two reasons, out of an abundance of caution, and secondly, it will have no impact on the functionality of the box, you do not require it for the box to work, but you will potentially see a CPU hit of 20-30% as the device approaches maximum user load. PR for the potential SSL issue is 562746. ... View more