Multi-hosted IP addresses allow the same traffic IP to be hosted on multiple traffic managers at the same time. This can provide benefits to traffic distribution and reduce the number of IPs needed to run a service.
For a background on Traffic Manager's Fault Tolerance and Clustering approach, please refer to the document Feature Brief: Clustering and Fault Tolerance in Traffic Manager.
How do Multi-Hosted IP addresses work?
Multi-hosted IPs make use of multicast MAC addresses. When Traffic Manager observes an ARP for the target traffic IP address, it responds with a multicast MAC address that is calculated based on the value of the multicast IP address used for clustered communications:
The upstream switch will relay packets destined to the traffic IP address to that MAC address; because it's a multicast MAC, the switch will learn which nodes are using that MAC address and forward the traffic to each (problems with switches learning the location of MAC address - check out this solution: Why can't users connect to my multi-hosted IPs?).
The zcluster kernel module ( Kernel Modules for Linux Software) implements a filter in the hosts TCP stack that partitions the traffic to that traffic IP address and discards all but the host's share of the traffic. The method used to determine the shares that each host takes is stable and guarantees statistically-perfect distribution of traffic. It handles failover gracefully, redistributing the failed traffic share evenly between the remaining traffic managers, while ensuring that the remaining traffic managers don't need to rebalance their own shares to preserve the statistically-perfect distribution, and it does so in a stable fashion that does not require any per-connection synchronization or inter-cluster negotiation.
In more detail
Suppose you have 4 traffic managers in a cluster, all listening on the same user-specified IP address. These traffic managers all ARP the same multicast MAC address so the switch forwards all incoming traffic to that IP to all traffic managers.
The zcluster kernel module will:
- Take source IP, source port, destination IP, destination port (this is sufficient to identify a TCP or UDP session) and hash them together
- Give its local traffic manager a fixed quarter of the hash space and silently discard the remaining 3-quarters
This means that every connection is handled by just one traffic manager and connections are evenly distributed (on a statistical basis) between traffic managers.
If one or more traffic managers fail, then the distribution method makes three guarantees:
- The hash space is balanced perfectly equally between the running traffic managers
- When a traffic manager fails or recovers, the only part of the hash space that is redistributed is the portion relating to that traffic manager (i.e. it’s not necessary to move any of the hash space between running traffic managers to rebalance)
- The method only depends on the instantaneous state of the cluster
This means that the only ‘synchronization’ is the shared view of the health of the cluster (i.e. for the method to be stable, each running traffic manager has the same view as to which traffic managers are running and which are failed). Traffic Manager's health broadcasts ensure that this is the case (apart from possible momentary differences when one or more traffic managers fail or recover).
A worked example
Suppose you have four traffic managers, A, B, C and D. They each get 1/4 of the hash space.
- Traffic manager D fails: its 1/4 is shared three ways between A, B and C (so the hash space is still balanced)
- Traffic manager C fails: its original 1/4 and the 1/3 * 1/4 it inherited from D are both split half and half between A and B
- Traffic manager D recovers: Traffic managers A and B stop listening for 1/3 of their traffic (i.e. all of the traffic they inherited from D and 1/3rd of the traffic they inherited from C), and D starts listening for the portions that A and B released
Observations
Multi-hosted Traffic IP addresses are useful when you want to ensure even distribution of traffic across a cluster (for example, to spread SSL processing load) or when you have a very limited number of public IP addresses at your disposal.
They do replicate ingress traffic across multiple ports in the internal network. The implication of this is that each Traffic Manager needs the same ingress bandwidth as the upstream switch; this is rarely a significant problem.
There is a built-in limit of 8 - the maximum number of traffic managers that can be present in a multi-cast traffic IP group.
Using Multi-Hosted traffic IP groups with IP transparency to your back-end servers is challenging. You need to configure appropriate source-based routing on each back-end server so that it directs the egress traffic to the correct node in the cluster.
