1. The Issue
When using perpetual licensing on Traffic Manager instances which are clustered, the failure of one of the instances results in licensed throughput capability being lost until that instance is recovered.
2. The Solution
Automatically adjust the bandwidth allocation across cluster members so that wasted or unused bandwidth is used effectively.
3. A Brief Overview of the Solution
An SSC holds the configuration for the Traffic Manager cluster members.
The Traffic Managers are configured to execute scripts on two events being raised, the machinetimeout event and the allmachinesok event.
Those scripts make REST calls to the SSC in order to dynamically and automatically amend the Traffic Manager instance configuration held for the two cluster members.
4. The Solution in a Little More Detail
4.1. Move to an SSC Licensing Model
If you're currently running Traffic Managers with perpetual licenses, then you'll need to move from the perpetual licensing model to the SSC licensing model. This effectively allows you to allocate bandwidth and features across multiple Traffic Managers within your estate. The SSC has a "bucket" of bandwidth along with configured feature sets which can be allocated and distributed across the estate as required, allowing for right-sizing of instances, features and also allowing multi-tenant access to various instances as required throughout the organisation.
Instance Hosts and Instance resources are configured on the SSC, after which a Flexible License is uploaded on each of the Traffic Manager instances which you wish to be licensed by the SSC, and those instances "call home" to the SSC regularly in order to assess their licensing state and to obtain their feature set.
For more information on SSC, visit the Riverbed website pages covering this product, here - SteelCentral Services Controller for SteelApp Software.
There's also a Brochure attached to this article which covers the basics of the SSC.
4.2. Traffic Manager Configuration and a Bit of Bash Scripting!
The SSC has a REST API that can be accessed from external platforms able to send and receive REST calls. This includes the Traffic Manager itself.
To carry out automated bandwidth allocation on cluster members, we'll need to carry out the following;
- a. Create a script which can be executed on the Traffic Manager, which will issue REST calls in order to change the SSC configuration for the cluster members in the event of a cluster member failure.
- b. Create another script which can be executed on the Traffic Manager, which will issue REST calls to reset the SSC configuration for the cluster members when all of the cluster members are up and operational.
- c. Upload the two scripts to be used, on to the Traffic Manager cluster.
- d. Create a new event and action on the Traffic Manager cluster which will be initiated when a cluster member fails, calling the script mentioned in point a above.
- e. Create a new event and action on the Traffic Manager cluster which will be initiated when all of the cluster members are up and operational, calling the script mentioned in point b above.
4.2.a. The Script to Re-allocate Bandwidth After a Cluster Member Failure
This script, called Cluster_Member_Fail_Bandwidth_Allocation and attached, is shown below.
Script Function:
- Determine which cluster member has executed the script.
- Make REST calls to the SSC to allocate bandwidth according to which cluster member is up and which is down.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
|
#!/bin/bash # # Cluster_Member_Fail_Bandwidth_Allocation # ---------------------------------------- # Called on event: machinetimeout # # Checks which host calls this script and assigns bandwidth in SSC accordingly # If demo-1 makes the call, then demo-1 gets 999 and demo-2 gets 1 # If demo-2 makes the call, then demo-2 gets 999 and demo-1 gets 1 # # Grab the hostname of the executing host Calling_Hostname=$(hostname -f) # If demo-1.example.com is executing then issue REST calls accordingly if [ $Calling_Hostname == "demo-1.example.com" ] then # Set the demo-1.example.com instance bandwidth figure to 999 and # demo-2.example.com instance bandwidth figure to 1 curl -k --basic -H "Content-Type: application/json" -H "Accept: application/json" -u adminuser:adminpassword -d \ '{"bandwidth":999}' \ curl -k --basic -H "Content-Type: application/json" -H "Accept: application/json" -u adminuser:adminpassword -d \ '{"bandwidth":1}' \ fi # If demo-2.example.com is executing then issue REST calls accordingly if [ $Calling_Hostname == "demo-2.example.com" ] then # Set the demo-2.example.com instance bandwidth figure to 999 and # demo-1.example.com instance bandwidth figure to 1 curl -k --basic -H "Content-Type: application/json" -H "Accept: application/json" -u adminuser:adminpassword -d \ '{"bandwidth":999}' \ curl -k --basic -H "Content-Type: application/json" -H "Accept: application/json" -u adminuser:adminpassword -d \ '{"bandwidth":1}' \ fi |
There are some obvious parts to the script that will need to be changed to fit your own environment. The hostname validation, the admin username and password in the REST calls and the SSC name, port and path used in the curl statements. Hopefully from this you will be able to see just how easy the process is, and how the SSC can be manipulated to contain the configuration that you require.
This script can be considered a skeleton, as can the other script for resetting the bandwidth, shown later.
4.2.b. The Script to Reset the Bandwidth
This script, called Cluster_Member_All_Machines_OK and attached, is shown below.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
#!/bin/bash # # Cluster_Member_All_Machines_OK # ------------------------------ # Called on event: allmachinesok # # Resets bandwidth for demo-1.example.com and demo-2.example.com - both get 500 # # Set both demo-1.example.com and demo-2.example.com bandwidth figure to 500 curl -k --basic -H "Content-Type: application/json" -H "Accept: application/json" -u adminuser:adminpassword -d \ '{"bandwidth":500}' \ curl -k --basic -H "Content-Type: application/json" -H "Accept: application/json" -u adminuser:adminpassword -d \ '{"bandwidth":500}' \ |
Again, there are some parts to the script that will need to be changed to fit your own environment. The admin username and password in the REST calls and the SSC name, port and path used in the curl statements.
4.2.c. Upload the Bash Scripts to be Used
On one of the Traffic Managers, upload the two bash scripts that will be needed for the solution to work. The scripts are uploaded in the Catalogs > Extra Files > Action Programs section of the Traffic Manager, and can then be referenced from the Actions when they are created later.
4.2.d. Create a New Event and Action for a Cluster Member Failure
On the Traffic Manager (any one of the cluster members), create a new event type as shown in the screenshot below - I've created Cluster_Member_Down, but this event could be called anything relevant. The important factor here is that the event is raised from the machinetimeout event.
Once this event has been created, an action must be associated with it.
Create a new external program action as shown in the screenshot below - I've created one called Cluster_Member_Down, but again this could be called anything relevant. The important factor for the action is that it's an external program action and that it calls the correct bash script, in my case called Cluster_Member_Fail_Bandwidth_Allocation.
4.2.e. Create a New Event and Action for All Cluster Members OK
On the Traffic Manager (any one of the cluster members), create a new event type as shown in the screenshot below - I've created All_Cluster_Members_OK, but this event could be called anything relevant. The important factor here is that the event is raised from the allmachinesok event.
Once this event has been created, an action must be associated with it.
Create a new external program action as shown in the screenshot below - I've created one called All_Cluster_Members_OK, but again this could be called anything relevant. The important factor for the action is that it's an external program action and that it calls the correct bash script, in my case called Cluster_Member_All_Machines_OK.
5. Testing
In order to test the solution, simply DOWN Traffic Manager A from an A/B cluster. Traffic Manager B should raise the machinetimeout event which will in turn execute the Cluster_Member_Down event and associated action and script, Cluster_Member_Fail_Bandwidth_Allocation.
The script should allocate 999Mbps to Traffic Manager B, and 1Mbps to Traffic Manager A within the SSC configuration.
As the Flexible License on the Traffic Manager polls the SSC every 3 minutes for an update on it's licensed state, you may not see an immediate change to the bandwidth allocation of the Traffic Managers in questions.
You can force the Traffic Manager to poll the SSC by removing the Flexible License and re-adding the license again - the re-configuration of the Flexible License will then force the Traffic Manager to re-poll the SSC and you should then see the updated bandwidth in the System > Licenses (after expanding the license information) page of the Traffic Manager as shown in the screenshot below;
To test the resetting of the bandwidth allocation for the cluster, simply UP Traffic Manager B. Once Traffic Manager B re-joins the cluster communications, the allmachinesok event will be raised which will execute the All_Cluster_Members_OK event and associated action and script, Cluster_Member_All_Machines_OK.
The script should allocate 500Mbps to Traffic Manager B, and 500Mbps to Traffic Manager A within the SSC configuration.
Just as before for the failure event and changes, the Flexible License on the Traffic Manager polls the SSC every 3 minutes for an update on it's licensed state so you may not see an immediate change to the bandwidth allocation of the Traffic Managers in questions.
You can force the Traffic Manager to poll the SSC once again, by removing the Flexible License and re-adding the license again - the re-configuration of the Flexible License will then force the Traffic Manager to re-poll the SSC and you should then see the updated bandwidth in the System > Licenses (after expanding the license information) page of the Traffic Manager as before (and shown above).
6. Summary
Please feel free to use the information contained within this post to experiment!!!
If you do not yet have an SSC deployment, then an Evaluation can be arranged by contacting your Partner or Brocade Salesman. They will be able to arrange for the Evaluation, and will be there to support you if required.
