Gathering statistics from a cluster of Traffic Managers

[#1]

This article describes how to gather activity statistics across a cluster of traffic managers using Perl, SOAP::Lite and Traffic Manager's SOAP Control API.

Overview

Each local Traffic Manager tracks a very wide range of activity statistics. These may be exported using SNMP or retrieved using the System/Stats interface in Traffic Manager's SOAP Control API.

When you use the Activity monitoring in Traffic Manager's Administration Interface, a collector process communicates with each of the Traffic Managers in your cluster, gathering the local statistics from each and merging them before plotting them on the activity chart.

'Aggregate data across all traffic managers'

However, when you use the SNMP or Control API interfaces directly, you will only receive the statistics from the Traffic Manager machine you have connected to. If you want to get a cluster-wide view of activity using SNMP or the Control API, you will need to poll each machine and merge the results yourself.

Using Perl and SOAP::Lite to query the traffic managers and merge activity statistics

The following code sample determines the total TCP connection rate across the cluster as follows:

1. Connect to the named traffic manager and use the getAllClusterMachines() method to retrieve a list of all of the machines in the cluster;
2. Poll each machine in the cluster for its current value of TotalConn (the total number of TCP connections processed since startup);
3. Sleep for 10 seconds, then poll each machine again;
4. Calculate the number of connections processed by each traffic manager in the 10-second window, and calculate the per-second rate accurately using high-res time.

The code:

#!/usr/bin/perl -w

use SOAP::Lite 0.6;
use Time::HiRes qw( time sleep );
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME}=0;

my $userpass = "admin:admin";      # SOAP-capable authentication credentials
my $adminserver = "stingray:9090"; # Details of an admin server in the cluster
my $sampletime = 10;               # Sample time (seconds)

sub getAllClusterMembers( $$ );
sub makeConnections( $$$ );
sub makeRequest( $$ );

my $machines = getAllClusterMembers( $adminserver, $userpass );
print "Discovered cluster members ". ( join ", ", @$machines ) . "\n";

my $connections = makeConnections( $machines, $userpass,
   "http://soap.zeus.com/zxtm/1.0/System/Stats/" );

# sample the value of getTotalConn
my $start = time();
my $res1 = makeRequest( $connections, "getTotalConn" );
sleep( $sampletime-(time()-$start) );

my $res2 = makeRequest( $connections, "getTotalConn" );

# Determine connection rate per traffic manager
my $totalrate = 0;
foreach my $z ( keys %{$res1} ) {
   my $conns   = $res2->{$z}->result - $res1->{$z}->result;
   my $elapsed = $res2->{$z}->{time} - $res1->{$z}->{time};

   my $rate = $conns / $elapsed;
   $totalrate += $rate;
}

print "Total connection rate across all machines: " .
      sprintf( '%.2f', $totalrate ) . "\n";

sub getAllClusterMembers( $$ ) {
    my( $adminserver, $userpass ) = @_;

    # Discover cluster members
    my $mconn =  SOAP::Lite
         -> ns('http://soap.zeus.com/zxtm/1.0/System/MachineInfo/')
         -> proxy("https://$userpass\@$adminserver/soap")
         -> on_fault( sub  {
              my( $conn, $res ) = @_;
              die ref $res?$res->faultstring:$conn->transport->status; } );

    $mconn->proxy->ssl_opts( SSL_verify_mode => 0 );  

    my $res = $mconn->getAllClusterMachines();

    # $res->result is a reference to an array of System.MachineInfo.Machine objects

    # Pull out the name/port of the traffic managers in our cluster
    my @machines = grep [email protected]://(.*?)/@[email protected],
       map { $_->{admin_server}; } @{$res->result};

    return \@machines;
}

sub makeConnections( $$$ ) {
    my( $machines, $userpass, $ns ) = @_;
    my %conns;

    foreach my $z ( @$machines ) {
       $conns{ $z } = SOAP::Lite
         -> ns( $ns )
         -> proxy("https://$userpass\@$z/soap")
         -> on_fault( sub  {
              my( $conn, $res ) = @_;
              die ref $res?$res->faultstring:$conn->transport->status; } );

       $conns{ $z }->proxy->ssl_opts( SSL_verify_mode => 0 );
    }
    return \%conns;
}

sub makeRequest( $$ ) {
    my( $conns, $req ) = @_;
    my %res;
    foreach my $z ( keys %$conns ) {
       my $r = $conns->{$z}->$req();
       $r->{time} = time();
       $res{$z}=$r;
    }

    return \%res;
}

Running the script

$ ./getConnections.pl

Discovered cluster members stingray1-ny:9090, stingray1-sf:9090

Total connection rate across all machines: 5.02

[mattwater]

Owen, I have tried running your example and have recieved the following error.

C:\Scripts\Stingray>getConnections.pl

500 Can't connect to mystingray:9090 (certificate verify failed) at C:\Scripts\Stingray\getConnections.pl line 49.

[owen]

Hi Mathew,

Apologies for the problem... some perl installations are a lot more strict about not accepting self-signed SSL certificates than others... I've updated the code in the article above to add in the necessary steps to persuade it to accept Stingray's admin server certificate.

The script should work fine for you now

regards

Owen

[mattwater]

Thanks Owen, your knocking them out the park for me today.

Have to run but will test tomorrow.

[mattwater]

I just tried the new script and its failing on or around line 51

401 Unauthorised .... getConnections.pl line 51.

[owen]

Hi Matt - check that you've edited the authentication credentials in line 8:

   my $userpass = "admin:admin";      # SOAP-capable authentication credentials

This value should be usernameassword, corresponding to a user account on the Stingray cluster that has permissions to use the SOAP API.

regards

Owen

[mattwater]

Sorry for the late reply, holidays

All fixed, pointing the script to the STM I created the account on helps.