Popular news and blogging sites such as Slashdot and Digg have huge readerships. They are community driven and allow their members to post articles on various topics ranging from hazelnut chocolate bars to global warming. These sites, due to their massive readership, have the power to generate huge spikes in the web traffic to those (un)fortunate enough to get mentioned in their articles. Fortunately Traffic Manager and TrafficScript can help.
If the referenced site happens to be yours, you are faced with dealing with this sudden and unpredictable spike in bandwidth and request rate, causing:
Traffic Manager has the ability to shape traffic in two important ways. Firstly, you can restrict the amount of bandwidth any client or group of clients are allowed to consume. This is commonly known as "Bandwidth Management" and in Traffic Manager it's configured by using a bandwidth class. Bandwidth classes are used to specify the maximum bits per second to make available. The alternative method is to limit the number of requests that those clients or group of clients can make per second and/or per minute. This is commonly known as "Rate Shaping" and is configured within a rate class.
Both Rate Shaping and Bandwidth Management classes are configured and stored within the catalog section of Traffic Manager. Once you have created a class it is ready for use and can be applied to one or more of your Virtual Servers. However the true power of these Traffic Shaping features really becomes apparent when you make use of them with TrafficScript.
I would class an Abusive Referer as any site on the internet that refers enough traffic to your server to overwhelm it and effectively deny service to other users. This abuse is usually unintentional, the problem lies in the sheer number of people wanting to visit your site at that one time. This slashdot effect can be detected and dealt with by a TrafficScript rule and either a Bandwidth or a Rate Class.
Take a look at the TrafficScript below for an example of how you could stop a site (in this instance slashdot) from from using a large proportion or all of your available bandwidth.
$referrer = http.getHeader( "Referer" ); if( string.contains( $referrer, "slashdot" ) ) { http.addResponseHeader( "Set-Cookie", "slashdot=1" ); response.setBandwidthClass( "slashdot" ); } if( http.getCookie( "slashdot" ) ) { response.setBandwidthClass( "slashdot" ); }
In this example we are specifically targeting slashdot users and preventing them from using more bandwidth than we have allotted them in our "slashdot" bandwidth class. This rule requires you to know the name of the site you want protection from, but this rule or similar could be modified to defend against other high traffic sites.
# Referer whitelist. These referers are never rate limited. $whitelist = "localhost 172.16.121.100"; # Referers that are allowed to pass a higher number of clients. $highTraffic = "google mypartner.com"; # How many queued requests are allowed before we track users. $shapeQueue = 2; # Retrieve the referer and strip out the domain name part. $referer = http.getheader("Referer"); $referer = String.regexsub($referer, ".*?://(.*?)/.*", "$1", "i" ); # Check to see if this user has already been given an abuse cookie. # If they have we'll force them into a bandwidth class if ( $cookie = http.getCookie("AbusiveReferer") ) { response.setBandwidthClass("AbusiveReferer"); } # If the referer is whitelisted then exit. if ( String.contains( $whitelist, $referer ) ) { break; } # Put the incoming users through the busy or standard rate classes # and check the queue length for their referer. if ( String.contains( $highTraffic, $referer ) ) { $backlog = rate.getbacklog("BusyReferer", $referer); rate.use("BusyReferer", $referer); } else { $backlog = rate.getbacklog("StandardReferer", $referer); rate.use("StandardReferer", $referer); }
# If we have exceeded our backlog limit, then give them a cookie # this will enforce bandwidth shaping for subsequent requests. if ( $backlog > $shapeQueue ) { http.setResponseCookie("AbusiveReferer", $referer); response.setBandwidthClass("AbusiveReferer"); }
In order for the TrafficScript to function optimally, you must enter your servers own domainname(s) into the white list. If you do not, then the script will perform rate shaping on everyone surfing your website!
You also need to set appropriate values for the BusyReferer and StandardReferer shaping classes. Remember we're only counting the clients entry to the site, so Perhaps you want to set 10/minute as a maximum standard rate and then 20/minute for your BusyReferer rate.
In this script we also use a bandwidth class for when things get busy. You will need to create this class, called "AbusiveReferer" and assign it an appropriate amount of bandwidth. Users are only put into this class when their referer is exceeding the rate of referrals set by the relevant rate class.
Rate Shaping classes can be given a context so you can apply the class to a subset of users, based on a piece of key data. The second script uses context to create an instance of the Rate Shaping class for each referer. If you do not use context, then all referers will share the same instance of the rate class.
Traffic Manager can use bandwidth and rate shaping classes to control the number of requests that can be made by any group of clients. In this article, we have covered choosing the class based on the referer, which has allowed us to restrict the rate at which any one site can refer visitors to us. These examples could be modified to base the restrictions on other data, such as cookies, or even extended to work with other protocols. A good example would be FTP, where you could extract the username from the FTP logon data and apply a bandwidth class based on the username.