Augmenting the Web with Stingray Traffic Manager (04/05/2010)

[aknox]

Augmenting the Web with Stingray Traffic Manager (04/05/2010)

This article illustrates how you can use Stingray Traffic Manager to personalize and enhance the websites you visit. Augmented browsing allows you to get the content you want presented to you in the way that you want - a feature that is becoming increasingly rare as more and more data becomes available on the Internet.

By using Stingray Traffic Manager to augment websites you can see the effects in any browser without having to install extra plugins. Even better - all this can be achieved with a free development licensed version of Stingray Traffic Manager!

Although you would traditionally expect to find Stingray Traffic Manager deployed in front of a series of web servers, making sure that everything runs smoothly for the users accessing a website, it can be used in many other scenarios involving network traffic. Many of these applications don't require the traffic manager to process large volumes of traffic and so are perfectly suited for use with a development license that is restricted only by the number of requests that can be processed every second.

This article explores one such application: using TrafficScript to personalize and augment websites.

Augmented Browsing

The basic principle of augmented browsing is that a website generates and delivers content to you, which is then examined and rewritten using rules that you specify, before being given to the browser to render. Perhaps the most well known example of augmented browsing technology is

<a href="http://www.greasespot.net/" target="_blank">*Greasemonkey*</a>

. Greasemonkey is a fantastic tool for post-processing websites, however it is not available in all browsers and requires an extension to be installed in others. There are also some inconsistencies when running the same scripts under different browsers. Using Stingray Traffic Manager to augment websites offers a browser-independent solution that doesn't require any add-ons and can be used by anyone accessing the web through your traffic manager.

Some of the possible uses for augmented browsing include:

2. Altering the style of a website - for example changing the colors, or highlighting certain parts of the website.
4. Enhancing existing content, such as expanding shortened URLs, or adding links to related information.
6. Adding new content - for example adding buttons to social networking sites that perform multiple operations, such as accepting or rejecting all invitations.
8. Filtering out unwanted parts of a website, such as adverts, comments, videos, or explicit words.

Configuring Stingray Traffic Manager

First of all, you'll need a copy of the traffic manager! If you don't already have one, you can obtain a copy of it along with a free developer license

<a href="http://www.zeus.com/downloads/developers/" target="_blank">*here*</a>

. Then next step is to configure the traffic manager to proxy your web requests - as follows:

Creating a Service

When you have the traffic manager up and running, create a new virtual server by going to Services > Virtual Servers. In theCreate a new virtual server section, enter a name, such as 'HTTP Proxy', set the protocol to HTTP, the port to 8080 (this port is typically used by HTTP proxies) and the default traffic pool to 'discard'.

You will now have a service that, when started, will receive requests on port 8080 and discard them. Not very useful! Instead you need to tell the service to send all requests to their target URLs. This will be done using TrafficScript, but first the traffic manager will need to know how to handle the requests that are being sent out - for example, how long it should wait for responses.

Creating a Pool

To provide these settings you will need to create a pool. To do this, go to Services > Pools and in the Create a new Pool section, enter 'HTTP Proxy Pool' for the name and enter any valid hostname and port for the nodes, such as 'localhost:5678', as no requests will actually be sent there. Leave the monitor field set to 'No Monitor'. The default pool settings will suffice for now, so no further changes are necessary.

Proxying Requests

TrafficScript can now be used to forward the incoming requests to the correct location. Create a new TrafficScript rule for the HTTP Proxy service you created earlier by going to Services > Virtual Servers > HTTP Proxy > Rules and clicking Manage Rules in Catalog under Request Rules. In the Create a new rule section, enter 'Forward Proxy' as the name and select 'Use TrafficScript Language', then click 'Create Rule' (leave the 'associate with virtual server' check box ticked). This will create a new rule and associate it with your HTTP Proxy service.

The service is going to be receiving requests from your browser that look like this:

GET

<a title="http://www.zeus.com/about/index.html" href="http://www.zeus.com/about/index.html" target=_blank>*http://www.zeus.com/about/index.html*</a>

HTTP/1.1

What it needs to do to operate as an HTTP proxy is to discover the hostname from the URL (in this case,

<a title="www.zeus.com" href="http://www.zeus.com/" target=_blank>*www.zeus.com*</a>

) and forward the request there with the protocol and hostname removed from the Request-URI and the hostname added as a Hostheader field as follows:

GET /about/index.html HTTP/1.1

Host:

<a title="www.zeus.com" href="http://www.zeus.com/" target=_blank>*www.zeus.com*</a>

The following code will achieve that - just copy and paste it into the rule and click 'Update':

• $url = http.getRawURL();
• if( string.startsWith( $url, "/" ) ) {
• # Not a typical request to a proxy - get the destination
• # from the Host header instead
• $host = http.getHostHeader();
• } else if( string.startsWith( $url, "http://" )
• || string.startsWith( $url, "https://" ) ) {
• # A typical request to a proxy - strip off the protocol
• # and the path so we just have the hostname
• $host = string.substring( $url, string.find( $url, "://" ) + 3,
• string.length( $url ) - 1 );
• $host = string.substring( $host, 0, string.find( $host, "/" ) - 1 );
• # Set the host header to be the hostname
• http.setHeader( "Host", $host );
• }
• # Find out to which port we should be sending the request
• $port = 80;

if( string.regexmatch( $host, "^(.*)
d+)$" ) ) {

2. $host = $1;
4. $port = $2;
6. }
10. # Strip off the protocol and hostname from the Request-URI
12. http.setPath( http.getPath() );
16. # Forward the request
18. pool.use( "HTTP Proxy Pool", $host, $port );
19. 
    
    

The rule simply grabs the hostname (either from the Request-URI as discussed above, or from the Host header if it doesn't exist), sets the Request-URI to be just the path portion of the incoming URL and then forwards the request to the destination host and port using the settings specified in the 'HTTP Proxy Pool' pool you created earlier.

Configuring the Virtual Server

There are a few additional settings needed to make this service forward requests properly. In Services > Virtual Servers > HTTP Proxy > Connection Management expand the HTTP-Specific Settings section and set add_cluster_ip to No. Under Cookie Settings set cookie!domain to 'Rewrite the domain to the host header of the request'. Under Location Header settings setlocation!rewrite to 'Nothing'. These settings are useful only when sending requests to internal servers.

Streaming Responses

You're almost done setting things up now! The final task is to make sure that responses are streamed back efficiently to the browser. If the contents of the responses are going to be modified then the body data of the response must be read. By default, the traffic manager will read in the entire body of the response before making any modifications to it so your browser won't be able to start rendering the content immediately.

To remedy this, you can use the HTTP streaming feature of TrafficScript to modify data as it arrives and immediately stream it to the browser.

Create a response rule for your service by going to Services > Virtual Servers > HTTP Proxy > Rules and clicking Manage Rules in Catalog under Response Rules. Under Create a new rule enter the name 'Proxy Response' and select 'Use TrafficScript Language' again. Paste the following code into the rule and save it:

2. if( string.startswith( http.getResponseHeader( "Content-Type" ), "text/html" ) ) {
4. # Start streaming all HTML responses
6. http.stream.startResponse( http.getResponseCode(),
8. http.getResponseHeader( "Content-Type" ) );
12. while( 1 ) {
16. # Get the next line of data
18. $data = http.stream.readResponse( 4096, "\n" );
22. # See if we've reached the end of the response
24. if( $data == "" ) {
26. break;
28. }
32. # ... Here is where we will make modifications to the response ...
36. # Write the data to the client
38. http.stream.writeResponse( $data );
40. }
44. # Stop streaming the response
46. http.stream.finishResponse();
48. }

This rule will stream data to the browser as it is received, but will also allow you to make modifications to it, line-by-line, as it arrives.

Finally, start up the service by going to the front page of the traffic manager and clicking the play button next to the HTTP Proxy service. Now you're all set to use the traffic manager as a proxy!

Configuring your Browser to use the Proxy

In your browser's advanced settings, enter the traffic manager's hostname and port 8080 (or whichever port you chose to run the service on) as the HTTP proxy settings and start browsing the web. If everything worked then you should be able to browse the web as normal - but you will also see some activity in the status applet within the traffic manager's user interface. If something has gone wrong then double check that the rules match those presented above, and that all the necessary settings have been changed - and feel free to leave a comment!