The Pulse Secure Virtual Traffic Manager includes a useful benchmarking tool named 'zeusbench' that we use for internal performance testing and as a load generation tool for training courses. The first incarnation of ZeusBench was donated to the Apache project a long time ago, and is now well known as ApacheBench. The new incarnation starts from a completely fresh codebase; it has a new 'rate' mode of operation, is less resource intensive and is often more accurate in its reporting.
You’ll find ZeusBench in the admin/bin directory of your Traffic Manager installation ($ZEUSHOME). Run zeusbench with the -–help option to display the comprehensive help documentation:
$ $ZEUSHOME/admin/bin/zeusbench --help
zeusbench generates HTTP load by sending requests to the desired server. It can run in two different modes:
When run with the –c N option, zeusbench simulates N concurrent HTTP clients. Each client makes a request, reads the response and then repeats, as quickly as possible.
Concurrency mode is very stable and will quickly push a web server to its maximum capacity if sufficient concurrent connections are used. It's very difficult to overpower a web server unless you select far too many concurrent connections, so it's a good way to get stable, repeatable transactions-per-second results. This makes it suitable for experimenting with different performance tunings, or looking at the effect of adding TrafficScript rules.
When run with the –r N option, zeusbench sends new HTTP requests at the specified rate (requests per second) and reads responses when they are returned.
Rate mode is suitable to test whether a web server can cope with a desired transaction rate, but it is very easy to overwhelm a server with requests. It's great for testing how a service copes with a flash-crowd - try running one zeusbench instance for background traffic, then fire off a second instance to simulate a short flash crowd effect. Rate-based tests are very variable; it's difficult to get repeatable results when the server is overloaded, and it’s difficult to determine the maximum capacity of the server (use a concurrency test for this).
The charts below illustrate two zeusbench tests against the same service; one where the concurrency is varied, and one where the rate is varied:
Measuring transactions-per-second (left hand axis, blue) and response times (right hand axis, red) in concurrency and rate-based tests
The concurrency-based tests apply load in a stable manner, so are effective at measuring the maximum achievable transactions-per-second. However, they can create a backlog of requests at high concurrencies, so the response time will grow accordingly.
The rate-based tests are less prone to creating a backlog of requests so long as the request rate is lower then the maximum transactions-per-second. For lower request rates, they give a good estimate of the best achievable response time, but they quickly overload the service when the request rate nears or exceeds the maximum sustainable transaction rate.
Keepalives can make a very large difference to the nature of a test. By default, zeusbench will open a TCP connection and use it for one request and response before closing it. If zeusbench is instructed to use keepalives (with the –k flag), it will reuse TCP connections indefinitely; the –k N option can specify the number of times a connection is reused before it is closed.
We’ve just scratched the surface of the options that zeusbench offers. zeusbench gives you great control over timeouts, the HTTP requests that it issues, the ability to ramp up concurrency or rate values over time, the SSL parameters and more.
$ $ZEUSHOME/admin/bin/zeusbench --help
for the detailed help documentation
When running a benchmark, it is always wise to sanity-check the results you obtain by comparing several different measurements. For example, you can compare the results reported by zeusbench with the connection counts charted in the Traffic Manager Activity Monitor. Some discrepancies are inevitable, due to differences in per-second counting, or differences in data transfers and network traffic counts.
Benchmarking requires a careful, scientific approach, and you need to understand what load you are generating and what you are measuring before drawing any detailed conclusions. Furthermore, the performance you measure over a low-latency local network may be very different to the performance you can achieve when you put a service online at the edge of a high-latency, lossy WAN.
For more information, check out the document:Tuning Virtual Traffic Manager
The article Dynamic Rate Shaping of Slow Applications gives a worked example of the use of zeusbench to determine how to rate-shape traffic to a slow application.
Also note that many Traffic Manager licenses impose a maximum bandwidth limit (development licenses are typically limited to 1Mbits); this will obviously impede any benchmarking attempts. You can commence a managed evaluation and obtain a license key that gives unrestricted performance if required.