httperf is a benchmarking tool that measures the HTTP request throughput of a web server. The way it achieves this is by sending requests to the server at a fixed rate and measuring the rate at which replies arrive. Running the test several times and with monotonically increasing request rates, one can see the reply rate level off when the server becomes saturated, i.e., when it is operating at its full capacity.

autobench is a Perl wrapper around httperf. It runs httperf a number of times against a Web server, increasing the number of requested connections per second on each iteration, and extracts the significant data from the httperf output, delivering a CSV format file which can be imported directly into a spreadsheet for analysis/graphing.

openload is a load testing tool for Web applications. It simulates a number of concurrent users and it measures transactions per second (a transaction is a completed request to the Web server) and response time.

Introduction httperf
httperf is a tool to measure web server performance. It speaks the HTTP protocol both in its HTTP/1.0 and HTTP/1.1 flavors and offers a variety of workload generators. While running, it keeps track of a number of performance metrics that are summarized in the form of statistics that are printed at the end of a test run. The most basic operation of httperf is to generate a fixed number of HTTP GET requests and to measure how many replies (responses) came back from the server and at what rate the responses arrived.

EXAMPLES
httperf –hog –server www
This command causes httperf to create a connection
to host www, send a request for the root document
(http://www/), receive the reply, close the connec-
tion, and then print some performance statistics.

httperf –hog –ser www –num-conn 100 –rate 10 –timeout
5
Like above, except that a total of 100 connections
are created and that connections are created at a
fixed rate of 50 per second. Note that option
“–server” has been abbreviated to “–ser”.

httperf –hog –ser=www –wsess=10,5,2 –rate 1 –timeout
5
Causes httperf to generate a total of 10 sessions
at a rate of 1 session per second. Each session
consists of 5 calls that are spaced out by 2 sec-
onds.

httperf –client=0/1 –server=www.example.com –port=80 –uri=/ –rate=10 –send-buffer=4096 –recv-buffer=16384 –num-conns=500 –num-calls=1

Total: connections 500 requests 500 replies 500 test-duration 50.354 s

Connection rate: 9.9 conn/s (100.7 ms/conn, <=8 concurrent connections)
Connection time [ms]: min 449.7 avg 465.1 max 2856.6 median 451.5 stddev 132.1
Connection time [ms]: connect 74.1
Connection length [replies/conn]: 1.000

Request rate: 9.9 req/s (100.7 ms/req)
Request size [B]: 65.0

Reply rate [replies/s]: min 9.2 avg 9.9 max 10.0 stddev 0.3 (10 samples)
Reply time [ms]: response 88.1 transfer 302.9
Reply size [B]: header 274.0 content 54744.0 footer 2.0 (total 55020.0)
Reply status: 1xx=0 2xx=500 3xx=0 4xx=0 5xx=0

CPU time [s]: user 15.65 system 34.65 (user 31.1% system 68.8% total 99.9%)
Net I/O: 534.1 KB/s (4.4*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

The 3 arguments I specified on the command line are:

* server: the name or IP address of your Web site (you can also specify a particular URL via the –uri argument)
* rate: specifies the number of HTTP requests/second sent to the Web server — indicates the number of concurrent clients accessing the server
* num-conns: specifies how many total HTTP connections will be made during the test run — this is a cumulative number, so the higher the number of connections, the longer the test run

Here is a detailed interpretation of an httperf test run. In short, the main numbers to look for are the connection rate, the request rate and the reply rate. Ideally, you would like to see that all these numbers are very close to the request rate specified on the command line. If the actual request rate and the reply rate start to decline, that’s a sign your server became saturated and can’t handle any new connections. That could also be a sign that your client became saturated, so that’s why it’s better to test your client against a fast Web site in order to gauge how many outgoing HTTP requests can be sustained by your client.

Introduction Autobench

Autobench is a simple Perl script that facilitates multiple runs of httperf and automatically increases the HTTP request rate. Configuration of autobench can be achieved for example by means of the ~/.autobench.conf file. Here is how my file looks like:

# Autobench Configuration File

# host1, host2
# The hostnames of the servers under test
# Eg. host1 = iis.test.com
# host2 = apache.test.com

host1 = testhost1
host2 = testhost2

# uri1, uri2
# The URI to test (relative to the document root). For a fair comparison
# the files should be identical (although the paths to them may differ on the
# different hosts)

uri1 = /
uri2 = /

# port1, port2
# The port number on which the servers are listening

port1 = 80
port2 = 80

# low_rate, high_rate, rate_step
# The ‘rate’ is the number of number of connections to open per second.
# A series of tests will be conducted, starting at low rate,
# increasing by rate step, and finishing at high_rate.
# The default settings test at rates of 20,30,40,50…180,190,200

low_rate = 10
high_rate = 50
rate_step = 10

# num_conn, num_call
# num_conn is the total number of connections to make during a test
# num_call is the number of requests per connection
# The product of num_call and rate is the the approximate number of
# requests per second that will be attempted.

num_conn = 200
#num_call = 10
num_call = 1

# timeout sets the maximimum time (in seconds) that httperf will wait
# for replies from the web server. If the timeout is exceeded, the
# reply concerned is counted as an error.

timeout = 60

# output_fmt
# sets the output type – may be either “csv”, or “tsv”;

output_fmt = csv

## Config for distributed autobench (autobench_admin)
# clients
# comma separated list of the hostnames and portnumbers for the
# autobench clients. No whitespace can appear before or after the commas.
# clients = bench1.foo.com:4600,bench2.foo.com:4600,bench3.foo.com:4600

clients = localhost:4600

The only variable I usually tweak from one test run to another is num_conn, which I set to the desired number of total HTTP connections to the server for that test run. In the example file above it is set to 200.

I changed the default num_call value from 10 to 1 (num_call specifies the number of HTTP requests per connection; I like to set it to 1 to keep things simple). I started my test runs with low_rate set to 10, high_rate set to 50 and rate_step set to 10. What this means is that autobench will run httperf 5 times, starting with 10 requests/sec and going up to 50 requests/sec in increments of 10.

When running the following command line…

# autobench –single_host –host1=www.example.com –file=example.com.csv

# autobench –single_host –host1 example.com –uri1 /10K –quiet –low_rate 20 –high_rate 200 –rate_step 20 –num_call 1 –num_conn 500 –timeout 5 –file result.tsv

You can creat a flow chart using the values in the result.tsv

Introduction Openload
OpenWebLoad is a tool for load testing web applications. It aims to be easy to use and providing near real-time performance measurements of the application under test. This is particulary useful when you are doing optimization as you can see the impact of your changes almost immediately.

OpenWebLoad is (currently) a commandline tool, that you execute from a prompt like this:

openload [options] http://testapp.site.com 10

The 2 parameters are:

* The url of the web page you want to test.
* Number of simultanous clients to simulate. This is optional and defaults to 5.
* A number of options is also available. See here for a detailed description of all the options.

You will then get output similar to this:

$ openload localhost 10
URL: http://localhost:80/
Clients: 10
MaTps 355.11, Tps 355.11, Resp Time 0.015, Err 0%, Count 511
MaTps 339.50, Tps 199.00, Resp Time 0.051, Err 0%, Count 711
MaTps 343.72, Tps 381.68, Resp Time 0.032, Err 0%, Count 1111
MaTps 382.04, Tps 727.00, Resp Time 0.020, Err 0%, Count 1838
MaTps 398.54, Tps 547.00, Resp Time 0.018, Err 0%, Count 2385
MaTps 425.78, Tps 670.90, Resp Time 0.014, Err 0%, Count 3072

Total TPS: 452.90
Avg. Response time: 0.021 sec.
Max Response time: 0.769 sec

Where:

* MaTps: a 20 second moving average of TPS.
* Tps: (Transactions Per Second) is the number of completed requests during that second.
* Resp Time: the average response time in seconds for the elapsed second.
* Err: the percentage of responses that was erronous, i.e. didn’t return a HTTP 200 Ok staus.
* Count: the total number of completed requests.
* Total TPS is the average TPS for the whole run, i.e. (Total completed requests) / (Total elapsed time).
* Avg. Response time: the overall average response time in seconds.
* Max Response time: the highest response time during this run.