Performance Testing Each Linux Distribution as an Internet Server

Each distribution was tested using KeyLabs' FireBench%99 firewall performance test tool. For years, KeyLabs has used FireBench to create realistic traffic loads to tax and evaluate standalone firewall performance. However, for this evaluation, the typical pass-through test network was modified so that we could focus on the performance on each Linux distribution as an Internet server, not specifically as a firewall. The goal of this test was to specify an objective for the Linux server and then configure each Linux distribution to best match that objective. The objective assigned to the Linux distributions in the test was to provide basic Internet server services. Therefore, each distribution in this comparison was configured with basic HTTP, FTP and firewall services.

The same PC configuration was used for each Linux distribution, which consisted of a Compaq DeskPro EN 815 with a Pentium III 866-MHz CPU and 256 Mbytes RAM. Each system's disk partitions were wiped before installation. IPTables was selected as the firewall of choice for all distributions except for Slackware, which was using IPTables' predecessor, IPChains. We also enabled each distributions' preferred Web and ftp servers. All other services, with the exception of basic kernel processes and system logs, were disabled. As per FireBench specifications, each system's firewall was configured to conform to predetermined security objectives, which were to block all traffic except for HTTP and ftp. These security objectives were identical for all systems in this evaluation, however, the specific firewall rules required to achieve those security objectives may vary.

FireBench measures four different performance metrics--connections per second, concurrent connections, FTP throughput and HTTP throughput.

The Connections-Per-Second test measures the firewall's ability to correctly handle high connection rates. For the purposes of this test a connection is counted if it can be opened, pass traffic and close without error. The methodology for the connections-per-second test follows this basic looped process: 1) Open a TCP/IP connection using port 80 to a Web server on the Linux system under test, 2) Send a minimal HTTP 1.0 HEAD request to the server for the main default page, 3) Check that it received the correct header from the server within one secon, 4) Close the connection.

The Concurrent Connections test benchmarks the maximum number of active connections that the firewall can attain. Each thread opens a TCP/IP connection using port 80 to the Web server on the Linux system being tested and then performs the following actions in a loop: 1) Send a HTTP 1.1 GET request with Keep-Alive enabled to the server for a 100-byte text file; 2) Check that the text file was received within three seconds; 3) Wait for three second minus the time it took to receive the file. Each second FireBench records the number of active connects. The number of connections is increased over time until no more connections can be added.

FTP and HTTP Throughput tests measure how many bytes per second can be transferred from the Linux system under test via the given protocol (i.e. ftp or HTTP). The following looped process is followed: 1) Open a TCP/IP connection, 2) Either log into the ftp server and retrieve a 20-Mbyte binary file or send an HTTP 1.0 GET request to the server for a 20 Mbyte binary file, 3) Log the number of bytes transferred every second, 4) Close the connection after retrieving the file.

To view more details on the FireBench testing tool, download the Firewall Benchmarking with FireBench II white paper from KeyLabs' Web site, http://www.keylabs.com/portal.

All testing for this shootout was performed at KeyLabs%99, the IT industry's premier technology assurance solution provider. Since its inception in 1996, KeyLabs has led the testing industry in the development of a full suite of custom network testing services that includes e-commerce stress and security testing, performance analysis, scalability analysis and proof-of-concept testing. In addition, KeyLabs develops and manages industry certification programs for software and hardware vendors.