Performance Testing Each Linux Distribution as an Internet Server
The same PC configuration was used for each Linux distribution, which consisted of a Compaq DeskPro EN 815 with a Pentium III 866-MHz CPU and 256 Mbytes RAM. Each system's disk partitions were wiped before installation. IPTables was selected as the firewall of choice for all distributions except for Slackware, which was using IPTables' predecessor, IPChains. We also enabled each distributions' preferred Web and ftp servers. All other services, with the exception of basic kernel processes and system logs, were disabled. As per FireBench specifications, each system's firewall was configured to conform to predetermined security objectives, which were to block all traffic except for HTTP and ftp. These security objectives were identical for all systems in this evaluation, however, the specific firewall rules required to achieve those security objectives may vary.
FireBench measures four different performance metrics--connections per second, concurrent connections, FTP throughput and HTTP throughput.
The Connections-Per-Second test measures the firewall's ability to correctly handle high connection rates. For the purposes of this test a connection is counted if it can be opened, pass traffic and close without error. The methodology for the connections-per-second test follows this basic looped process: 1) Open a TCP/IP connection using port 80 to a Web server on the Linux system under test, 2) Send a minimal HTTP 1.0 HEAD request to the server for the main default page, 3) Check that it received the correct header from the server within one secon, 4) Close the connection.
The Concurrent Connections test benchmarks the maximum number of active connections that the firewall can attain. Each thread opens a TCP/IP connection using port 80 to the Web server on the Linux system being tested and then performs the following actions in a loop: 1) Send a HTTP 1.1 GET request with Keep-Alive enabled to the server for a 100-byte text file; 2) Check that the text file was received within three seconds; 3) Wait for three second minus the time it took to receive the file. Each second FireBench records the number of active connects. The number of connections is increased over time until no more connections can be added.
FTP and HTTP Throughput tests measure how many bytes per second can be transferred from the Linux system under test via the given protocol (i.e. ftp or HTTP). The following looped process is followed: 1) Open a TCP/IP connection, 2) Either log into the ftp server and retrieve a 20-Mbyte binary file or send an HTTP 1.0 GET request to the server for a 20 Mbyte binary file, 3) Log the number of bytes transferred every second, 4) Close the connection after retrieving the file.
To view more details on the FireBench testing tool, download the Firewall Benchmarking with FireBench II white paper from KeyLabs' Web site, http://www.keylabs.com/portal.
All testing for this shootout was performed at KeyLabs%99, the IT industry's premier technology assurance solution provider. Since its inception in 1996, KeyLabs has led the testing industry in the development of a full suite of custom network testing services that includes e-commerce stress and security testing, performance analysis, scalability analysis and proof-of-concept testing. In addition, KeyLabs develops and manages industry certification programs for software and hardware vendors.