World Series of Linux: Our Methodology

The players came from two camps: Debian and RPM. On the Debian side of the competition, engineers evaluated Ubuntu 7.10 Gutsy Gibbon, Freespire, and Xandros Professional Edition. On the RPM side, engineers looked at SUSE Linux Enterprise Desktop 10, Fedora 7, and PCLinux OS.

Test Center engineers considered installing each distribution in a virtual machine, and then decided against it because if there were any problems, it would be difficult to determine whether the issue was with the distribution or with the VM. Instead, engineers installed each distribution one at a time.

The tests were originally designed and split up between two rounds. For the preliminary round, each distribution had to complete a list of tasks. The distribution that successfully completed the tasks, had the easiest interface, and the best user support advanced to the final round. A winner was selected from each league, from the Debian-based systems and from the RPM-based systems, to face-off in the final World Series competition. The final round had only three tasks, and the distribution was ranked depending on how easy it was to complete them.

In reality, testing spread out to three rounds, and the final round was expanded. Since the RPM-distributions struggled to complete the preliminary round, they were re-evaluated on a different test box as part of another round of testing. The final round was expanded to include all the tasks from the first preliminary round as well as the three "advanced" tasks.

For the initial round of testing, engineers focused on installation and basic office setup. The goal was to see how easily and effortlessly the operating system installed. Engineers ranked the installation process, the software installed, and the operating system's look and feel. After confirming that the system had a working network connection, engineers downloaded updates, installed a printer, and printed a document. A third-party software was installed. Finally, engineers played a CD, a DVD, and Internet video to determine multimedia support. The test concluded by taking screenshots and saving them onto a USB disk drive plugged into the box.

Finding software programs to download was a little tricky since all the distributions came with most applications bundled into the system, such as OpenOffice, Pidgin instant messaging client, GIMP image editor, and GNOME screen capture software. Engineers selected Wireshark, a network troubleshooting tool from wireshark.org.

For the second ad hoc round, the printing and networking tests were repeated on a different system for two distributions. The winner of this ad hoc round moved on to the final round.

For the final round which pit the two winners against each other, the focus was on advanced features, such as creating and managing users, setting up wireless networks, mounting networked (Samba) Windows shares, and installing peripherals.