10 Must-Have Linux Tools

Enter lsof, a utility that provides verbose output on current network connections and the files associated with them. Securing the network is easier once the solution provider learns what program is operating on an open port, which daemons have established connections, and what ports are open on that server. Ports that are open and accepting connections show up in lsof with the word LISTEN. The word ESTABLISHED indicates that a connection on the given port has been made.

Like screen, lsof is usually installed by default. The command lsof -v will indicate whether it exists on the system. If it's not there, lsof is available as a package for most distributions.

Running lsof by itself will output all open files corresponding to every active process on the box. This can be quite lengthy, so adding the -h flag will give a rundown in a more manageable size. In the output, lsof returns the process ID (PID), user running the command, the file descriptor, type of connection, device number, the Internet protocol and the name of the file or Internet address. The information does more than list network connections. For example, in an lsof output, there may be multiple processes associated with the sshd daemon. Looking at the user field indicates who is actually logged into the box.

The -i flag lists all open files associated with Internet connections. This option is useful in identifying unnecessary security risks and shutting them down. The search can also specify a particular port, service or host name using techniques such as lsof -i :22, lsof -i :smtp or lsof -i @techbuilder.org.

9. Who you gonna call? Sometimes, something is wrong because nothing is happening. There's no error message printed on the screen. Perhaps the program is hung trying to read a particular file or spinning on disk access way more than it should. For those situations, by tracing calls the program makes to each process and the subsequent calls made by those processes, solution providers can look at the path the program is taking and where it's getting stuck.

Linux has strace, which lets the solution provider see what system calls a process is making. In many cases, a program may fail because it is unable to open a file or because of insufficient memory, and that will be evident while sifting through strace-generated data. Applications causing a segfault should be run with strace to see if there are memory issues.

The strace command is useful to determine if something hangs during a DNS lookup. Every now and then, a system will hang just for a minute while running a program (say, Telnet). Performing strace on the program shows that the problem was the program trying to do a reverse DNS lookup on an IP address. The solution provider can now take appropriate action.

10. Know thy server: Perhaps a new name server has been deployed on the network. Or perhaps there are some problems with the DNS servers. Solution providers can try ping and other tests to check if the servers are up and running. However, dig would be more useful to diagnose DNS problems, such as discovering if all the servers are responding. Since dig returns output that looks just like an actual bind zone file, solution providers can make sure the name servers all have the same configuration.

There are several record types dig can query on, including IN for Internet, NS for name server, MX for mail server and SOA (Start of Authority). Looking at dig-generated output, solution providers can see information such as the primary name server and mail server priority. For solution providers wondering about mail server priority, dig will show which of the mail servers have higher priority in delivering mail. If the one with the higher priority fails or can't connect, the second highest mail server will then deliver the mail.

Using the txt option with dig would also indicate whether the mail server has an SPF (Sender Policy Framework) record. Servers without SPF records may have trouble delivering mail to Hotmail and other mail systems that perform spam filtering using complete SPF records.

Host is similar to dig, but not as comprehensive. However, the command host -a [domain] shows the complete resolve of everything -- name servers, mx servers, etc. -- associated with the domain. With both dig and host, solution providers can query simple to very complex and detailed DNS servers.