Making Networks More Secure, Part 2

Ethereal Open Source protocol analyzer

This week I'll explain how Ethereal can help you recognize network scans and identify protocol misconfigurations -- particularly vexing when setting up virtual private networks (VPNs). I'll also show how Ethereal can help you work with application users, who often don't know what ports their software uses when interacting with the Internet.

For those who of you who do not want to revisit Part 1, or who've never seen that Recipe, here's a recap of some basic information: A protocol analyzer is a software tool that tunes into packets of data as they move across networks. When properly used, it helps demonstrate why secure software is a good idea. It also helps to secure networks in a wide variety of ways.

For system builders and networks, a protocol analyzer can be an invaluable diagnostic and troubleshooting tool. In fact, nearly all kinds of network-security software -- including routers, firewalls, intrusion detection systems, and security scanners -- work by examining inbound or outbound network traffic to decide if it should be blocked or allowed to continue on its way. Protocol analyzers do the same thing; but instead of acting on what they find, they display the contents of the traffic so humans can inspect it and possibly act on it. But there's nothing automatic about a protocol analyzer. That's why it's so smart to understand this as a passive yet powerful inspection tool.

What humans can observe or learn as they inspect the network traffic that a protocol analyzer collects -- called a "packet trace" or "packet capture" -- often relates to network-security issues. Although protocol analyzers have many uses outside the area of network security (for example, debugging software that interacts with a network), they have lots of security applications, too.

In this second part of the Recipe, I'll describe how certain well-known denial of service (DoS) attacks look to a protocol analyzer. I'll also discuss other cases when an incomplete TCP handshake can provide helpful diagnostic information. You'll also see what a port scan looks like -- it's a common reconnaissance technique that can signal impending attack -- and learn how to identify ports that applications use, or when port mismatches sometimes occur.

In this Recipe, I will show how to apply Ethereal in two scenarios, both of which have numerous security applications:

• TCP handshake: Many application-layer IP protocols use TCP for transport. Careful observation of what's going on as TCP connections are requested, negotiated, and established -- called the TCP handshake or the three-way handshake (more on this soon) -- can detect potential DoS attacks. It can also help diagnose problems when seeking legitimate service connections.
• Port observation: Protocol analyzers will cheerfully display all TCP and UDP ports that incoming or outgoing traffic seeks to access. When regular increasing sweeps over valid port addresses occur, port scans may be underway -- and are often preludes to attacks. Likewise, when you are trying to use custom or non-standard applications, or when you're looking for potential malware (worms or Trojans) at work, finding certain ports active or requests for access to same can be informative.

Regardless of how you choose to use Ethereal -- or train your clients to use it themselves -- you must first install the software on a machine running on the network segment you wish to monitor; attach an Ethereal-running laptop; and launch the software. All three steps are covered in detail in Part 1 of this Recipe.

Of course, you'll also need a copy of Ethereal. All the necessary download links are available at Ethereal.com, including mirrors all over the world as well as at the organization's home site.

Checking the TCP Handshake

First, here's what you might see and what it can tell you. The TCP handshake is called "three-handed" because it involves a three-step process between two hosts. These three steps involve the following:

• The initiating host sends a packet with its SYN (synchronize sequence number) flag set, and a corresponding sequence number to get things going. This is the equivalent of saying, "Let's start a conversation and number all exchanges starting with the sequence number provided." This opening sally also includes an MSS (maximum segment size) value to help both hosts avoid unwanted segment fragmentation, as well as source and destination port numbers on which the conversation should be conducted.
• The receiving host replies with its own starting sequence number (SYN), an explicit acknowledgement (ACK) to permit communication to begin, its own MSS, and a confirmation of the source and destination port numbers originally sent.
• The initiating host completes the three-way handshake by sending an ACK to acknowledge receipt of the receiving host's sequence number and segment size information.

Now that we have our components assembled, let's look at the four steps involved in capturing TCP handshake traffic. (Later, I'll explain how to zero in on only the information you want or need to see.)

• Start Ethereal from the Start menu, a shortcut, or whatever method suits you best. The executable is named ethereal.exe so you can even launch it from the command line if you like. Define a display filter that reads tcp in the Filter text box in the lower left-hand corner of the default display.
• Start packet capture on your Ethereal machine by clicking the Capture menu, then the Start selection. The defaults for the program are fine for this exercise, so you needn't make any other changes.
• From either the machine where Ethereal is running or another machine, initiate some kind of TCP-based service. My example uses FTP, but any protocol that uses TCP will do.
• Stop the packet capture by clicking the Stop button. Either save only those packets that include the data on display, or simply print the display information for later reuse. Figure 1 below shows what such a display will look like. You shouldn't need to capture more than five to 10 seconds' worth of traffic for this example.

Fig. 1: View this screen-shot in a separate window.

Figure 1 shows the three phases of the TCP handshake in the lines numbered 4, 5, and 6 on the left-hand margin of the screen shot. They are:

• Line 4 shows the local host on my subnet, 172.16.1.33, initiating the handshake with the FTP server at 206.224.65.195. The "Info" fields on the right-hand side show a typical starting sequence number of 0, a TCP Window size of 16K, and an MSS of 1460.
• Line 5 shows the remote host on the Internet at 206.224.65.195 returning the initial ACK along with the SYN. It returns a TCP Window size of 8760, and a maximum segment size of 1460 (same as the original).
• Line 6 shows the local host, 172.16.1.33, completing the handshake with a final ACK, an increment of the sequence number to 1, and a TCP Window size of 17,520.

The conclusion of the sequence, of course, is the opening of an FTP connection. That's why line 7 in the screen-shot above shows the initial step involved in logging into the FTP server to establish an open connection. But before FTP can run, TCP must hook up client and server. That's what you see happening in lines 4 through 6.

Not all TCP handshakes go forward to completion. Sometimes, they are refused by the destination host for any of a number of reasons: "no connections available," "host busy," and "connection refused" are all possible error messages. Sometimes no ACK occurs at all.

Other times -- and this is where DoS attacks unfortunately come into play -- the initiating host deliberately does not respond to the other host's ACK. When this occurs, the second host must keep a potential TCP session open and wait for a timeout period (typically one or two seconds) to expire before the second host can release that connection so the resources necessary to open the connection can then be reused.

A simple DoS attack occurs when a single host sends a large number of SYN packets to open a TCP connection, but then fails to respond to the other host's ACK to complete the handshake. A distributed denial of service (DDoS) attack occurs when multiple hosts send large numbers of SYN packets to open TCP connections, but never respond to any ACKs from the target host. Either way, if a computer is attacked, the protocol analyzer will show a large number of SYN and SYN, ACK packets without any closing ACK packets to finish the handshake. People trying to use a host that's under attack will find it inaccessible, because TCP/IP stack implementations set limits on the total number of open connections they can handle. DoS attacks deliberately overwhelm such machines with more requests for connections than they can handle, thereby swamping the host's IP stack. This explains why this type of attack is also called a "SYN Flood."

How do you deal with a DoS attack? The best way is to work with your ISP or whomever is upstream from the Internet connection for the host under attack. Although it's possible to configure a firewall to block input from the address (or addresses) where such attacks originate, this requires handling lots of incoming data. Far better to block DoS traffic as far upstream as possible -- and to stop it at its source as well.

The TCP handshake packets can also be used for diagnostic purposes when you're setting up a VPN connection. Handshake failure is a common symptom of misconfiguration. Where the handshake fails often indicates potential sources of problems. So this is well worth capturing and inspecting.

In general, if the VPN connection fails after the initial SYN packet, with no answering SYN, ACK from the target, there are several potential problems:

• Port selection: Double-check against the documentation, or with somebody who knows what the server is expecting.
• Authentication: Make sure certificates are valid, or dig further into traces to see if authentication server requests are completing successfully.
• Security associations (SA) : Make sure the client has a valid SA defined, and that the server's definitions include that valid SA in what's allowed.
• Errors in address restrictions: Make sure the client's source address is recognized as a legal endpoint for a VPN session.

If, on the other hand, the VPN connection proceeds through the second step and hands up at the third step, there's probably some kind of access control or other higher-level issue preventing the connection from occurring. In most cases, careful inspection of packet traces and reflection on where things are breaking down will tell you what to check -- and, possibly, what to fix.

Port Observation

Watching the ports that higher-layer protocols use can also be very informative. This requires looking past the handshake, if TCP is involved (since there is no handshake for UDP), and checking source and destination port addresses. This kind of information can be revealing in many security-related instances. Here's how to conduct a systematic port scan:

• Start Ethereal from either the Start menu, a shortcut, or whatever method suits you best. The executable is named ethereal.exe, so you can even launch it from the command line, if you like. Define a display filter that reads not arp && not http and ip.src == a.b.c.d in the Filter text box in the lower left-hand corner of the default display. You'll have to run a sample capture once to get the source address for the Symantec server where the scan originates. When I ran this capture, that address was 206.205.10.200, but that's likely to change.
• Start packet capture on your Ethereal machine by clicking the Capture menu. Then click the Start selection. The defaults for the program are fine for this exercise, so you needn't make any other changes.
• Fire up a Web browser and go to this Symantec page. Scroll down the Web page until you see the "check for security risks" graphic. Click on the graphic. Next, click the Start button under "Security Scan." Wait until the scan is complete before you proceed to the next step. (Warning: If your Internet connection is protected by a firewall, you may need to disable the firewall while the scan is underway to produce the desired results.)
• Stop the packet capture and look at the trace. Scroll partway down, and you should see something like Figure 2, below.

Fig. 2: View this screen-shot in a separate window.

Figure 2 shows a typical pattern for an ongoing port scan or port probe: a regular, usually increasing series of destination port addresses being accessed by a sequence of source port addresses outside the well-known port address number range (75xx and 78xx in the capture data shown).

What' s important here is a pattern of attempts to initiate TCP handshakes on a large number of ports in a short span of time. This is what indicates a scan is probably underway. Further inspection of such a trace will also tell you which local ports are responding -- not good, but it does happen. I had to disable my firewall to get this to work, because it was rejecting all such attempts, and it warned me that scans were underway!

You will no doubt notice scans on your Internet interface -- I get scanned 20-plus times a day, every day, on my Road Runner cable segment, which acts like a big local Ethernet segment. But beyond that, you will also find port information useful for other purposes as well.

In fact, while I was working on this article, a colleague confessed that his company tried to use the exact same configuration for a remote access VPN in the U.S. and Japan, thinking that because it worked in one place, it would automatically work in the other. Wrong! It wasn't until they put a protocol analyzer on a configuration that should have been working that they discovered the port addresses associated with the service used different pairs in the U.S. and in Japan. Further research showed this was a telco-based choice, and that a change of telcos from one location to the other made all the difference. Once the right port IDs were configured in the client software, the remote access VPN service worked like a charm.

Another great use for port identification is when applications outside those covered in the IANA (Internet Assigned Numbers Authority) port-address list are used -- or when such applications switch to non-standard ports, as can sometimes be the case. With a protocol analyzer in hand, there's no need to guess about port addresses. As long as you know the IP address for the host where the application traffic originates, you can use an address filter -- ip.src == a.b.c.d, where you'd replace the nonsense address shown with the real thing to get the desired results, of course -- to look for TCP or UDP packets. Then check source and destination port addresses. By the way, the same technique works admirably when looking for signs of presence of Trojans or other malware that may seek to, or succeed in, opening ports for unwanted uses.

All in all, system builders should find a protocol analyzer a valuable addition to their toolbox. If you attach a properly-equipped laptop to a network segment or an Internet connection, you can quickly gauge the hostility of the surrounding environment, and learn all kinds of other useful things. Also, showing customers potential security is more convincing than talking about them, and a protocol analyzer is great for "show and tell."

Sidebar: Port Address Information

If you're using a protocol analyzer and need to figure out which ports are which, you'll find the following resources incredibly helpful.

The Pine Mountain Group, one of a small number of organizations that offers technical certifications for those interested in protocol analysis, has published a pocket reference that's handy when working with a protocol analyzer. It's called the NetAnalyst Reference Guide. In addition to a reasonably comprehensive list of port numbers, it also includes protocol information, numerous decodes, IP addressing info galore, and more. Worth checking out!

This is Part 2 of a two-part Recipe. Part 1, by the same author, explains the basics of using Ethereal.

ED TITTEL is a technology writer who has contributed to more than 100 computer books, a trainer, and a consultant who specializes in IT certification and information security. He has no commercial affiliation with any vendor or product described in this article.

Did you try this Recipe? If so, how did it work out? To discuss this or other TechBuilder Recipes, start or join a discussion thread on the Recipe Forum. Membership is required. Not yet a member? Becoming a member is free, secure, and fast. Join TechBuilder today.