How to Handle a Security Crisis: Nine Key Steps

The lists I offered in "Seven Common Sense Rules For Secure Websites" provide good discussion fodder for staff meetings when thinking about being proactive about security. But it's a different world when your organization is forced to be reactive about security, such as after an intrusion has been detected. Here are some guidelines for that situation, too:

The key to handling an attack is simple: Don't panic. It's very likely that by the time you discover the intrusion, most of the damage already has been done. In fact, it probably has been going on for weeks or months. The chance that you've discovered a break-in that happened just an hour ago is slim to none.

In that light, take a deep breath and begin developing a carefully thought-out strategy for dealing with the break-in. You need to avoid tipping off the intruder by announcing the break-in or by performing any other activity that would seem abnormal to someone who may have been watching your site's operations for many weeks. Hint: Performing a system backup usually is a good idea at this point and (hopefully!) will appear to be a normal activity to the intruder.

This also is a good time to remind yourself that some studies have shown that 60 percent of security incidents involve an insider. Be very careful with whom you discuss the incident until you're sure you have all the facts.

Here's a nine-step plan that may assist you in your time of crisis:

Step 1: Don't panic. In many cases, a problem isn't noticed until hours or days after it took place. Another few hours or days won't affect the outcome. The difference between a panicky response and a rational response will make a difference, however. Many recovery situations are exacerbated by the destruction of important log, state and tracking information during an initial panic.

Step 2: Decide on an appropriate level of response. No one benefits from an over-hyped security incident. Proceed calmly. Identify the staff and resources that must participate and leave others to assist with the post-mortem after it's all over.

Step 3: Hoard all available tracking information. Check accounting files and logs. Try to determine where the original breach occurred. Perform a backup of all your systems. Make sure you physically write-protect backup tapes if you put them in a drive to read them.

Step 4: Assess your degree of exposure. Determine what crucial information (if any) has "left" the company, and devise an appropriate mitigation strategy. Determine the level of future risk.

Step 5: Pull the plug. If necessary and appropriate, disconnect compromised machines from the network. Close known holes and stop the bleeding. The vulnerabilities database from ISS provides some good technical suggestions on what to actually do with the systems that were broken into. It can be found at http://xforce.iss.net

Step 6: Devise a recovery plan. With a creative colleague, draw up a recovery plan on a nearby whiteboard. This procedure is most effective when performed away from a keyboard. Focus on putting out the fire and minimizing the damage. Avoid assessing blame or creating excitement. In your plan, don't forget to address the psychological fallout your user community may experience.

Step 7: Communicate the recovery plan. Educate users and management about the effects of the break-in, the potential for future problems and your preliminary recovery strategy. Be open and honest. Security incidents are a fact of life in a modern networked environment. They are not a reflection on your ability as a systems administrator or on anything else worth being embarrassed about. Openly admitting that you have a problem is 90 percent of the battle, as long as you can demonstrate that you have a plan to remedy the situation.

Step 8: Implement the recovery plan. You know your systems and networks better than anyone. Follow your plan and your instincts. Speak with a colleague at a similar institution (preferably one who knows you well) to keep yourself on the right track.

Step 9: Report the incident to authorities. If the incident involved outside parties, you should report the matter to CERT. They can be reached by fax at (412) 268-6989 or by e-mail at cert@cert.org. Provide as much information as you can.

A standard form is available from www.cert.org to help jog your memory. Here are some of the more useful pieces of information you might provide.

The names, hardware types and OS versions of the compromised machines.

The list of patches that had been applied at the time of the incident.

A list of accounts that are known to have been compromised.

The names and IP addresses of any remote hosts that were involved.

Contact information, if you know it, for the administrators of remote sites.

Relevant log entries or audit information.

If you believe that a previously undocumented software problem may have been involved, you should report the incident to your vendor as well.

No matter what your security situation, don't let your organization be surprised that being a "player" in the networked world today requires constant investment in and continuous innovation of your security policies, staff and infrastructure. After all, there's no free lunch in this arena.

[selected excerpts from UNIX System Administration Handbook Copyright 2000, Prentice Hall PTR. Used with permission.]