Assembling Your Incident Response Team
Creating an incident response team means assembling a group of individuals to work and train together. Many organizations do not have the luxury of a dedicated incident response team, and use people who have another primary job function. It is critical that, as with any functional team, the IRT team needs to practice its tradecraft, improve its skills, and understand how to work together. If your team is not dedicated, it is recommended to at least set aside dedicated time on a regular schedule for the team members to get together and work on mock scenarios. The incident response team should include forensic investigators, corporate communications and public relations teams, network and system administrators, and legal representatives so that everybody is aware of who the IRT members are and how to work with them. You don’t want to be doing introductions while under the pressure of a major cyber incident.
When you are choosing members of the incident response team, they should include individuals who have experience in how technical systems can be breached and those that understand how to configure and manage such systems. Breach experience could be associated with a job title like penetration tester while managing a system could be a job title like network engineer. Additionally, the incident response team needs managerial, leadership, marketing, and legal representatives to accurately gauge the situation and make appropriate calls that may be financially or politically sensitive. If you are wondering why marketing personnel would be involved, the short answer is that they are probably better able to spin a situation in the most favorable manner for a negative situation like a cyberbreach. What you don’t want is an upset analyst complaining about the situation to the local news.
Here is a list of roles that should be included in a incident response program:
C-level sponsor
Technical members who have knowledge of virtual, physical, and cloud technology
Managers who are authorized to make decisions that impact changes to technology and services
Legal representatives who can advise on the impact of a manager’s decisions
Marketing personnel representing a public-facing message
When to Engage the Incident Response Team
The first sign of trouble will likely be determined by people who are not on your incident response team. Many breaches are reported by customers, outside organizations, or general employees. Brian Krebs, who runs the extremely popular and well-respected website Krebs on Security (https://krebsonsecurity.com/), has reported on many data breaches on his site. It is rumored that some organizations first learn they are victims of data breaches because they have read the news on Brian’s website. This has led to the slogan “Brian Krebs is my IDS.”
This story unfortunately highlights how many organizations cannot accurately determine whether attackers have infiltrated their networks. Organizations need to understand when and why they should engage their incident response team. Engaging the team too soon can be costly for the organization and burn out team members. Engaging a team too late could mean attackers may have a chance to fully compromise a network and hide their tracks. There should be rules or triggers that require your organization to engage in incident response as well as a simple method for people to report an incident. Our recommendation is to centralize the method of contact to one email or phone number because people involved in an incident will likely be panicking and need to quickly find the IR resource. This is why the United States makes its emergency number simple to remember: just dial 911. We talk more about contact lists shortly.
The OODA loop provides a good reference process to understand how to quickly react to rapidly unfolding cybersecurity events. It is often used in cybersecurity in areas of threat intelligence. OODA, which stands for observe, orient, decide, and act, was developed by military strategist and United States Air Force Colonel John Boyd (https://en.wikipedia.org/wiki/OODA_loop). The process can be implemented as a workflow and is still used today by many incident response specialists when dealing with various types of crisis situations, including nontechnical ones. Figure 4-1 shows the OODA loop.
)
Figure 4-1 OODA Loop
From an incident response standpoint, OODA can be used as starting point for teams when they are investigating a situation. In Figure 4-2, we applied typical actions that an incident response team may take and the relationship of those actions in regard to the OODA loop.
)
Figure 4-2 OODA Loop with Cyber Correlations
When an IRT team initially responds to an incident, they should observe and understand the security event. We mentioned some of these steps earlier in the chapter in relation to initial assessment and interview guidelines.
As a technical member of the IRT team, a network engineer should pay close attention to the security events that are observable. This includes evaluating logs, SIEMs, and device alerts. Another technique we recommend when you are observing the environment is to note the applications that are being used and research any vulnerabilities or security notices for those applications. Most security logs do not show how attackers exploit them to compromise the network. As a network engineer and forensics specialist, you need to look beyond what the logs tell you and conduct additional, and sometimes manual, research. There are many ways to search for possible exploits. Normally, a Google search with the application and exact version number can reveal them. Additionally, sites such as https://cve.mitre.org/ are good places to look for common vulnerabilities and exposure. We also recommend searching the Exploit Database at www.exploit-db.com/ or even PasteBin at https://pastebin.com/. Figure 4-3 shows the search option within the exploit-db website. When you use these search engines, don’t just limit your searches to applications; try emails, IPs, ASNs, and other information belonging to the organization to see whether there are any data leaks.
)
Figure 4-3 Exploit-db.com Search Example
Let’s look at the typical actions conducted by attackers and how those actions correspond to the OODA model. In Figure 4-4, some actions of cyber attackers have been added into the OODA loop. These actions are loosely based on the Lockheed Martin Cyber Kill Chain Model and describe actions taken by attackers during a cyberattack. You can see the attacker’s potential techniques correlate nicely with opportunities that IRT members can use to investigate and respond to different aspects of an attack. Not every attack will follow a structured OODA loop, but this is a way to model a generalized attack process with how an IRT should operate.
)
Figure 4-4 Revisiting the OODA Loop
Outstanding Items that Often Get Missed with Incident Response
We have already given you a few reference frameworks you should consider when building an incident response program. A few outstanding items often get overlooked and you should be aware of them before we get into how a team will respond to an incident. These items may sound simple, and perhaps obvious, but the lack of planning in organizations around these items can become a major problem. First, let’s look at how the team is engaged.
Phone Tree and Contact List
Years ago, when I was first starting off my career, I was working at an IT helpdesk for a major global organization answering questions around VPN issues, resetting passwords, and performing other IT-related tasks. I normally worked a 2 a.m. to 10 a.m. shift, so I could continue taking classes at the university. One by-product of working an odd shift was that the IT helpdesk was one of the few phone numbers that was answered 24 hours a day. We did, however, get all types of unexpected calls that were really outside our expertise. One early morning, I got a call from the company operator. She didn’t know what other number to dial and remembered the IT helpdesk was managed 24 hours per day. One of the top executives of this corporation was a passenger on a hijacked commercial airliner, and the criminals were demanding a ransom for the life of the executive. Remember, when people are in a crisis, they will likely not be able to recall complicated numbers or processes. You should consider making your method of contact as simple as possible.
Often customers roll their eyes at me when I discuss creating a solid phone tree as part of an incident response plan. Hopefully, you will never be in a similar situation to the one I was in. In that situation, our IT team’s simple contact method was the only thing the executive could remember. I was probably not the best person to help the executive on the hijacked plane. Anybody would have been better than the late-night desktop support person. Then again, IT is expected to be able to fix anything. By the way, the executive as well as most of the passengers were fine.
A good incident response and communication plan should have a readily available contact list. This is sometimes done as a phone tree. A phone tree is simply a list of people who need to be notified when a breach occurs. The list may include people outside your organization such as third-party interests and contractors. The list normally specifies who is optional and who is required to contact. It should have multiple contact numbers, emails, or other methods of contacting the individuals because important people tend to be busy and mobile. Because criminals are never mindful of our schedules, most lists should include what to do if a person cannot be reached, how often to retry, and what the backup plan is. It is important to remember that contacting people can take valuable time away from your experts investigating the breach. This is why it is not uncommon for organizations to outsource the call list to a service company.
One final contact concept is feedback as the incident takes place. Leadership and stakeholders will likely want to be updated continuously on the status of an incident as it is being resolved. The incident response preparedness plan should include instructions on how the team can provide feedback and updates to company and data stakeholders. If this step is not defined, it is likely the incident response team will be overrun by update requests. If the plan includes a feedback loop, everyone will understand how updates are received and delivered, thus cutting down on the individual updates the incident response team must do, enabling them to concentrate on investigating the incident.
Facilities
Similar to phone trees, small details surrounding facilities can often be overlooked. It can be surprising to walk into a facility and realize that you need to get to a camera mounted from the ceiling to analyze its hard drives, or you need to unmount an access point installed on a brick wall. What happens when investigators arrive in the middle of the night at an unfamiliar location? Will they be able to get access to the data center or be stuck in the parking lot while the business is destroyed by a cyber incident? Cyber forensic investigators may need access to areas and systems that are not normally available to unauthorized individuals or are hard to reach. Long delays in physical access could make the difference between preserving evidence or it being destroyed. Organizations need to prepare for quickly granting access to such areas when required. Information regarding how to access areas such as the data center, remote backup locations, technology located in the ceiling (wireless access points, for example), and so on should be included in the incident response plan.
Another question to ask is when all team members are required to be at the same physical location or if members can work remotely over secure channels such as VPN. If they are working remotely, how will information be shared without exposing results to the risk of contamination? Are the facilities able to accommodate the entire team if called in for an emergency? Some organizations make sure their facilities have beds and showers, along with caterers and food service available 24/7 to ensure their teams have everything in place to investigate complicated incidents. In most cases, you do not have to go to this extreme, but it may be wise to prepare for the worst-case situation or at least have something documented.