NetFlow for Cybersecurity

NetFlow for Cybersecurity and Incident Response

NetFlow is a tremendous security tool. It provides anomaly detection and investigative capabilities that can be helpful in incident response. The Cisco Cyber Threat Defense (CTD) solution uses NetFlow as the primary security visibility tool. Complete visibility is one of the key requirements when identifying and classifying security threats.

The first step in the process of preparing your network and staff to successfully identify security threats is achieving complete network visibility. You cannot protect against or mitigate what you cannot view/detect. You can achieve this level of network visibility through existing features on network devices you already have and on devices whose potential you do not even realize. In addition, you should create strategic network diagrams to clearly illustrate your packet flows as well as where, within the network, you may enable security mechanisms to identify, classify, and mitigate the threat. Remember that network security is a constant war. When defending against the enemy, you must know your own territory and implement defense mechanisms in place. Your goal should be to eliminate any network blind spots.

NetFlow as an Anomaly Detection Tool

You can use NetFlow as an anomaly detection tool. Anomaly-based analysis keeps track of network traffic that diverges from “normal” behavioral patterns. You must define what is considered to be normal behavior. You can use anomaly-based detection to mitigate DDoS attacks and zero-day outbreaks. DDoS attacks are often used maliciously to consume the resources of your hosts and network that would otherwise be used to serve legitimate users. The goal with these types of attacks is to overwhelm the victim network resources, or a system’s resources such as CPU and memory. In most cases, this is done by sending numerous IP packets or forged requests.

Particularly dangerous is when an attacker builds up a more powerful attack with a more sophisticated and effective method of compromising multiple hosts and installing small attack daemons. This is what many call zombies, or bot hosts/nets. Subsequently, an attacker can launch a coordinated attack from thousands of zombies onto a single victim. This daemon typically contains both the code for sourcing a variety of attacks and some basic communications infrastructure to allow for remote control.

Typically, an anomaly-detection system monitors network traffic and alerts and then reacts to any sudden increase in traffic and any other anomalies.

NetFlow, along with other mechanisms such as syslog and SNMP, can be enabled within your infrastructure to provide the necessary data used for identifying and classifying threats and anomalies. Before implementing these anomaly-detection capabilities, you should perform traffic analysis to gain an understanding of general traffic rates and patterns. In anomaly detection, learning is generally performed over a significant interval, including both the peaks and valleys of network activity.

Incident Response and Network Security Forensics

NetFlow is often compared to a phone bill. When police want to investigate criminals, for instance, they often collect and investigate their phone records. NetFlow provides information about all network activity that can be very useful for incident response and network forensics. This information can help you discover indicators of compromise (IOC).

The following six-step methodology on security incident handling has been adopted by many organizations, including service providers, enterprises, and government organizations:

• Step 1. Preparation

• Step 2. Identification

• Step 3. Containment

• Step 4. Eradication

• Step 5. Recovery

• Step 6. Lessons learned

NetFlow plays a crucial role in the preparation and identification phases. Information collected in NetFlow records can be used as part of identifying, categorizing, and scoping suspected incidents as part of the identification. NetFlow data also provides great benefits for attack traceback and attribution. In addition, NetFlow provides visibility into what is getting into your network and what information is being exfiltrated out of your network.

Figure 4-11 shows an example of how a botnet is performing a DDoS attack against the corporate network, while at the same time communicating with an internal host in the call center. NetFlow in this case can be used as an anomaly-detection tool for the DDoS attack and also as a forensics tool to potentially find other IOCs of more sophisticated attacks that may be carried out incognito.

Figure 4-11 Detecting What Is Getting Into Your Network

Figure 4-12 shows how a “stepping-stone” attack is carried out in the corporate network. A compromised host in the engineering department is extraditing large amounts of sensitive data to an attacker in the Internet from a server in the data center.

Figure 4-12 Detecting What Is Getting Out of Your Network

You can also use NetFlow in combination with DNS records to help you detect suspicious and malicious traffic, such as the following:

• Suspicious requests to .gov, .mil, and .edu sites when you do not even do business with any of those entities

• Large amounts of traffic leaving the organization late at night to suspicious sites

• Traffic to embargoed countries that should not have any business partners or transactions

• Suspicious virtual private network (VPN) requests and VPN traffic

• Requests and transactions to sites without any content

• Pornography sites or any other sites that violate corporate policy

• Illegal file-sharing sites

Syslog and packet captures are also often used in network forensics; however, an area where these traditional network forensics tools fall short is in coverage. For instance, it is very difficult to deploy hundreds of sniffers (packet-capture devices) in the network of large organizations. In addition, the cost will be extremely high. When a security incident or breach is detected, the incident responders need answers fast! They do not have time to go over terabytes of packet captures, and they can definitely not analyze every computer on the network to find the root cause, miscreant, and source of the breach. You can use NetFlow to obtain a high-level view of what is happening in the network, and then the incident responder can perform a deep-dive investigation with packet captures and other tools later in the investigation. Sniffers then can be deployed as needed in key locations where suspicious activity is suspected. The beauty of NetFlow is that you can deploy it anywhere you have a supported router, switch, or Cisco ASA; alternatively, you can use Cisco NetFlow Generation Appliance (NGA). After the Lancope acquisition, Cisco also sells the StealthWatch FlowSensor product, which is a physical or virtual appliance that can generate NetFlow data when legacy Cisco network infrastructure components are not capable of producing line-rate, unsampled NetFlow data.

NetFlow can fill in some of the gaps and challenges regarding the collection of packet captures everywhere in the network. It is easier to store large amounts of NetFlow data because it is only a transactional record. Therefore, administrators can keep a longer history of events that occurred on their networks. Historical records can prove very valuable when investigating a breach. Network transactions can show you where an initial infection came from, what command-and-control channel was initiated by the malware, what other computers on the internal network were accessed by that infected host, and whether other hosts in the network reached out to the same attacker or command-and-control system.

The logging facility on Cisco IOS routers, switches, Cisco ASA, and other infrastructure devices allows you to save syslog messages locally or to a remote host. By default, routers send logging messages to a logging process. The logging process controls the delivery of logging messages to various destinations, such as the logging buffer, terminal lines, a syslog server, or a monitoring event correlation system such as Cisco Prime Infrastructure or Splunk. You can set the severity level of the messages to control the type of messages displayed, in addition to a timestamp to successfully track the reported information. Every security professional and incident responder knows how important it is to have good logs. There is no better way to find out what was happening in a router, switch, or firewall at the time an attack occurred. However, like all things, syslog has limitations. You have to enable the collection of logs from each endpoint; so in many environments, syslog coverage is incomplete, and after a computer has been compromised, it is not possible to trust the logs coming from that device anymore. Syslog is extremely important, but it cannot tell you everything. Many network telemetry sources can also be correlated with NetFlow while responding to security incidents and performing network forensics, including the following:

• Dynamic Host Configuration Protocol (DHCP) logs

• VPN logs

• Network address translation (NAT) information

• 802.1x authentication logs

• Server logs (syslog)

• Web proxy logs

• Spam filters from email security appliances such as the Cisco Email Security Appliance (ESA)

• DNS server logs

• Active directory logs

Table 4-24 lists the different event types, their source, and their respective events that can be combined with NetFlow while you’re responding to security incidents and performing network forensics.

Table 4-24 Network Telemetry Type, Sources, and Events

Network forensics can be an intimidating topic for many security professionals. Everyone knows that forensic investigation may entail many other sources of information from end hosts, servers, and any affected systems. Each forensics team needs to have awareness of many different areas, such as the following:

• Having thorough knowledge of assets, risks, impact, and likelihood of events.

• Practicing incident response policies and procedures in mock events and collecting NetFlow on a regular basis to analyze what is happening in the network.

• Awareness of current vulnerabilities and threats.

• Understanding evidence handling and chain of custody. (Even NetFlow events can be used as evidence.)

• Enacting mitigation based on evidence collected.

• Knowing the documentation requirements for evidence, depending on your country and local laws.

• Understanding the analysis process during and after the incident.

• Having a framework for communications, both within the team and external to the team.

Using NetFlow for Data Leak Detection and Prevention

Many network administrators, security professionals, and business leaders struggle in the effort to prevent data loss within their organizations. The ability to identify anomalous behavior in data flows is crucial to detect and prevent data loss. The application of analytics to data collected via NetFlow can aid security professionals in detecting anomalous amounts of data leaving the organization and abnormal traffic patterns inside of the organization.

Using NetFlow along with identity management systems, you can detect who initiated the data transfer, the hosts (IP addresses) involved, the amount of data transferred, and the services used. In addition, you can measure how long the communications lasted as well as the frequency of the same connection attempts.

In the following scenario, a large retail organization is the victim of a major breach where attackers stole more than 100,000 credit card numbers. The retailer headquarters is in New York, NY and has two large secondary offices in Raleigh, North Carolina, and San Juan, Puerto Rico. This retailer also has more than 1000 stores in the United States and Canada. Figure 4-13 illustrates these offices and stores.

Figure 4-13 Retailer High-Level Network Topology

The breach was not detected for several months after the attackers had already penetrated the network. The retailer had firewalls and intrusion prevention devices, but those were not enough to detect or mitigate the attack. The attack was thought to be an inside job, because the malware that was extracting the credit card numbers was very sophisticated and tailored to such an organization. The breach was detected only because law enforcement contacted the victimized retailer, telling them that thousands of fraudulent credit card transactions had been detected on credit cards that were last legitimately used at their stores.

After the organization started their incident response and forensics investigation, they decided to deploy NetFlow in routers at the edge of the data center. The topology in Figure 4-14 illustrates the network at the New York headquarters and the two routers that were configured with NetFlow.

Figure 4-14 New York Headquarters NetFlow Routers

The data center has numerous servers that are dedicated for credit card processing applications (software), as illustrated in Figure 4-15.

Figure 4-15 Credit Card Processing Servers

After deploying NetFlow in their data center edge routers, the retailer observed that numerous DNS requests were being sent from the credit card processing servers to DNS servers outside of the country (United States). The most interesting fact was that such DNS servers were in embargoed countries where the retailer previously had never transacted any business. In addition, most of these DNS requests were being sent during off-hours (mostly around 2:00 to 4:00 a.m. local time).

The retailer was able to inspect NetFlow traffic and detect the country where the credit card information was sent by using the MaxMind Geolocation database. MaxMind provides IP intelligence to thousands of companies to locate their Internet visitors and perform analytics. MaxMind has a service called minFraud, which helps businesses prevent fraudulent online transactions and reduce manual review.

The attackers were sending stolen credit card data over DNS using tunneling. DNS is a protocol that enables systems to resolve domain names (for example, example.com) into IP addresses (for example, 93.184.216.34). DNS is not intended for a command channel or even tunneling. However, attackers have developed software that enables tunneling over DNS. Because traditionally DNS it is not designed for data transfer, it is less inspected in terms of security monitoring. Undetected DNS tunneling (otherwise known as DNS exfiltration) represents a significant risk to any organization.

In this case, the credit cards were base64 encoded and sent over DNS requests (tunneling) to cybercriminals abroad. Attackers nowadays use different DNS record types and encoding methods to exfiltrate data from victims’ systems and networks. The following are some examples of encoding methods:

• Base64 encoding

• Binary (8-bit) encoding

• NetBIOS encoding

• Hex encoding

Several utilities have been created to perform DNS tunneling (for the good and also the bad). The following are a few examples:

• DeNiSe: A Python tool for tunneling TCP over DNS.

• dns2tcp: Written by Olivier Dembour and Nicolas Collignon in C, dns2tcp supports KEY and TXT request types.

• DNScapy: Created by Pierre Bienaime, this Python-based Scapy tool for packet generation even supports SSH tunneling over DNS, including a SOCKS proxy.

• DNScat or DNScat-P: This Java-based tool created by Tadeusz Pietraszek supports bidirectional communication through DNS.

• DNScat (DNScat-B): Written by Ron Bowes, this tool runs on Linux, Mac OS X, and Windows. DNScat encodes DNS requests in NetBIOS or hex encoding.

• Heyoka: This tool, written in C, supports bidirectional tunneling for data exfiltration.

• Iodine: Written by Bjorn Andersson and Erik Ekman in C, Iodine runs on Linux, Mac OS X, and Windows, and can even be ported to Android.

• Nameserver Transfer Protocol (NSTX): Creates IP tunnels using DNS.

• OzymanDNS: Written in Perl by Dan Kaminsky, this tool is used to set up an SSH tunnel over DNS or for file transfer. The requests are base32 encoded, and responses are base64-encoded TXT records.

• psudp: Developed by Kenton Born, this tool injects data into existing DNS requests by modifying the IP/UDP lengths.

• Feederbot and Moto: This malware using DNS has been used by attackers to steal sensitive information from many organizations.

The retailer’s network security personnel were able to perform detailed analysis of the techniques used by the attackers to steal this information and discovered the types of malware and vulnerabilities being exploited in systems in the data center. Network telemetry tools such as NetFlow are invaluable when trying to understand what is happening (good and bad) in the network, and it is a crucial tool for incident response and network forensics.

Retailers or any organizations that process credit cards or electronic payments are often under regulation from the Payment Card Industry Data Security Standard (PCI DSS). PCI DSS was created to encourage and maintain cardholder data security and expedite the consistent use of data security methodologies. This standard enforces a baseline of technical and operational requirements. PCI DSS applies to the following:

• Banks

• “Brick-and-mortar” retailers and online retailers

• Merchants

• Processors

• Acquirers

• Issuers

• Service providers or any other organizations that store, process, or transmit cardholder data (CHD) and/or sensitive authentication data (SAD)

The PCI DSS defines general requirements for the following:

• Building and maintaining secure networks

• Protecting cardholder data

• Enforcing vulnerability management and patching programs

• Implementing adequate access control measures

• Consistently monitoring networks and their systems

• Guaranteeing the maintenance of information security policies

As you can see from this list, adequate monitoring of systems is an underlying and fundamental requirement. NetFlow, intrusion prevention systems, and others are often used to maintain this required visibility into what is happening in the network. Additional details about PCI DSS are covered in Chapter 7, “Compliance Frameworks.”

Many threat actors are seeking to steal intellectual property from many organizations and individuals. According to the Merriam-Webster dictionary, intellectual property is “something (such as an idea, invention, or process) that derives from the work of the mind or intellect; an application, right, or registration relating to this.” Intellectual property (and other forms of expression) is often protected by patent, trademark, and copyright (in addition to state and federal laws). In today’s world, espionage (both cyber and corporate) is a huge business. Many types of attackers (for example, corporations, cybercriminals, and nation states) are after information with independent economic value, such as the following:

• Blueprints

• Chemical formulas

• Research and development documents

• Marketing strategies

• Manufacturing processes

• Source code

• Songs

• Books

• Documentation guides

In 1996, to maintain the health and competitiveness of the U.S. economy, the United States Congress passed the Economic Espionage Act to protect trade secrets from bad actors.