IPSec Overview Part One: General IPSec Standards

Virtual Private Networks (VPNs) are becoming required expertise for network and security engineers, and IPSec is the most commonly used protocol when implementing VPNs. In this first article of a five-part series on the Cisco implementation of IPSec, Andrew Mason delves into the components that make up the IPSec protocol suite.

Internet Protocol Security (generally shortened to IPSec) is a framework of open standards that provides data confidentiality, data integrity, and data authentication between participating peers at the IP layer. IPSec can be used to protect one or more data flows between IPSec peers. Documented in a series of Internet RFCs, the overall IPSec implementation is guided by "Security Architecture for the Internet Protocol," RFC 2401. IPSec consists of two main protocols:

• Authentication Header (AH)
• Encapsulating Security Payload (ESP)

IPSec also uses other existing encryption standards to make up a protocol suite.

IPSec includes several standards that are supported by Cisco IOS and the PIX Firewall:

• IP Security Protocol
• Authentication Header (AH)
• Encapsulating Security Payload (ESP)
• DES Algorithm
• Triple DES Algorithm (2DES)
• Diffie-Hellman (D-H)
• Message Digest 5 (MD5)
• Secure Hash Algorithm-1 (SHA-1)
• Rivest, Shamir, and Adelman Signatures (RSA)
• Internet Key Exchange (IKE)
• Certificate authorities (CA)

These standards are described briefly in the following sections.

IP Security Protocol

The IP Security Protocol consists of the Authentication Header (AH) and the Encapsulating Security Payload (ESP).

Authentication Header (AH)

Authentication Header (AH) provides authentication and integrity to the datagrams passed between two systems. This is achieved by applying a keyed one-way hash function to the datagram to create a message digest. If any part of the datagram is changed during transit, this will be detected by the receiver when it performs the same one-way hash function on the datagram and compares the value of the message digest that the sender has supplied. The fact that the one-way hash also involves the use of a secret shared between the two systems means that authenticity can be guaranteed.

AH may also enforce anti-replay protection by requiring that a receiving host set the replay bit in the header to indicate that the packet has been seen. Without it, an attacker may be able to resend the same packet many times; for example, sending a packet that withdraws $100 from account X. Figure 1 shows two routers and confirms that the data between them is sent in cleartext.

Figure 1 Authentication Header (AH).

The AH function is applied to the entire datagram except for any mutable IP header fields that change in transit, such as Time To Live (TTL) fields that are modified by the routers along the transmission path. AH works as follows:

1. The IP header and data payload is hashed.

2. The hash is used to build a new AH header, which is appended to the original packet.

3. The new packet is transmitted to the IPSec peer router.

4. The peer router hashes the IP header and data payload, extracts the transmitted hash from the AH header, and compares the two hashes. The hashes must match exactly. If even one bit is changed in the transmitted packet, the hash output on the received packet will change and the AH header will not match.

This process can be seen in Figure 2.

Figure 2 AH authentication and integrity.

Encapsulating Security Payload (ESP)

Encapsulating Security Payload (ESP) is a security protocol used to provide confidentiality (encryption), data origin authentication, integrity, optional anti-replay service, and limited traffic-flow confidentiality by defeating traffic-flow analysis. Figure 3 shows that the data payload is encrypted with ESP.

Figure 3 Encapsulating Security Payload (ESP).

ESP provides confidentiality by performing encryption at the IP packet layer. It supports a variety of symmetric encryption algorithms. The default algorithm for IPSec is 56-bit DES. This cipher must be implemented to guarantee interoperability among IPSec products. Cisco products also support use of 3DES for strong encryption. Confidentiality may be selected independent of all other services.

DES Algorithm

DES is used to encrypt and decrypt packet data; it turns cleartext into ciphertext via an encryption algorithm. The decryption algorithm on the remote end restores cleartext from ciphertext. Shared secret keys enable the encryption and decryption. DES uses a 56-bit key, ensuring high-performance encryption.

Triple DES Algorithm (3DES)

Triple DES (or 3DES) is also a supported encryption protocol for use in IPSec on Cisco products. The 3DES algorithm is a variant of the 56-bit DES. 3DES operates similarly to DES in that data is broken into 64-bit blocks. 3DES then processes each block three times, each time with an independent 56-bit key. 3DES effectively doubles encryption strength over 56-bit DES.

Diffie-Hellman (D-H)

Diffie-Hellman (D-H) is a public-key cryptography protocol. It allows two parties to establish a shared secret key used by encryption algorithms (DES or MD5, for example) over an insecure communications channel. D-H is used within IKE (described later in this article) to establish session keys. 768-bit and 1024-bit D-H groups are supported in the Cisco routers and PIX Firewall. The 1024-bit group is more secure.

Message Digest 5 (MD5)

Message Digest 5 (MD5) is a hash algorithm used to authenticate packet data. Cisco routers and the PIX Firewall use the MD5 hashed message authentication code (HMAC) variant that provides an additional level of hashing. A hash is a one-way encryption algorithm that takes an input message of arbitrary length and produces a fixed-length output message. IKE, AH, and ESP can use MD5 for authentication.

Secure Hash Algorithm 1 (SHA-1)

Secure Hash Algorithm 1 (SHA-1) is a hash algorithm used to authenticate packet data. Cisco routers and the PIX Firewall use the SHA-1 HMAC variant, which provides an additional level of hashing. IKE, AH, and ESP can use SHA-1 for authentication.

Rivest, Shamir, and Adelman Signatures (RSA)

RSA is a public-key cryptographic system used for authentication. IKE on the Cisco router or PIX Firewall uses a D-H exchange to determine secret keys on each IPSec peer used by encryption algorithms. The D-H exchange can be authenticated with RSA signatures or pre-shared keys.

Internet Key Exchange (IKE)

IKE is a hybrid protocol that provides utility services for IPSec: authentication of the IPSec peers, negotiation of IKE and IPSec security associations, and establishment of keys for encryption algorithms used by IPSec.

Certificate Authorities (CA)

The certificate authority (CA) support of Cisco routers and the PIX Firewall allows the IPSec-protected network to scale by providing the equivalent of a digital identification card to each device. When two IPSec peers want to communicate, they exchange digital certificates to prove their identities (thus removing the need to manually exchange public keys with each peer or to manually specify a shared key at each peer). The digital certificates are obtained from a certificate authority. CA support on Cisco products uses RSA signatures to authenticate the CA exchange.

This brings us to the end of the first part of this five-part series of articles covering IPSec. Be sure to catch the next installment.