Home > Articles > Cisco Network Technology > General Networking > Network Security and Virtual Private Network Technologies

Network Security and Virtual Private Network Technologies

  • Sample Chapter is provided courtesy of Cisco Press.
  • Date: Oct 1, 2004.

Chapter Description

Network security can be a headache, especially when VPN technology is added to the mix. This chapter from the CCSP Self-Study text will help prepare you to deal with the complex security issues presented by virtual vrivate networks.


IPSec acts at the network layer, protecting and authenticating IP packets between participating IPSec devices (peers), such as PIX Firewalls, Cisco routers, Cisco VPN 3000 Concentrators, Cisco VPN Clients, and other IPSec-compliant products. IPSec is not bound to any specific encryption or authentication algorithms, keying technology, or security algorithms. IPSec is a framework of open standards. Because it isn't bound to specific algor-ithms, IPSec allows newer and better algorithms to be implemented without patching the existing IPSec standards. IPSec provides data confidentiality, data integrity, and data origin authentication between participating peers at the IP layer. IPSec is used to secure a path between a pair of gateways, a pair of hosts, or a gateway and a host.

IPSec spells out the rules for secure communications. IPSec, in turn, relies on existing algorithms to implement the encryption, authentication, and key exchange.

Some of the standard algorithms are as follows:

  • Data Encryption Standard (DES) algorithm—Used to encrypt and decrypt packet data.

  • 3DES algorithm—Effectively doubles encryption strength over 56-bit DES.

  • Advanced Encryption Standard (AES)—A newer cipher algorithm designed to replace DES. Has a variable key length between 128 and 256 bits. Cisco is the first industry vendor to implement AES on all its VPN-capable platforms.

  • Message Digest 5 (MD5) algorithm—Used to authenticate packet data.

  • Secure Hash Algorithm 1 (SHA-1)—Used to authenticate packet data.

  • Diffie-Hellman (DH)—A public-key cryptography protocol that allows two parties to establish a shared secret key used by encryption and hash algorithms (for example, DES and MD5) over an insecure communications channel.

Figure 1-6 shows four IPSec framework squares to be filled. When configuring security services to be provided by an IPSec gateway, you first must choose an IPSec protocol. The choices are ESP or ESP with AH. The second square is an encryption algorithm. Choose the encryption algorithm appropriate for the level of security desired: DES or 3DES. The third square is authentication. Choose an authentication algorithm to provide data integrity: MD5 or SHA. The last square is the DH algorithm group. Choose which group to use: DH1, DH2, or DH5. IPSec provides the framework, and the administrator chooses the algorithms used to implement the security services within that framework.

Figure 6Figure 1-6 IPSec Protocol Framework


IPSec security services provide four critical functions:

  • Confidentiality (encryption)—The sender can encrypt the packets before transmitting them across a network. By doing so, no one can eavesdrop on the communication. If intercepted, the communications cannot be read.

  • Data integrity—The receiver can verify that the data was transmitted through the Internet without being changed or altered in any way.

  • Origin authentication—The receiver can authenticate the packet's source, guaranteeing and certifying the source of the information.

  • Anti-replay protection—Anti-replay protection verifies that each packet is unique, not duplicated. IPSec packets are protected by comparing the sequence number of the received packets and a sliding window on the destination host, or security gateway. A packet whose sequence number is before the sliding window is considered late, or a duplicate. Late and duplicate packets are dropped.


The good news is that the Internet is a public network. The bad news is that the Internet is a public network. Clear-text data transported over the public Internet can be intercepted and read. To keep the data private, it can be encrypted. Through digital scrambling, the data is rendered unreadable.

For encryption to work, both the sender and receiver need to know the rules used to transform the original message into its coded form. Rules are based on an algorithm and a key. An algorithm is a mathematical function that combines a message, text, characters, or all three with a string of characters called a key. The output is an unreadable cipher string. Decryption is extremely difficult or impossible without the correct key.

In Figure 1-7, someone wants to send a financial document across the Internet. At the local end, the document is combined with a key and is run through an encryption algorithm. The output is cipher text. The cipher text is then sent through the Internet. At the remote end, the message is recombined with a key and is sent back through the encryption algorithm. The output is the original financial document.

Figure 7Figure 1-7 Encryption


Two types of encryption keys exist: symmetric and asymmetric. With symmetric key encryption, each peer uses the same key to encrypt and decrypt the data. With asymmetric key encryption, the local end uses one key to encrypt the traffic, and the remote end uses another key to decrypt it. Both are discussed in further detail later in this chapter.

Encryption Algorithms

The degree of security depends on the key's length. If someone tries to hack the key through a brute-force attack, guessing every possible combination, the number of possibilities is a function of the key's length. The time to process all the possibilities is a function of the computer's computing power. Therefore, the shorter the key, the easier it is to break. A 64-bit key with a relatively sophisticated computer can take approximately 1 year to break. A 128-bit key with the same machine can take roughly 1019 years to decrypt.

Some of the encryption algorithms are as follows:

  • DES—DES was developed by IBM. It uses a 56-bit key, ensuring high-performance encryption. DES is a symmetric key algorithm.

  • 3DES—The 3DES algorithm is a variant of the 56-bit DES. 3DES operates similarly to DES, in that data is broken into 64-bit blocks. 3DES then processes each block three times, each time with an independent 56-bit key. 3DES effectively doubles encryption strength over 56-bit DES. 3DES is a symmetric key algorithm.

  • AES—The National Institute of Standards and Technology (NIST) recently adopted AES to replace the existing DES encryption in cryptographic devices. AES provides stronger security than DES and is computationally more efficient than 3DES. AES offers three different key strengths: 128-, 192-, and 256-bit keys. Cisco now supports VPN encryption from version 4.0 of the VPN Concentrator software and the addition of a SEP-E module. The older SEP modules do not perform hardware encryption on AES, only on DES and 3DES.

  • RSA—Rivest, Shamir, and Adelman (RSA) encryption, shown in Figure 1-8, uses asymmetric keys for encryption and decryption. Each end, local and remote, generates two encryption keys: a private key and a public key. It keeps its private key and exchanges its public key with people with whom it wants to communicate. To send an encrypted message to the remote end, the local end encrypts the message using the remote's public key and the RSA encryption algorithm. The result is an unreadable cipher text. This message is sent through the insecure network. The remote end uses its private key and the RSA algorithm to decrypt the cipher text. The result is the original message. The only one who can decrypt the message is the destination that owns the private key. With RSA encryption, the opposite also holds true. The remote end can encrypt a message using its own private key. The receiver can decrypt the message using the sender's public key. This RSA encryption technique is used for digital signatures.

Figure 8Figure 1-8 RSA Encryption


Key Exchange

DES, 3DES, AES, and also the two authentication algorithms, MD5 and SHA-1, all require a symmetric shared secret key to perform encryption and decryption. The question is, how do the encrypting and decrypting devices get the shared secret key?

The keys can be sent by e-mail, courier, overnight express, or public key exchange. The easiest method is DH public key exchange. The DH key agreement is a public key exchange method that provides a way for two peers to establish a shared secret key that only they know, although they are communicating over an insecure channel.

Public key cryptosystems rely on a two-key system: a public key, which is exchanged between end users, and a private key, which is kept secret by the original owners. The DH public key algorithm states that if user A and user B exchange public keys, and a calculation is performed on their individual private key and one another's public key, the end result of the process is an identical shared key. The shared key is used to derive encryption and authentication keys.

Variations of the DH key exchange algorithm are known as DH groups 1 through 7. DH groups 1, 2, and 5 support exponentiation over a prime modulus with a key size of 768 bits, 1024 bits, and 1536 bits, respectively. Cisco 3000 Clients support DH groups 1, 2, and 5. DES and 3DES encryption support DH groups 1 and 2. AES encryption supports DH groups 2 and 5. In addition to these, the Certicom movianVPN Client supports group 7. Group 7 supports Elliptical Curve Cryptography (ECC), which reduces the time needed to generate keys. During tunnel setup, VPN peers negotiate which DH group to use.

Security is not an issue with DH key exchange. Although someone might know a user's public key, the shared secret cannot be generated, because the private key never becomes public.

DH Key Exchange

DH key exchange is a public key exchange method that provides a way for two IPSec peers to establish a shared secret key that only they know, although they are communicating over an insecure channel.

With DH, each peer generates a public/private key pair. The private key generated by each peer is kept secret and never shared. The public key is calculated from the private key by each peer and is exchanged over the insecure channel. Each peer combines the other's public key with its own private key and computes the same shared secret number. The shared secret number is then converted into a shared secret key. The shared secret key is never exchanged over the insecure channel.

The following steps are used to implement the DH process (see Figure 1-9):

  1. Each peer generates a large prime integer, p and q. Each peer sends the other its prime integer over the insecure channel. For example, peer A sends p to peer B. Each peer then uses the p and q values to generate g, a primitive root of p.

  2. Each peer generates a private DH key (peer A: Xa, peer B: Xb).

  3. Each peer generates a public DH key. The local private key is combined with the prime number p and the primitive root g in each peer to generate a public key, Ya for peer A and Yb for peer B. The formula for peer A is Ya = g^Xa mod p. The formula for peer B is Yb = g^Xb mod p. The exponentiation is computationally expensive. The ^ character denotes exponentiation (g to the Xa power); mod denotes modulus.

  4. The public keys Ya and Yb are exchanged in public.

  5. Each peer generates a shared secret number (ZZ) by combining the public key received from the opposite peer with its own private key. The formula for peer A is ZZ = (YbXa) mod p. The formula for peer B is ZZ = (YaXb) mod p. The ZZ values are identical in each peer. Anyone who knows p or g, or the DH public keys, cannot guess or easily calculate the shared secret value—largely because of the difficulty in factoring large prime numbers.

  6. Shared secret number ZZ is used to derive the encryption and authentication symmetric keys.

Figure 9Figure 1-9 DH Key Exchange


Data Integrity

The next VPN-critical function is data integrity. VPN data is transported over some form of insecure network, such as the Internet. Potentially, this data could be intercepted and modified. To guard against this, each message has a hash attached to it. This is called a Hash-based Message Authentication Code (HMAC). A hash guarantees the integrity of the original message. If the transmitted hash matches the received hash, the message has not been tampered with. However, if there is no match, the message was altered.

In Figure 1-10, someone is trying to send Terry Smith a check for $100. At the remote end, Alex Jones is trying to cash the check for $1000. As the check progressed through the Internet, it was altered. Both the recipient and the dollar amount were changed. In this case, the hashes do not match, so the transaction is no longer valid.

Figure 10Figure 1-10 Data Integrity


Data integrity is commonly known and talked about as authentication. The packets are authenticated using the hash that is attached to each packet. Two main algorithms facilitate data integrity within the IPSec framework—MD5 and SHA-1.

HMAC guarantees the message's integrity. At the local end, the message and a shared secret key are sent through a hash algorithm, which produces a hash value. Basically, a hash algorithm is a formula used to convert a variable-length message into a single string of digits of a fixed length. It is a one-way algorithm. A message can produce a hash, but a hash cannot produce the original message. It is analogous to dropping a plate on the floor. The plate can produce a multitude of pieces, but the pieces cannot be recombined to reproduce the plate in its original form. The message and hash are sent over the network.

At the remote end, a two-step process occurs, as shown in Figure 1-11. First, the received message and shared secret key are sent through the hash algorithm, resulting in a recalculated hash value. Second, the receiver compares the recalculated hash with the hash that was attached to the message. If the original hash and recalculated hash match, the message's integrity is guaranteed. If any of the original message is changed while in transit, the hash values are different.

Figure 11Figure 1-11 HMAC


The two common HMAC algorithms are as follows:

  • HMAC-MD5—HMAC-MD5 uses a 128-bit shared secret key. The variable-length message and the 128-bit shared secret key are combined and run through the HMAC-MD5 hash algorithm. The output is a 128-bit hash. The hash is appended to the original message and forwarded to the remote end.

  • HMAC-SHA-1—HMAC-SHA-1 uses a 160-bit secret key. The variable-length message and the 160-bit shared secret key are combined and run through the HMAC-SHA-1 hash algorithm. The output is a 160-bit hash. The hash is appended to the original message and forwarded to the remote end.

HMAC-SHA-1 is considered cryptographically stronger than HMAC-MD5. HMAC-SHA-1 is recommended when its slightly superior security is important.

Origin Authentication

In the middle ages, a seal guaranteed the authenticity of an edict. In modern times, a signed document is notarized with a seal and a signature. In the electronic era, a document is signed using the sender's private encryption key—a digital signature. A signature is authenticated by decrypting the signature with the sender's public key.

In Figure 1-12, the local device derives a hash and encrypts it with its private key. The encrypted hash (digital signature) is attached to the message and is forwarded to the remote end. At the remote end, the encrypted hash is decrypted using the local end's public key. If the decrypted hash matches the recomputed hash, the signature is genuine. A digital signature ties a message to a sender. The sender is authenticated. It is used during the initial establishment of a VPN tunnel to authenticate both ends to the tunnel.

Figure 12Figure 1-12 Digital Signature


The two common digital signature algorithms are RSA and Directory System Agent (DSA). RSA is used commercially and is the most common. DSA is used by U.S. Government agencies and is not as common.

When conducting business long distance, it is necessary to know who is on the other end of the phone, e-mail, or fax. The same is true of VPNs. The device on the other end of the VPN tunnel must be authenticated before the communication path is considered secure.

The three peer authentication methods are as follows:

  • Preshared keys—A secret key value entered into each peer manually that is used to authenticate the peer.

  • RSA signatures—Uses the exchange of digital certificates to authenticate the peers.

  • RSA encrypted nonces—Nonces (random numbers generated by each peer) are encrypted and then exchanged between peers. The two nonces are used during the peer authentication process.

Preshared Keys

With preshared keys, the same preshared key is configured on each IPSec peer. At each end, the preshared key is combined with other information to form the authentication key. Starting at the local end, the authentication key and the identity information (device-specific information) are sent through a hash algorithm to form hash_I. The local Internet Key Exchange (IKE) peer provides one-way authentication by sending hash_I to the remote peer. If the remote peer can independently create the same hash, the local peer is authenticated, as shown in Figure 1-13.

Figure 13Figure 1-13 Preshared Keys


The authentication process continues in the opposite direction. The remote peer combines its identity information with the preshared-based authentication key and sends them through a hash algorithm to form hash_R. hash_R is sent to the local peer. If the local peer can independently create the same hash from its stored information and preshared-based authentication key, the remote peer is authenticated. Each peer must authenticate its opposite peer before the tunnel is considered secure. Preshared keys are easy to configure manually but do not scale well. Each IPSec peer must be configured with the preshared key of every other peer with which it communicates.

RSA Signatures

With RSA signatures (see Figure 1-14), hash_I and hash_R not only are authenticated but also are digitally signed. Starting at the local end, the authentication key and identity information (device-specific information) are sent through a hash algorithm to form hash_I. hash_I is then encrypted using the local peer's private encryption key. The result is a digital signature. The digital signature and a digital certificate are forwarded to the remote peer. (The public encryption key for decrypting the signature is included in the digital certificate exchanged between peers.)

Figure 14Figure 1-14 RSA Signatures


At the remote peer, local peer authentication is a two-step process. First, the remote peer verifies the digital signature by decrypting it using the public encryption key enclosed in the digital certificate. The result is hash_I. Next, the remote peer independently creates hash_I from stored information. If the calculated hash_I equals the decrypted hash_I, the local peer is authenticated (as shown in the figure). Digital signatures and certificates are discussed in more detail in Chapter 5, "Configuring the Cisco VPN 3000 for Remote Access Using Digital Certificates."

After the remote peer authenticates the local peer, the authentication process begins in the opposite direction. The remote peer combines its identity information with the authentication key and sends them through a hash algorithm to form hash_R. hash_R is encrypted using the remote peer's private encryption key—a digital signature. The digital signature and certificate are sent to the local peer. The local peer performs two tasks: It creates hash_R from stored information, and it decrypts the digital signature. If the calculated hash_R and the decrypted hash_R match, the remote peer is authenticated. Each peer must authenticate its opposite peer before the tunnel is considered secure.

RSA Encrypted Nonces

RSA encrypted nonces require that each party generate a nonce—a pseudorandom number. The nonces are then encrypted and exchanged. Upon receipt of the nonce, each end formulates an authentication key made up of the initiator and responder nonces, the DH key, and the initiator and responder cookies. The nonce-based authentication key is combined with device-specific information and run through a hash algorithm, where the output becomes hash_I. The local IKE peer provides one-way authentication by sending hash_I to the remote peer. If the remote peer can independently create the same hash from stored information and its nonce-based authentication key, the local peer is authenticated.

After the remote end authenticates the local peer, the authentication process begins in the opposite direction. The remote peer combines its identity information with the nonce-based authentication key and sends them through a hash algorithm to form hash_R. Hash_R is sent to the local peer. If the local peer can independently create the same hash from stored information and the nonce-based key, the remote peer is authenticated. Each peer must authenticate its opposite peer before the tunnel is considered secure.

Anti-Replay Protection

IPSec uses anti-replay mechanisms to ensure that IP packets cannot be intercepted by a third party or man in the middle and then be changed and reinserted into the data stream. This is implemented in IPSec by the Authentication Header (AH) protocol and the Encapsulating Security Payload (ESP) protocol. The anti-replay mechanism works by keeping track of the sequence number allocated to each packet as it arrives at the VPN endpoint. When a security association is established between two VPN endpoints, the sequence counter is set to 0. The packets that are encrypted and transmitted over the VPN are sequenced starting from 1. Each time a packet is sent, the receiver of the packet verifies that the sequence number is not that of a previously sent packet. If the receiver receives a packet with a duplicate sequence number, the packet is discarded, and an error message is sent back to the transmitting VPN endpoint to log this event.


AH implements anti-replay by default, although ESP does it only when data authentication is turned on (for example, MD5 or SHA-1) in the IPSec transform-set.

7. IPSec Protocol Framework | Next Section Previous Section

Cisco Press Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from Cisco Press and its family of brands. I can unsubscribe at any time.


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about Cisco Press products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information

To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.


Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites; develop new products and services; conduct educational research; and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.


If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@ciscopress.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information

Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.


This site is not directed to children under the age of 13.


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information

If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.


Users can always make an informed choice as to whether they should proceed with certain services offered by Cisco Press. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.ciscopress.com/u.aspx.

Sale of Personal Information

Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents

California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure

Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact

Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice

We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020