About IPSec VPN Negotiations

The devices at either end of an IPSec VPN tunnel are IPSec peers. To build the VPN tunnel, IPSec peers exchange a series of messages about encryption and authentication, and attempt to agree on many different parameters. This process is known as VPN negotiations. One device in the negotiation sequence is the initiator and the other device is the responder.

VPN negotiations happen in two distinct phases: Phase 1 and Phase 2.

Phase 1

The main purpose of Phase 1 is to set up a secure encrypted channel through which the two peers can negotiate Phase 2. When Phase 1 finishes successfully, the peers quickly move on to Phase 2 negotiations. If Phase 1 fails, the devices cannot begin Phase 2.

Phase 2

The purpose of Phase 2 negotiations is for the two peers to agree on a set of parameters that define what traffic can go through the VPN, and how to encrypt and authenticate the traffic. This agreement is called a Security Association.

The Phase 1 and Phase 2 configurations must match for the devices on either end of the tunnel.

Phase 1 Negotiations

In Phase 1 negotiations, the two VPN gateway devices exchange credentials. The devices identify each other and negotiate to find a common set of Phase 1 settings to use. When Phase 1 negotiations are completed, the two devices have a Phase 1 Security Association (SA). This SA is valid for a specified amount of time. If the two VPN gateways do not complete Phase 2 negotiations before the Phase 1 SA expires, then they must complete Phase 1 negotiations again.

The Phase 1 negotiation process depends on which version of IKE the gateway endpoints use. IKE authenticates IPSec peers and negotiates IKE SAs during this phase, setting up a secure communications channel for negotiating IPSec SAs in Phase 2.

Phase 1 negotiations include these steps:

  1. The devices agree on the IKE version to use (IKEv1 or IKEv2). Each device can use IKEv1 or IKEv2. The IKE version for both devices must match.
  2. The devices exchange credentials.

    The credentials can be a certificate or a pre-shared key. Both gateway endpoints must use the same credential method, and the credentials must match.

  3. The devices identify each other.

    Each device provides a Phase 1 identifier, which can be an IP address, domain name, domain information, or an X500 name. The VPN configuration on each device specifies the Phase 1 identifier of the local and the remote device. The configurations must match.

  4. For IKEv1, the VPN gateways decide whether to use Main Mode or Aggressive Mode for Phase 1 negotiations.

    The VPN gateway that starts the IKE negotiations sends either a Main Mode proposal or an Aggressive Mode proposal. The other VPN gateway can reject the proposal if it is not configured to use that mode.

  • Main Mode ensures the identity of both VPN gateways, but can be used only if both devices have a static IP address. Main Mode validates the IP address and gateway ID.
  • Aggressive Mode is faster but less secure than Main Mode because it requires fewer exchanges between two VPN gateways. In Aggressive Mode, the exchange relies mainly on the ID types used in the exchange by both VPN gateways. Aggressive Mode does not ensure the identity of the VPN gateway. The IKEv1 Aggressive Mode vulnerability described in CVE-2002-1623 means that Aggressive Mode is less secure than Main Mode unless you configure a certificate.
  1. The VPN gateways agree on Phase 1 parameters.
  • Whether to use NAT traversal
  • Whether to use IKE Keep-Alive (between Fireboxes only)
  • Whether to use Dead Peer Detection (RFC 3706)

IKE Keep-Alive is an obsolete setting. We recommend DPD instead.

For IKEv2, NAT Traversal and DPD are always enabled, and IKE Keep-Alive is not supported.

  1. The VPN gateways agree on Phase 1 Transform settings. The settings in the Phase 1 transform on each IPSec device must exactly match, or IKE negotiations fail.

The items you can set in the Phase 1 transform are:

  • Authentication — The type of authentication (SHA-2, SHA-1, or MD5)
  • Encryption — The type of encryption algorithm (DES, 3DES, or AES) and key length
  • SA Life — The amount of time until the Phase 1 Security Association expires
  • Key Group — The Diffie-Hellman key group

Phase 2 Negotiations

After the two IPSec VPN gateways successfully complete Phase 1 negotiations, Phase 2 negotiations begin. The purpose of Phase 2 negotiations is to establish the Phase 2 SA (sometimes called the IPSec SA). The IPSec SA is a set of traffic specifications that tell the device what traffic to send over the VPN, and how to encrypt and authenticate that traffic.

Phase 2 negotiations include these steps:

  1. The VPN gateways use the Phase 1 SA to secure Phase 2 negotiations. The VPN gateways agree on whether to use Perfect Forward Secrecy (PFS).

    VPN encryption keys are changed at the interval specified by the Force Key Expiration setting. The interval is eight hours by default. To prevent SAs from using Phase 1 keys for Phase 2, PFS forces the DH calculation to happen a second time. This means that Phase 1 and Phase 2 always have different keys, which is harder to break unless you select a DH group lower than 14.
    We recommend that you use PFS to keep your data secure. If you want to use PFS, it must be enabled on both VPN gateways, and both gateways must use the same Diffie-Hellman key groups.

  1. The VPN gateways agree on a Phase 2 proposal.

    The Phase 2 proposal includes the algorithm to use to authenticate data, the algorithm to use to encrypt data, and how often to make new Phase 2 encryption keys.

    The items you can set in a Phase 2 proposal include:

  • Type — For a manual BOVPN, you can select the type of protocol to use: Authentication Header (AH) or Encapsulating Security Payload (ESP). Both AH and ESP encrypt the data and protect against spoofing and packet manipulation (replay detection). We recommend that you use ESP, because you can protect against spoofing in other ways. Managed BOVPNs, Mobile VPN with IKEv2, Mobile VPN with IPSec, and Mobile VPN with L2TP always use ESP.

The IPSec authentication process checks the sequence of encrypted packets to prevent replay attacks. The anti-replay window size for VPN connections is fixed to 32 packets and cannot be modified. In Fireware v12.2 or higher, you can disable anti-replay from the Fireware command line interface (CLI) to troubleshoot connection issues. If you disable anti-replay, you will be susceptible to replay attacks. For more information, go to the diagnose vpn command in the Fireware CLI Reference Guide.

  • Authentication — Authentication makes sure that the information received is exactly the same as the information sent. You can use SHA-1, SHA-2, or MD5 as the algorithm the VPN gateways use to authenticate IKE messages from each other. SHA-2 is the only secure option.
  • Encryption — Encryption keeps the data confidential. You can select DES, 3DES, or AES, or AES-GCM. AES and AES-GCM variants are the only secure options.
  • Force Key Expiration — To make sure Phase 2 encryption keys change periodically, specify a key expiration interval. The default setting is 8 hours. The longer a Phase 2 encryption key is in use, the more data an attacker can collect to use to mount an attack on the key. We recommend that you do not select the Traffic option because it causes high Firebox load, throughput issues, packet loss, and frequent, random outages. The Traffic option does not work with most third-party devices.

  1. The VPN gateways exchange Phase 2 traffic selectors (tunnel routes).

    You can specify the Phase 2 traffic selectors for the local and remote VPN gateway as a host IP address, a network IP address, or an IP address range. Phase 2 traffic selectors are always sent as a pair in a Phase 2 proposal: one indicates which IP addresses behind the local device can send traffic over the VPN, and the other indicates which IP addresses behind the remote device can send traffic over the VPN. This is also known as a tunnel route.

Related Topics

How IPSec VPNs Work