IPsec is becomming the Internet standard for securing IP packets. Instead of manually configuring all the encryption parameters, the keys are usually negatiated between the peers using an ISAKMP (Internet Security Association and Key Management Protocol)/Oakley protocol: IKE (Internet Key Exchange). This post goes into its details.

The main goal of IKE is to negotiate one or more SA’s (Security Associations) between the peers. SA’s contain all information needed to preform the crypto-operations such as encrypting and/or authenticating the packet. It specifies which cipher to use, which key to use, which mode (tunnel or transport) to use, …

Phase 2

Phase 2 is responsible to negotiate the SA’s themselves. This negotiation happens in Quick Mode (QM). The initiating peer sends one or more proposals of encryption parameters. These include:

  • The two sides of the tunnel. A side can be a host, subnet or name.
  • Which encapsulation mode to use: tunnel or transport
  • Which cipher (transform ID) to use: NULL, DES, 3DEs, AES, …
  • Which authentication algorithm to use: HMAC-MD5, HMAC-SHA1, …
  • The lifetime of the SA, specified in seconds and/or kilobytes

The receiving peer compares the list of proposals to its own list and picks the highest match. Along with the parameters, both sides exchange “key material”: a bunch of bits that form the encryption key.

After the 3 packet quick mode exchange, both sides have negotiated usually two SA’s (one in each direction).

Phase 1

Phase 1 is responsible to protect the Quick Mode exchange and to authenticate the remote peer. Authentication usually happens with a pre-shared key (PSK), but you can also use certificates and other mechanisms. Besides the authentication, a Diffie-Hellman key exchange takes place to form a secure channel over an insecure network. The channel is secured similar to a phase2 tunnel, with the same kind of parameters.


When debugging IPsec tunnels, things can get fairly difficult since all packets are encrypted. To get around this problem, I usually change the phase2 parameters to use ESP-NULL encryption. This is a special encryption algorithm that does just nothing: the packets are NOT encrypted.

Wireshark can be configured to “Attempt to detect/decode NULL encrypted ESP payloads”. Obviously, the confidentiality of the tunnel is compromised as well.

The above trick only makes the tunneled data visible on the wire; the IKE exchange is still protected by phase1. Just out of curiosity, I wanted to look into this exchange. Aparenly you can configure the linux isakmpd to dump the decrypted ISAKMP packets by passing it the -L option. The dump only contains the ISAKMP packets in decrypted form, not the tunneled data.

This is the topology used for the above pcap dumps.