IPsec secures IP traffic at the packet level
We recommend using IPsec when implementing an IPv6 network
When Vint Cerf devised the TCP/IP protocol 50 years ago, he should have paid more attention to its security. He should also have provided for a bigger address space. So said Vince himself when interviewed for IEEE Spectrum recently.
Most readers will be aware that IPv6 extends the address space from the 32 bits we had with IPv4 to 128 bits. Indeed, the bigger address space is the main reason for migrating to IPv6. Fewer people are aware, however, that IPv6 supports the encryption of connections with IPsec by default.
"IPsec makes use of the IPv6 extension headers, whereas with IPv4 use of the protocol involved a sort of hack," says Paul Wouters. Now Senior Security Architect at Aiven, Paul is also one of the 2 Security Area Directors at the IETF, author of various RFCs, and longtime developer of Libreswan and its predecessors. "IPsec isn't very visible to the end user, but it's used all over the place: data centres, clusters, military networks and virtualised environments all rely on IPsec. It's also used for mobile and internet telephony. For many health care providers, using IPsec is a legal requirement. And Airbus uses it to ensure that passengers can't get onto a plane's internal network. So IPsec is part of a huge number of IETF protocols."
IPsec was developed in the 1990s by the IETF's IP Security Working Group, with various US government organisations acting as cooperating sponsors. Defined in RFC 4301, the protocol secures both the headers and the contents (or 'payload') of the IP network packets (or 'datagrams'). IPsec therefore occupies an intermediate position between the Internet Protocol and the overlying transport protocols, such as TCP, UDP and SCTP.
Figure 1: IPsec occupies an intermediate position between the Internet Protocol and the overlying transport protocols.
IPv6
IPsec was developed in parallel with IPv6, and was initially even a mandatory feature of the latter protocol. However, RFC 6434 (now RFC 8504) changed IPsec's status from mandatory to recommended.
It has since been rumoured that, back in the nineties, the NSA deliberately frustrated the development and implementation of IPsec security, but Wouters regards that assertion as overblown. However, the Snowden leaks did show that IPsec security was an NSA target.
Although implementation wasn't ultimately made mandatory, IPsec is now supported by default by almost all current network equipment and operating systems. The end user is nevertheless barely aware of IPsec, because the protocol operates at a lower level of the network stack. IPsec becomes visible mainly when a virtual private network (VPN) is used, but it is also active in the context of communication between data centres, in mobile and internet telephony, and in cloud environments. Although IPsec encryption is performed at the IP packet level, the key and session management take place at a higher level. The protocol therefore also works well for connections with dynamic IP addresses (mobile users).
Protection levels and modes
IPsec originally offered 2 protection levels: the Authentication Header (AH) and Encapsulating Security Payload (ESP). Each of those could be used either in transport mode or in tunnel mode. Let's look at the latter first. In transport mode, the original datagram header is still used, while in tunnel mode, the original datagram (header plus payload) is encapsulated within a complete new header (derived from the straightforward IP Encapsulation within IP protocol defined in RFC 2003).
Transport mode is therefore intended for host-to-host connections (point-to-point), while tunnel mode is mainly for host-gateway (remote access) and gateway-gateway (site-to-site) connections. An example of the latter is an LAN-to-LAN VPN for traffic between an organisation's different sites.
In all cases, the decision as to whether packets will be encrypted, dropped or forwarded without encryption is made at the host-gateway interface in line with the configured policies, broadly as happens with a packet filter.
ESP and AH
So, to the difference between AH and ESP: Defined in RFC 4302, AH uses an HMAC/digital signature to assure the integrity not only of the payload, but also of the fixed components of the packet header. Fixed components are components such as the source and destination addresses, which do not change. By contrast, components such as the TTL/hop limit and the ToS field (the Traffic Class in IPv6) can be modified in transit and cannot therefore be secured using AH.
ESP (defined in RFC 4303) doesn't secure the headers, as AH does, but concentrates on the IP packets' payloads. However, the biggest difference is that ESP also encrypts the entire payload, and therefore assures both the integrity and the confidentiality of the messages.
To be completely clear, if you use ESP in tunnel mode – by far the most common IPsec usage mode – both the payload and the entire header of the encapsulated packet are protected. Whereas, if you use AH in tunnel mode, the fixed components of the outer header are protected as well. However, no one actually does that: if you ever still encounter the AH protocol in practice, it's used in transport mode in combination with pure IP-in-IP tunnelling (defined in RFC 1853) to secure host-to-host connections.
Figure 1: IPsec originally offered 2 protection levels and 2 usage modes.
More security features
What AH and ESP have in common is that both authenticate the source – not explicitly, but insofar as only the sender possesses the private key used to sign/encrypt the message.
Forward secrecy is achieved by replacing the session keys every hour. For total security, instead of basing a new session key on the existing seed, you should always perform a fresh key exchange on the basis of DH (perfect forward secrecy).
Another way of increasing security is to frustrate traffic analysis by using tunnel mode and making all the packets the same length (by providing padding) or inserting 'empty' packets (made up entirely of padding).
The recipient can also protect themselves against replay attacks by additionally validating the (IPsec) serial number of the packet (included in the hash calculation). However, it's not really necessary to do that, because UDP is packet-based, implying that the overlying application layer must be capable of handling retransmits securely. Furthermore, TCP has its own serial number (the basis of the secure end-to-end connections).
Finally, there is IPComp, a data compression technology (defined in RFC 3173). Compression has to be done before ESP encryption: the compression of encrypted messages is pointless, since good encryption ensures that no further information can be distilled out of the cypher text. However, IPComp is rarely used, since compression has often already been performed at the application level.
ESP as a replacement for AH
Whereas previously AH and ESP could be used together (or encapsulated) to assure both the integrity of the header and the confidentiality of the payload, the arrival of IPsec version 3 has made that unnecessary. The current version of ESP (now supported by all implementations) incorporates header security. The use of AH for new installations is therefore discouraged.
For equipment and software that supports IPsec, the implementation of ESP is now mandatory, while AH is optional. If you nevertheless want to dispense with encryption for any reason, the answer is to use ESP with the NULL encryption option.
If you're wondering why AH and ESP are so similar, the explanation is linked to the fact that AH is slightly older, but developed in the same period as ESP. At that time, the export of (strong) cryptography from the US was prohibited, leading to use of the 2 technologies side by side.
Internet Key Exchange
For IPsec configuration and the coordination and exchange of key material, hosts and gateways use IKEv2, a key management protocol defined in RFC 7296. The IPsec policies of the hosts and gateways involved form the basis of the key management procedures.
When a connection is established, the systems involved first consult their IKEv2 servers. They can then agree the encryption protocols and keys to be used (the so-called Security Associations).
While IPsec operates on the internet layer and forms part of the network stack (in kernel space), IKE is a separate protocol that makes use of a daemon running on UDP port 500 (in userland).
The IPSECKEY record
The algorithms available for hashing, encryption and compression in ESP and AH are defined in RFC 8221. Those available for IKEv2 are defined in RFC 8247. The latest updates to both those RFCs are to be found in RFC 9395.
When establishing an IPsec connection, it's possible to use not only symmetric keys but also PKI certificates (X.509) or raw public keys. Defined in RFC 4025, the IPSECKEY record – for which DNSSEC security is essential – was devised specifically for the latter purpose.
It should be noted that systems at the lowest level often don't know the other party's name, but just their IP address, making IKE/IPSECKEY a use case for the implementation of DNSSEC on Reverse DNS (rDNS).
Problems with NAT
Although the variable components of the headers are not included in AH/ESP hash calculations, problems can nevertheless arise with the fixed components sometimes. For example, NAT – which we have with good reason often described as an undesirable workaround – modifies an IP packet's source and destination addresses, which have traditionally been considered to be fixed.
If there's a NAT gateway in the way, ESP tunnels can be used only in combination with NAT Traversal. RFC 3947 and 3948 define how to encapsulate ESP and IKE traffic in UDP (with a dedicated unsecured header) and direct it via port 4500 (ESPinUDP). If your network stack also supports ESPinTCP (defined in RFC 8229), you can use that protocol to bypass firewalls (which often block UDP traffic) by using an available TCP port (as with TLS over port 443 or even HTTPS encapsulation).
Perimeter vulnerabilities
However, we would warn against using tunnels to bypass firewalls and gateways, because they're known to result in perimeter vulnerabilities. It's better to make NAT part of the IPsec gateway (or to locate the NAT system immediately in front of the IPsec system).
"On the internet as originally conceived, all hosts could reach each other directly using their public IP addresses," explains Wouters. "That all changed when NAT and then CGNAT came along. What ESPinTCP now enables us to do wasn't possible before. In the past, that led to the development of protocols such as OpenVPN, which necessarily operates in userland and is therefore much slower than a network protocol operating in kernel space. The modern Wireguard protocol does operate in kernel space and has a packet format very similar to ESP's. However, Wireguard is less mature than IPsec and exhibits various inefficiencies that slow it down. Also, there's no support for DHCP or DNS, in contrast to IKEv2. The same goes for AES-GCM (defined in RFC 8452), which is now a standard feature of almost every processor."
Not a simple protocol
As you'll have gathered, IPsec definitely isn't a simple protocol. In fact, people used to say that the US government agencies involved in its development deliberately made IPsec as complicated as possible, to enable them to exploit implementation vulnerabilities further down the line. Similarly, RFC 4301 is not an accessible document, and therefore unsuitable as an introduction to IPsec's basic concepts.
The most readable and comprehensive explanation of IPsec that we've come across is NIST's 'Guide to IPsec VPNs', whose lead author is Wouters. As well as explaining the technology, the NIST guide provides useful information about planning and implementing IPsec. We recommend the use of IPsec, particularly when implementing or upgrading an IPv6 network.