🛡️ 08-02: IPsec Fundamentals¶

📌 What Is IPsec?¶

IPsec (Internet Protocol Security) is a suite of protocols that adds security services — confidentiality, integrity, and authentication — directly at the IP layer (Layer 3). Because it operates at the network layer rather than inside an individual application, IPsec can protect any traffic riding on top of IP (TCP, UDP, ICMP, anything) without the applications themselves needing to know anything about encryption.

This is IPsec's defining trait: it's transparent to applications. A web browser, an email client, or a legacy application written decades before encryption was a concern can have its traffic protected by IPsec without a single line of code changing — the protection happens below it, at the IP layer.

💡 Contrast this with TLS (covered in Module 09), which protects traffic at a higher layer and requires the application (or at least the socket library) to be TLS-aware. IPsec's "it just works for everything" property is exactly why it's the dominant technology for site-to-site VPNs, where you want to protect all traffic between two networks indiscriminately.

IPsec is actually a suite — a collection of cooperating protocols — not a single algorithm. The three pieces you need to understand are:

AH (Authentication Header) — integrity and authentication, no encryption
ESP (Encapsulating Security Payload) — encryption plus integrity and authentication
IKE (Internet Key Exchange) — negotiates and establishes the keys the other two use

🔏 AH vs. ESP¶

AH — Authentication Header¶

AH provides:

Integrity — proof the packet wasn't modified in transit
Authentication — proof the packet really came from the claimed sender
Anti-replay protection — proof this isn't an old, captured packet being resent

What AH does not provide: confidentiality. AH does not encrypt the payload at all — anyone capturing the packet can still read its contents. AH only guarantees that whatever they read hasn't been tampered with and really did come from who it claims.

💡 A useful mental model: AH is like a tamper-evident seal on a glass box. Everyone can see what's inside (no confidentiality), but if anyone tries to swap or alter the contents, the seal breaks and that's detectable.

Because AH provides no encryption, it's rarely used on its own in modern deployments — in most real-world scenarios, if you're protecting data at all, you want confidentiality too, which means ESP.

ESP — Encapsulating Security Payload¶

ESP provides everything AH does (integrity, authentication, anti-replay) plus encryption of the payload. This makes ESP the workhorse of nearly all real IPsec deployments today.

Confidentiality — the payload is encrypted (using a symmetric cipher such as AES — see the Module 07 lesson on block ciphers for how the encryption itself works)
Integrity/authentication — a MAC (message authentication code) is attached so tampering or spoofing is detectable
Anti-replay — sequence numbers prevent an attacker from capturing and resending old packets

Feature	AH	ESP
Confidentiality (encryption)	❌ No	✅ Yes
Integrity	✅ Yes	✅ Yes
Authentication	✅ Yes	✅ Yes
Anti-replay	✅ Yes	✅ Yes
Protects IP header itself	✅ Yes (in transport mode, most of it)	❌ No (only the payload)
Common in practice today	Rare	Very common

💡 A subtlety worth remembering: because AH authenticates parts of the outer IP header too, it breaks under NAT (Network Address Translation) — if a NAT device rewrites the source IP address in transit, AH's integrity check fails, because the header it authenticated no longer matches. ESP doesn't have this problem (in transport mode, at least, and with NAT-Traversal extensions), which is another reason ESP dominates in the real world where NAT is everywhere.

🤝 Security Associations (SAs)¶

Before AH or ESP can protect a single packet, both ends of the connection need to agree on a common set of parameters: which encryption algorithm to use, which key, which integrity algorithm, and so on. This agreed-upon bundle of parameters is called a Security Association (SA).

Think of an SA as a contract between two IPsec peers that says: "For traffic matching this description, we will use this encryption algorithm, this key, this integrity algorithm, and this mode." Every IPsec-protected connection needs at least one active SA in each direction.

Key properties of an SA:

Unidirectional — an SA only protects traffic in one direction. A typical two-way IPsec conversation needs two SAs, one for each direction.
Uniquely identified by a triple: the destination IP address, the security protocol (AH or ESP), and a Security Parameter Index (SPI) — essentially an ID number that tells the receiving device which SA (and thus which keys/algorithms) to use to process an incoming packet.
Time-limited — SAs expire and are renegotiated periodically (both by time and by amount of data protected), which limits how much data any one key ever protects.

💡 If AH and ESP are the "workers" doing the actual protecting, an SA is the "instructions" telling them exactly how to do their job for a specific flow of traffic. Without an established SA, there's no shared understanding of which key or algorithm to use, and no protected communication can happen.

🔑 IKE — Internet Key Exchange¶

SAs need keys, and manually configuring identical secret keys on every device by hand doesn't scale and is a security risk (the key has to be transported somewhere safe first). IKE (Internet Key Exchange) is the protocol that automates this: it lets two IPsec peers securely negotiate and establish the SAs (and the keys inside them) over an untrusted network, without ever transmitting the actual secret key in the clear.

IKE typically works in two phases:

Phase 1 — the two peers authenticate each other and establish a secure, encrypted channel between themselves (this channel is sometimes called the IKE SA or ISAKMP SA). This is where the heavy cryptographic key-agreement work happens.
Phase 2 — using the secure channel from Phase 1, the peers negotiate the actual IPsec SAs (the AH/ESP parameters) that will protect the real data traffic.

The mathematical trick that makes Phase 1 possible — letting two parties who have never met agree on a shared secret over a network that an eavesdropper is watching — is Diffie-Hellman key exchange. IKE doesn't invent its own key-agreement math; it relies on Diffie-Hellman underneath. For the full explanation of how two parties can derive the same secret number while an attacker watching every message still can't compute it, see 07-05: Diffie-Hellman Key Exchange.

💡 A simple way to remember the layering: Diffie-Hellman is the mathematical primitive that creates a shared secret; IKE is the protocol that orchestrates when and how that primitive gets used (including authentication of the peers, so you know you did Diffie-Hellman with the right party and not an attacker); and the SA is the resulting agreement that AH/ESP then use to actually protect packets.

🔄 Putting It Together: The IPsec Flow¶

Two devices (e.g., two VPN gateways) want to establish a protected connection.
IKE Phase 1 runs: the devices authenticate each other and, using Diffie-Hellman math, derive a shared secret without ever sending it directly over the wire.
IKE Phase 2 runs inside the now-secure IKE channel: the devices agree on the actual SAs — which protocol (AH or ESP), which encryption/integrity algorithms, and derived session keys.
Data traffic now flows, protected according to those SAs — typically ESP, providing both confidentiality and integrity.
Periodically, SAs expire and Phase 2 (sometimes Phase 1 too) re-runs to establish fresh keys, limiting the exposure of any single key.

📌 Key Takeaways¶

IPsec protects traffic at the IP layer, so it works transparently underneath any application or transport protocol.
AH provides integrity and authentication only — no encryption, so no confidentiality.
ESP provides encryption plus integrity and authentication, and is the protocol used in almost all real-world IPsec deployments.
AH breaks under NAT because it authenticates parts of the outer IP header; ESP generally survives NAT (especially with NAT-Traversal).
A Security Association (SA) is the agreed-upon bundle of algorithms, keys, and parameters that AH/ESP use to protect a specific, one-directional flow of traffic.
IKE automates the negotiation and establishment of SAs, using Diffie-Hellman underneath to safely derive shared secrets over an untrusted network — see 07-05: Diffie-Hellman Key Exchange.
IKE runs in two phases: Phase 1 sets up a secure channel between the peers; Phase 2 uses that channel to negotiate the actual IPsec SAs for data traffic.
SAs are time-limited and get renegotiated periodically, which limits how much traffic any single key ever protects.