IPSec Explained: How It Works, What It's Used For, and Why It's Still Relevant

What Is IPSec?

IPSec — short for Internet Protocol Security — is a suite of protocols for securing IP communications. Unlike TLS, which encrypts at the application layer, IPSec works at the network layer. That means it authenticates and encrypts every IP packet that travels between two endpoints, including headers in tunnel mode, not just the application payload.

It has been in production since the mid-1990s and remains the dominant protocol for enterprise VPNs, site-to-site tunnels, and corporate remote access. When your IT department provisions a VPN for remote workers, there is a good chance it runs IKEv2/IPSec under the hood, even if the client branding hides that fact.

IPSec is not a single protocol. It is a framework combining key exchange (IKE), authentication (AH), and encryption (ESP) into a coordinated system. Understanding each component individually makes the whole architecture significantly less opaque — and makes troubleshooting much faster.

The Core Components of IPSec

IKE — Internet Key Exchange

Before any encrypted data can flow, both endpoints must agree on encryption algorithms, authentication methods, and exchange cryptographic keys. This negotiation is handled by IKE (Internet Key Exchange).

IKEv1 (the original) worked in two phases with many message exchanges and was notoriously complex to configure and debug. IKEv2 (RFC 7296, current standard) streamlined the process, reduced the number of message roundtrips, added built-in NAT traversal, and improved mobility support for roaming clients. Every new IPSec deployment should use IKEv2.

IKE runs on UDP port 500. When NAT is detected between endpoints, IKEv2 automatically switches to UDP port 4500 and wraps ESP packets in UDP — this is NAT Traversal (NAT-T). If you are troubleshooting an IPSec tunnel that will not connect, confirming that ports 500 and 4500 are open in both directions is the first step.

ESP — Encapsulating Security Payload

ESP is the component that actually provides encryption and integrity for your data. It takes the original packet (or just its payload, depending on mode) and wraps it in an encrypted envelope with an authentication tag.

ESP handles both confidentiality (encryption, so no one can read the content) and integrity (so tampering is detected). Modern ESP implementations use AES-256-GCM, which provides both encryption and integrity authentication in a single pass. AES-GCM is also hardware-accelerated in most modern CPUs and network ASICs, which is why IPSec on enterprise hardware performs so well.

ESP uses IP protocol number 50. This is not a TCP or UDP port — it is a Layer 3 protocol identifier, like TCP (protocol 6) or UDP (protocol 17). Firewalls must permit protocol 50 explicitly, not just port-based rules. When NAT-T is active, ESP is encapsulated in UDP port 4500 instead.

AH — Authentication Header

AH provides authentication and integrity but no encryption. It signs the IP header and payload to prove the packet came from the legitimate sender and was not modified in transit. Because AH authenticates the IP header and NAT modifies IP headers by design, AH breaks across any NAT device. For this reason, AH is rarely used in modern deployments — ESP with authentication covers the same use cases without the NAT incompatibility.

Tunnel Mode vs. Transport Mode

Tunnel Mode

In tunnel mode, the entire original IP packet — header and payload — is encrypted and encapsulated inside a new outer IP packet. The outer packet's source and destination are the tunnel endpoint addresses (VPN gateways or routers). The original inner IP addresses are invisible to the transit network.

Tunnel mode is used for site-to-site VPNs connecting office networks and for remote-access VPNs where clients connect to a corporate gateway. It provides the strongest privacy — not just the content but also the internal source and destination addresses are hidden from anyone observing the transit path.

Transport Mode

In transport mode, only the payload of the original packet is encrypted. The original IP header remains visible. The outer IP addresses are the actual endpoints; there are no separate tunnel endpoint addresses.

Transport mode is primarily used for host-to-host encryption between servers in the same data center or organization — for example, encrypting all traffic between two application servers on the same internal network. It has lower overhead than tunnel mode but does not hide the communicating parties' IP addresses.

How an IPSec Connection Is Established

The connection process has two phases, each producing a Security Association (SA):

Phase 1 — IKE SA: The two endpoints authenticate each other (using pre-shared keys, X.509 certificates, or EAP for users) and establish a secure, authenticated channel for Phase 2 negotiation. The output is an IKE SA — a bidirectional control channel. Authentication methods: pre-shared keys are simple but hard to rotate at scale; certificates with a proper PKI are the correct choice for production. Phase 1 parameters include encryption algorithm, integrity algorithm, Diffie-Hellman group, and lifetime.

Phase 2 — IPSec SA: Using the secure channel from Phase 1, the endpoints negotiate the actual data protection parameters: encryption algorithm, integrity algorithm, key lifetime, and traffic selectors (which source/destination subnets the SA protects). Phase 2 produces two unidirectional IPSec SAs — one for each direction of traffic.

Keys are periodically renegotiated — by default approximately every hour or every gigabyte — to limit exposure if a session key is ever compromised. Enabling Perfect Forward Secrecy (PFS) means each rekeying generates fresh Diffie-Hellman key material independent of previous session keys. Even if an attacker records encrypted traffic and later compromises one key, they cannot decrypt past or future sessions.

IPSec vs. SSL VPN vs. WireGuard

Why IPSec Remains Dominant in Enterprise Environments

Despite WireGuard's simpler design and performance advantages on consumer hardware, large enterprises continue deploying IPSec for several concrete reasons:

• Hardware acceleration at scale: Enterprise routers and firewalls from Cisco, Juniper, Fortinet, and Palo Alto include dedicated IPSec crypto engines. At high throughput (multi-gigabit), hardware-accelerated AES-GCM makes IPSec extremely fast. The performance advantage of WireGuard on commodity hardware narrows significantly against dedicated crypto silicon.
• Vendor interoperability: IPSec is an IETF standard. Any certified IPSec implementation interoperates with any other. Connecting 300 branch offices that run a mix of Cisco, Juniper, and pfSense hardware works because they all speak the same protocol. WireGuard is still building out the enterprise ecosystem.
• Feature completeness: IPSec supports traffic selectors (fine-grained control over which traffic the tunnel protects), QoS marking, redundancy (IKEv2 MULTI-SA), RADIUS/LDAP authentication, and XAUTH. These enterprise-required features are mature and well-tested in IPSec implementations.
• Regulatory compliance: FIPS 140-2 certified IPSec implementations satisfy requirements in HIPAA, PCI-DSS, and US government networks (CNSA suite). WireGuard's ChaCha20-Poly1305, while cryptographically strong, does not yet have the same breadth of certifications in regulated industries.

Common IPSec Issues and Fixes

Phase 1 Will Not Complete

The most common cause is mismatched IKE parameters. Both endpoints must agree on exactly the same encryption algorithm, integrity algorithm, DH group, and authentication method. Check the IKE debug logs on both sides — they will identify which parameter caused the negotiation to fail. Mismatched pre-shared keys and mismatched DH groups are the two most frequent culprits. Also verify that UDP 500 and 4500 are permitted in both directions.

Phase 2 Establishes But No Traffic Passes

Almost always a traffic selector (proxy ID or crypto ACL) mismatch. The source and destination subnet definitions in Phase 2 must be identical on both endpoints. A common mistake: one side defines 10.0.1.0/24 and the other defines 10.0.1.0/255.255.255.0 in a format the parser handles differently. Both sides log traffic selector mismatch — check the IKE logs.

NAT Traversal Problems

Standard ESP does not work through NAT because NAT modifies the IP header that ESP authenticates. IKEv2 detects NAT automatically during the IKE handshake and activates NAT-T, wrapping ESP in UDP port 4500. Ensure both endpoints have NAT-T enabled (it is default-on in most implementations) and that UDP 4500 is permitted through intermediate firewalls.

MTU and Fragmentation Issues

IPSec adds 60-80 bytes of overhead per packet. Large packets exceed the effective MTU of the tunnel interface, triggering fragmentation. If intermediate firewalls block ICMP type 3 code 4 (Fragmentation Needed), Path MTU Discovery fails silently. Fix: set the tunnel interface MTU to 1400 or lower and configure TCP MSS clamping to prevent TCP from negotiating oversized segments. This eliminates the fragmentation problem at its source.

IPSec Firewall Ports

• UDP 500 — IKE initial negotiation (both directions)
• UDP 4500 — IKE NAT traversal + NAT-T encapsulated ESP (both directions)
• IP Protocol 50 — ESP (when not using NAT-T; not a port number)
• IP Protocol 51 — AH (rarely needed in modern deployments)

IPSec on Mobile Devices

iOS and Android both include built-in IKEv2/IPSec clients. Enterprise MDM solutions push VPN profiles to employee devices without requiring additional app installation. The VPN client is part of the operating system, managed and updated by the OS vendor.

IKEv2's MOBIKE extension (RFC 4555) handles the mobile roaming case. When a phone switches from Wi-Fi to LTE, MOBIKE allows the IKEv2 session to update its IP address without tearing down and re-establishing the full SA. Reconnection typically takes under a second. This protocol-level roaming support, tested across thousands of device and carrier combinations over many years, gives IPSec an advantage in enterprise mobile deployments where reliability matters more than raw performance.

Real-World IPSec Deployments

Most corporate remote access clients — Cisco AnyConnect (in IKEv2 mode), Palo Alto GlobalProtect, Juniper Pulse Secure, and Fortinet FortiClient — use IPSec as the transport. The branding hides the protocol, but the underlying mechanism is IKEv2 negotiating an SA with the corporate gateway firewall.

Site-to-site tunnels between offices are almost universally IPSec. Whether the endpoints are Cisco ASAs, Fortinet FortiGates, or open-source pfSense routers, they speak the same IPSec standard. Cloud provider VPN gateways (AWS Site-to-Site VPN, Azure VPN Gateway, GCP Cloud VPN) also run IPSec, making hybrid cloud connectivity straightforward for any organization with existing IPSec infrastructure.

Pro Tips

• Always use IKEv2 instead of IKEv1. IKEv2 is faster, more secure, handles NAT automatically, and supports mobile roaming. IKEv1 is legacy and should only be used when forced by older hardware that cannot be upgraded.
• Enable PFS for Phase 2. Perfect Forward Secrecy ensures each rekeying generates independent key material. Compromising one session key exposes only that session, not past or future traffic. The DH group computation cost is negligible on modern hardware.
• Use certificate-based authentication in production. Pre-shared keys are simple to configure but require secure distribution and rotation. A compromised PSK can be used to impersonate either endpoint. A PKI with certificate revocation provides far stronger identity guarantees.
• Set DH group 14 (2048-bit) as your minimum. DH groups 1, 2, and 5 are considered cryptographically weak and should never be used in new configurations. Group 14 (2048-bit MODP) is the current acceptable minimum; groups 19 and 20 (256-bit and 384-bit ECDH) offer better performance at equivalent security.
• Enable IKE debug logging on both sides before troubleshooting. IKE negotiation logs identify exactly which parameter mismatched, which proposal was rejected, and where the handshake stalled. Without logs, IPSec troubleshooting requires guessing. With logs, most issues are resolved in minutes.
• Test your tunnel with traffic that traverses the full path. A successful Phase 2 negotiation does not guarantee traffic flows. Generate actual test traffic from hosts behind each gateway after the tunnel comes up to confirm end-to-end connectivity before declaring the deployment complete.

Check your current connection details and see what protocol your network is using — instant results, no login needed.