What is tunnel mode and transport mode in IPsec?
IPSec tunnel mode—the entire original IP datagram is encrypted, and it becomes the payload in a new IP packet. with tunnel mode, an attacker can only determine the tunnel endpoints and not the true source and destination of the tunneled packets, even if they are the same as the tunnel endpoints.
IPSec transport mode—only the IP payload is encrypted, and the original IP headers are left intact.
Tunnel mode and Transport mode
When using ESP you can specify one of two modes, in which ESP operates in. Tunnel mode encrypts the whole packet. Tunnel mode is used for site to site VPN, when securing communication between security gateways, concentrators, firewalls, etc. Tunnel mode provides security for the entire original IP packet, that is the headers and the payload.
The other mode ESP can operate in is Transport mode, which is not as secure as it only encrypts the data portion and not the whole packet unlike tunel tunnel mode.
Transport mode encrypts the data portion of the packet. It works between two different workstations running some kind of VPN software. Transport mode protects payload of packet and the high layer protocols. Transport mode leaves the original IP addresses in open clear text. Using transport mode the final destination is not a gateway or router, generally the host itself. Transport mode provides security to the higher layer protocols only.