Internet Protocol Security (IPSec) is a collection of standards that was designed specifically to create secure end-to-end secure connections. The standards were developed by the Internet Engineering Task For (IETF) to secure communications over both public and private networks, though it is particularly beneficial to public networks. In this article I'll explain to you some of the fundamentals of IPSec, how it is used, and what products use it.

IPSec Basics
IPSec is framework that is built into various security products to provide end-to-end security in wide area networking communications. Using strong encryption, and public key cryptography, IPSec can secure data links that would otherwise be insecure and susceptible to exploitation.

IPSec is a bundle of protocols and algorithms and is a flexible framework that allows vendors who build it into their products to select the algorithms, keys, and authentication methods they want to use. One should assume that two different implementations of IPSec are not necessarily the same as far as protocols and algorithms go.

The bundle of protocols, hashing, and encryption algorithms used in IPSec include:

• IKE [Internet Key Exhange protocol]
• ISAKMP [Internet Security Assocation and Key Management Protocol]
• ESP [Encapsulating Security Payload]
• AH [Authentication Header protocol]
• ESP [Encapsulating Security Payload protocol]
• STS [Station-to-Station protocol]
• HMAC [Hash Message Authentication Code]
• MD5 [Message Digest 5]
• SHA-1 [Security Hash Algorithm]
• 3DES [Triple Data Encryption Standard]
• XAUTH [Extended Authentication]
• AES [Advanced Encryption Standard]

Though I won't be discussing these protocols and algorithms in much detail in this article, I have noted them in the event that you may want to research these individual components of IPSec yourself. To understand IPSec better, the two protocols worth understanding first are AH and ESP. AH is used to authenticate users, and ESP applies cryptographic protections that provide authentication, integrity, and confidentiality of messages.

There are two modes of operation for IPSec: transport mode and tunnel mode. In transport mode, only the payload of the message is encrypted. In tunnel mode, the payload, the header, and the routing information are all encrypted. Needless to say, using IPSec is transport mode is far more risky that using it in tunnel mode.

IPSec VPNs are network connections that are based on public and private key cryptography. Users of IPSec implementations are issued public keys and private keys that are associated with their respective identity. When a message is sent from one user to another, it is automatically signed with the user's private key. The receiver uses the sender's public key to decrypt the message. VPN endpoints essentially act as databases that manage and distribute keys and security associations in similar ways that a Certificate Authority (CA) does.

Benefits of IPSec
IPSec is typically used to attain confidentiality, integrity, and authentication in the transport of data across insecure channels. Though it's original purpose was to secure traffic across public networks, it's implementations are often used to increase the security of private networks as well, since organizations cannot always be sure if weaknesses in their own private networks are susceptible to exploitation. If implemented properly, IPSec provides a private channel for sending and exchanging vulnerable data whether the data is email, ftp traffic, news feeds, partner and supply chain data, medical records, or any other type of TCP/IP based data.

Limitations of IPSec
Though IPSec is the most ubiquitous, and probably the best existing framework for the implementation of VPNs, there are certain limitations associated with it. When implemented in transport mode, IPSec is susceptible to replay attacks. Due to limitations of ISAKMP, Neils and Schneier have suggested that it is likely that IPSec is also susceptible to man-in-the-middle attacks.

IPSec session hijacking can occur when an authenticating header is not used. In this type of attack, malicious data can be insert into the payload, say an rm -r command (on a Unix system) that would remove every file on the recipient filesystem.

Because IPSec traffic is routable, IPSec implementations may also be susceptible to source routing exploits, depending on security safeguards (or lack thereof) that have been put in place on the routers over which it travels. When used in tunnel mode, IPSec is not as vulnerable to routing exploits since the routing information is encrypted.

Steve Bellovin of AT&T Research has pointed out that many of the weaknesses of IPSec are inherent to the limitations of the encryption modes used in the implementation (*1). One can conclude that if the embedded encryption modes used in the IPSec framework were stronger, IPSec would be more secure.

Though IPSec is currently not part of IPv4, it is part of IPv6. The good news is that some of the weaknesses in IPSec have been corrected in IPv6. In IPv4, fragmentation fields in the IP header are allowed to change. In IPv4, when IPSec is used in transport mode, a hacker could potentially intercept a packet, change the fragmentation field introducing malicious data, and then insert the packet back into the data stream. In IPv6, intermediate routers are not supposed to allow packet fragmentation.

IPSec Technology Challengers
For many products that could use IPSec, some of them instead use an alternative encryption technology known as Secure Sockets Layer (SSL). The difference between SSL and IPSec is that IPSec works at the network layer, and secures entire networks, and SSL works at the application layer, and secures applications. IPSec and SSL are both used to provide confidentiality of data, and authentication, but they achieve these goals in significantly different ways.

SSL was originally designed by Netscape to secure (HTTP) traffic passing through web browsers and is a session layer protocol. Unlike IPSec, SSL is based on a client/server model and is typically used for host-to-host secure transport. Because IPSec works at the network layer, it can be used to secure subnet-to-subnet, network-to-network, or network-to-host communications. This means that IPSec traffic can be routed, while SSL traffic cannot.

While many people see SSL as a technology competitor to IPSec, this view is not entirely accurate. In most cases, IPSec and SSL are used to solve different types of problems. Also, while IPSec based connections require a substantial amount of planning and implementation time, SSL implementations are relatively quick to use, and sometimes require no planning at all, depending on what browser someone might be using and how it is currently configured.

IPSec Market Implications
As wireless networks become more ubiquitous, so to will the use of VPNs. Wrapping a VPN around your wireless network is just about the best way to secure it. Since wireless access points are layer 2 devices, when you use a wireless network, you can secure it using a VPN in exactly the same way you secure a wire-based network. (Remember that a wireless access point is simply a transceiver that at one end always plugs back into a wire.) If you have an enterprise class wireless network, your best bet at securing it is with an IPSec based VPN.

Signature based intrusion detection systems can only work on unencrypted links making them virtually unusable on IPSec based connections. Host-based intrusion prevention systems work just as you would expect them to in an IPSec based infrastructure, since host-based intrusion prevention systems are not implemented on network links. Since IPSec and network based intrusion detection systems cannot interoperate with each other, the adoption of wireless networks will likely create less demand for network based intrusion detection systems, and more demand for host based intrusion prevention systems.

While some vendors see IPSec and SSL going head-to-head to solve security problems, they actually both have their own place in the world of information technology infrastructure.

Products that Use IPSec
IPSec is used most commonly in firewalls, VPNs, and authentication products. VPNs are the dominate users of IPSec, and of the firewalls that use IPSec, use it because they also have built-in VPN capabilities. However, there are other products on the market that either come bundled with IPSec, or make use of it in some way - such as secure operating systems, developer toolkits, and encryption accelerator products.