Setting ip IPsec tunnel from Linux to Cisco PIX

In this post I am going to put down my experience setting up a IPsec tunnel from a Linux router to a Cisco PIX device. I'll explain the setup, the solution, and the pitfalls encountered along the way.

Setup

We have a LAN using IP subnet 192.168.0.0/22, connected to the Internet through a Linux router, whose external IP address is 80.1.1.1 (address made up for the sake of example, of course). The goal was to connect to a network within customer's LAN through IPsec tunnel. The destination net was 194.1.1.2/32 (yes, the netmask here is 32, since this actually was a single host; but I'd like to talk about it as a subnet, because in the case of a true subnet our solution would be the same, just netmask would have less bits). Even though the destination's IP address is "external-looking", it is blocked off the Internet by a firewall and is only reachable through IPsec via an external gateway whose address is 194.1.1.1. The following crude ASCII art diagram illustrates the setup:

                              80.1.1.1     
   Our LAN     |------------|/              
---------------+Linux router+--------------|
192.168.0.0/22 |------------|              |
                                           |
            IPsec tunnel                   |
    ---------------------------------------|
    |194.1.1.1
    |         \|---------|  Dest. subnet
    |----------+Cisco PIX+----------------
               |---------|  194.1.1.2/32

Additional complication was that our IP address range 192.168.0.0/22 was in use in the customer's LAN, thus we could not allow our packets to go into their network with their original source IP addresses. To circumvent this, our first step was to set up SNAT to masquerade all our outgoing packets as coming from IP address 172.16.1.1.

The Cisco PIX was managed by the customer, and thus outside of our control. I was responsible to set up our Linux router.

Solution

Now I will jump forward and describe the solution that worked.

Some theory on IPsec and indispensable guidance on setting up IPsec in Linux can be found at Linux IPsec HOWTO. The solution below was based on the information obtained from the HOWTO.

SNAT

As mentioned above, our first step was to set up SNAT for our packets going off to the customer's net:

iptables -t nat -A POSTROUTING -d 194.1.1.2/32 -j SNAT --to-source 172.16.1.1

Note that this is independent of IPsec. It just tells the kernel to masquerade any packets going out to 194.1.1.2/32 as coming from source address 172.16.1.1 (which does not need to correspond to any physical network interface of the router). Before any tunnel was set up, I could test the SNAT by pinging 194.1.1.2 and watching the source address of outgoing packets with tcpdump.

Kernel security policy

The next step is to set up kernel's security policy. Simply put, we need to tell the kernel that whenever it sees a packet going to 194.1.1.2/32 it has to send it through tunnel that runs from 80.1.1.1 (our external address) to 194.1.1.1 (customer's external address). This is achieved with the following script:

#!/bin/sh

SRCNET="172.16.1.1/32"
DSTNET="194.1.1.2/32"
TUNNEL_LOCAL="80.1.1.1"
TUNNEL_REMOTE="194.1.1.1"

/sbin/setkey -c  >/dev/null 2>&1 << EOF

spdadd $SRCNET $DSTNET any -P out ipsec
            esp/tunnel/$TUNNEL_LOCAL-$TUNNEL_REMOTE/require
            ;

spdadd $DSTNET $SRCNET any -P in ipsec
            esp/tunnel/$TUNNEL_REMOTE-$TUNNEL_LOCAL/require
           ;
EOF

Note that we only tell the kernel to use ESP and not AH. Many sources would tell you to use both, and this is a good thing to do; it's just that in our case, for some reason, the customer insisted on using ESP only. Also note that the SRCNET is set to 172.16.1.1/32, as the SNAT we set up earlier will modify the packets' source addresses before IPsec sees them. Had we not used SNAT, SRCNET would have been 192.168.0.0/22.

Now the kernel knows when to use the tunnel, but to create the tunnel a security association should be set up between our router and the remote Cisco box. This is the job of racoon daemon, and I'm moving on to describe its configuration.

Racoon and ISAKMP

Racoon speaks ISAKMP protocol in order to create the tunnel. A detailed description if ISAKMP is outside the scope of this post, but it is important to know that creation of the tunnel proceeds in two phases. At phase 1 racoon negotiates a preliminary secure connection to the remote site called ISAKMP Security Association (SA). At phase 2, using the existing ISAKMP SA, it is able to privately talk to the remote site and negotiate the parameters of the IPsec tunnel itself. At phase 2 the actual IPSec Security Association (SA) is created.

There are many details of ISAKMP that racoon needs to negotiate with the remote end of the tunnel. Description of them all is outside of our scope now. Our customer wanted to use the following parameters:

Phase 1:

• Authentication Method: RSA-Key
• Diffie-Hellman Group: 5 (1536 bit)
• Encryption Algorithm: AES-256
• Data Integrity Algorithm: SHA-1
• Use aggressive mode: No
• Lifetime: 86400s (1 day)

Phase 2:

• Encapsulation (ESP or AH): ESP
• Encryption Algorithm: AES-256
• Authentication Algorithm: SHA-1
• Perfect Forward Secrecy: No
• Lifetime: 3600s (1 hour)

Let's to go over these requirements and try to make sense of them.

Phase 1:

• Authentication Method: RSA-Key. There are several possible authentication methods, such as pre-shared key or RSA keys (or, rather, X509 certificates). We are going to use certificates.
• Diffie-Hellman Group: 5 (1536 bit). Diffie-Hellman algorithm is used to exchange session keys between the IPsec peers. The "group" is one the algorithm's parameters. The important thing is that both peers use the same value.
• Encryption Algorithm: AES-256. This just means that out of the multitude of encryption algorithms they want to use AES with 256-bit long keys.
• Data Integrity Algorithm: SHA-1. Similarly this just means that of all the hash algorithms they want to use SHA-1.
• Use aggressive mode: No. Aggressive mode is an optional feature of Phase 1 negotiation that "reduces the number of round-trips at the expense of not providing identity protection" (RFC2408). They don't want to use it. I guess they care about identity protection.
• Lifetime: 86400s (1 day). This means that the ISAKMP SA that we establish at Phase 1 will have lifetime of 1 day, and after that will have to be renegotiated.

Phase 2:

• Encapsulation (ESP or AH): ESP. IPsec uses two protocols, ESP and AH. For some reason, they only want to use ESP (this is why when setting up kernel security policy above I told it to use ESP only).
• Encryption Algorithm: AES-256. Just using the same encryption algorithm as in Phase 1.
• Authentication Algorithm: SHA-1. Just using the same hash algorithm as in Phase 1.
• Perfect Forward Secrecy: No. Perfect Forward Secrecy is an optional feature of Phase 2 negotiation, which means that "disclosure of longterm secret keying material [RSA private key in our case] does not compromise the secrecy of the exchanged keys from previous communications". For whatever reason, they don't want to use it.
• Lifetime: 3600s (1 hour). This means that the IPsec SA that we establish at Phase 2 will have lifetime of 1 hour, and after that will have to be renegotiated, so that if anyone is eavesdropping on us, they won't have much time to brute-force our keys.

The above requirements translate into the following racoon config that can be added to /etc/racoon/racoon.conf. The comments in the config below explain how config directives correspond to the requirements.

# The following section tells racoon how to conduct Phase 1 
# negotiation to the remote peer 194.1.1.1
remote 194.1.1.1
{
  # We are NOT using aggressive mode, therefore the mode is "main"
  exchange_mode main;
  # The peer identifies themselves by their IP address. See below in "Pitfalls" 
  # section on why this directive was necessary.
  peers_identifier address;
  # We will not verify the peer's certificate. See below in "Pitfalls" 
  # section on why this directive was necessary.
  verify_cert off;
  # The following line specifies the location of our certificate and private key
  certificate_type x509 "ipsec.cer" "ipsec.key";

  proposal {
    # for encryption algorithm we will use AES 256
    encryption_algorithm aes 256;
    # for hash algorithm we will use SHA1
    hash_algorithm sha1;
    # we will authenticate with X509 certificate (called rsasig here)
    authentication_method rsasig;
    # We will use Diffie-Hellman group 5
    dh_group 5;
    # Lifetime of phase 1 association will be 24 hours
    lifetime time 86400 sec;
  }
}

# The following section tells racoon how to conduct Phase 2
# negotiation for packets exchanged between subnets 172.16.1.1/32 and
# 194.1.1.2/32. Notice that here we use "internal" source and destination
# subnet addresses, and NOT tunnel endpoint addresses.
sainfo address 172.16.1.1/32 any address 194.1.1.2/32 any
{
  # Lifetime of Phase 2 association will be 1 hour
  lifetime time 1 hour ;
  # for encryption algorithm we will use AES 256
  encryption_algorithm aes 256 ;
  # for authentication algorithm we will use SHA1
  authentication_algorithm hmac_sha1;
  # For compression algorithm we will use "deflate". This wasn't in the specs,
  # but it's the default
  compression_algorithm deflate ;
}

Pitfalls

IPsec is a complex protocol and many are the pitfalls on the road to a successful IPsec tunnel.

Parameters

Parameters offered by the initiator side (such as encryption algorithms, lifetimes, Diffie-Hellman group, etc.) must be exactly the same as expected by the remote side. In theory ISAKMP is supposed to negotiate the parameters, but in practice I found that even the slightest difference led to failed negotiation. The problem is aggravated by the fact that different implementation of IPsec (in our case that of Linux and Cisco PIX) use different configuration syntax and slightly different terminology, so that translating the requirements of Cisco PIX into the configuration language of racoon is not always straightforward.

Identification

In IPsec each party identifies itself to the other party and presents credentials to prove their identity. The protocol allows for several forms of identification, such as:

• By IP address (i.e. a peer says "I am the owner of this IP address")
• By DNS name (i.e. a peer says "I am the owner of this DNS name")
• By distinguished name from an X509 certificate (i.e. a peer says "My DN is as follows")

etc. When you configure a tunnel, you need to specify which form of identification you use. When your credential is an X509 certificate, the best choice is identification by distinguished name, because the certificate is exactly the kind of credential that proves that the DN belongs to you. This form of identification is achieved by the following line in racoon configuration:

my_identifier asn1dn; 

However, our customer configured its Cisco to identify itself by IP address. How does one prove with an X509 certificate that the IP address is his? Well, racoon expects that the certificate in this case will have an subjectAltName attribute with the value "IP:xx.xx.xx.xx" (where xx.xx.xx.xx is the peer's IP address). Unfortunately, our customer's certificate did not have such attribute. To work around, I had to turn off the certificate validation with "verify_cert off;" directive.

Tunnel stability

Unfortunately, the tunnel proved to be unstable. Approximately once in several days the remote end would terminate the ISAKMP association. This is not a problem in itself, since racoon is capable of re-negotiating the association when needed. However, apparently the remote end also terminates the IPsec association, but racoon does not notice this; thus it continues to use the IPsec association that the remote end no longer accepts and no data comes through.

In about an hour's time racoon will expire the IPsec connection and re-negotiation a new one, so the tunnel will be re-established. If you can't wait, you can force the renegotiation. One way to do it is to delete all security policy entries with setkey -F command. Another is to remove the IPsec association with racoonctl command (see manpage for details).