I’ve been having a problem for a while now: my Android Ice Cream Sandwich (ICS) devices were unable to connect to my L2TP/IPSec PSK VPN. This happened on both my Asus Transformer Prime (TF201) running ICS 4.0.3 and my shiny new Samsung Galaxy S3 running ICS 4.0.4

In order to do my job, I need to connect to my VPN in order to have a secure connection to the LAN, internal servers and my desktop computer. So this problem was major. Yesterday night, I finally took a few minutes to diagnose the problem.

My VPN server is running on Ubuntu 10.04 LTS and is setup using OpenSwan (l2tpd & pluto ipsec). I checked the /var/log/pluto.log while trying to connect my Android device to the VPN, and noticed these errors:

"L2TP-PSK-NAT"[16] 11.22.33.44#15: byte 7 of ISAKMP NAT-OA Payload must be zero, but is not
"L2TP-PSK-NAT"[16] 11.22.33.44#15: malformed payload in packet
| payload malformed after IV
| 7c 98 58 d1 bd 64 bd 43 6f c3 5d 7c 19 e3 23 ef
"L2TP-PSK-NAT"[16] 11.22.33.44#15: sending notification PAYLOAD_MALFORMED to 11.22.33.44:4500


After a quick Google Search on “ics l2tp ipsec vpn malformed payload in packet”, I found the following bug report on Google Code: Issue 23124: Can’t connect to VPN (nexus s – ice cream sandwich).
 

Comment 203 from strawman is particularly interesting as the person identified the bug and provides a fix. Basically, Google is not following the RFC by the book and OpenSwan doesn’t tolerate this error. The user fixed the building of the ISAKMP NAT-OA packet (byte 7) in /system/bin/racoon and uploaded his patch.

This involves copying the “racoon” binary onto your Android system, so you will need root to do so:

1. Download the new “racoon” binary from Comment 203 in the Google Code link above
2. Copy the “racoon” binary onto your Android device’s Internal SD-Card (I will assume this is /mnt/sdcard/ in these instructions)
3. Open a Terminal or an ADB Shell session
4. Get superuser (root) access:
   su
5. Mount the /system filesystem as RW, you need root for this:
   mount -o remount,rw /system
6. Make a backup of the existing “racoon” binary:
   cd /system/bin
cp racoon racoon.original
7. Copy the new “racoon” in /system/bin:
   cp /mnt/sdcard/racoon .
8. Set the proper permissions on the new “racoon”:
   chmod 755 racoon
chgrp shell racoon
9. Remount the /system filesystem as RO:
   mount -o remount,ro /system

That’s it, you are all done. Now, try connecting to your L2TP/IPSec PSK VPN with your Android device again, and within a few seconds, you should finally be able to connect successfully.

In case something goes wrong, the patch doesn’t work, or if a future OTA upgrade fails, just remember to delete the “new” racoon and rename the original one back to its proper name.

P.S.: All credit goes to strawman on Comment 203. The new “racoon” binary can be downloaded from the Google Code page linked above.