More Ubiquiti USG VPN wonkyness_

March 11, 2020 @15:00

The Problem

So I came into the office this morning and noticed that my Ubiquiti USG-3 had upgraded itself from v4.4.44.5213844 to v4.4.50.5272448 and the VPN tunnel was down. I flailed at a few obvious things, reset the clock since it had lost connection to my NTP server, made sure the configuration didn't get wiped, made sure my certificates appeared to be in place. Everything checked out OK and the logs weren't showing anything so I went and cranked up the debug level in /etc/strongswan.d/charon-logging.conf changing the level of both ike and net to 2 and restarting the daemon with ipsec stop and ipsec start. After a bit I noticed the following in /var/log/charon.log:

Mar 11 10:02:15 16[IKE] <peer-[REDACTED]-tunnel-0|1> no trusted ECDSA public key found for '[REDACTED]'

The Theory

I'm not 100% sure what happened, but I have a theory after a couple hours of troubleshooting. I downgraded back to v4.4.44 and the tunnel still didn't work so I poked around more and noticed that my certificates were no longer validating when using openssl verify. My internal CA has two levels, the root signs a CA for general certificates and a separate CA only used for VPN authentication. This means that if someone gets access to a certificate for SSL use, they cannot actually use that to authenticate to my VPN. It turns out that when you issue the CLI command to set the CA certificate in the USG a script copies that file to /etc/ipsec.d/cacerts.

set vpn ipsec site-to-site peer [REDACTED] authentication x509 ca-cert-file /config/auth/ca.crt

I had flailed around in a previous post trying to fix a different issue and it looks like I changed the ca-cert-file several times. Since this doesn't clean up any of the previous files and strongswan(8) will load all the files it finds in /etc/ipsec.d/cacerts the tunnel was working fine.

The Fix

I looked around and the UniFi software provides no way to add multiple CA files in the configuration so I just manually copied my intermediate CA cert to /etc/ipsec.d/cacerts and restarted the tunnel with restart vpn from the CLI.

That worked.

So to validate my theory I then upgraded back up to v4.4.50 and sure enough it broke again because the upgrade blew away the contents of the filesystem and just copied only the root CA back as per the configuration. I was able to manually replace the intermediate CA again and restart the vpn and it came back up.

Conclusion

While my tunnel is back up, I now am painfully aware that the next software update will break it until I can put my intermediate CA certificate back into place. This isn't a particularly good position to be in with a remote site. Couple that with the fact that the strongSwan version is ancient (5.2.2 was released in 2015 and likely is missing a pile of bug fixes and security updates I cannot continue to recommend the USG platform. You don't lose much in the UniFi system by not having it and it is just too much risk to use at a remote site to make it worthwhile. I'll continue to use mine since I'm in the office most days but I'll probably go back to MikroTik for places I don't want to use OpenBSD.

Subscribe via RSS. Send me a comment.