The Problem
So I came into the office this morning and noticed that my Ubiquiti USG-3
had upgraded itself from
v4.4.44.5213844
to
v4.4.50.5272448
and the VPN tunnel was down. I flailed at a few obvious things, reset the
clock since it had lost connection to my NTP server, made sure the
configuration didn't get wiped, made sure my certificates appeared to be in
place. Everything checked out OK and the logs weren't showing anything so
I went and cranked up the debug level in /etc/strongswan.d/charon-logging.conf
changing the level of both ike and net to 2 and restarting the daemon with
ipsec stop and ipsec start. After a bit I noticed the following in
/var/log/charon.log:
Mar 11 10:02:15 16[IKE] <peer-[REDACTED]-tunnel-0|1> no trusted ECDSA public key found for '[REDACTED]'
The Theory
I'm not 100% sure what happened, but I have a theory after a couple hours of
troubleshooting. I downgraded back to v4.4.44 and the tunnel still didn't work
so I poked around more and noticed that my certificates were no longer
validating when using openssl verify. My internal CA has two levels, the
root signs a CA for general certificates and a separate CA only used for
VPN authentication. This means that if someone gets access to a certificate
for SSL use, they cannot actually use that to authenticate to my VPN.
It turns out that when you issue the CLI command to set the CA certificate in
the USG a script copies that file to /etc/ipsec.d/cacerts.
set vpn ipsec site-to-site peer [REDACTED] authentication x509 ca-cert-file /config/auth/ca.crt
I had flailed around in a previous post
trying to fix a different issue and it looks like I changed the
ca-cert-file several times. Since this doesn't clean up any
of the previous files and strongswan(8) will load all the files
it finds in /etc/ipsec.d/cacerts the tunnel was working fine.
The Fix
I looked around and the UniFi software provides no way to add
multiple CA files in the configuration so I just manually
copied my intermediate CA cert to /etc/ipsec.d/cacerts and
restarted the tunnel with restart vpn from the CLI.
That worked.
So to validate my theory I then upgraded back up to v4.4.50 and sure enough it broke again because the upgrade blew away the contents of the filesystem and just copied only the root CA back as per the configuration. I was able to manually replace the intermediate CA again and restart the vpn and it came back up.
Conclusion
While my tunnel is back up, I now am painfully aware that the next software update will break it until I can put my intermediate CA certificate back into place. This isn't a particularly good position to be in with a remote site. Couple that with the fact that the strongSwan version is ancient (5.2.2 was released in 2015 and likely is missing a pile of bug fixes and security updates I cannot continue to recommend the USG platform. You don't lose much in the UniFi system by not having it and it is just too much risk to use at a remote site to make it worthwhile. I'll continue to use mine since I'm in the office most days but I'll probably go back to MikroTik for places I don't want to use OpenBSD.