Background
I have several physical locations linked together with VPN tunnels. The
central VPN server runs OpenBSD with iked(8). I also have several
roaming clients (iOS and macOS) that terminate client access tunnels to
this system so I am loathe to make breaking changes to it. The site to
site tunnels run a gif(8) tunnel in IP-over-IP mode to provide a
layer 3 routable interface on top of the IKEv2 tunnel. My internal tunnels
run ospfd(8) and ospf6d(8) to exchange routes and my
external site to site tunnels run bgpd(8). Most of my internal
sites use OpenBSD as endpoints so configuration is painfully simple,
however in my office at work I have been using a MikroTik RouterBoard
RB951-2HnD. This has worked well enough but lately it has been showing
its age, randomly requiring manual intervention to re-establish tunnels and
flirting with periods of unexplainable high latency.
Notes
This is not meant to be a comprehensive HOWTO. I doubt your setup will be close enough to mine to translate directly but hopefully you will find some useful information since this isn't a particularly well documented use case for the Ubiquiti UniFi USG product line.
It is also worth noting that under the covers the USG runs the same EdgeOS as their EdgeRouter line of products with the caveat that the controller will overwrite the configuration any time it provisions the device. Fortunately Ubiquiti has foreseen this and provides a way to provide advanced configuration via a JSON file on the controller.
I manage all of my sites from a centralized UniFi controller instance, so I need the VPN to work before I can swap out the RouterBoard for the USG. This is an overview of how I did that.
Pre-Deployment
Since I already have a working VPN tunnel at the site I already had all the X.509 certificates and IP addresses needed to configure the new router. Starting at home, where the controller is located I plugged in the USG WAN port to my LAN and connected my laptop to the USG LAN port. I was able to adopt the gateway into the controller with no trouble.
I fiddled around with the config until I got it working and stuffed the changes into the config.gateway.json file. Finally I blew the device away and forgot it from the controller. It is important at this point to reload the certificates into the factory defaulted router (put them in /config/auth) before adopting the gateway in the controller. The gateway will go into a reboot loop much the same way as if you typo-ed the config.gateway.json file if it cannot find the files. Once the certificates were loaded, I re-adopted the gateway and the configuration was applied.
I was then able to take it into work and swap the MicroTik.
Configuration
I will simply annotate the config.gateway.json file inline to explain how this all ended up going together.
{
"service": {
"dns": {
"forwarding": {
"options": [
"domain=work.ub3rgeek.net"
]
}
},
Set the DNS domain name handed out by the gateway, not strictly needed in this context, but handy.
"nat": {
"rule": {
"6004": {
"description": "VPN Link Local NAT",
"destination": {
"address": "!172.16.0.0/15"
},
"log": "disable",
"outbound-interface": "tun0",
"outside-address": {
"address": "192.168.197.97"
},
"source": {
"address": "172.16.0.0/15"
},
"type": "source"
}
}
}
},
NAT any traffic coming from the tunnel or IPSec endpoint addresses to the canonical address of the router. This prevents local daemons from selecting the wrong source IP (most frequently done by syslogd).
"interfaces": {
"loopback": {
"lo": {
"address": [
"172.16.197.96/32"
]
}
},
This is the IPSec endpoint, I use policy based IPSec so this needs to exist somewhere so the traffic can get picked up by the kernel and sent across the tunnel.
"tunnel": {
"tun0": {
"address": [
"172.17.197.198/30"
],
"description": "ub3rgeek vpn",
"encapsulation": "ipip",
"ip": {
"ospf": {
"network": "point-to-point"
}
},
"local-ip": "172.16.197.96",
"mtu": "1420",
"multicast": "enable",
"remote-ip": "172.16.197.32",
"ttl": "255"
}
}
},
This sets up the IP-over-IP tunnel. Note I could not get the OSPF session to come up for the life of me using my normal /32 addressed tunnel so I switched to a /30. After that OSPF came right up. If you debug ospf events and get complaints that the peer address of tun0 is not an ospf address, then you might be hitting this too.
"protocols": {
"ospf": {
"area": {
"0.0.0.0": {
"network": [
"192.168.197.96/28",
"172.17.197.196/30",
"10.10.10.0/24"
]
}
},
"parameters": {
"abr-type": "cisco",
"router-id": "192.168.197.97"
},
"passive-interface": [
"eth0",
"eth1"
]
}
},
This is rather straightforward, I'm redistributing the local networks and the tunnel address. This is a pretty simple OSPF configuration. Since I have no routers on the Ethernet end of things I set both interfaces to passive.
"vpn": {
"ipsec": {
"auto-firewall-nat-exclude": "enable",
"esp-group": {
"ub3rgeek": {
"compression": "disable",
"lifetime": "3600",
"mode": "tunnel",
"pfs": "dh-group14",
"proposal": {
"1": {
"encryption": "aes256",
"hash": "sha256"
}
}
}
},
"ike-group": {
"ub3rgeek": {
"ikev2-reauth": "no",
"key-exchange": "ikev2",
"lifetime": "28800",
"proposal": {
"1": {
"dh-group": "14",
"encryption": "aes256",
"hash": "sha256"
}
}
}
},
"site-to-site": {
"peer": {
"69.55.65.182": {
"authentication": {
"id": "bdr01.work.ub3rgeek.net",
"mode": "x509",
"remote-id": "bdr01.colo.ub3rgeek.net",
"x509": {
"ca-cert-file": "/config/auth/ca.crt",
"cert-file": "/config/auth/server.crt",
"key": {
"file": "/config/auth/server.key"
}
}
},
"connection-type": "initiate",
"ike-group": "ub3rgeek",
"ikev2-reauth": "inherit",
"local-address": "default",
"tunnel": {
"0": {
"allow-nat-networks": "disable",
"allow-public-networks": "disable",
"esp-group": "ub3rgeek",
"local": {
"prefix": "172.16.197.96/32"
},
"protocol": "all",
"remote": {
"prefix": "172.16.197.32/32"
}
}
}
}
}
}
}
}
}
This is the real meat and potatoes of the configuration. It corresponds to the following configuration on the OpenBSD side of things.
#
# bdr01.work.ub3rgeek.net
#
ikev2 "work" passive esp \
from 172.16.197.96 to 172.16.197.96 \
peer $PEER_WORK \
ikesa enc aes-256 \
auth hmac-sha2-256 \
prf hmac-sha2-256 \
group modp2048 \
childsa enc aes-256 \
auth hmac-sha2-256 \
group modp2048 \
srcid bdr01.colo.ub3rgeek.net dstid bdr01.work.ub3rgeek.net \
lifetime 360m bytes 32g
Conclusion
In the end I am very happy about the whole thing. The USG is pretty slick and for simple configurations I imagine it is super easy to get going, and other than the lack of documentation for some of the things that aren't exposed in the controller UI it was not too hard to figure out. I would suggest if you are stuck trying to figure out the cli, you might want to explore the EdgeOS or Vyatta (the upstream Open Source project the EdgeOS is based on) documentation. I found those helpful.
🍺