Where to terminate Site-to-Site VPN Tunnels?

When using a multilayer firewall design it is not directly clear on which of these firewalls remote site-to-site VPNs should terminate. What must be considered in such scenarios? Differentiate between partners and own remote offices? Or between static and dynamic peer IPs? What about the default routes on the remote sites?
Following is a discussion about different approaches and some best practices. Since not all concepts work with all firewall vendors, the following strategies are separated by common firewalls, i.e., Cisco ASA, Fortinet FortiGate, Juniper ScreenOS, Palo Alto.

(This is one of many VPN tutorials on my blog. Have a look at this full list.)
Of course, if there is only a single firewall in place, this discussion is not necessary at all. All VPN tunnels must solely terminate on this single firewall. You’re done. But most customers have at least a two-firewall strategy which is not a bad idea at all. While the first firewall is merely for stopping all unused IP connections and for allowing connections to the DMZ, the second firewall has next-gen features such as APT scanning, URL filtering, user recognition, etc. Normally, both firewalls have a default route directed to the Internet (if no proxies are used).

Who has a Static S2S Tunnel?

• Remote Offices: Locations that are owned by the same company. Mostly coming from static IP addresses. If the remote office has no local Internet breakout, it has a default route back to the headquarter. That is: all Internet traffic must be processed by the second, next-generation firewall (or web-proxy). The network behind this remote office can be considered as internal, but on another location. It should be treated in the same way os other internal traffic.
  
• Home Offices: Users that are working from home. Normally coming from dynamic IP addresses. No local Internet breakout -> all traffic must traverse through the second NGFW, too. (I.e., same as “remote office”, but from a dynamic IP address.)
• Partners: Other companies that must access certain services in the DMZ (such as servers or proxies). Almost coming from static IP addresses. No routing/ACLs for accessing the internal networks are required. That is: This traffic must not go through the second firewall.

The Problems

1. The first firewall has a default route to the Internet. Otherwise, no connections could be made from/to the firewall at all.
2. But for the remote- and home-offices, the traffic coming out of the VPN-interface need a default route to the second firewall (and not directly to the Internet). If all VPNs are terminated on the first firewall, this is not the case. -> The best approach is to have an own routing instance for the tunnel-interfaces with a default route to the second firewall.
3. Only a few firewall appliances implement the concept of “virtual routers” (Juniper ScreenOS, Palo Alto). For the FortiGate, policy-based forwarding can be used. For the Cisco ASA, none of these concepts work. Only a workaround can be used there (if it is not an option to buy a better firewall).

(Note that for some remote access VPN solutions, it is default to have two virtual routers / routing tables in the box. This is true, for example, for the Pulse Secure, formerly Juniper SA/MAG. With these devices, traffic coming out from the VPN has another default route than Internet traffic with its default route to the Internet.)
All following concepts describe the case in which the first firewall is from vendor X. It is not related to the second firewall.

Concept 1: Two Virtual Routers on the first firewall (Juniper, Palo Alto)

Both firewall vendors (Juniper ScreenOS and Palo Alto) have virtual routers implemented. That means, the “tunnel-interface” for the VPN can be on another virtual router with another default route. While the default virtual router can point to the Internet (for all outgoing connections and for terminating the VPN), the second virtual router (with the tunnel-interface in it) can point to the second firewall.

Concept 2: Policy-Based Routing/Forwarding (FortiGate)

Unluckily, the FortiGate has not virtual routers, but only virtual domains (vdoms). This is great for separating firewall instances, but does not solve the VPN problem here. However, it is possible to configure a policy route: All traffic coming out of the tunnel-interface has a “new” default route to the second firewall.

Concept 3: Partners on First, Remote Office on Second Firewall (Cisco ASA)

Cisco ASA has neither virtual routers nor policy-based routing. The only “concept” is to terminate the partners on the Cisco ASA and to allow the remote offices to access the second firewall for VPN termination. That is, the static IP addresses from the remote offices must be allowed in an ACL on the Cisco ASA to reach the second firewall directly. Note that it is NOT recommended to allow “from any” to the second firewall, which means that VPN connections from home offices (coming from dynamic IP addresses) will not work in this scenario. That’s definitely a major drawback.