Double Firewall Hopping with PfSense

[trobinson667]

Double Firewall Hopping with PfSense

Greetings, everyone. I’ve recently had to do some housekeeping in my lab environment as I prepare to expand, and write up some additional chapters for Suricata: An Operator’s Guide. My lab environment is kind of sprawling across my home network with little enclaves everywhere, but I figured I would write about my malware detonation lab, and IoT containment zone (if I ever get IoT devices to mess around with), and some new discoveries I made, trying to nest one firewall behind another. The goal of this blog post is to show you how to route through two pfSense firewalls to go from 10.0.0.16 (shadowchamber) to the host hard-light.kaidacorp.local at 172.16.50.2. Please bear in mind that this blog post is pfSense-centric. I would hazard to guess that most of these config settings are applicable to Opnsense, but I’m not really an Opnsense fan, so I don’t use it often. Your mileage may vary. For the visual learners out there, here’s a simplified network diagram made with draw.io:

This is a simplified representation of what my home network, and one of my many homelab enclaves looks like. The dotted green line from 10.0.0.16 to 172.16.50.2 depicts our source, and our destination. For this demonstration

Malware Lab Network (so far)

Our local, physical network being served by a comcast router’s DHCP server (10.0.0.0/24). My local box is a Windows 10 machine at 10.0.0.16. My physical pfSense Gateway is at 10.0.0.253 (armageddonheart.kaidacorp.local). 10.0.0.253 is the IP address of the WAN interface. This pfSense box, a relatively cheap no-name ali-express box provides DNS (with pfblockerNG), NTP, DHCP (to LAN, OPT1 and OPT2 networks) and also serves to segment my home network from my “malware analysis” network, and all of its supporting infrastructure. I have three subnets that the pfSense router brokers access to: 10.1.0.0/24 (LAN), 10.2.0.0/24 (OPT1), and 10.3.0.0/24 (OPT2).

10.1.0.0/24 hosts two relatively cheap ali-express physical machines that I use occasionally for malware detonation. 10.2.0.0/24 is a “supporting infrastructure” network that hosts a FOG (free open ghost) server, as well as one of my three Proxmox systems, (nemesis.kaidacorp.local, 10.2.0.2), which is the hardware running the FOG VM (onslaught.kaidacorp.local, 10.2.0.3). The FOG VM hosts images of both the Physical Linux (Ubuntu 24.04 – Liandry) and Windows (Windows 11 – Berserk) system. Both systems have packet capture software, (tshark, tcpdump), and mitmproxy running, with SSL Key Log File generation enabled. I have some customizations in place on my Windows and Linux box, but that’s a story for another time. The baseline images are copied via PXE booting, and fog’s PXE agents. At any time, I can force either physical machine to boot from PXE and grab a clean image after I’m done running malware. I can also restore the “clean” image, update it, and collect a new image at any time as well. It was a little difficult to figure out how to get PXE booting to work across network segments, but with correct DHCP options, and some configurations to change where NFS is binding, and ensuring the boxes can reach the NFS share where the images are at, I managed to make things work.

That leaves 10.3.0.0/24. I have a gl-iNet box (redline.kaidacorp.local) acting in bridge mode behind the pfSense firewall. The device is configured to get its DNS there, and pfblockerNG is in full effect. This is the wi-fi network where I plan to do weird IoT things in the future, but also, I connect my phone to this wi-fi network because DNS-based blocking is extremely effective for Android-based devices. I like reaping those benefits, but I’m not necessarily ready to make every machine on the 10.0.0.0/24 network (that aren’t depicted) do outbound DNS blocking, as it could break a lot of our IoT and smart devices.

Network Expansions

I’m in the process of expanding 10.2.0.0/24 to contain another subnet through a pfSense VM hosted on nemesis (anh-5.kaidacorp.local, 10.2.0.4 -- WAN interface). the LAN interface (172.16.50.1) provides access 172.16.50.0/24. I’m planning on adding a Windows VM, and three Linux VMs. Two of the Linux VMs and the Windows VM will be to use virtual machines for malware analysis, and exploit dev (one exploit “pitcher” and two “catcher” VMs – One Linux, One Windows), while the final Linux VM will host polarproxy for demonstrating TLS/SSL decryption.

I have a bunch of little enclaves very similar to this all over my home network with different VMs, doing different things, so I’m very familiar with route -p add x.x.x.x mask x.x.x.x [gateway IP] that can be used in Windows to persist static routes. In order to add routes in Windows, you’ll need a command prompt that is UAC elevated (“Run as Administrator”). As for Linux, there are a multitude of ways to do this, depending on your distro and what network management suite they use if they’re too good for /etc/network/interfaces and post-up. So far, I’ve had to work with Raspbians udhcpd and ubuntu’s netplan. In most cases, this results in dropping the ip route add x.x.x.x/xx via x.x.x.x [dev xxx] command into the correct config file somewhere. Note that in most cases with Linux, running ip route add, or adding the ip route add command to these most network configuration files typically requires root, su or sudo for root privileges to modify the routing table or files necessary.

Why does any of this Matter? Static Route Limitations

The whole reason I’m bringing any of this up, is that it wasn’t until recently that I realized that static routes have a pretty severe limitation: Without invoking things that shouldn’t be invoked, you cannot and should not try to nest static routes. In most cases, it simply won’t work. Let me give you an example. My ultimate goal is to reach 172.16.50.2. I need to hop through the gateway at 10.0.0.253, then the second gateway at 10.2.0.4 to get there. I want to reach 10.2.0.0/24 via 10.0.0.253 in Windows, and I want this to be a permanent static route, because my default gateway is a comcast gateway that can’t do routing protocols. so I run:

route -p add 10.2.0.0 mask 255.255.255.0 10.0.0.253

No problemo. Here’s your persistent route.

Fun fact: Windows stores route -p routes in the registry for persistence:

Any ipv4 routes persisted in Windows via route -p add are stored in HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\PersistentRoutes as REG_SZ strings. Know that persistent routes can be added manually without the use of the route command.

So we add one static route, but in order to reach 172.16.50.0/24, I have to go through 10.2.0.4 as the gateway to that network. You’d normally think that you’d just do route -p add 172.16.50.0 mask 255.255.255.0 10.2.0.4, or ip route add 172.16.50.0/24 via 10.2.0.4 and that you’d be off and running to the races. But unfortunately, that’s not correct. Static routes can only point to the next network via a direct hop or a direct route to a device on the same subnet as one of your network interfaces unless you want to learn how to do proxy arp. I did not want to learn how to do this, so here’s the solution: You’d have to establish that static route to 172.16.50.0/24 via the firewall/router at 10.2.0.4 on the first-hop router/firewall (in this case, 10.0.0.253) itself.

pfSense : Gateways and Static Route Configuration

pfSense has a weird way of establishing static routes, requiring you to define a gateway and a route separately. Here’s how to do it:

Log in to the first pfSense router in the path to your destination (again, in this case, its armageddonheart.kaidacorp.local (10.0.0.253), and once logged in, navigate to System > Routing. The default tab you should end up on is the gateways page. Assume that the anh5_lan entry isn’t there, and click the Add button.

Head over to System > Routing. The default tab that should be displayed is the Gateways tab. So if that isn’t your landing tab, click on it. Click the Add button.

Next, we have to create a gateway. On this page, you’ll want to fill out the Interface, Address Family, Name, Gateway, and Description input boxes. In my case here is what I entered:

• Interface: OPT 1 – the 10.2.0.0/24 network is allocated to the OPT1 interface.
• Address Family: IPv4
• Name: anh5_lan – The name doesn’t matter, pick something that defines what this gateway is for.
• Gateway: 10.2.0.4 – This is the IP address on the OPT1 network thats serves as the gateway for the destination we want to reach.
• Description: Gateway to reach ANH-5 firewall’s LAN networks (172.16.50.0/24) – something to remind you as to why you created this gateway in the first place. Be detailed.

Click Save at the bottom of the page, followed by Apply Settings at the top of the Gateways tab that you’re automatically redirected to.

Pay attention to the fields with red boxes around them. They include: the Interface, Address Family, Name, Gateway, and Description input boxes. At a minimum, I recommend filling all of those boxes out. As for the other settings? You can investigate those on your own if you’d like. My route worked without fiddling with them.

Next, we have to go to the Static Routes tab. By default, the page will be empty. Click the Add button.

Click on the Static Routes tab. Ignore the fact that the route I need is already here, and click the Add button.

On the Edit Route Entry page, All of the drop-downs and input boxes need to be filled out. Here is what I had to enter for my use case:

• Destination Network: 172.16.50.0. and in the subnet mask drop-down: /24.
• Gateway: anh5_LAN - 10.2.0.4 – Should reflect the name and the IP address of the gateway you just made.
• Description: anh5_lan – a brief description of the purpose this route serves.

Do not tamper with the Disable this static route checkbox. It should be unchecked by default. If for some reason it is checked, make sure it is unchecked. With all that out of the way, click the Save button, and back on the Static Routes tab, click Apply Changes at the top of the page.

I’m not bothering with red boxes here. All of the input boxes and drop-downs need to be filled out. The only thing that should NOT be touched is the Disable this static route checkbox. It should be unchecked. If for any reason it’s checked, make sure to uncheck it. Save and Apply Settings. Almost done with routing.

Establishing a New Route on your Windows/Linux host

The next step you’ll need to do, is create another static route. In my case, I need to use the static route on 10.0.0.253 for me to ultimately reach 172.16.50.0/24, so this is what my route -p add command looked like for windows:

route -p add 172.16.50.0 mask 255.255.255.0 10.0.0.253

For linux users, you’ll be using the ip route command:

ip route add 172.16.50.0/24 via 10.0.0.253 (Note: you might need to add dev [interface name] to the end of this command to ensure the system doesn’t assign the route to the wrong network interface (e.g. ip route add 172.16.50.0/24 via 10.0.0.253 dev ens18))

If you still have the OLD static route to 172.16.50.0/24 (or whatever your target network is) via 10.2.0.4 (or whatever the gateway device’s IP is), you’ll need to delete that route first before you add the correct route.

• Windows: route -p delete 172.16.50.0
• Linux: ip route delete 172.16.50.0/24

If you haven’t already, delete the incorrect static route, THEN create the correct static route. This output is from the route print command on Windows, showing a route to 172.16.50.0/24 via 10.0.0.253. The correct route, based on the gateway and static route we just made on pfSense.

Firewall Adjustments

I want to be able to SSH to the VM hard-light.kaidacorp.local at 172.16.50.2. In order to do that, I need to make sure I have firewall rules configured in two places: The “WAN” interface on the 10.0.0.253 pfSense firewall (armageddonheart.kaidacorp.local), and on the “WAN” interface of the 10.2.0.4 pfSense firewall (anh-5.kaidacorp.local).

I’m not going to give you a full primer on how to add firewall rules on pfSense however I’m going to give you a couple of notes:

• Under Interfaces > WAN scroll all the way down and uncheck the Block private networks and loopback addresses check box at the bottom of the page. As always, click Save, then Apply Changes. You’ll need to do this for both firewalls.

This setting must be unchecked on both pfSense firewalls. If you’re using if you are using other firewall distributions or appliances that are not pfSense, you might need to hunt through the interface or default firewall rules to see if this is a configuration setting that can be modified. Chances are if you’re using a routing device without firewall software or access control lists, this is something you don’t have to care about.

How do I reach the pfSense web interface from the WAN interface? Normally you can’t do that. access the console of the pfsense device. selection option 8 to open up a root shell. run the command pfctl -d to temporarily disable the firewall. Create a firewall rule on the WAN interface to allow access to the WAN ip address on port 443 (https) from your workstation’s IP address (or host alias). And while you’re there, disable this checkbox. If at any time you lose connectivity to the web interface before you can put these changes into place, run pfctl -d on the console again to temporarily disable the pfSense firewall. Note that every time you Apply Changes to the firewall, pfSense reactivates the firewall, requiring you to re-run pfctl -d on the console if you’re not done.

• Make use of Firewall > Aliases and set up an alias for system(s) you want to have access to your lab network(s). Do this for both firewalls you’re creating firewall rules for. In this case, I set up a host(s) alias on both of my firewalls named “admin systems”. For now, it only consists of my Windows box at 10.0.0.16 (shadowchamber), but if you’re sharing your lab environment, or want to be able to access it from multiple hosts (e.g. you have to share your VM environment, or maybe you want both your local workstation and/or a jump box to have direct network access). It makes it much easier for granting lab network access to additional hosts.

pfSense has the concept of Aliases under the Firewall > Aliases page. I recommend creating a host(s) alias containing IP addresses of hosts that will need access to the lab environment. Using an alias for multiple IP addresses/hostnames will make rule management much easier. Instead of having to create new rules to allow a new IP addresses into the lab network(s), just modify the aliases for existing rules. Here is my admin_systems host alias that I am using on both firewalls.

• You’ll need to create two firewall rules – one on each firewall you’re using to reach the target host on the target network. Remember that my two firewalls are at 10.0.0.253 (armageddonheart.kaidacorp.local), and 10.2.0.4 (anh-5.kaidacorp.local).
• On the WAN interface of armageddonheart (10.0.0.253), create a firewall rule where the source is the admin_systems alias, and the destination is 172.16.50.0/24 on port 22/tcp, ipv4.

This is the firewall rule on the WAN (10.0.0.253) Interface on armageddonheart.kaidacorp.local. It allows admin_systems alias (currently just 10.0.0.16) to access 172.16.50.0/24 on port 22/tcp over ipv4.

• On the WAN interface of anh-5 (10.2.0.4), create an identical firewall rule where the source is the admin_systems alias, and the destination can either be the LAN subnets pre-built alias, or you can input a network (e.g. 172.16.50.0/24) on port 22/tcp, ipv4.

This is the firewall rule on the WAN (10.2.0.4) Interface on anh-5.kaidacorp.local. Once again, it allows the admin_systems alias (just 10.0.0.16 currently) to access the LAN subnets pre-built alias on port 22/tcp over ipv4. Note that you can substitute the LAN subnets alias for the subnet that needs to be reached (e.g., 172.16.50.0/24)

Results

Boom SSH connectivity. I’m lazy as can be, so I recommend that once SSH connectivity is established, set up key-based auth, and once you confirm key-based auth has been achieved, reconfigure /etc/ssh/sshd_config. Set PasswordAuthentication to no, add the AuthenticationMethods directive to your configuration, and set it to just publickey. Restart sshd, reconnect. Now your SSH server will only accept key-based auth.

And here we are. Access to 172.16.50.2 has been established, and it only took a mountain of firewall and routing work to get here.

If you’re brave and you want to set up something like OSPF, you’ll need the FRR package, and to find a tutorial to set all that up because I’m out. Maybe if I set the comcast router to bridge mode, and get another pfSense firewall for my local network, I’ll consider it, but for now, persistent static routes it is.

Troubleshooting

• Establish where you are trying to go, and all the hops needed to get there. For example: I’m starting at 10.0.0.16 (10.0.0.0/24). I want to reach 172.16.50.2 (172.16.50/24). I have to go through 10.0.0.253 as the first-hop, then 10.2.0.4 as the second-hop to get there.
• You’ll need a minimum of three static routes to get here:
  • Two routes on the windows host:
  • route -p add 10.2.0.0 mask 255.255.255.0 10.0.0.253 or ip route add 10.2.0.0/24 via 10.0.0.253
  • route -p add 172.16.50.0 mask 255.255.255.0 10.0.0.253 or ip route add 172.16.50.0/24 via 10.0.0.253
• A correctly defined gateway and static route on the first firewall (10.0.0.253)
  • The gateway must define the gateway IP (10.2.0.4) address and correct interface on the first firewall (OPT1) necessary to reach the second firewall.
  • The static route must define the gateway to be used (10.2.0.4), and the correct destination network on the second firewall (172.16.50.0/24)
• Ensure the Block private networks and loopback addresses checkbox under Interfaces > WAN on both firewalls is unchecked
• Double check the source and dest IP addresses, networks and/or aliases, protocols, port numbers, and address families used in the firewall rules on both firewalls are correct
  • Firewall 1 – Interface: WAN source: admin_systems/10.0.0.16 dest: 172.16.50.0/24 protocol: tcp port: 22 address family: ipv4
  • Firewall 2 – Interface: WAN source: admin_systems/10.0.0/16 dest: LAN subnets/172.16.50.0/24 tcp port: 22 address family: ipv4
• Verify the sshd service is installed and enabled (ss -alnt). Double check the target IP address (ip -br a – 172.16.50.2)
• If you have any reason to suspect them, check to make sure that there are not any host-based firewall rules on either the source host (10.0.0.16) or the destination host (172.16.50.2) that may be blocking the connection
• Verify all systems are powered on, and fully configured
• Verify physical network connectivity (network cables, NICs, etc.)