Category: DevOps

Note: All IP addresses in this article have been modified to protect client privacy

When managing virtual machines across different subnets in a node with Virtualizor environment, you might encounter networking challenges that aren’t immediately obvious. I recently tackled such an issue in a Hetzner where VMs with IPs from different subnets couldn’t communicate with each other.

The Problem

Here was my setup:

• A node Virtualizor with a main IP address
• Three different subnets allocated:
  • 192.168.1.0/29 (connected to natbr8)
  • 10.10.20.0/29 (connected to natbr7)
  • 172.16.5.0/29 (connected to natbr5)
• VMs with IPs from different subnets (specifically 10.10.20.6 and 192.168.1.4) couldn’t talk to each other

While some might suggest that Hetzner blocks cross-subnet communication by default, the reality is more nuanced. Hetzner doesn’t inherently block such communication – the issue is that proper routing configuration is needed to enable it.

The Solution

After troubleshooting, I found that solving this problem required configuring several networking components:

1. Host-level NAT and Forwarding Rules

First, I needed proper NAT masquerading for traffic between subnets:

# NAT rules for cross-subnet communication
iptables -t nat -A POSTROUTING -s 192.168.1.0/29 -d 10.10.20.0/29 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 10.10.20.0/29 -d 192.168.1.0/29 -j MASQUERADE

Then I needed to ensure packet forwarding between network bridges:

# Allow forwarding between bridges
iptables -A FORWARD -i natbr7 -o natbr8 -j ACCEPT
iptables -A FORWARD -i natbr8 -o natbr7 -j ACCEPTiptables -A FORWARD -o natbr7 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -o natbr8 -m state --state RELATED,ESTABLISHED -j ACCEPT

2. IP Routing Rules

Special routing rules were needed to correctly handle traffic between the specific VMs:

# Add routing rules for specific VMs
ip rule add from 10.10.20.6 to 192.168.1.4
ip rule add from 192.168.1.4 to 10.10.20.6

3. VM-level Configuration

Inside the VM with IP 192.168.1.4, I added a specific route:

ip route add 10.10.20.6 via 192.168.1.1 dev eth0

The VM with IP 10.10.20.6 already had appropriate routing via its default gateway:

192.168.1.0/29 via 10.10.20.1 dev eth0

How It Works

With this configuration, here’s how traffic flows:

1. When VM 192.168.1.4 sends a packet to VM 10.10.20.6:
   • The packet gets routed through gateway 192.168.1.1
   • The host applies NAT masquerading
   • The packet is forwarded from natbr8 to natbr7
   • The packet arrives at VM 10.10.20.6
2. When VM 10.10.20.6 sends a packet to VM 192.168.1.4:
   • The packet gets routed through gateway 10.10.20.1
   • The host applies NAT masquerading
   • The packet is forwarded from natbr7 to natbr8
   • The packet arrives at VM 192.168.1.4

Lessons Learned

This experience taught me several important things about cloud networking:

1. Provider policies aren’t always the culprit – While some cloud providers do restrict cross-subnet communication, often the issue is just proper configuration.
2. Layer by layer troubleshooting is essential – Working through each networking layer (VM routes, host forwarding, NAT, etc.) methodically led to the solution.
3. VM-level routing matters – Even with correct host configuration, each VM needs to know how to route packets to other subnets.
4. Documentation is crucial – After fixing the issue, documenting the solution thoroughly saved time when I needed to replicate or modify the setup later.

For anyone facing similar issues in Hetzner or other cloud environments, I recommend examining your routing tables, NAT rules, and forwarding configurations at both the host and VM levels. The problem is almost always solvable with the right networking configuration.

Do you face similar networking challenges in your infrastructure?