Daily Debugging: PVE Cross-VLAN Ping Fails (Wrong Default Gateway + OpenWrt LAN NAT)

This is a practical troubleshooting post. Facts first, evidence second, conclusions last.

TL;DR

Symptom: Cross-VLAN ping times out. The target host shows echo reply sent, yet the client never receives it.
Root cause #1 (decisive): PVE intentionally kept OpenWrt (192.168.22.13) as the default gateway for QoS/traffic handling, but had no explicit routes for other internal VLAN subnets, so return traffic to internal networks followed the default route into OpenWrt and never traversed the primary L3 gateway (classic asymmetric path).
Root cause #2 (amplifier): OpenWrt had masq=1 on the LAN zone, applying SNAT to internal east-west traffic and making paths/sessions unpredictable.
Fix:
- Keep PVE default gateway = OpenWrt (192.168.22.13) for QoS, but add explicit routes for internal subnets (e.g., 192.168.11.0/24, 192.168.183.0/24) via the primary gateway (192.168.22.1)
- Disable masquerade on OpenWrt lan zone
Prevention: Route internally, NAT only on real WAN egress; if a side gateway must be the default route (QoS/proxy), explicitly split internal routing vs Internet default; use tcpdump + ip route get as a standard SOP.

Background (Topology and Roles)

Multi-VLAN home network:

Cloud Gateway Fiber (primary L3 gateway)
- Inter-VLAN routing
- Example subnets:
  - VLAN11: 192.168.11.0/24 (GW 192.168.11.1)
  - VLAN22 (PVE/Lab): 192.168.22.0/24 (GW 192.168.22.1)
  - VLAN183: 192.168.183.0/24 (GW 192.168.183.1)
OpenWrt (secondary / side gateway)
- 192.168.22.13/24 on br-lan
- fw3 + iptables, plus proxy tooling (OpenClash/Passwall)
PVE node: 192.168.22.12/24 on vmbr0
Test hosts:
- 192.168.183.235 (VLAN183)
- 192.168.11.29 (VLAN11)

The intended design is simple:

Inter-VLAN traffic should be routed by the primary gateway.
NAT should only happen on a true WAN egress, not on internal VLAN-to-VLAN paths.

ASCII topology (printable via shell)

This script does not change any config. It prints the topology and the two key paths (bad vs fixed) so readers can visualize the data flow:

bash
#!/usr/bin/env bash
set -euo pipefail

# Print the topology and the two paths (bad vs fixed).
cat <<'EOF'

                    +------------------------------+
                    |   Cloud Gateway Fiber (L3)   |
                    |   VLAN11: 192.168.11.1       |
                    |   VLAN22: 192.168.22.1       |
                    |   VLAN183: 192.168.183.1     |
                    +---------------+--------------+
                                    |
                                  VLAN22
                                    |
                   +----------------+----------------+
                   |                                 |
        +----------v-----------+          +----------v-----------+
        |  OpenWrt (side GW)   |          |        PVE           |
        |  192.168.22.13       |          |  192.168.22.12       |
        |  QoS / traffic mgmt  |          |  vmbr0               |
        +----------------------+          +----------------------+

Clients:
  VLAN183 host: 192.168.183.235
  VLAN11  host: 192.168.11.29

Problem #1 (before):
  192.168.183.235 -> Fiber -> PVE
  PVE reply -> (default gw) OpenWrt -> [NAT/proxy/unknown] -> ???   (Fiber never sees the reply)

Fix idea:
  Keep PVE default gw = OpenWrt (for QoS),
  but route internal subnets via Fiber.

Problem #2 (amplifier):
  OpenWrt LAN masquerade (SNAT) rewrites east-west traffic,
  making internal routing/session behavior unpredictable.

EOF

Incidents and Impact

Incident #1: VLAN183 could not ping PVE

Client 192.168.183.235 → ping 192.168.22.12 timed out
On PVE, captures showed:
- ICMP echo request arriving
- ICMP echo reply being generated
On the primary gateway (Fiber), captures on br22/br0 showed requests only, no replies

Meaning: the reply never traversed the primary gateway.

Incident #2: After fixing #1, VLAN11 still could not ping PVE

192.168.11.29 could not ping 192.168.22.12
But it could ping OpenWrt 192.168.22.13

This pointed to OpenWrt NAT/firewall behavior affecting internal reachability.

Investigation (Evidence-driven)

1) Confirm the primary gateway is fine

On Fiber:

It could ping both 192.168.183.235 and 192.168.22.12.
ip route showed both subnets as directly connected.

Conclusion: core L3 routing on Fiber was not broken.

2) Hop-by-hop capture: find where the reply disappears

On PVE: request/reply both exist → PVE is replying.
On Fiber (br22/br0): request only → reply never reached Fiber.

Conclusion: the return path is not going through the primary gateway.

3) Let `ip route get` tell the truth

On PVE:

ip route showed: default via 192.168.22.13 dev vmbr0
ip route get 192.168.183.235 showed: via 192.168.22.13

Root cause #1: PVE had no explicit route to internal VLAN subnets, so return traffic to 192.168.183.0/24 naturally followed the default route into OpenWrt. Once that happens, any NAT/proxy/misroute on OpenWrt can prevent the primary gateway from ever seeing the reply.

That creates asymmetric routing:

Forward: 183 → Fiber → 22 → PVE (OK)
Return: PVE → OpenWrt → (NAT/proxy/unknown) (uncontrolled)

4) Validate OpenWrt firewall/NAT configuration

After confirming OpenWrt uses fw3 + iptables (no nftables), the key finding was:

lan zone had masq='1'

Root cause #2: OpenWrt applied SNAT to internal traffic, rewriting sources to 192.168.22.13 and making internal routing/session behavior unstable, especially with proxy redirection.

Fix (Minimal changes)

Fix #1: Keep default gateway = OpenWrt, but route internal subnets via Fiber

Constraint: PVE default route must remain 192.168.22.13 (OpenWrt) for QoS/traffic handling. In that case, do not touch the default route. Instead, explicitly route internal VLAN subnets via the primary L3 gateway (192.168.22.1).

Temporary test (examples for VLAN11/VLAN183; add others as needed):

bash
# Keep default via OpenWrt for QoS / traffic management
# default via 192.168.22.13

# Route internal VLANs via the primary L3 gateway (Fiber)
ip route replace 192.168.183.0/24 via 192.168.22.1 dev vmbr0
ip route replace 192.168.11.0/24  via 192.168.22.1 dev vmbr0

Verification tips:

ip route get 192.168.183.235 should show via 192.168.22.1
ICMP replies should be visible on Fiber interfaces (e.g., br22/br0)

Persist it in Proxmox networking (commonly /etc/network/interfaces for vmbr0) with post-up:

bash
post-up ip route replace 192.168.183.0/24 via 192.168.22.1 dev vmbr0
post-up ip route replace 192.168.11.0/24  via 192.168.22.1 dev vmbr0
post-down ip route del 192.168.183.0/24 via 192.168.22.1 dev vmbr0 || true
post-down ip route del 192.168.11.0/24  via 192.168.22.1 dev vmbr0 || true

Verification:

Fiber sees ICMP replies on br22/br0.
192.168.183.235 can ping 192.168.22.12 again.

Fix #2: Disable masquerade on OpenWrt LAN zone

bash
uci set firewall.@zone[0].masq='0'
uci commit firewall
/etc/init.d/firewall restart