
> This is a practical troubleshooting post. Facts first, evidence second, conclusions last.

## TL;DR

- **Symptom**: Cross-VLAN `ping` times out. The target host shows `echo reply` sent, yet the client never receives it.
- **Root cause #1 (decisive)**: PVE **intentionally** kept OpenWrt (`192.168.22.13`) as the default gateway for QoS/traffic handling, but had **no explicit routes for other internal VLAN subnets**, so return traffic to internal networks followed the default route into OpenWrt and never traversed the primary L3 gateway (classic asymmetric path).
- **Root cause #2 (amplifier)**: OpenWrt had `masq=1` on the **LAN zone**, applying **SNAT** to internal east-west traffic and making paths/sessions unpredictable.
- **Fix**:
  - **Keep** PVE default gateway = OpenWrt (`192.168.22.13`) for QoS, but add explicit routes for internal subnets (e.g., `192.168.11.0/24`, `192.168.183.0/24`) via the primary gateway (`192.168.22.1`)
  - Disable masquerade on OpenWrt `lan` zone
- **Prevention**: Route internally, NAT only on real WAN egress; if a side gateway must be the default route (QoS/proxy), explicitly split internal routing vs Internet default; use `tcpdump` + `ip route get` as a standard SOP.

---

## Background (Topology and Roles)

Multi-VLAN home network:

- **Cloud Gateway Fiber (primary L3 gateway)**
  - Inter-VLAN routing
  - Example subnets:
    - VLAN11: `192.168.11.0/24` (GW `192.168.11.1`)
    - VLAN22 (PVE/Lab): `192.168.22.0/24` (GW `192.168.22.1`)
    - VLAN183: `192.168.183.0/24` (GW `192.168.183.1`)
- **OpenWrt (secondary / side gateway)**
  - `192.168.22.13/24` on `br-lan`
  - `fw3 + iptables`, plus proxy tooling (OpenClash/Passwall)
- **PVE node**: `192.168.22.12/24` on `vmbr0`
- **Test hosts**:
  - `192.168.183.235` (VLAN183)
  - `192.168.11.29` (VLAN11)

The intended design is simple:

- Inter-VLAN traffic should be routed by the primary gateway.
- NAT should only happen on a true WAN egress, not on internal VLAN-to-VLAN paths.

---

## ASCII topology (printable via shell)

This script does not change any config. It prints the topology and the two key paths (bad vs fixed) so readers can visualize the data flow:

```bash
#!/usr/bin/env bash
set -euo pipefail

# Print the topology and the two paths (bad vs fixed).
cat <<'EOF'

                    +------------------------------+
                    |   Cloud Gateway Fiber (L3)   |
                    |   VLAN11: 192.168.11.1       |
                    |   VLAN22: 192.168.22.1       |
                    |   VLAN183: 192.168.183.1     |
                    +---------------+--------------+
                                    |
                                  VLAN22
                                    |
                   +----------------+----------------+
                   |                                 |
        +----------v-----------+          +----------v-----------+
        |  OpenWrt (side GW)   |          |        PVE           |
        |  192.168.22.13       |          |  192.168.22.12       |
        |  QoS / traffic mgmt  |          |  vmbr0               |
        +----------------------+          +----------------------+

Clients:
  VLAN183 host: 192.168.183.235
  VLAN11  host: 192.168.11.29

Problem #1 (before):
  192.168.183.235 -> Fiber -> PVE
  PVE reply -> (default gw) OpenWrt -> [NAT/proxy/unknown] -> ???   (Fiber never sees the reply)

Fix idea:
  Keep PVE default gw = OpenWrt (for QoS),
  but route internal subnets via Fiber.

Problem #2 (amplifier):
  OpenWrt LAN masquerade (SNAT) rewrites east-west traffic,
  making internal routing/session behavior unpredictable.

EOF
```

## Incidents and Impact

### Incident #1: VLAN183 could not ping PVE

- Client `192.168.183.235` → `ping 192.168.22.12` timed out
- On PVE, captures showed:
  - `ICMP echo request` arriving
  - `ICMP echo reply` being generated
- On the primary gateway (Fiber), captures on `br22`/`br0` showed requests only, no replies

**Meaning**: the reply never traversed the primary gateway.

### Incident #2: After fixing #1, VLAN11 still could not ping PVE

- `192.168.11.29` could not ping `192.168.22.12`
- But it could ping OpenWrt `192.168.22.13`

This pointed to OpenWrt NAT/firewall behavior affecting internal reachability.

---

## Investigation (Evidence-driven)

### 1) Confirm the primary gateway is fine

On Fiber:

- It could ping both `192.168.183.235` and `192.168.22.12`.
- `ip route` showed both subnets as directly connected.

Conclusion: core L3 routing on Fiber was not broken.

### 2) Hop-by-hop capture: find where the reply disappears

- On PVE: request/reply both exist → PVE is replying.
- On Fiber (`br22`/`br0`): request only → reply never reached Fiber.

Conclusion: the return path is not going through the primary gateway.

### 3) Let `ip route get` tell the truth

On PVE:

- `ip route` showed: `default via 192.168.22.13 dev vmbr0`
- `ip route get 192.168.183.235` showed: `via 192.168.22.13`

**Root cause #1**: PVE had **no explicit route** to internal VLAN subnets, so return traffic to `192.168.183.0/24` naturally followed the default route into OpenWrt. Once that happens, any NAT/proxy/misroute on OpenWrt can prevent the primary gateway from ever seeing the reply.

That creates asymmetric routing:

- Forward: `183 → Fiber → 22 → PVE` (OK)
- Return: `PVE → OpenWrt → (NAT/proxy/unknown)` (uncontrolled)

### 4) Validate OpenWrt firewall/NAT configuration

After confirming OpenWrt uses `fw3 + iptables` (no nftables), the key finding was:

- `lan` zone had `masq='1'`

**Root cause #2**: OpenWrt applied SNAT to internal traffic, rewriting sources to `192.168.22.13` and making internal routing/session behavior unstable, especially with proxy redirection.

---

## Fix (Minimal changes)

### Fix #1: Keep default gateway = OpenWrt, but route internal subnets via Fiber

Constraint: PVE default route must remain `192.168.22.13` (OpenWrt) for QoS/traffic handling. In that case, do not touch the default route. Instead, explicitly route internal VLAN subnets via the primary L3 gateway (`192.168.22.1`).

Temporary test (examples for VLAN11/VLAN183; add others as needed):

```bash
# Keep default via OpenWrt for QoS / traffic management
# default via 192.168.22.13

# Route internal VLANs via the primary L3 gateway (Fiber)
ip route replace 192.168.183.0/24 via 192.168.22.1 dev vmbr0
ip route replace 192.168.11.0/24  via 192.168.22.1 dev vmbr0
```

Verification tips:

- `ip route get 192.168.183.235` should show `via 192.168.22.1`
- ICMP replies should be visible on Fiber interfaces (e.g., `br22/br0`)

Persist it in Proxmox networking (commonly `/etc/network/interfaces` for `vmbr0`) with `post-up`:

```bash
post-up ip route replace 192.168.183.0/24 via 192.168.22.1 dev vmbr0
post-up ip route replace 192.168.11.0/24  via 192.168.22.1 dev vmbr0
post-down ip route del 192.168.183.0/24 via 192.168.22.1 dev vmbr0 || true
post-down ip route del 192.168.11.0/24  via 192.168.22.1 dev vmbr0 || true
```

Verification:

- Fiber sees ICMP replies on `br22/br0`.
- `192.168.183.235` can ping `192.168.22.12` again.

### Fix #2: Disable masquerade on OpenWrt LAN zone

```bash
uci set firewall.@zone[0].masq='0'
uci commit firewall
/etc/init.d/firewall restart
```

Verification:

- `192.168.11.29` can ping `192.168.22.12`.
- Internal traffic keeps real source IPs, making debugging and ACLs sane.

---

## Prevention

- **Route internally; NAT only on real WAN**
  - `lan`/internal zones: `masq=0`
  - `wan` egress: `masq=1`
- **If a side gateway must be the default route (QoS/proxy), split routing explicitly**
  - Default route can stay on OpenWrt, but internal subnets must have explicit routes via the primary L3 gateway (or use policy routing by destination).
  - Keep east-west traffic on predictable L3 routing. Do not let it fall into NAT/proxy black boxes.
- **Standard SOP (Standard Operating Procedure)**
  - Meaning: for “cross-VLAN ping fails”, do not guess. Follow the same evidence-driven checklist every time.
  - **Step 1: packet captures answer 3 questions**: did the request arrive, did the target reply, and at which hop did the reply disappear?
    - On PVE: `tcpdump -ni vmbr0 icmp`
    - On the primary gateway (both VLAN interfaces): `tcpdump -ni br22 icmp` and `tcpdump -ni br0 icmp`
  - **Step 2: verify routing decisions (no imagination)**
    - On PVE: `ip route` + `ip route get 192.168.183.235`
    - On OpenWrt/Fiber: `ip route get 192.168.183.235` (confirm the actual next hop)
  - **Step 3: simplify the data path first**
    - Restore pure routing with real source IPs and predictable return paths before layering QoS/proxy/policy routing.
- **Make proxy behavior explicit**
  - Exclude internal subnets from transparent proxy/tunnel redirection.

---

## One-line lesson

If ICMP looks like “ghost packets”, you most likely built a black box with the wrong default gateway and the wrong NAT — not a broken protocol stack.


A classic multi-VLAN connectivity incident: ICMP times out while the target clearly replies locally. The real causes are asymmetric routing from a wrong PVE default gateway and an OpenWrt misconfiguration that enables SNAT (masquerade) on the LAN zone. Fixed with hop-by-hop captures and minimal config changes.

Daily Debugging: PVE Cross-VLAN Ping Fails (Wrong Default Gateway + OpenWrt LAN NAT)

Comments