I needed to lend a Singapore server a temporary network egress for one weekend. The egress had to be my Ubuntu VM inside a local K3s cluster, not the Singapore server itself. It also had to survive terminal disconnects and transient network failures, stop after exactly four days, and leave no reusable credential behind.

The final design was small:

• OpenSSH remote dynamic forwarding exposed a SOCKS5 listener only on the Singapore server's loopback interface.
• A systemd service on the Ubuntu VM kept the SSH connection alive and restarted it after failures.
• An absolute systemd timer stopped and disabled the service at the deadline.
• Live socket inspection showed destination IPs and ports without pretending that HTTPS payloads were readable.
• Cleanup revoked the public key first, then removed the local private key and every tunnel artifact.

This post documents the complete path, including the mistakes. All public addresses, usernames, and key material below are sanitized examples.

TL;DR

The command that made the Ubuntu VM the egress was:

ssh -N -T \
  -i ~/.ssh/sg-egress-ed25519 \
  -o BatchMode=yes \
  -o IdentitiesOnly=yes \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=~/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  -o ServerAliveInterval=30 \
  -o ServerAliveCountMax=3 \
  -o ExitOnForwardFailure=yes \
  -R 127.0.0.1:1080 \
  sg-egress@203.0.113.10

The important detail is that -R 127.0.0.1:1080 has no fixed destination. With modern OpenSSH, that creates a remote dynamic SOCKS listener. Applications on the Singapore server connect to 127.0.0.1:1080; the Ubuntu-side SSH client then opens the destination connections, so the Internet sees the Ubuntu VM's upstream public address.

Verification from the Singapore server:

curl --proxy socks5h://127.0.0.1:1080 https://ifconfig.me/ip

If the result matches the Ubuntu VM's direct egress IP and differs from the Singapore server's direct IP, the data path is correct.

Requirement and safety boundaries

This was intentionally a narrow, temporary capability:

1. Only processes on the Singapore server could use the proxy.
2. The SOCKS port was never exposed on 0.0.0.0.
3. The SSH key belonged to an unprivileged account that could only open the required remote listener.
4. The tunnel recovered automatically, but the deadline could not slide after a reboot.
5. The key expired server-side at the same deadline and was revoked before its private half was deleted.

Binding the proxy to 127.0.0.1 matters. Using this instead:

-R 0.0.0.0:1080

would request a public listener. The client address alone is not a sufficient guard: GatewayPorts yes on the SSH server forces remote forwards onto wildcard addresses even when the client requests 127.0.0.1. I therefore enforced GatewayPorts no and PermitListen 127.0.0.1:1080 for the tunnel account, then verified the live listener.

Topology and data path

The Ubuntu VM ran in KubeVirt on K3s. Its SSH service was published through a NodePort on the worker node.

flowchart LR
    Mac["Operator Mac"]
    Node["K3s worker<br/>192.168.22.31:30022"]
    VM["KubeVirt Ubuntu VM<br/>SSH :22"]
    SG["Singapore server<br/>127.0.0.1:1080"]
    Internet["Internet destination"]

    Mac -->|"SSH administration"| Node
    Node -->|"NodePort to VM :22"| VM
    VM -->|"SSH control connection :22<br/>remote dynamic forwarding"| SG
    SG -->|"SOCKS5 request inside tunnel"| VM
    VM -->|"new outbound connection"| Internet

The direction is easy to misread. The SSH control connection starts on the Ubuntu VM and goes to Singapore, but proxied application traffic starts on Singapore and exits from Ubuntu.

Confirm that I was entering the guest, not virt-launcher

The KubeVirt pod contained the QEMU process. Running kubectl exec into virt-launcher would enter the compute container, not the Ubuntu guest OS.

I first confirmed the VMI and its SSH Service:

kubectl -n kubevirt-vms get vmi ubuntu-vm -o wide
kubectl -n kubevirt-vms get service ubuntu-vm-ssh -o wide

The Service mapped NodePort 30022 to guest port 22, so guest access was:

ssh -p 30022 vmuser@192.168.22.31

For one-off administration I also used a local port-forward:

kubectl -n kubevirt-vms port-forward service/ubuntu-vm-ssh 2222:22
ssh -p 2222 vmuser@127.0.0.1

These ports have different meanings:

• 30022 is the persistent K3s NodePort.
• 2222 was a temporary port on my Mac created by kubectl port-forward.
• 1080 was the SOCKS listener on the Singapore server.

Also note the CLI detail: ssh uses lowercase -p for a port; scp uses uppercase -P.

Prepare a dedicated SSH credential and deadline

The private key lived only on the Ubuntu VM and was mode 0600:

install -d -m 700 ~/.ssh
install -m 600 /secure/input/sg-egress-ed25519 ~/.ssh/sg-egress-ed25519
ssh-keygen -lf ~/.ssh/sg-egress-ed25519

I calculated the deadline once and derived the OpenSSH key-expiry format from the same value:

deadline=$(date -u -d '+96 hours' '+%Y-%m-%d %H:%M:%S UTC')
key_expiry=$(date -u -d "$deadline" '+%Y%m%d%H%M%SZ')
public_key=$(ssh-keygen -y -f ~/.ssh/sg-egress-ed25519)
printf 'deadline=%s\nkey_expiry=%s\n' "$deadline" "$key_expiry"
printf 'expiry-time="%s" %s sg-egress-four-day\n' "$key_expiry" "$public_key"

I transferred only the final public-key line through an authenticated administrative channel. On the Singapore server, a separate administrator created a dedicated account and installed that line as /home/sg-egress/.ssh/authorized_keys:

sudo useradd --create-home --user-group --shell /usr/sbin/nologin sg-egress
sudo install -d -o sg-egress -g sg-egress -m 700 /home/sg-egress/.ssh
sudo install -o sg-egress -g sg-egress -m 600 \
  /secure/input/sg-egress-authorized-key \
  /home/sg-egress/.ssh/authorized_keys

I also installed /etc/ssh/sshd_config.d/sg-egress.conf:

Match User sg-egress
    AuthenticationMethods publickey
    PasswordAuthentication no
    KbdInteractiveAuthentication no
    AllowAgentForwarding no
    AllowStreamLocalForwarding no
    AllowTcpForwarding remote
    GatewayPorts no
    PermitListen 127.0.0.1:1080
    PermitTTY no
    PermitTunnel no
    PermitUserRC no
    X11Forwarding no
    MaxSessions 0

Match all

MaxSessions 0 denies shell, command, and subsystem sessions while still permitting forwarding. AllowTcpForwarding remote rejects local forwards, and PermitListen limits the only allowed remote listener. I validated and reloaded the server configuration before using the key:

sudo sshd -t
sudo systemctl reload ssh
sudo sshd -T -C user=sg-egress,host=localhost,addr=127.0.0.1 | \
  grep -E '^(allowagentforwarding|allowstreamlocalforwarding|allowtcpforwarding|gatewayports|maxsessions|permitlisten|permittty|permittunnel|permituserrc|x11forwarding) '

Strict host-key checking also needs an explicit trust bootstrap. From an already authenticated administrative console on Singapore, I recorded the ED25519 host-key fingerprint:

sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

On the Ubuntu VM, running as vmuser, I scanned the public host key into a dedicated file but refused to install it until its fingerprint exactly matched the value obtained through that trusted channel. The tunnel never reads or modifies the shared known_hosts file:

pinned_hosts=$(mktemp ~/.ssh/sg-egress-known_hosts.XXXXXX)
trap 'test ! -e "$pinned_hosts" || unlink "$pinned_hosts"' EXIT

ssh-keyscan -t ed25519 203.0.113.10 > "$pinned_hosts"
scanned_fingerprint=$(ssh-keygen -lf "$pinned_hosts" | awk 'NR == 1 { print $2 }')
read -r -p 'Trusted ED25519 SHA256 fingerprint: ' trusted_fingerprint

if [ -z "$scanned_fingerprint" ] || \
   [ -z "$trusted_fingerprint" ] || \
   [ "$scanned_fingerprint" != "$trusted_fingerprint" ]; then
  printf 'host-key fingerprint mismatch\n' >&2
  exit 1
fi

chmod 600 "$pinned_hosts"
mv "$pinned_hosts" ~/.ssh/sg-egress-known_hosts
trap - EXIT

ssh-keygen -F 203.0.113.10 -f ~/.ssh/sg-egress-known_hosts
ssh-keygen -lf ~/.ssh/sg-egress-known_hosts

ssh-keyscan only collected the presented key; it was not the trust source. The independently obtained fingerprint was the trust source. Every tunnel command explicitly uses this file and ignores global host-key files, so exactly the verified key is trusted without changing unrelated SSH state.

Do not paste a production private key into a ticket, chat, shell history, or repository. Use a dedicated short-lived key and an authenticated transfer mechanism. If a private key is ever disclosed, revocation is mandatory; deleting one local copy is not enough.

Why the first -R idea was wrong

The first mental model was a reverse SSH login port:

ssh -N -R 127.0.0.1:10022:127.0.0.1:22 sg-egress@203.0.113.10

That makes Singapore's 127.0.0.1:10022 forward to the Ubuntu VM's SSH port. It lets the Singapore server log back into the VM, but it does not make Ubuntu a general Internet egress.

The corrected command removed the fixed destination:

ssh -N -T \
  -i ~/.ssh/sg-egress-ed25519 \
  -o BatchMode=yes \
  -o IdentitiesOnly=yes \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=~/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  -o ServerAliveInterval=30 \
  -o ServerAliveCountMax=3 \
  -o TCPKeepAlive=yes \
  -o ConnectTimeout=15 \
  -o ExitOnForwardFailure=yes \
  -R 127.0.0.1:1080 \
  sg-egress@203.0.113.10

Now 1080 is a SOCKS5 endpoint, and destinations are selected per client request.

Make it survive a weekend with systemd

A foreground terminal is not a reliability mechanism. I installed this unit as /etc/systemd/system/sg-egress-socks.service:

[Unit]
Description=Reverse SOCKS5 tunnel using Ubuntu VM as SG egress
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=0

[Service]
Type=simple
User=vmuser
Environment=LANG=C
ExecStart=/usr/bin/ssh -N -T \
  -i /home/vmuser/.ssh/sg-egress-ed25519 \
  -o BatchMode=yes \
  -o IdentitiesOnly=yes \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=/home/vmuser/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  -o ServerAliveInterval=30 \
  -o ServerAliveCountMax=3 \
  -o TCPKeepAlive=yes \
  -o ConnectTimeout=15 \
  -o ExitOnForwardFailure=yes \
  -R 127.0.0.1:1080 \
  sg-egress@203.0.113.10
Restart=always
RestartSec=5s
TimeoutStopSec=15s
NoNewPrivileges=yes
PrivateTmp=yes

[Install]
WantedBy=multi-user.target

ExitOnForwardFailure=yes is essential. Without it, SSH could stay connected even when it failed to bind the remote port, leaving systemd with a green process and a broken service.

Before enabling the unit, I validated it:

systemd-analyze verify /etc/systemd/system/sg-egress-socks.service

Then:

sudo systemctl daemon-reload
sudo systemctl enable --now sg-egress-socks.service
systemctl is-enabled sg-egress-socks.service
systemctl is-active sg-egress-socks.service

Stop after exactly 96 hours

RuntimeMaxSec=4d looks attractive, but combining a runtime limit with an automatic restart policy creates ambiguous deadline behavior. I wanted a deadline that survived reboots and never moved. The deadline and key_expiry values were already calculated from the same instant when the key was prepared; the timer had to consume that exact deadline rather than a copied example timestamp.

The stop action was /etc/systemd/system/sg-egress-socks-stop.service:

[Unit]
Description=Stop and disable the SG egress SOCKS tunnel

[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl disable sg-egress-socks.service
ExecStart=/usr/bin/systemctl stop sg-egress-socks.service

I rejected an empty or expired value, checked how systemd parsed it, and generated the timer directly from the shell variable:

test -n "${deadline:-}" || {
  printf 'deadline is not set\n' >&2
  exit 1
}
deadline_epoch=$(date -u -d "$deadline" '+%s')
test "$deadline_epoch" -gt "$(date -u '+%s')" || {
  printf 'deadline is not in the future: %s\n' "$deadline" >&2
  exit 1
}
systemd-analyze calendar "$deadline"

sudo tee /etc/systemd/system/sg-egress-socks-stop.timer >/dev/null <<EOF
[Unit]
Description=Stop the SG egress SOCKS tunnel after the four-day window

[Timer]
OnCalendar=$deadline
AccuracySec=1s
Persistent=yes
Unit=sg-egress-socks-stop.service

[Install]
WantedBy=timers.target
EOF

Persistent=yes ensures that a missed deadline fires after the VM returns. The stop action also disables the tunnel unit, so a later reboot cannot reopen it. The matching expiry-time on the Singapore server prevents a new SSH authentication after the same deadline even if the client-side timer fails.

sudo systemctl daemon-reload
sudo systemctl enable --now sg-egress-socks-stop.timer
systemctl list-timers --all sg-egress-socks-stop.timer

Verify the actual egress

I compared three observations:

# On the Ubuntu VM
curl -4fsS https://ifconfig.me/ip

# Directly on the Singapore server
curl -4fsS https://ifconfig.me/ip

# On Singapore, through the reverse SOCKS tunnel
curl -4fsS \
  --proxy socks5h://127.0.0.1:1080 \
  https://ifconfig.me/ip

The expected relationship is:

SG via SOCKS IP == Ubuntu direct IP
SG direct IP     != Ubuntu direct IP

I also checked the listener on Singapore:

ss -lntp 'sport = :1080'

It had to show 127.0.0.1:1080, not 0.0.0.0:1080.

Finally, I killed the SSH main process once and watched systemd recover. The PID changed, NRestarts increased, and the SOCKS test worked again seven seconds later. This tested the failure mode I actually cared about instead of merely checking that the initial start succeeded.

Observe live traffic without inventing visibility

The SSH client process opens destination sockets on the Ubuntu VM. Its live connections can be inspected with:

tunnel_pid=$(systemctl show sg-egress-socks.service -p MainPID --value)
sudo lsof -nP -a -p "$tunnel_pid" -i

For a persistent terminal view:

tmux new-session -A -s sg-egress-monitor
watch -n 1 'pid=$(systemctl show sg-egress-socks.service -p MainPID --value); printf "tunnel_pid=%s\n" "$pid"; if [ "$pid" != 0 ]; then sudo -n lsof -nP -a -p "$pid" -i; else printf "service is not running\n"; fi'

Detach with Ctrl-b, then d.

This view has strict limits:

• It shows active destination IPs and ports.
• It may miss very short connections between one-second samples.
• It does not reconstruct historical flows.
• HTTPS payloads, credentials, and full URLs remain encrypted.
• DNS names may be visible only when ordinary DNS or additional auditing captures them.

If historical accounting is a real requirement, enable consented flow logging before the window starts. A live lsof screen is not an audit log.

Failures encountered

1. One IP-check endpoint failed

api.ipify.org:443 returned Connection refused, and a Cloudflare endpoint timed out from the Ubuntu egress. Other independent endpoints returned the same valid public address.

The lesson was not to change the tunnel because one measurement target failed. Test DNS, TCP reachability, and at least two unrelated endpoints before blaming the data path.

2. Remote port 1080 was occupied

The service later accumulated 1,428 failures over roughly two hours:

Error: remote port forwarding failed for listen port 1080

ExitOnForwardFailure=yes made each failed bind visible to systemd, which retried every five seconds. The evidence proved a port conflict on Singapore; it did not prove whether the owner was a stale SSH session or another manually started tunnel.

The correct investigation was:

ss -lntp 'sport = :1080'
ps -fp <listener-pid>

Do not kill a listener based only on a port number. Identify the owning process and connection first. Once the conflicting listener disappeared, the managed service acquired 1080 and stayed active.

3. A multiline paste broke the tmux monitor

The shell received actual newlines inside the service name and options:

sg-egress-
socks.service
-p
MainPID

That produced socks.service: not found, -p: not found, and an empty PID that made lsof -a invalid. Terminal visual wrapping is harmless; embedded newline characters are not. Replacing the stored tmux command with one literal shell line fixed the monitor.

4. A green control connection does not imply proxy traffic

The monitor always showed one established socket from Ubuntu to Singapore port 22. That was the SSH control connection. Additional destination sockets appeared only while a client actually used the SOCKS proxy.

During a ten-second sample, Singapore had no established client to 127.0.0.1:1080 and no speed-test process. The correct conclusion was simply “the proxy is idle now,” not “the monitoring is broken.”

Complete revocation and cleanup

The order matters. Stop the active tunnel first, revoke the public key through a separate administrative account, verify that the old key cannot create a new tunnel, and only then delete the private key.

The remote scripts below use stdin for their bodies, so they cannot also use stdin for a sudo password. Before changing any state, I verified that the separate administrator had pre-existing non-interactive sudo access:

ssh \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=~/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  admin@203.0.113.10 \
  sudo -n true

If that check fails, stop here and run the privileged command bodies directly from an already authenticated Singapore management console. Do not send a sudo password through the script stream.

1. Stop the active tunnel but keep the private key

sudo systemctl disable --now sg-egress-socks-stop.timer
sudo systemctl disable --now sg-egress-socks.service
sudo systemctl stop sg-egress-socks-stop.service

Stopping first closes the already-authenticated SSH connection. Removing an authorized key does not terminate sessions that are already established.

2. Remove only the matching public key on Singapore

From the Ubuntu VM, derive the public blob without printing private material. Use a separate administrative account on Singapore to pass it as one controlled argument, require exactly one match, and replace authorized_keys atomically. The restricted tunnel key cannot and must not execute this cleanup script:

key_blob=$(ssh-keygen -y -f ~/.ssh/sg-egress-ed25519 | awk '{print $2}')

ssh \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=~/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  admin@203.0.113.10 \
  sudo -n bash -s -- "$key_blob" <<'REMOTE'
set -eu
key_blob=$1
auth_file=/home/sg-egress/.ssh/authorized_keys
matches=$(awk -v key_blob="$key_blob" \
  'index($0, key_blob) { count++ } END { print count+0 }' \
  "$auth_file")

if [ "$matches" -ne 1 ]; then
  printf 'refusing to edit: expected 1 key, found %s\n' "$matches" >&2
  exit 1
fi

temp_auth=$(mktemp /home/sg-egress/.ssh/authorized_keys.cleanup.XXXXXX)
trap 'test ! -e "$temp_auth" || unlink "$temp_auth"' EXIT

awk -v key_blob="$key_blob" \
  'index($0, key_blob) == 0' \
  "$auth_file" > "$temp_auth"

chmod --reference="$auth_file" "$temp_auth"
chown --reference="$auth_file" "$temp_auth"
mv "$temp_auth" "$auth_file"
REMOTE

An OpenSSH public-key blob contains a restricted base64 alphabet, but the match-count guard remains important: cleanup must not silently remove zero keys or multiple keys.

Then verify that the old key cannot authenticate a new forwarding connection. Because the managed service is already stopped, port 1080 is free; require an explicit Permission denied (publickey) result rather than treating a timeout or network error as successful revocation:

timeout 10s ssh -N -T \
  -i ~/.ssh/sg-egress-ed25519 \
  -o BatchMode=yes \
  -o IdentitiesOnly=yes \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=~/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  -o ConnectTimeout=5 \
  -o ExitOnForwardFailure=yes \
  -R 127.0.0.1:1080 \
  sg-egress@203.0.113.10

The expected result is Permission denied (publickey).

The administrator can now remove the dedicated account and its scoped SSH configuration:

ssh \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=~/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  admin@203.0.113.10 \
  sudo -n bash -s <<'REMOTE'
set -eu
userdel -r sg-egress
unlink /etc/ssh/sshd_config.d/sg-egress.conf
sshd -t
systemctl reload ssh
REMOTE

3. Remove the Ubuntu-side runtime artifacts

tmux kill-session -t sg-egress-monitor

sudo unlink /etc/systemd/system/sg-egress-socks.service
sudo unlink /etc/systemd/system/sg-egress-socks-stop.service
sudo unlink /etc/systemd/system/sg-egress-socks-stop.timer
sudo systemctl daemon-reload

unlink ~/.ssh/sg-egress-ed25519
unlink ~/.ssh/sg-egress-known_hosts

Remove any explicitly known temporary copies as well. Avoid broad globs when credentials are involved.

System journals were retained as an audit trail. Selectively erasing one unit's records is not a normal journal operation, while vacuuming the whole journal would destroy unrelated operational evidence.

4. Final checks

systemctl show sg-egress-socks.service -p LoadState --value
systemctl list-timers --all 'sg-egress-socks*'
pgrep -af '[s]sh.*-R.*1080'
tmux has-session -t sg-egress-monitor

The final state was:

• all three units: not-found
• no matching timer
• no ssh -R process
• no monitor tmux session
• no private key, dedicated host-key file, or temporary unit copies on Ubuntu
• no dedicated sg-egress account or scoped SSH configuration on Singapore
• the old key rejected by Singapore
• no listener on Singapore 127.0.0.1:1080

What I would keep for the next temporary egress

1. Treat reverse login forwarding and reverse dynamic forwarding as different tools.
2. Bind temporary proxies to loopback unless broader exposure is explicitly required.
3. Combine SSH keepalives with systemd restart and ExitOnForwardFailure=yes.
4. Use an absolute persistent deadline, then disable the service at expiry.
5. Verify the egress by comparing direct and proxied public addresses.
6. Decide before launch whether live observation or historical auditing is required.
7. Revoke the server-side public key before deleting the local private key.
8. Preserve logs, but remove credentials and runtime artifacts.

The tunnel itself was one line. Making it reliable, bounded, observable, and revocable was the actual engineering work.

Microsoft Store failed to initialize while browsers and other desktop applications continued to work. The obvious reactions—reinstalling Store, resetting the whole network stack, or replacing Clash—would all have been too destructive.

Nothing was broken in the Store package, the relevant Windows services, or the Clash core. The failure was on the first edge of the data path: Microsoft Store runs inside an AppContainer and lacked permission to reach the Clash proxy through the local loopback interface.

The minimal fix was to grant a loopback exemption to Microsoft Store. Clash Verge kept both TUN mode and the Windows system proxy enabled.

TL;DR

• Symptom: Microsoft Store failed to initialize when Clash Verge used both TUN mode and the Windows system proxy.
• Proxy state: The system proxy was 127.0.0.1:7897, and verge-mihomo was listening on that port.
• Eliminated causes: The Store package and its related Windows services were healthy.
• Root cause: Microsoft Store was missing from the AppContainer loopback exemption list and could not reach the local Clash proxy.
• Fix: Run this command as Administrator:

CheckNetIsolation.exe LoopbackExempt -a -n=Microsoft.WindowsStore_8wekyb3d8bbwe

Incident context

The failure occurred with both Clash Verge entry points enabled:

• TUN mode: enabled
• Windows system proxy: enabled
• System proxy address: 127.0.0.1:7897
• Clash core: verge-mihomo

Microsoft Store displayed this message:

Sorry about that. Something went wrong and Microsoft Store failed to initialize. Try refreshing or come back later.

The important signal was that regular desktop applications still had network access. A machine-wide outage or a stopped Clash core was therefore unlikely.

Start with the data path

With the system proxy enabled, the expected request path was:

Microsoft Store (AppContainer)
        │
        │ connect to 127.0.0.1:7897
        ▼
Clash / verge-mihomo
        │
        ▼
Microsoft services

The broken edge was the first one, not the connection from Clash to the Internet.

Windows isolates AppContainer applications from the local loopback interface by default. A Win32 application reaching 127.0.0.1:7897 does not prove that Microsoft Store can do the same. Pointing the system proxy at a local listener exposes that difference.

Clash Fake-IP records only prove that a request entered Clash's DNS or proxy pipeline. Seeing an address from 198.18.0.0/15 is not enough to blame Fake-IP itself. Changing the DNS mode would have treated a symptom without proving the cause.

Evidence: eliminate corruption before changing anything

1. Verify that Clash is listening

Check port 7897 in PowerShell:

Get-NetTCPConnection -LocalPort 7897 -State Listen

In this incident, verge-mihomo was listening normally. The proxy process and its local entry point were healthy.

2. Check the Microsoft Store package

Get-AppxPackage -Name Microsoft.WindowsStore

The package status was Ok. With no evidence of package corruption, uninstalling or re-registering Store would have added risk without addressing the data path.

3. Check related Windows services

Get-Service AppXSvc, BITS, InstallService, wuauserv

No service failure was found. Package corruption and stopped update services were now low-priority hypotheses.

4. Inspect the loopback exemption list

CheckNetIsolation.exe LoopbackExempt -s

The list did not contain:

microsoft.windowsstore_8wekyb3d8bbwe

The evidence now formed a complete chain: Store had to reach 127.0.0.1:7897, the proxy was listening, but the AppContainer did not have loopback access.

Minimal fix: exempt Microsoft Store only

Open PowerShell or Windows Terminal as Administrator and run:

CheckNetIsolation.exe LoopbackExempt -a -n=Microsoft.WindowsStore_8wekyb3d8bbwe

Verify the result:

CheckNetIsolation.exe LoopbackExempt -s

The output should now include:

microsoft.windowsstore_8wekyb3d8bbwe

This grants loopback access to Microsoft Store only. It does not require disabling Clash TUN or the Windows system proxy, and it is narrower than relaxing isolation for unrelated applications.

If the local proxy is no longer used, remove the exemption with:

CheckNetIsolation.exe LoopbackExempt -d -n=Microsoft.WindowsStore_8wekyb3d8bbwe

Clear stale state

After fixing the permission, clear state left behind by the failed requests.

1. Flush the DNS cache

ipconfig /flushdns

2. Reset the Microsoft Store cache

Close Microsoft Store completely, then run:

wsreset.exe

wsreset.exe clears Store state and restarts the application. It does not grant loopback access, so running it before fixing the permission may only reproduce the same failure after a clean restart.

Why 0x80073D02 was not the root cause

The event log also contained 0x80073D02. It indicates that Windows could not update Microsoft Store while the application was still using resources required by the update.

That was a separate issue:

• The loopback restriction explained why Store could not reach the local proxy.
• 0x80073D02 explained why a Store update could not finish while Store was running.

The correct response to the error code was to close Store and allow the update to finish. It was not evidence that the Store package was corrupted.

State after the fix

• Microsoft Store package status remained Ok
• Microsoft Store opened and refreshed normally
• The AppContainer loopback exemption was present
• Clash Verge TUN mode remained enabled
• The Windows system proxy remained 127.0.0.1:7897
• verge-mihomo continued running

If the problem returns

Use this order instead of reinstalling Store immediately:

1. Close Microsoft Store completely.
2. Confirm that Clash Verge and verge-mihomo are running.
3. Confirm that 127.0.0.1:7897 is listening.
4. Run CheckNetIsolation.exe LoopbackExempt -s and verify the Store exemption.
5. Run wsreset.exe, then reopen Store.
6. If it still fails, inspect Clash connection logs for Microsoft domain rules and test the selected proxy node.

A major Windows upgrade or system reset is a good reason to check the exemption again. Normal Clash configuration updates should not alter Windows AppContainer loopback permissions.

Takeaway

The misleading shortcut was to equate “Store cannot open” with “Store is broken.” A better troubleshooting sequence is:

1. Draw the request path.
2. Verify every node.
3. Identify the broken edge.
4. Change only that edge.

Here, the broken edge was Microsoft Store → 127.0.0.1:7897. A single-application loopback exemption fixed it. Reinstalling Store, disabling TUN, or changing Clash DNS mode was unnecessary.

The machine I upgraded was not a disposable test box.

It is my daily GNOME desktop and development environment, but it also runs PostgreSQL, Redis, Docker, Ollama, Traefik, Open WebUI, and several other services. Moving from NixOS 25.11 to 26.05 changes systemd, D-Bus, Docker, GNOME, NetworkManager, glibc, and more. If I had simply run nixos-rebuild switch, a failure would have left me wondering whether the operating system, a database, a container image, or the live desktop session was responsible.

The most valuable result of this upgrade was therefore not a list of renamed NixOS options. It was a safer upgrade process: reduce the number of moving parts, build everything in advance, prepare the rollback before deployment, and validate the new system with real services and real data.

TL;DR

This was the final upgrade sequence:

flowchart TB
    A["Phase 1: Reduce the variables<br/>Capture the baseline · Freeze application versions"]
    B["Phase 2: Produce the candidate system<br/>Update the Flake · Migrate configuration · Build everything · Diff closures"]
    C["Phase 3: Deploy with a way back<br/>Back up stateful data · Install only the boot entry · Reboot"]
    D["Phase 4: Validate the real system<br/>Test in layers · Observe delayed jobs and logs"]

    A --> B --> C --> D

Six principles are worth carrying into future upgrades:

1. system.stateVersion is not the currently installed NixOS release. Do not bump it automatically.
2. Keep the operating-system upgrade separate from application upgrades. Temporarily pin container images to exact digests.
3. A successful nix eval does not prove that the system can be built. Build the complete system closure.
4. When several foundational components change at once, prefer a boot deployment over a live switch.
5. System generations and data backups recover from different classes of failure. You need both.
6. Validation must outlive the first few minutes after boot. Timers and delayed jobs may fail later.

Why I Did Not Run nixos-rebuild switch

NixOS has excellent declarative configuration and generation rollback, but neither eliminates every runtime risk.

The candidate system included all of these changes at once:

• D-Bus moved from dbus-daemon to dbus-broker.
• The initrd switched to systemd.
• systemd, Docker, GNOME, and NetworkManager crossed release boundaries.
• The glibc update affected PostgreSQL collation metadata.
• Home Manager required several option migrations.
• Containers still using latest or main could have upgraded their applications during the reboot.

switch activates services inside the currently running system. That is convenient for a small configuration change. It is less attractive when foundational services, the desktop session, and stateful applications all change together. The result can be an awkward intermediate state: the target system on disk is new, some processes still belong to the old environment, and other services have already restarted.

I used the prebuilt result instead:

sudo ./result-26.05/bin/switch-to-configuration boot

This installs the new generation as the next boot target without hot-switching every service in the current session. The old system stays intact until the backups and checks are complete, and the reboot creates a clean boundary: a complete old system before it, and a complete new system after it.

Step 1: Capture a Baseline Before Changing Versions

Before touching the Flake, I recorded:

• the output of nixos-version, plus the kernel, Nix, and systemd versions;
• available space on / and /boot;
• failed systemd units;
• versions and health of PostgreSQL, Redis, Docker, and other stateful services;
• running containers and their resolved image digests;
• the current system generation and at least one older bootable generation.

This may not feel like progress, but it determines whether two important questions can be answered later: What actually changed, and did this problem already exist before the upgrade?

Do not bump stateVersion

I deliberately kept the existing values:

system.stateVersion = "25.05";
home.stateVersion = "24.05";

stateVersion is not a display field for the installed release. It declares which historical compatibility behavior the system should preserve, and it can affect service data directories, defaults, and data formats.

Updating nixpkgs to 26.05 does not mean that system.stateVersion should also become 26.05. Unless you intend to review and perform the associated state migrations, leaving it unchanged is usually the safe choice.

Protect the rollback path first

My maintenance job previously ran this every week:

nix-collect-garbage -d

The -d flag deletes every non-current generation, which directly conflicts with the goal of retaining the old system after an upgrade. I changed it to:

nix-collect-garbage --delete-older-than 30d

NixOS can create rollback generations, but that does not mean your garbage-collection policy will preserve them. Rollback capability is itself something that must be maintained.

Step 2: Freeze Applications to Reduce the Number of Variables

Before the upgrade, Open WebUI, Dockhand, and Traefik used main, latest, or ordinary version tags. If Docker restarted after those tags had moved in the registry, an operating-system upgrade could silently become three application upgrades as well, potentially including application database migrations.

I resolved the image used by each running container and pinned that exact digest in the configuration:

image = "ghcr.io/example/app@sha256:<digest>";

After reboot, the applications would therefore run the same images that had already been working on the old system.

The lesson is not that tags should never be used. It is that one change window should deal with one layer of the stack. Finish and validate the operating-system upgrade first, then upgrade the applications as separate, reviewable changes. Otherwise, the troubleshooting space grows exponentially.

Step 3: Migrate the Configuration, Not Just the Option Names

The direct part of the 25.11-to-26.05 migration was updating the Flake releases:

nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";

home-manager = {
  url = "github:nix-community/home-manager/release-26.05";
  inputs.nixpkgs.follows = "nixpkgs";
};

I then followed evaluation errors and deprecation warnings through the configuration. A few cases were particularly instructive.

Move systemd-resolved to structured settings

The old extraConfig, dnssec, and fallbackDns options became structured settings:

services.resolved.settings.Resolve = {
  DNSSEC = false;
  DNSOverTLS = "no";
  MulticastDNS = "no";
  LLMNR = "yes";
  Cache = "yes";
  FallbackDNS = [
    "1.1.1.1"
    "8.8.8.8"
  ];
};

This looks like a syntax-only migration, but it still requires runtime verification. After reboot, I used resolvectl status to inspect the effective resolver, per-link DNS configuration, and resolv.conf mode.

Select the Ollama CPU package explicitly

The previous configuration used a Boolean acceleration setting:

services.ollama.acceleration = false;

The new release expresses that choice through the package:

services.ollama.package = pkgs.ollama-cpu;

When migrating configuration, preserve the original intent instead of merely searching for a new option with a similar name. The behavior I needed to retain here was “do not enable GPU acceleration.”

Pin the PostgreSQL server and client to the same major version

The PostgreSQL service was already pinned to version 15:

services.postgresql.package = pkgs.postgresql_15;

The system package list, however, still contained the unversioned postgresql package. Under the new nixpkgs release, that resolved to PostgreSQL 17. The full system build then found file collisions between the PostgreSQL 15 and 17 outputs in system-path.

The fix was to pin the CLI tools as well:

environment.systemPackages = with pkgs; [
  postgresql_15
];

Pinning a service does not necessarily pin every related tool in the system. For databases, compilers, and language runtimes with major-version compatibility boundaries, inspect the closure for accidental second versions.

Remove Flake outputs that are no longer delivered

The ThinkPad configuration and Home Manager package both built successfully, but nix flake check --no-build still found an obsolete NUC/Proxmox configuration that no longer evaluated under 26.05.

It could not stop the ThinkPad from booting, but it proved that the Flake was no longer internally consistent. After confirming that the machine had been retired, I removed its output, input, and host configuration.

An output with no owner and no continuous validation is not a harmless archive for possible future use. It is hidden maintenance cost waiting for the next upgrade.

Step 4: Build Everything—Do Not Treat eval as Acceptance

I first confirmed that the target system evaluated:

nix eval --raw \
  .#nixosConfigurations.nixos.config.system.build.toplevel.drvPath

Then I performed a real build of the complete system:

nix build \
  .#nixosConfigurations.nixos.config.system.build.toplevel \
  --print-build-logs \
  -o result-26.05

I also built the standalone Home Manager activation package:

nix build \
  .#homeConfigurations.cheverjohn.activationPackage \
  --no-link

An evaluation proves that module merging and expression evaluation succeed. It does not catch every problem, including:

• two packages installing the same file into system-path;
• an unavailable upstream download for proprietary software;
• a Home Manager activation package that fails to build;
• an error while generating a service script;
• a NAR missing from a regional binary cache with no fallback substituter.

This system required more than a thousand derivations, and its closure grew from roughly 31.6 GiB to 34.0 GiB. A full build costs more than an evaluation, but that cost is best paid while the old system is still healthy—not after rebooting into the new one.

Diff the closures, not just the source code

Once the build completed, I ran:

nix store diff-closures /run/current-system ./result-26.05

A Git diff tells me what I wrote in the configuration. A closure diff tells me what will actually run. It quickly exposed:

• major-version changes in the kernel, systemd, and Docker;
• whether PostgreSQL had moved unexpectedly;
• replacements of runtimes such as Node.js;
• the actual changes to D-Bus, the desktop, and firmware.

In a declarative system, the final closure—not the source diff—is the deployment artifact.

Step 5: Back Up the Data Before Installing the Boot Entry

An old generation can roll back system configuration and software. It cannot undo database writes or an application data migration. Before deployment, I therefore backed up:

• PostgreSQL with pg_dumpall;
• Redis after checking persistence and saving an RDB snapshot;
• bind-mounted state for file-based applications such as Open WebUI, with the relevant containers stopped;
• the current configuration, lock file, and important state inventories.

There is an important boundary here:

• generation rollback handles a system that does not boot or has broken configuration;
• data backup handles a new service that has already modified persistent state.

Preparing only one of them is not a complete recovery plan.

Only after the backups were complete did I install the prebuilt result into systemd-boot. I also verified that the boot menu still contained a NixOS 25.11 generation.

Step 6: Validate the Rebooted System in Layers

Reaching the desktop only proves that the machine booted. It does not prove that the upgrade is finished. I divided acceptance testing into several layers.

System identity

nixos-version
uname -r
readlink -f /run/current-system
systemctl is-system-running
systemctl --failed

The first task is to confirm that the running system is the intended store path, rather than an old generation selected by mistake.

Stateful services

pg_isready -U postgres
redis-cli ping
curl http://127.0.0.1:11434/api/version

An active process is not enough. I checked versions, existing data, persistence status, and real API responses.

Containers

For each container, I checked:

• health status;
• whether .Config.Image still contained the pinned digest;
• restart count;
• the external HTTP endpoint;
• application logs for migration failures.

Desktop and network

This layer covered the GNOME session, Home Manager user services, Wi-Fi, DNS, Bluetooth, and external connectivity. A desktop system cannot be accepted by testing system services alone: user units and desktop portals can also break across releases.

Two Problems That Appeared Only After Reboot

PostgreSQL collation warnings after the glibc update

The new system uses glibc 2.42, while the existing PostgreSQL cluster recorded collation version 2.40. Connections produced a collation version mismatch warning.

I did not react to the warning by immediately changing metadata. First, I performed read-only checks of the recorded and actual versions for each database and confirmed that there were no user tables or indexes. With a pre-upgrade backup already available, I refreshed the metadata:

ALTER DATABASE postgres REFRESH COLLATION VERSION;
ALTER DATABASE template1 REFRESH COLLATION VERSION;

If a database contains indexes that depend on locale-aware ordering, merely refreshing the version number is not enough. The affected objects must first be assessed and rebuilt. REFRESH COLLATION VERSION does not rebuild indexes for you.

ExecStart is not a shell command line

The system initially looked healthy after reboot. At midnight, however, a timer activated drop-caches.service; the service failed and changed the overall system state to degraded. Its existing configuration was:

serviceConfig.ExecStart =
  "${pkgs.coreutils}/bin/sync; echo 3 > /proc/sys/vm/drop_caches";

systemd does not pass ExecStart= through a shell by default. The ; and > characters do not mean command sequencing and redirection. systemd treated sync; as part of the executable name and returned 203/EXEC.

If shell behavior is genuinely required, use a NixOS service script:

systemd.services.drop-caches = {
  serviceConfig.Type = "oneshot";
  script = ''
    ${pkgs.coreutils}/bin/sync
    echo 3 > /proc/sys/vm/drop_caches
  '';
};

The better solution is probably to remove this timer entirely. Linux already manages the page cache, and regularly forcing it to be dropped usually does little beyond reducing cache hit rates.

NixOS 26.05 did not introduce this mistake. The broken configuration had existed for some time; the real timer execution simply fell outside my earlier observation windows. The lesson is that upgrade acceptance must cover delayed timers, not just the system state immediately after boot.

At the time of writing, this service remains a known follow-up item. I did not turn “the core services work” into the less honest claim that “the system has no problems.”

Do Not Let Every Red Log Line Drive the Upgrade

The new boot also logged duplicate D-Bus service names from dbus-broker, an Fcitx/PAM warning, and several BIOS, ACPI, and PMC messages.

They were worth recording, but the word error alone was not enough to classify them as blocking. I used five questions instead:

1. Does the message correspond to a failed unit?
2. Does an independent functional check fail?
3. Does the error repeat or trigger retries?
4. Was it introduced by the upgrade, or did it already exist?
5. Would fixing it expand the scope of the current change?

For example, dbus-broker printed many duplicate-name warnings, but GNOME, portals, and the keyring all worked. I recorded the messages as non-blocking noise instead of restructuring the entire desktop package set during the upgrade window.

Being rigorous does not mean changing configuration whenever a log line turns red. It means supporting every conclusion with state and functional evidence.

A Reusable Upgrade Checklist

Before the upgrade

• Create a dedicated upgrade branch.
• Record the system, kernel, Nix, systemd, and database versions.
• Check /, /boot, failed units, and current generations.
• Inventory every stateful service and its data directory.
• Record the exact digest of every running container image.
• Keep system.stateVersion and home.stateVersion unchanged.
• Confirm that garbage collection will not remove the rollback generation too soon.

Configuration and build

• Match the nixpkgs and Home Manager release branches.
• Handle removed, renamed, and semantically changed options.
• Pin database server and CLI tools to the intended major version.
• Freeze container images so applications do not upgrade at the same time.
• Build the complete NixOS system closure.
• Build the standalone Home Manager activation package.
• Run the formatter, git diff --check, and Flake checks.
• Review the effective version changes with nix store diff-closures.

Deployment and recovery

• Back up PostgreSQL, Redis, and file-based application state.
• Verify that backups are readable and record which services must stop before restoration.
• Deploy with boot to avoid a mixed live transition of foundational services.
• Confirm that an old generation remains in the boot menu.
• Define the conditions for system rollback and data restoration.

After reboot

• Confirm that /run/current-system points to the target closure.
• Check system and user units.
• Verify database connectivity, versions, data, and persistence.
• Verify container digests, health, restart counts, and HTTP endpoints.
• Test the desktop, Wi-Fi, DNS, Bluetooth, and external connectivity.
• Check the glibc/PostgreSQL collation version.
• Check failed units again after timers have had time to run.
• Keep backups and the old generation until the observation period ends.

Closing Thoughts

The NixOS 25.11-to-26.05 upgrade ultimately succeeded. The new kernel, systemd, dbus-broker, GNOME, Docker, PostgreSQL, Redis, Ollama, and Home Manager all work, while the old generation and pre-upgrade backups remain available.

What gives me confidence is not merely that this particular upgrade did not fail. It is that the process did not depend on luck:

• the build happened while the old system was still usable;
• application upgrades were moved out of this change window;
• both system rollback and data recovery existed before deployment;
• every post-reboot conclusion was supported by a command, a state check, or a real request;
• the one failed unit remained visible instead of being ignored for the sake of a clean conclusion.

The strength of NixOS is not just that configuration can be written as code. More importantly, it lets us decompose a risky upgrade into a sequence of reviewable, buildable, verifiable, and reversible state transitions. The tooling makes that possible; whether the upgrade is truly safe still depends on how we design the process.

TL;DR

• Symptom: https://traefik.example.local/dashboard/ shows Not secure in Chrome/Safari; Firefox works fine.
• Root cause: macOS Trust validator rejects wildcard cert *.example.local for subdomain traefik.example.local (OpenSSL accepts it, Apple Trust does not).
• Fix: Issue explicit hostname certificate for traefik.example.local instead of betting on wildcard.
• In one sentence: Validator rules determine everything, not "the cert is broken."

This was not "the certificate is broken." It was "the certificate is validated by a different engine." OpenSSL said OK. Apple Trust said no. Chrome/Safari follow Apple Trust.

This post has two parts: TLS basics + the real troubleshooting chain.

TLS basics (only the essentials)

Let’s keep the core pieces:

• Certificate chain: leaf → intermediate (optional) → root CA. Browsers trust roots, not leaves.
• SAN (Subject Alternative Name): hostname matching is done here; CN is legacy.
• Trust store: macOS uses System Keychain; Firefox has its own store; Chrome/Safari use system trust.
• Validation: the same cert can be accepted by one validator and rejected by another.

One sentence: “Valid” does not mean “accepted by your current validator.”

Problem Symptoms and Impact Scope

Accessing https://traefik.example.local/dashboard/#/ showed Not secure in Chrome/Safari.

Impact scope:

• ❌ Chrome (relies on macOS System Keychain)
• ❌ Safari (relies on macOS System Keychain)
• ✅ Firefox (uses its own cert store, unaffected)

Reproduction prerequisites:

• macOS version: Sonoma 14.x+ (other versions may behave differently)
• CA already installed in macOS Keychain and set to "Always Trust"
• Certificate issued by mkcert (version 1.4.4)
• Service running on 10.0.0.100

I needed three answers:

1. Does DNS point to the right server?
2. Is the cert chain complete and SAN correct?
3. Does the system trust engine accept the hostname?

Hypothesis Checklist (in elimination order)

Before troubleshooting, list possible root causes:

• H1: CA not properly installed in system trust store

  • Validation: Check macOS Keychain, confirm CA cert exists and is "Always Trust"
  • Expected: If CA missing, all browsers would fail (but Firefox works, so low probability)

• H2: Incomplete certificate chain (missing intermediate cert)

  • Validation: openssl s_client -showcerts to check chain depth
  • Expected: Self-signed CA usually has only two levels (leaf + root), mkcert default is complete

• H3: SAN does not include target domain

  • Validation: openssl x509 -text to check Subject Alternative Name field
  • Expected: If SAN missing or mismatched, OpenSSL would also fail (but curl passed, so low probability)

• H4: Validator rule differences (OpenSSL vs Apple Trust)

  • Validation: Use security verify-cert to simulate macOS system validation
  • Expected: This is the most likely root cause — different validators have different hostname matching rules for wildcards

Validate each hypothesis in order.

Validation Flow Visualization

This diagram shows the 4-layer validation progression (no execution, conceptual only):

#!/usr/bin/env bash
# Validation flow: 4-layer check to identify which layer rejects

cat <<'EOF'

┌─────────────────────────────────────────────────────────┐
│  Layer 1: DNS Resolution                                │
│  dig traefik.example.local +short                       │
│  ✅ Result: 10.0.0.100                                   │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│  Layer 2: Certificate Content (SAN)                     │
│  openssl s_client + grep "Subject Alternative Name"     │
│  ✅ Result: DNS:*.example.local, DNS:example.local      │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│  Layer 3: OpenSSL Validator (Linux/curl)                │
│  curl -Iv https://traefik.example.local/                │
│  ✅ Result: SSL certificate verify ok                   │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│  Layer 4: Apple Trust Validator (macOS)                 │
│  security verify-cert -n traefik.example.local          │
│  ❌ Result: Host name mismatch                          │
└─────────────────────────────────────────────────────────┘

Key finding: Layer 3 ✅ but Layer 4 ❌
→ Different validators have different hostname matching rules for wildcards
→ Chrome/Safari rely on Apple Trust (Layer 4)

EOF

The troubleshooting chain (in order)

1) DNS check

dig traefik.example.local +short

Expected output:

10.0.0.100

Observation: DNS points correctly, network layer is fine.

Conclusion: Network connectivity ruled out, proceed to certificate layer.

2) Inspect the cert chain and SAN

openssl s_client -connect traefik.example.local:443 -servername traefik.example.local -showcerts 2>/dev/null | openssl x509 -text -noout | grep -A2 "Subject Alternative Name"

Expected output (key part):

X509v3 Subject Alternative Name:
    DNS:*.example.local, DNS:example.local

Observation: SAN contains wildcard *.example.local, which should theoretically match traefik.example.local per RFC 6125.

Conclusion: Certificate fields are correct, H3 (SAN missing) eliminated.

3) OpenSSL validation (passes)

curl -Iv https://traefik.example.local/ 2>&1 | grep -E "SSL certificate verify|subject:"

Expected output:

* SSL certificate verify ok.
* subject: CN=*.example.local

Observation: OpenSSL validator considers cert valid, hostname matching passed.

Conclusion: OpenSSL finds no issues, H2 (incomplete chain) eliminated.

4) macOS system validation (fails)

# Export cert to temp file first (if not already done)
echo | openssl s_client -connect traefik.example.local:443 -servername traefik.example.local 2>/dev/null | \
  openssl x509 > /tmp/traefik-leaf.pem

# macOS system-level validation
security verify-cert -c /tmp/traefik-leaf.pem -p ssl -n traefik.example.local -L -v

Expected output (key error):

...certificate verification failed
...Host name mismatch

Observation: Apple Trust explicitly rejects this wildcard match for traefik.example.local.

Conclusion: Apple Trust considers this SAN unable to match the target domain. This is the pivotal point: OpenSSL OK, Apple Trust not OK. Chrome/Safari rely on Apple Trust. H4 (validator rule differences) confirmed.

Root cause

Not a missing CA, not a broken chain. The real cause:

macOS hostname validation did not accept the wildcard cert for traefik.example.local.

This is not theory; it is the system validator's explicit result.

Minimal, reliable fix

Stop relying on the wildcard. Issue an explicit hostname certificate.

mkcert -cert-file traefik.example.local.pem -key-file traefik.example.local-key.pem "traefik.example.local"

If you want to keep wildcard + apex too:

mkcert -cert-file traefik.example.local.pem -key-file traefik.example.local-key.pem \
  "traefik.example.local" "*.example.local" "example.local"

Then replace the Traefik TLS cert and restart the service.

A reusable checklist

• DNS resolution is correct (dig)
• SAN includes the target hostname (openssl x509 -text)
• OpenSSL validation (curl -Iv)
• macOS validation (security verify-cert)
• Browser trust store alignment

Retrospective and Prevention

One-Sentence Lesson

The most valuable evidence: The Host name mismatch output from security verify-cert — it directly proved the issue was not "CA trust" but "hostname validation rules."

If you remember one thing: Don't judge cert correctness by "it opens on my machine." Different validators (OpenSSL / Apple Trust / Firefox NSS) can reach different conclusions on the same cert.

Prevention Checklist (for next time you issue certs)

For internal self-signed CA (mkcert/self-built CA):

• Prefer explicit hostnames over betting on wildcard cross-platform compatibility
• If you must use wildcards, pre-validate with security verify-cert on macOS
• Keep issuance command records (with complete SAN list) for easy re-issuance

For public certs (Let's Encrypt, etc.):

• Request both example.com and *.example.com if you need to cover apex and subdomains
• Validate on at least 3 platforms: Linux (OpenSSL) / macOS (Apple Trust) / Windows (Schannel)

Validation template (reusable script):

#!/usr/bin/env bash
# Quick validation of cert behavior across validators

DOMAIN="traefik.example.local"

echo "==> OpenSSL validation"
curl -Iv https://${DOMAIN}/ 2>&1 | grep -E "SSL certificate verify|subject:"

echo "==> macOS validation (must run on macOS)"
# Export cert first
echo | openssl s_client -connect ${DOMAIN}:443 -servername ${DOMAIN} 2>/dev/null | \
  openssl x509 > /tmp/cert-to-verify.pem
# System validation
security verify-cert -c /tmp/cert-to-verify.pem -p ssl -n ${DOMAIN} -L -v

echo "==> Check SAN"
openssl s_client -connect ${DOMAIN}:443 -servername ${DOMAIN} </dev/null 2>/dev/null | \
  openssl x509 -text -noout | grep -A2 "Subject Alternative Name"

Final Conclusion

"I already installed the CA" is not the end of the story. What matters is which validator you are actually using.

This time the fix is simple and clean: issue an explicit cert for the host and the warning disappears.

This post summarizes what worked (and what hurt) when I deployed multiple self-hosted services in this repo (e.g. dockhand, vaultwarden, Checkmate, Miniflux). It focuses on repeatable defaults, pitfalls, and a practical debugging workflow.

Goals and non-negotiables

• Domain-only access: Do not expose host ports for business services. Avoid IP:port bypassing auth or leaking admin panels.
• Single entrypoint: Only open 80/443 externally. Let Traefik handle reverse proxying, TLS, and security headers.
• Shared middleware: Run shared MongoDB / Redis / PostgreSQL once and reuse them across apps to reduce waste.
• Never break userspace: Any change must not break existing online services (routing/port/network/auth conflicts).

Repo layout and my deployment defaults

• One app per directory: e.g. dockhand/, vaultwarden/, Checkmate/, v2/ (Miniflux).
• Each app has a runnable docker-compose.yml: avoid scattered upstream examples under contrib/.
• Shared middleware is a separate project: middlewares/ runs postgres / mongodb / redis.
  • Shared middleware joins internal networks only and exposes no public ports.
  • Apps join shared networks via external networks.
• One Traefik network: all services that Traefik should reach join an external network like traefik_proxy.
• Prefer Docker labels (docker provider): use file provider only for a few non-container targets or special cases.

Standard deployment playbook (the core tricks)

1) Do not publish host ports

For public-facing services:

• Use expose: (container-network visible only) instead of ports: (host visible).
• Route traffic via Traefik using labels.

This immediately reduces:

• Port conflicts (e.g. host 3000 already in use).
• IP:port bypass risks.

2) Traefik labels (domain routing + TLS + security headers)

Typical label set (example):

• traefik.enable=true
• traefik.http.routers.<name>.rule=Host(`example.com`)
• traefik.http.routers.<name>.entrypoints=websecure
• traefik.http.routers.<name>.tls.certresolver=letsencrypt
• traefik.http.routers.<name>.middlewares=security-headers@file
• traefik.http.services.<name>.loadbalancer.server.port=<container_port>

3) Multi-network containers must pin the Traefik network

When a container joins both:

• traefik_proxy (for Traefik)
• infra_backend (for databases/caches)

Traefik may pick the wrong container IP (from a network Traefik cannot reach), resulting in 504 Gateway Timeout.

Fix: add this label to the service:

• traefik.docker.network=traefik_proxy

This was the typical cause for Checkmate: adding it removed the 504 immediately.

Shared middleware (middlewares/)

Design principles

• High cohesion, low coupling: middleware provides network services only.
• Least privilege: no public ports by default; internal networks only.
• Centralized persistence: volumes managed in middlewares/ for easier backup/migration.

Practical notes

• Use a stable shared network name (e.g. infra_backend), and let apps join with external: true.
• Use service names in connection strings (e.g. postgres / mongodb).
• Prefer separate DB/schema/user per app to avoid data pollution.

App-specific pitfalls I hit this time

Dockhand (high-risk admin surface)

• Do not publish host ports, domain + Traefik only.
• Always add auth: protect it with Traefik basicAuth.
• Keep it simple; complexity is how incidents are born.

Vaultwarden (Bitwarden-compatible)

This is where it is easy to “be clever” and break clients:

• Do not put BasicAuth on the entire site: official clients and browser extensions will fail.
• Correct approach: protect /admin only.
  • Client paths like /, /api, /identity must not be blocked by BasicAuth.
• If “the UI loads but login fails”, check:
  • whether Cloudflare is challenging API requests (see below)
  • whether /api is treated as ServerConfig (may require rewriting to /api/config in Traefik)

Checkmate (multi-network + MongoDB)

• Sharing MongoDB + multiple networks often triggers the Traefik wrong-network issue:
  • symptoms: 504
  • fix: traefik.docker.network=traefik_proxy

Miniflux (rss reader)

• Miniflux requires PostgreSQL. No MongoDB/Redis is needed (unless you add RSSHub/Browserless, etc.).
• HEAD / returning 405 can be normal as long as GET works; do not misdiagnose it as downtime.

Cloudflare pitfall: clients/extensions cannot log in

I hit the same issue twice:

• Vaultwarden browser extension: Cloudflare challenge → 403
• Reeder (Google Reader API): Cloudflare challenge → 403

Common signals:

• response headers contain server: cloudflare + cf-mitigated: challenge
• requests never reach Traefik/app logs

Mitigation options (recommended order):

• A: set the subdomain to DNS-only (grey cloud): the most robust, no client-side hacks needed.
• B: keep proxy (orange cloud), but skip challenge for this host/path:
  • at minimum allow /api, /identity, /reader/api/0, /accounts/ClientLogin, etc.

Key takeaway:

• “It works in a browser” does not mean “it works in a client”. Browsers can pass challenges; clients usually cannot.

Debug playbook (fast to slow)

1) Decide whether the request reaches your server

• Cloudflare 403 challenge: request never reaches your server. Fix Cloudflare/WAF first.
• Traefik 404/504: request reaches Traefik, but routing/network/backend is wrong.
• Backend 4xx/5xx: request reaches the app. Inspect app logs/config.

2) Traefik-side debugging

• Check access logs: which router/service handled the request and what status code it returned.
• Multi-network containers: suspect missing traefik.docker.network first.
• Same host defined twice: suspect file-provider vs docker-provider conflicts (delete legacy file routes).

3) App-side debugging

• docker logs <container>: verify startup, listening port, DB connectivity.
• Verify you did not accidentally use ports: and expose/conflict host ports.

Final security checklist (before going public)

• No business ports published on the host (only keep 80/443)
• Admin surfaces must be authenticated (dockhand, vaultwarden /admin, etc.)
• Databases/caches are never exposed to the public internet
• No Traefik routing conflicts (avoid defining the same host in both file-provider and docker-provider)
• Cloudflare must not challenge API paths (clients/extensions/third-party integrations will break)

This is a troubleshooting log template. Facts first, evidence second, conclusions last.

TL;DR

• Symptoms: The DNS service becomes unresponsive; the whole machine stutters/freezes; sometimes it looks like storage I/O issues or a “disk drop”.
• Pattern: Random, usually after 1+ day of continuous running.
• Recovery: Power off and let it cool down for a while, then it works again.
• Current conclusion (probabilistic): ____ (e.g., thermals / power / storage-controller reset).

Impact

• Direct impact: ____ (e.g., LAN-wide DNS timeouts).
• Indirect impact: ____ (services that depend on name resolution).
• Time window: ____ (start time / duration / auto-recovery?).

Environment (reproducible)

Hardware

• CPU: Intel® N100 (4C/4T, 6MB cache, up to 3.4GHz)
• GPU: Intel® UHD Graphics
• RAM: 8GB LPDDR5 (single channel, onboard, 4800MHz)
• Storage: UFS 2.1 256G
• Wireless: Intel AX200/201 (Wi‑Fi 6 / Bluetooth 5.2)
• Ethernet: 2.5G RJ45 ×1 (PoE IEEE 802.3at)
• Ports: USB 3.2 Gen2 Type‑A ×2; USB 3.2 Gen2 Type‑C ×1 (Alt DP/PD); HDMI ×1
• Power: 65W USB‑C Power Delivery adapter

Software

• Actual OS: Linux (current kernel: 6.8.12-5-pve; factory OS: Windows 11 Home)
• Kernel (if Linux): 6.8.12-5-pve
• Bootloader (GRUB): grub2 2.06-13+pmx2
• DNS stack: ____ (dnsmasq / unbound / AdGuard Home / Pi‑hole / other)
• Deployment: ____ (systemd / Docker / other)
• Other services: ____
• Logging policy: ____ (write frequency / rotation / same disk?).

Problem statement (facts only)

• First seen: ____
• Frequency: about every ____ hours/days (random).
• Observable symptoms during failure:
  • DNS: ____ (timeouts / refusal / high latency)
  • System: ____ (SSH/RDP reachable? CPU/memory spike?)
  • Storage: ____ (mount disappears / read-only remount / I/O error / device reset)
  • Network: ____ (packet loss? NIC reset?)
• Recovery: ____ (power off for ____ minutes; does reboot also work: ____)
• Load correlation: ____ (QPS / logging / other tasks).

Immediate mitigation (timeline)

• T+0: ____ (e.g., switch upstream DNS)
• T+5m: ____ (stop DNS or high-IO workloads)
• T+10m: ____ (collect logs and metrics snapshots)
• Result: ____

Evidence model (align everything by timestamp)

• One timeline: failure start time, temperature peak, I/O errors, service timeouts.
• One core question: is DNS dead, or is the storage/system dead and DNS is just the first visible symptom?

Hypotheses (prioritized)

• H1: Thermals / overheating → throttling or protection → instability
  • Evidence: ____
• H2: Power instability (PD/PoE/adapter) → controller reset
  • Evidence: ____
• H3: Storage media/controller issue → I/O stall or device drop
  • Evidence: ____
• H4: Software resource exhaustion (FD/memory/log amplification/IO saturation)
  • Evidence: ____
• H5: Kernel/driver bug (power management, device PM policy)
  • Evidence: ____

Data to capture (during failure)

• Temps/fans: ____ (sensors, /sys/class/thermal/, Windows tools)
• System logs: ____ (journalctl, Windows Event Viewer)
• Storage logs: ____ (I/O error, timeout, reset, read-only remount)
• Resource metrics: ____ (CPU/load/iowait/memory/disk throughput)
• DNS metrics: ____ (QPS, cache hit ratio, upstream latency, error rate)
• Client checks: ____ (dig/nslookup outputs)

Key evidence (3–5 snippets)

[time] ____

[time] ____

[time] ____

Investigation (Attempt → Observation → Conclusion)

Attempt A: ____

• Change: ____
• Observation: ____
• Conclusion: ____

Attempt B: ____

• Change: ____
• Observation: ____
• Conclusion: ____

Attempt C: ____

• Change: ____
• Observation: ____
• Conclusion: ____

Final conclusion (probability, not absolute claims)

• Most likely root cause: ____
• Secondary factors: ____
• Ruled out: ____ (with evidence)

Fix plan (short-term vs long-term)

Short-term mitigation

• ____ (e.g., improve cooling)
• ____ (e.g., cap power / disable aggressive power saving)
• ____ (e.g., reduce disk writes / configure log rotation)

Long-term fix

• ____ (e.g., replace power adapter / replace storage / BIOS update)
• ____ (e.g., airflow redesign / thermal pads)
• ____ (e.g., kernel/OS change)

Validation

• Uptime: ____ days without recurrence
• Temperature: CPU ≤ ____°C; storage ≤ ____°C
• Logs: no I/O errors / resets / read-only remount

Retrospective

• Biggest learning: ____
• What I will do earlier next time: ____ (monitoring, alerting, auto log capture)
• Open questions: ____

Attempt plan: Monitor 192.168.22.18 (Minisforum) via a monitoring LXC (192.168.22.19) on PVE 192.168.22.12

Goal: turn “it feels like overheating / disk drop / DNS died” into an evidence-aligned timeline: temps + kernel/storage errors + real DNS query success/latency + host reachability.

Note: never put passwords into configs or the repo. Any credentials should live in root-only local files on the monitoring host.

Extra pitfall: port 22 is reachable, but ssh still hangs / drops (more common across subnets)

This one is misleading: nc says 22/tcp open, so people assume “network is fine”. But ssh can still stall during handshake and eventually time out (or die with Broken pipe).

Topology (what happened in this incident)

• Client subnet: 192.168.11.0/24
• Target host: 192.168.22.12 (PVE)
• Same-subnet control point: 192.168.22.18 (SSH from here to .12 is stable)
• Correlated signal: DNS server (e.g. 192.168.22.53:53) experienced timeouts — reverse lookups can amplify SSH handshake latency.

Symptoms (typical)

• nc -vz -G 2 192.168.22.12 22 reports succeeded (TCP handshake is OK)
• ssh root@192.168.22.12 hangs and later shows:
  • ssh_dispatch_run_fatal: Connection to 192.168.22.12 port 22: Operation timed out
  • or Read from remote host 192.168.22.12: Operation timed out / client_loop: send disconnect: Broken pipe

Quick triage (split “port is open” into smaller truths)

1. Check whether the SSH banner is returned immediately (protocol only, no auth):

nc -v 192.168.22.12 22

You should quickly see SSH-2.0-OpenSSH_.... If the TCP connect succeeds but the banner is slow/missing, the issue is likely in sshd/system path (DNS reverse lookup stalls, high load, policy/ACL causing unstable data path), not “port blocked”.

2. Capture where SSH stalls (useful evidence):

ssh -vvv -o ConnectTimeout=5 -o ServerAliveInterval=5 -o ServerAliveCountMax=1 root@192.168.22.12

Minimal fix (eliminate DNS/GSSAPI special cases first)

If you can SSH from a same-subnet host (e.g. .22.18) into .22.12, prefer the server-side fix:

1. On .22.12, edit /etc/ssh/sshd_config and make sure you have:

• UseDNS no
• GSSAPIAuthentication no

2. Validate and restart:

sshd -t && systemctl restart ssh
systemctl status ssh --no-pager

Practical rationale: when DNS/reverse lookup is flaky, UseDNS yes can block parts of the handshake/audit path. Disabling it removes an entire class of “it depends on DNS today” failures.

If it is still flaky: continue with the “cross-subnet triad”

• Return path routing (the target must know how to route back to 192.168.11.0/24):

ip route get 192.168.11.12

• Firewall/ACL/state tracking: allow not only 22/tcp inbound but also the correct return direction and conntrack behavior (especially with gateways/NAT/policy routing).
• MTU/fragmentation: tunnels/PPPoE/VLAN paths can yield “SYN works but data is unstable”; validate with DF-sized pings on Linux and temporarily lower MSS/MTU to confirm.

Architecture (minimal, but sufficient)

• 192.168.22.19 (monitoring LXC on PVE 192.168.22.12): Prometheus + Grafana + blackbox_exporter (optional Pushgateway)
• 192.168.22.18 (target host): nodeexporter (host metrics) + (optional) smartctlexporter + custom textfile metrics (kernel/I/O error counter)

Data model:

• Time series (Prometheus): scrape every 15s, keep at least 48h around failures.
• Active probing (blackbox_exporter): probe .18/.54 from .19 to separate “host down” from “service down”.
• Alerting: pin the incident timestamp immediately.

Attempt D: Bring up the monitoring stack (Prometheus + Grafana + blackbox_exporter)

D0. Pre-checks (network & ports)

From 192.168.22.19, verify reachability to 192.168.22.18:

ping 192.168.22.18
Test-NetConnection 192.168.22.18 -Port 22

Ports used (defaults):

• .18:9100 node_exporter
• .18:9633 smartctl_exporter (optional)
• .19:9090 Prometheus
• .19:3000 Grafana
• .19:9115 blackbox_exporter
• .19:9091 Pushgateway (optional)

D1. Deploy node_exporter on 192.168.22.18 (systemd, stable)

Run on .18:

export VER="1.7.0"
curl -fsSL -o /tmp/node_exporter.tar.gz "https://github.com/prometheus/node_exporter/releases/download/v${VER}/node_exporter-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/node_exporter.tar.gz
install -m 0755 /tmp/node_exporter-${VER}.linux-amd64/node_exporter /usr/local/bin/node_exporter

useradd --system --no-create-home --shell /usr/sbin/nologin nodeexp || true
mkdir -p /var/lib/node_exporter/textfile_collector
chown -R nodeexp:nodeexp /var/lib/node_exporter

cat >/etc/systemd/system/node_exporter.service <<'EOF'
[Unit]
Description=Prometheus Node Exporter
After=network-online.target
Wants=network-online.target

[Service]
User=nodeexp
Group=nodeexp
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=:9100 \
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now node_exporter
systemctl status node_exporter --no-pager

Verify (from .12 or any LAN host):

curl -fsS http://192.168.22.18:9100/metrics | head

D2. (Recommended) Add a kernel/storage error counter as a custom metric

The keywords that matter: I/O error|timeout|reset|remount read-only|EXT4-fs error|BTRFS error|blk_update_request

Use node_exporter textfile collector and a timer:

cat >/usr/local/bin/collect_kernel_error_metrics.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

OUT="/var/lib/node_exporter/textfile_collector/kernel_errors.prom"
TMP="$(mktemp)"

COUNT="$(journalctl -k --since "10 min ago" --no-pager 2>/dev/null | \
  grep -Eic 'I/O error|timeout|reset|remount read-only|EXT4-fs error|BTRFS error|blk_update_request|nvme|ufs' || true)"

NOW="$(date +%s)"

{
  echo "# HELP kernel_error_events_10m Number of kernel error-like events in last 10 minutes"
  echo "# TYPE kernel_error_events_10m gauge"
  echo "kernel_error_events_10m ${COUNT}"
  echo "# HELP kernel_error_scrape_time_seconds Last collect time"
  echo "# TYPE kernel_error_scrape_time_seconds gauge"
  echo "kernel_error_scrape_time_seconds ${NOW}"
} >"${TMP}"

mv "${TMP}" "${OUT}"
EOF

chmod +x /usr/local/bin/collect_kernel_error_metrics.sh

cat >/etc/systemd/system/kernel-error-metrics.service <<'EOF'
[Unit]
Description=Collect kernel error metrics for node_exporter textfile collector

[Service]
Type=oneshot
ExecStart=/usr/local/bin/collect_kernel_error_metrics.sh
EOF

cat >/etc/systemd/system/kernel-error-metrics.timer <<'EOF'
[Unit]
Description=Run kernel error metrics collector every 60s

[Timer]
OnBootSec=30s
OnUnitActiveSec=60s
AccuracySec=1s

[Install]
WantedBy=timers.target
EOF

systemctl daemon-reload
systemctl enable --now kernel-error-metrics.timer

Verify:

curl -fsS http://192.168.22.18:9100/metrics | grep -E '^kernel_error_events_10m|^kernel_error_scrape_time_seconds' || true

D3. (Optional) smartctl_exporter for disk health

If the storage device does not expose SMART/health, this may not work. Still worth trying.

apt-get update
apt-get install -y smartmontools

export VER="0.12.0"
curl -fsSL -o /tmp/smartctl_exporter.tar.gz "https://github.com/prometheus-community/smartctl_exporter/releases/download/v${VER}/smartctl_exporter-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/smartctl_exporter.tar.gz
install -m 0755 /tmp/smartctl_exporter-${VER}.linux-amd64/smartctl_exporter /usr/local/bin/smartctl_exporter

useradd --system --no-create-home --shell /usr/sbin/nologin smartctl-exp || true

cat >/etc/systemd/system/smartctl_exporter.service <<'EOF'
[Unit]
Description=Prometheus Smartctl Exporter
After=network-online.target
Wants=network-online.target

[Service]
User=smartctl-exp
Group=smartctl-exp
ExecStart=/usr/local/bin/smartctl_exporter --web.listen-address=:9633
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now smartctl_exporter

D4. Deploy Prometheus + Grafana + blackbox_exporter on 192.168.22.12 (PVE) via LXC (recommended)

Since 192.168.22.12 is also a PVE host, avoid piling Docker onto the host OS. The simplest operationally is: one Debian LXC container running the monitoring stack with systemd.

D4.1 Create a Debian LXC container on PVE (.12)

You can do it in the PVE Web UI, or via CLI like this (replace <CTID>, <GW>; LXC IP is pinned to 192.168.22.19):

# Run on the PVE host 192.168.22.12
pct create <CTID> local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname monitoring \
  --cores 2 --memory 2048 --swap 512 \
  --rootfs local-lvm:8 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.22.19/24,gw=<GW> \
  --unprivileged 1

pct set <CTID> -features keyctl=1,nesting=1
pct start <CTID>

D4.2 Install Prometheus in the container (systemd)

Enter the container:

pct exec <CTID> -- bash

Inside the container:

apt-get update
apt-get install -y ca-certificates curl tar

useradd --system --no-create-home --shell /usr/sbin/nologin prometheus || true
mkdir -p /etc/prometheus /var/lib/prometheus
chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

export VER="2.54.1"
curl -fsSL -o /tmp/prometheus.tar.gz "https://github.com/prometheus/prometheus/releases/download/v${VER}/prometheus-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/prometheus.tar.gz
install -m 0755 /tmp/prometheus-${VER}.linux-amd64/prometheus /usr/local/bin/prometheus
install -m 0755 /tmp/prometheus-${VER}.linux-amd64/promtool /usr/local/bin/promtool
cp -r /tmp/prometheus-${VER}.linux-amd64/consoles /etc/prometheus/
cp -r /tmp/prometheus-${VER}.linux-amd64/console_libraries /etc/prometheus/
chown -R prometheus:prometheus /etc/prometheus

Create /etc/prometheus/prometheus.yml (scrape .18 + active probes; DNS is the AdGuardDNS LXC at .54):

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "minisforum-node"
    static_configs:
      - targets: ["192.168.22.18:9100"]

  - job_name: "minisforum-smartctl"
    static_configs:
      - targets: ["192.168.22.18:9633"]

  - job_name: "blackbox-icmp"
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
          - 192.168.22.18
          - 192.168.22.54
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

  - job_name: "blackbox-tcp"
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
          - 192.168.22.18:22
          - 192.168.22.18:9100
          - 192.168.22.54:53
          - 192.168.22.54:3000
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

Create the systemd service:

cat >/etc/systemd/system/prometheus.service <<'EOF'
[Unit]
Description=Prometheus
After=network-online.target
Wants=network-online.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now prometheus

D4.3 Install blackbox_exporter in the container (systemd)

useradd --system --no-create-home --shell /usr/sbin/nologin blackbox || true
mkdir -p /etc/blackbox_exporter
chown -R blackbox:blackbox /etc/blackbox_exporter

export VER="0.25.0"
curl -fsSL -o /tmp/blackbox.tar.gz "https://github.com/prometheus/blackbox_exporter/releases/download/v${VER}/blackbox_exporter-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/blackbox.tar.gz
install -m 0755 /tmp/blackbox_exporter-${VER}.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter

cat >/etc/blackbox_exporter/blackbox.yml <<'EOF'
modules:
  icmp:
    prober: icmp
    timeout: 5s

  tcp_connect:
    prober: tcp
    timeout: 5s
EOF

chown -R blackbox:blackbox /etc/blackbox_exporter

cat >/etc/systemd/system/blackbox_exporter.service <<'EOF'
[Unit]
Description=Prometheus Blackbox Exporter
After=network-online.target
Wants=network-online.target

[Service]
User=blackbox
Group=blackbox
ExecStart=/usr/local/bin/blackbox_exporter \
  --config.file=/etc/blackbox_exporter/blackbox.yml \
  --web.listen-address=:9115
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now blackbox_exporter

D4.4 Install Grafana in the container (systemd)

Do not commit any Grafana admin password. Install it, then set the password manually.

apt-get install -y apt-transport-https software-properties-common wget gpg
mkdir -p /etc/apt/keyrings
wget -qO- https://apt.grafana.com/gpg.key | gpg --dearmor >/etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" >/etc/apt/sources.list.d/grafana.list
apt-get update
apt-get install -y grafana
systemctl enable --now grafana-server

Reset admin password (run in the container):

grafana-cli admin reset-admin-password

Attempt E: Deploy AdGuardDNS on 192.168.22.18 via LXC (fixed IP: 192.168.22.54) + real DNS query probing

Goal: isolate DNS service from the host and pin the DNS IP to 192.168.22.54 so monitoring targets never drift.

E0. Create the AdGuardDNS LXC container on PVE (.18) with IP 192.168.22.54

Example (replace <CTID>, <GW>):

# Run on the PVE host 192.168.22.18
pct create <CTID> local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname adguarddns \
  --cores 1 --memory 1024 --swap 256 \
  --rootfs local-lvm:4 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.22.54/24,gw=<GW> \
  --unprivileged 1

pct set <CTID> -features keyctl=1,nesting=1
pct start <CTID>

E1. Install AdGuard Home inside the container (as AdGuardDNS)

Enter the container:

pct exec <CTID> -- bash

Inside the container:

apt-get update
apt-get install -y ca-certificates curl tar

export VER="0.107.57"
curl -fsSL -o /tmp/adguardhome.tar.gz "https://github.com/AdguardTeam/AdGuardHome/releases/download/v${VER}/AdGuardHome_linux_amd64.tar.gz"
tar -C /opt -xzf /tmp/adguardhome.tar.gz

/opt/AdGuardHome/AdGuardHome -s install
systemctl status AdGuardHome --no-pager

First-time setup:

• Admin UI: http://192.168.22.54:3000
• DNS: 192.168.22.54:53 (UDP/TCP)

E1. Real DNS query probe using dig + Pushgateway (credential-free)

Use .12 to run dig periodically and push two metrics: success and latency.

1. Optionally install Pushgateway on the monitoring side (the .12 monitoring LXC) via systemd:

apt-get update
apt-get install -y ca-certificates curl tar

useradd --system --no-create-home --shell /usr/sbin/nologin pushgateway || true
mkdir -p /etc/pushgateway /var/lib/pushgateway
chown -R pushgateway:pushgateway /etc/pushgateway /var/lib/pushgateway

export VER="1.10.0"
curl -fsSL -o /tmp/pushgateway.tar.gz "https://github.com/prometheus/pushgateway/releases/download/v${VER}/pushgateway-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/pushgateway.tar.gz
install -m 0755 /tmp/pushgateway-${VER}.linux-amd64/pushgateway /usr/local/bin/pushgateway

cat >/etc/systemd/system/pushgateway.service <<'EOF'
[Unit]
Description=Prometheus Pushgateway
After=network-online.target
Wants=network-online.target

[Service]
User=pushgateway
Group=pushgateway
ExecStart=/usr/local/bin/pushgateway --web.listen-address=:9091
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now pushgateway
systemctl status pushgateway --no-pager

2. Add a Prometheus scrape job (use 127.0.0.1 since Pushgateway runs on the same container):

- job_name: "pushgateway"
  honor_labels: true
  static_configs:
    - targets: ["127.0.0.1:9091"]

3. Create a probe script (DNS IP is pinned to 192.168.22.54):

cat >/opt/monitoring/dns_probe_push.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

DNS_IP="192.168.22.54"
NAME="www.baidu.com"
PUSH_URL="http://127.0.0.1:9091/metrics/job/dns_probe/instance/${DNS_IP}"

OUT="$(dig +tries=1 +time=2 +stats @"${DNS_IP}" "${NAME}" A 2>/dev/null || true)"

if echo "${OUT}" | grep -qE '^;; ANSWER SECTION:'; then
  SUCCESS=1
else
  SUCCESS=0
fi

LAT_MS="$(echo "${OUT}" | awk -F': ' '/^;; Query time:/{print $2}' | awk '{print $1}' || true)"
LAT_MS="${LAT_MS:-0}"

cat <<METRICS | curl -fsS --data-binary @- "${PUSH_URL}" >/dev/null
# TYPE dns_probe_success gauge
dns_probe_success ${SUCCESS}
# TYPE dns_probe_latency_ms gauge
dns_probe_latency_ms ${LAT_MS}
METRICS
EOF

chmod +x /opt/monitoring/dns_probe_push.sh

4. Schedule it (during investigation, 15s is useful):

crontab -e
*/1 * * * * /opt/monitoring/dns_probe_push.sh
*/1 * * * * sleep 15; /opt/monitoring/dns_probe_push.sh
*/1 * * * * sleep 30; /opt/monitoring/dns_probe_push.sh
*/1 * * * * sleep 45; /opt/monitoring/dns_probe_push.sh

E2. (Optional) AdGuard Home internal stats (requires admin API creds; never commit them)

If you need AdGuard-specific stats (blocked/allowed/qps/upstream latency), run an exporter on .12 that talks to AdGuard’s admin API. Store credentials in a root-only local file (e.g. /opt/monitoring/secrets/adguard.env, chmod 600) and keep it out of Git.

This is a practical troubleshooting post. Facts first, evidence second, conclusions last.

TL;DR

• Symptom: Cross-VLAN ping times out. The target host shows echo reply sent, yet the client never receives it.
• Root cause #1 (decisive): PVE intentionally kept OpenWrt (192.168.22.13) as the default gateway for QoS/traffic handling, but had no explicit routes for other internal VLAN subnets, so return traffic to internal networks followed the default route into OpenWrt and never traversed the primary L3 gateway (classic asymmetric path).
• Root cause #2 (amplifier): OpenWrt had masq=1 on the LAN zone, applying SNAT to internal east-west traffic and making paths/sessions unpredictable.
• Fix:
  • Keep PVE default gateway = OpenWrt (192.168.22.13) for QoS, but add explicit routes for internal subnets (e.g., 192.168.11.0/24, 192.168.183.0/24) via the primary gateway (192.168.22.1)
  • Disable masquerade on OpenWrt lan zone
• Prevention: Route internally, NAT only on real WAN egress; if a side gateway must be the default route (QoS/proxy), explicitly split internal routing vs Internet default; use tcpdump + ip route get as a standard SOP.

Background (Topology and Roles)

Multi-VLAN home network:

• Cloud Gateway Fiber (primary L3 gateway)
  • Inter-VLAN routing
  • Example subnets:
    • VLAN11: 192.168.11.0/24 (GW 192.168.11.1)
    • VLAN22 (PVE/Lab): 192.168.22.0/24 (GW 192.168.22.1)
    • VLAN183: 192.168.183.0/24 (GW 192.168.183.1)
• OpenWrt (secondary / side gateway)
  • 192.168.22.13/24 on br-lan
  • fw3 + iptables, plus proxy tooling (OpenClash/Passwall)
• PVE node: 192.168.22.12/24 on vmbr0
• Test hosts:
  • 192.168.183.235 (VLAN183)
  • 192.168.11.29 (VLAN11)

The intended design is simple:

• Inter-VLAN traffic should be routed by the primary gateway.
• NAT should only happen on a true WAN egress, not on internal VLAN-to-VLAN paths.

ASCII topology (printable via shell)

This script does not change any config. It prints the topology and the two key paths (bad vs fixed) so readers can visualize the data flow:

#!/usr/bin/env bash
set -euo pipefail

# Print the topology and the two paths (bad vs fixed).
cat <<'EOF'

                    +------------------------------+
                    |   Cloud Gateway Fiber (L3)   |
                    |   VLAN11: 192.168.11.1       |
                    |   VLAN22: 192.168.22.1       |
                    |   VLAN183: 192.168.183.1     |
                    +---------------+--------------+
                                    |
                                  VLAN22
                                    |
                   +----------------+----------------+
                   |                                 |
        +----------v-----------+          +----------v-----------+
        |  OpenWrt (side GW)   |          |        PVE           |
        |  192.168.22.13       |          |  192.168.22.12       |
        |  QoS / traffic mgmt  |          |  vmbr0               |
        +----------------------+          +----------------------+

Clients:
  VLAN183 host: 192.168.183.235
  VLAN11  host: 192.168.11.29

Problem #1 (before):
  192.168.183.235 -> Fiber -> PVE
  PVE reply -> (default gw) OpenWrt -> [NAT/proxy/unknown] -> ???   (Fiber never sees the reply)

Fix idea:
  Keep PVE default gw = OpenWrt (for QoS),
  but route internal subnets via Fiber.

Problem #2 (amplifier):
  OpenWrt LAN masquerade (SNAT) rewrites east-west traffic,
  making internal routing/session behavior unpredictable.

EOF

Incidents and Impact

Incident #1: VLAN183 could not ping PVE

• Client 192.168.183.235 → ping 192.168.22.12 timed out
• On PVE, captures showed:
  • ICMP echo request arriving
  • ICMP echo reply being generated
• On the primary gateway (Fiber), captures on br22/br0 showed requests only, no replies

Meaning: the reply never traversed the primary gateway.

Incident #2: After fixing #1, VLAN11 still could not ping PVE

• 192.168.11.29 could not ping 192.168.22.12
• But it could ping OpenWrt 192.168.22.13

This pointed to OpenWrt NAT/firewall behavior affecting internal reachability.

Investigation (Evidence-driven)

1) Confirm the primary gateway is fine

On Fiber:

• It could ping both 192.168.183.235 and 192.168.22.12.
• ip route showed both subnets as directly connected.

Conclusion: core L3 routing on Fiber was not broken.

2) Hop-by-hop capture: find where the reply disappears

• On PVE: request/reply both exist → PVE is replying.
• On Fiber (br22/br0): request only → reply never reached Fiber.

Conclusion: the return path is not going through the primary gateway.

3) Let ip route get tell the truth

On PVE:

• ip route showed: default via 192.168.22.13 dev vmbr0
• ip route get 192.168.183.235 showed: via 192.168.22.13

Root cause #1: PVE had no explicit route to internal VLAN subnets, so return traffic to 192.168.183.0/24 naturally followed the default route into OpenWrt. Once that happens, any NAT/proxy/misroute on OpenWrt can prevent the primary gateway from ever seeing the reply.

That creates asymmetric routing:

• Forward: 183 → Fiber → 22 → PVE (OK)
• Return: PVE → OpenWrt → (NAT/proxy/unknown) (uncontrolled)

4) Validate OpenWrt firewall/NAT configuration

After confirming OpenWrt uses fw3 + iptables (no nftables), the key finding was:

• lan zone had masq='1'

Root cause #2: OpenWrt applied SNAT to internal traffic, rewriting sources to 192.168.22.13 and making internal routing/session behavior unstable, especially with proxy redirection.

Fix (Minimal changes)

Fix #1: Keep default gateway = OpenWrt, but route internal subnets via Fiber

Constraint: PVE default route must remain 192.168.22.13 (OpenWrt) for QoS/traffic handling. In that case, do not touch the default route. Instead, explicitly route internal VLAN subnets via the primary L3 gateway (192.168.22.1).

Temporary test (examples for VLAN11/VLAN183; add others as needed):

# Keep default via OpenWrt for QoS / traffic management
# default via 192.168.22.13

# Route internal VLANs via the primary L3 gateway (Fiber)
ip route replace 192.168.183.0/24 via 192.168.22.1 dev vmbr0
ip route replace 192.168.11.0/24  via 192.168.22.1 dev vmbr0

Verification tips:

• ip route get 192.168.183.235 should show via 192.168.22.1
• ICMP replies should be visible on Fiber interfaces (e.g., br22/br0)

Persist it in Proxmox networking (commonly /etc/network/interfaces for vmbr0) with post-up:

post-up ip route replace 192.168.183.0/24 via 192.168.22.1 dev vmbr0
post-up ip route replace 192.168.11.0/24  via 192.168.22.1 dev vmbr0
post-down ip route del 192.168.183.0/24 via 192.168.22.1 dev vmbr0 || true
post-down ip route del 192.168.11.0/24  via 192.168.22.1 dev vmbr0 || true

Verification:

• Fiber sees ICMP replies on br22/br0.
• 192.168.183.235 can ping 192.168.22.12 again.

Fix #2: Disable masquerade on OpenWrt LAN zone

uci set firewall.@zone[0].masq='0'
uci commit firewall
/etc/init.d/firewall restart

Verification:

• 192.168.11.29 can ping 192.168.22.12.
• Internal traffic keeps real source IPs, making debugging and ACLs sane.

Prevention

• Route internally; NAT only on real WAN
  • lan/internal zones: masq=0
  • wan egress: masq=1
• If a side gateway must be the default route (QoS/proxy), split routing explicitly
  • Default route can stay on OpenWrt, but internal subnets must have explicit routes via the primary L3 gateway (or use policy routing by destination).
  • Keep east-west traffic on predictable L3 routing. Do not let it fall into NAT/proxy black boxes.
• Standard SOP (Standard Operating Procedure)
  • Meaning: for “cross-VLAN ping fails”, do not guess. Follow the same evidence-driven checklist every time.
  • Step 1: packet captures answer 3 questions: did the request arrive, did the target reply, and at which hop did the reply disappear?
    • On PVE: tcpdump -ni vmbr0 icmp
    • On the primary gateway (both VLAN interfaces): tcpdump -ni br22 icmp and tcpdump -ni br0 icmp
  • Step 2: verify routing decisions (no imagination)
    • On PVE: ip route + ip route get 192.168.183.235
    • On OpenWrt/Fiber: ip route get 192.168.183.235 (confirm the actual next hop)
  • Step 3: simplify the data path first
    • Restore pure routing with real source IPs and predictable return paths before layering QoS/proxy/policy routing.
• Make proxy behavior explicit
  • Exclude internal subnets from transparent proxy/tunnel redirection.

One-line lesson

If ICMP looks like “ghost packets”, you most likely built a black box with the wrong default gateway and the wrong NAT — not a broken protocol stack.

As AI assistants become increasingly integrated into our daily workflows, understanding how to properly leverage tools like Claude can significantly enhance your productivity and decision-making. This comprehensive guide will help you maximize the value you get from Claude while maintaining good practices and realistic expectations.

Understanding Claude's Core Strengths

Before diving into specific use cases, it's crucial to understand what Claude excels at:

1. Text Analysis and Processing

Claude can quickly analyze, summarize, and extract insights from large volumes of text. Whether you're dealing with research papers, meeting notes, or technical documentation, Claude can help you identify key points and patterns.

2. Creative and Technical Writing

From drafting emails to writing code, Claude can assist with various forms of content creation while maintaining your personal voice and style preferences.

3. Problem-Solving and Research

Claude can help break down complex problems into manageable components and provide structured approaches to finding solutions.

4. Learning and Explanation

Need something explained? Claude can adapt explanations to your level of expertise and learning style.

Essential Best Practices

Start with Clear, Specific Prompts

Good: "Help me write a professional email declining a meeting request for Thursday afternoon, suggesting alternative times next week, and maintaining a collaborative tone."

Poor: "Write an email for me."

The more context and specificity you provide, the better Claude can tailor its response to your needs.

Iterate and Refine

Don't expect perfect results on the first try. Use follow-up prompts to:

• Ask for clarification or expansion on specific points
• Request different approaches or styles
• Provide feedback on what works and what doesn't

Verify Important Information

While Claude is knowledgeable, always verify critical facts, especially for:

• Medical or legal advice
• Financial decisions
• Current events or rapidly changing information
• Technical specifications or requirements

Practical Use Cases for Individuals

Professional Development

Email Communication:

• Draft professional emails with appropriate tone
• Summarize long email threads
• Translate complex technical concepts for non-technical audiences

Document Creation:

• Create structured reports and presentations
• Write job application materials
• Develop project plans and timelines

Learning and Skill Development:

• Get explanations of complex topics
• Practice interview questions
• Learn new programming languages or frameworks

Personal Productivity

Research and Planning:

• Compare options for major purchases
• Plan travel itineraries
• Research topics of personal interest

Creative Projects:

• Brainstorm ideas for hobbies or side projects
• Get feedback on creative writing
• Plan events or celebrations

Daily Life Management:

• Create meal plans and shopping lists
• Organize and prioritize tasks
• Draft important personal correspondence

Advanced Techniques

Context Management

Provide relevant background information at the start of conversations:

• Your role and expertise level
• The specific context or constraints you're working within
• Your goals and success criteria

Chain of Thought Prompting

For complex problems, ask Claude to "think step by step" or "explain your reasoning." This often leads to more thorough and accurate responses.

Role-Playing Scenarios

Ask Claude to take on specific roles or perspectives:

• "Act as a senior software engineer reviewing my code"
• "Respond as someone unfamiliar with this topic would"
• "Take the perspective of a potential customer"

Template Creation

Develop reusable prompts for recurring tasks:

• Meeting summary templates
• Email response frameworks
• Problem-solving methodologies

What Claude Cannot Do

Understanding limitations is crucial for effective use:

Real-Time Information

Claude's training has a knowledge cutoff and cannot access current web information or real-time data.

Personal Data Access

Claude cannot access your personal files, emails, or private information unless you explicitly share it in the conversation.

Execute Actions

Claude cannot directly perform actions in external systems, send emails, or make purchases on your behalf.

Replace Human Judgment

While Claude can provide insights and suggestions, important decisions should always involve human judgment and expertise.

Maintaining Privacy and Security

Information Sharing

• Avoid sharing sensitive personal information (SSNs, passwords, financial details)
• Be cautious with proprietary or confidential business information
• Consider anonymizing data when seeking help with specific scenarios

Data Retention

• Understand the platform's data retention policies
• Don't rely on Claude to remember information between separate conversations
• Keep your own records of important insights or decisions

Building an Effective Workflow

1. Define Your Use Cases

Identify specific, recurring tasks where Claude can add value to your routine.

2. Develop Standard Prompts

Create and refine prompt templates for your most common use cases.

3. Set Realistic Expectations

Understand that Claude is a tool to augment your capabilities, not replace your thinking and judgment.

4. Measure Impact

Track how Claude usage affects your productivity and the quality of your work.

5. Stay Updated

AI capabilities evolve rapidly. Stay informed about new features and best practices.

Common Pitfalls to Avoid

Over-Reliance

Don't become dependent on Claude for basic thinking or decision-making. Use it to enhance your capabilities, not replace them.

Prompt Laziness

Avoid vague or minimal prompts. Investing time in clear communication yields better results.

Ignoring Context

Remember to provide sufficient context for complex or specialized topics.

Assuming Perfection

Always review and validate Claude's outputs, especially for important tasks.

Conclusion

Claude can be a powerful ally in both professional and personal contexts when used thoughtfully and strategically. The key to success lies in understanding its strengths and limitations, developing clear communication patterns, and maintaining appropriate skepticism and verification practices.

Start with simple use cases, build your skills gradually, and always remember that Claude is most effective when it augments your human intelligence rather than attempting to replace it. With practice and the right approach, you'll find Claude becomes an invaluable tool in your personal and professional toolkit.

Remember: the goal isn't to use Claude for everything, but to use it for the right things in the right way.

AI has been moving at a breakneck pace. Many people are getting hooked on barking orders at AI and then discovering piles of error‑ridden code it produced (and in truth, you may not fully understand the code the AI wrote either).

If you do understand AI, sure—you know how to use prompt engineering. Still, I want to reinforce the fundamentals of software engineering.

Object‑oriented programming, design principles, design patterns, coding conventions, and refactoring.

Programming paradigms/styles

The mainstream programming paradigms/styles are:

• Procedural programming
• Object‑oriented programming (OOP)
• Functional programming

Among them, OOP remains the most widely adopted paradigm. Most popular programming languages today are object‑oriented or provide strong OO support.

Design principles

Definition: distilled practical experience for code design.

These principles often sound abstract and are described vaguely.

The hard part: understand the original intent, the specific problems they solve, and the scenarios where they apply.

Common principles include:

• SOLID principles

  You might wonder why SOLID uniquely comes with five additional “sub‑principles.”

  I had the same question, so I looked it up.

  SOLID is called the cornerstone of object‑oriented design because it offers a concrete, actionable toolkit—rather than vague ideas—for solving real problems in software development.

  Each sub‑principle targets a different layer of design pitfalls. It stands out precisely because it is not a single fuzzy rule, but a collection of five specific, operable design principles.

  • Single Responsibility Principle (SRP)
  • Open/Closed Principle (OCP)
  • Liskov Substitution Principle (LSP)
  • Interface Segregation Principle (ISP)
  • Dependency Inversion Principle (DIP)

• KISS principle (Keep It Simple, Stupid)

• DRY principle (Don’t Repeat Yourself)

• YAGNI principle (You Aren’t Gonna Need It)

• Law of Demeter (LOD)

Design patterns

Definition: recurring solutions or design approaches distilled from frequently encountered problems in software development.

Most design patterns are fundamentally about extensibility.

The hard part: knowing which problem each pattern addresses, recognizing typical application scenarios, and—importantly—avoiding overuse.

What’s included: there are 23 classic design patterns.

As languages evolve:

• Some patterns (e.g., Singleton) have aged poorly and even become anti‑patterns.
• Some are now built into languages (e.g., Iterator).
• New patterns keep emerging (e.g., Monostate).

The 23 classic patterns are grouped into three categories: creational, structural, and behavioral.

Creational

Common: Singleton, Factory (Factory Method and Abstract Factory), Builder.

Less common: Prototype.

Structural

Common: Proxy, Bridge, Decorator, Adapter.

Less common: Facade, Composite, Flyweight.

Behavioral

Common: Observer, Template Method, Strategy, Chain of Responsibility, Iterator, State.

Less common: Visitor, Memento, Command, Interpreter, Mediator.

Coding conventions

Definition: primarily concerned with readability. Compared with design principles and design patterns, conventions are more concrete and closer to code‑level details. Even if you’re unfamiliar with design principles or patterns, at minimum you should master basic coding conventions.

For example: how to name variables, classes, and functions; how to write comments; keep functions short; avoid too many parameters; and so on.

There are plenty of classic books to learn from—“Refactoring,” “Code Complete,” and “Clean Code,” to name a few.

Each convention is straightforward and specific. Just remember them and follow them. Unlike design principles, they require less personal interpretation.

Refactoring

As long as a project is alive and people keep working on it, the software will evolve. New features will inevitably push older code to be refactored. Refactoring is the practical way to keep code quality from decaying to an unrecoverable state.

The toolkit for refactoring includes everything mentioned earlier: paradigms, design principles, design patterns, and coding conventions.

While design patterns can improve extensibility, overuse or misuse increases complexity and harms readability.

In early development, unless absolutely necessary, don’t over‑design or apply complex patterns.

Instead, when code shows concrete issues, refactor to address those issues using the appropriate principles and patterns.

This effectively prevents premature over‑engineering.

Must‑know points

As follows:

• The purpose (why), target (what), timing (when), and methods (how) of refactoring
• Techniques for safe refactoring: unit tests and code testability
• Two scales of refactoring: large‑scale/high‑level and small‑scale/low‑level

How the five pieces fit together

The relationship among OOP, design principles, design patterns, coding conventions, and refactoring is as follows:

• OOP, with its rich features (encapsulation, abstraction, inheritance, polymorphism), enables sophisticated design ideas and is the foundation for implementing many principles and patterns.
• Design principles guide design decisions and help determine whether a particular pattern should be applied in a given scenario. For example, the Open/Closed Principle underpins patterns such as Strategy and Template Method.
• Design patterns are recurring solutions to common design problems in software development. Their main goal is to improve extensibility. In terms of abstraction, principles are more abstract; patterns are more concrete and executable.
• Coding conventions primarily address readability. Compared with principles and patterns, conventions are more specific, detail‑oriented, and directly actionable. Continuous small‑scale refactoring relies heavily on conventions as its theoretical backbone.

Ultimately, this article is about one thing: writing high‑quality code. Once you trace it back to that, how to do many things—and how to implement them in code—becomes clear.

mindmap
  root((Writing High-Quality Code))
    Object-Oriented
      Encapsulation, Abstraction, Inheritance, Polymorphism
      OOP vs Procedural Programming
      OO Analysis, Design, Programming
      Interface vs Abstract Class
      Program to an Interface, Not an Implementation
      Favor Composition Over Inheritance
      Anemic vs Rich Domain Model
    Design Principles
      SOLID Principles
        SRP Single Responsibility
        OCP Open/Closed
        LSP Liskov Substitution
        ISP Interface Segregation
        DIP Dependency Inversion
      Others
        DRY Principle
        KISS Principle
        YAGNI Principle
        Law of Demeter (LOD)
    Coding Conventions
      20 Quick Rules to Improve Code Quality
    Refactoring
      Purpose, Target, Timing, Methods
      Unit Tests and Testability
      Large-Scale (High-Level)
      Small-Scale (Low-Level)
    Design Patterns
      Creational
        Common
          Singleton
          Factory (Factory Method & Abstract Factory)
          Builder
        Less Common
          Prototype
      Structural
        Common
          Proxy
          Bridge
          Decorator
          Adapter
        Less Common
          Facade
          Composite
          Flyweight
      Behavioral
        Common
          Observer
          Template Method
          Strategy
          Chain of Responsibility
          Iterator
          State
        Less Common
          Visitor
          Memento
          Command
          Interpreter
          Mediator

mindmap
  root((Writing High-Quality Code))
    Object-Oriented
      Encapsulation, Abstraction, Inheritance, Polymorphism
      OOP vs Procedural Programming
      OO Analysis, Design, Programming
      Interface vs Abstract Class
      Program to an Interface, Not an Implementation
      Favor Composition Over Inheritance
      Anemic vs Rich Domain Model
    Design Principles
      SOLID Principles
        SRP Single Responsibility
        OCP Open/Closed
        LSP Liskov Substitution
        ISP Interface Segregation
        DIP Dependency Inversion
      Others
        DRY Principle
        KISS Principle
        YAGNI Principle
        Law of Demeter (LOD)
    Coding Conventions
      20 Quick Rules to Improve Code Quality
    Refactoring
      Purpose, Target, Timing, Methods
      Unit Tests and Testability
      Large-Scale (High-Level)
      Small-Scale (Low-Level)
    Design Patterns
      Creational
        Common
          Singleton
          Factory (Factory Method & Abstract Factory)
          Builder
        Less Common
          Prototype
      Structural
        Common
          Proxy
          Bridge
          Decorator
          Adapter
        Less Common
          Facade
          Composite
          Flyweight
      Behavioral
        Common
          Observer
          Template Method
          Strategy
          Chain of Responsibility
          Iterator
          State
        Less Common
          Visitor
          Memento
          Command
          Interpreter
          Mediator

PS: A bunch of keywords for further exploration:

System architecture principles High cohesion and low coupling Law of Demeter (principle of least knowledge) Composition over inheritance Separation of concerns Design by contract Cloud‑native design principles Resilience design principles Observability principles Immutable infrastructure Service autonomy Declarative configuration Security by design Defense in depth Principle of least privilege Secure by default Separation of duties Fail‑safe

When legacy systems are initially created and coded, the final system performance and maintainability depend entirely on the developer's expertise.

When you inherit a system with the following issues, this article might help you:

1. No monitoring
2. No alerting
3. No business success rate metrics
4. Frequent optimistic locking failures
5. Black box system response time (yes, your business users only vaguely tell you the system is slow, but don't know specifically where it's slow)
6. Complex log troubleshooting

Yes, these are the headache-inducing problems I've encountered over the past month.

Now that the month is up, I'm starting to take action and perform a series of performance optimizations on this system.

First, you need to understand the overall situation of the system. For optimization, you should at least know what you're optimizing. Business users should also know where exactly their system is slow.

So what you need to do is establish system baselines.

Establishing System Baselines

System baselines are quantitative descriptions of key performance indicators when the system is running normally.

They are the starting point for system performance optimization and the standard reference for measuring optimization effectiveness. Simply put, system baselines are your system's "health checkup report," recording system performance under various conditions. (1, 2)

System Baselines Include the Following

A complete system baseline should include the following aspects:

Performance Metrics

• Response time (average, P95, P99)
• Throughput (QPS/TPS)
• Latency
• Concurrent processing capability

Resource Usage

• CPU utilization
• Memory usage and allocation
• Disk I/O
• Network I/O
• Connection pool usage status

Business Metrics

• Business success rate
• Error rate
• Key business process completion time

System Stability Metrics

• Mean Time Between Failures (MTBF)
• Mean Time To Recovery (MTTR)
• Lock contention situations (such as the optimistic lock failure rate you mentioned)

Taking this opportunity during the May Day holiday to thoroughly analyze one of the most famous papers: "MapReduce: Simplified Data Processing on Large Clusters."

Of course, this article is deeply influenced by the article from "Muniao's Notes," so my article is purely a combination of two articles plus my own thoughts.

I've always heard senior colleagues talk about how powerful MapReduce distributed computing is, so today's the opportunity to dive in!

Introduction

MapReduce in the paper is actually a concept. But in another context, it can also be a programming model, as well as a distributed system implementation supporting that model. I found an article that explains this concept better:

MapReduce is a concept proposed in Google's 2004 paper (Google internally wrote the first version starting in 2003).

In Google's context, MapReduce is both a programming model and a distributed system implementation supporting that model. Its introduction enables developers without distributed systems background to leverage large-scale clusters for high-throughput processing of massive data with relative ease.

This article also has a sentence explaining the problem-solving approach for applying this technology: Finding pain points in requirements (such as how to maintain, update, and rank massive indexes), performing high-level abstractions of key processing flows (sharding Map, on-demand Reduce), for efficient system implementation (so-called tailored solutions).

Among these, finding an appropriate computational abstraction is the most difficult part, requiring both intuitive understanding of requirements and extremely high computer science literacy.

The quote above is from the referenced article from "Muniao's Notes."

Returning to the paper, we can see that on the first page, Google's experts clearly explained what this is.

As a reaction to this complexity, we designed a new abstraction that allows us to express the simple computations we were trying to perform but hides the messy details of parallelization, fault-tolerance, data distribution and load balancing in a library.

This means we abstracted something to express a computational method that can hide many conceptual details (parallelization, fault tolerance, data distribution, and load balancing).

This abstraction is inspired by the map and reduce primitives present in Lisp and many other functional languages.

We realized that most of our computations involved applying a map operation to each logical "record" in our input in order to compute a set of intermediate key/value pairs, and then applying a reduce operation to all the values that shared the same key, in order to combine the derived data appropriately.

Most of our computations involve applying a map operation to each logical record in the input to compute a set of intermediate key/value pairs, then applying a reduce operation to all values sharing the same key to appropriately combine the derived data.

Our use of a functional model with user-specified map and reduce operations allows us to parallelize large computations easily and to use re-execution as the primary mechanism for fault tolerance.

We use a functional model with user-specified map and reduce operations, enabling easy parallelization of large computations.

I find the last sentence very interesting:

use re-execution as the primary mechanism for fault tolerance.

Using "re-execution" as the primary mechanism for fault tolerance.

The paper's Abstract outlines the content:

1. Section 1 is the Introduction above
2. Section 2 describes the basic programming model and gives several examples, Programming Model
3. Section 3 describes an implementation of the MapReduce interface tailored towards our cluster-based computing environment, Implementation
   • Implementation of MapReduce interface customized for cluster computing environments
4. Section 4 describes several refinements of the programming model that we have found useful - Several programming model improvements
5. Section 5 has performance measurements of our implementation for a variety of tasks - Performance measurements of implementing various tasks
6. Section 6 explores the use of MapReduce within Google including our experiences in using it as the basis for a rewrite of our production indexing system - MapReduce applications within Google
7. Section 7 discusses related and future work

Programming Model

The Map key is a normal key, and the value can be thought of as a string array.

MapReduce, simply put, consists of two functions: the map function and the reduce function.

The Map function receives an input pair and generates a set of intermediate key/value pairs, then the MapReduce library combines all intermediate values associated with the same key.

Example

Below is pseudo-code, grandfather-level, from the original paper:

map(String key, String value):
    // key: document name
    // value: document contents
    for each word w in value:
      EmitIntermediate(w, "1");

reduce(String key, Iterator values):
    // key: a word
    // values: a list of counts
    int result = 0;
    for each v in values:
      result += ParseInt(v);
    Emit(AsString(result));

This is a classic MapReduce word count implementation, one of the most common examples in the MapReduce programming model.

• key: document name
• value: complete document content (text string)

EmitIntermediate represents the function provided by the MapReduce framework for outputting intermediate key/value pairs. Each call to this function produces a key-value pair: (word, "1"), indicating that the word appeared once.

For example, processing document content "hello world hello":

• First word "hello" → EmitIntermediate("hello", "1")
• Second word "world" → EmitIntermediate("world", "1")
• Third word "hello" → EmitIntermediate("hello", "1")

Map phase intermediate results:

("hello", "1")
("world", "1")
("hello", "1")

Shuffle Stage (Automatically Completed by Framework)

Between Map and Reduce, the MapReduce framework automatically performs the Shuffle operation:

1. Collect all mapper outputs
2. Sort by key (word)
3. Group all values with the same key together

So the above example after Shuffle:

("hello", ["1", "1"])
("world", ["1"])

Reduce Function

After Shuffle comes the Reduce function's work. The pseudo-code above essentially performs accumulation, which needs no further explanation.

Complete Execution Flow

Here's a larger example showing the entire MapReduce execution flow:

Assume three documents:

• document1.txt: "hello world"
• document2.txt: "hello mapreduce"
• document3.txt: "mapreduce world example"

Map Phase (Parallel Execution)

Mapper 1 processes document1.txt:

EmitIntermediate("hello", "1")
EmitIntermediate("world", "1")

Mapper 2 processes document2.txt:

EmitIntermediate("hello", "1")
EmitIntermediate("mapreduce", "1")

Mapper 3 processes document3.txt:

EmitIntermediate("mapreduce", "1")
EmitIntermediate("world", "1")
EmitIntermediate("example", "1")

Shuffle Phase (Automatically Completed by Framework):

("hello", ["1", "1"])
("world", ["1", "1"])
("mapreduce", ["1", "1"])
("example", ["1"])

Reduce Phase (Parallel Execution):

Reducer processes "hello":

result = 0
result += 1 = 1
result += 1 = 2
Emit("2")  # Output ("hello", "2")

Similarly for other words...

Final Results

("hello", "2")
("world", "2")
("mapreduce", "2")
("example", "1")

MapReduce Framework's Role

In this process, the MapReduce framework is responsible for:

1. Splitting input data into multiple shards, assigning them to different Mappers
2. Executing multiple Map tasks in parallel
3. Performing Shuffle operations, reorganizing and sorting intermediate results
4. Executing multiple Reduce tasks in parallel
5. Collecting and integrating Reduce outputs
6. Handling task failures and retries
7. Optimizing data locality, processing data on nodes where it resides when possible

This pattern allows developers to focus on business logic (Map and Reduce functions) without worrying about complex issues like parallelization, distributed computing, and fault tolerance.

More Examples

Here are more examples:

Distributed Grep

Working Principle

In this example:

• Map function checks each line of input text, emitting the line if it matches a specified pattern
• Reduce function is a simple identity function that directly copies intermediate results to output

Application Value

This pattern is very suitable for quickly finding text lines with specific patterns in large-scale distributed file systems. It fully utilizes MapReduce's parallel processing capabilities and is extremely efficient when searching for specific error information in TB or even PB-level log files.

Count of URL Access Frequency

Working Principle

• Map function: Processes web request logs, emitting <URL, 1> key-value pairs for each URL
• Reduce function: Sums all counts for the same URL, outputting <URL, total count>

Application Value

This is a fundamental operation in web analytics, crucial for understanding website traffic distribution, identifying popular content, and detecting abnormal access patterns. In large websites, daily log data can reach several TB, and MapReduce can effectively handle this scale of data.

Reverse Web-Link Graph

Working Principle

• Map function: Analyzes web content, outputting <target URL, source URL> for each discovered link
• Reduce function: Collects all source URLs for a target URL, outputting <target URL, source URL list>

Application Value

Reverse link graphs are one of the core data structures for modern search engines, used in the following scenarios:

• Foundation data for PageRank and other web importance algorithms
• Analyzing citation relationships between websites
• Discovering influential content creators
• Providing backlink analysis tools for webmasters

Building a complete web reverse link graph is a computation-intensive task, and the MapReduce model is very suitable for this naturally parallelizable problem.

Term-Vector per Host

Working Principle

• Map function: Analyzes document content, extracts hostname from URL, outputs <hostname, document term vector>
• Reduce function: Merges all term vectors for the same host, filters low-frequency terms, outputs <hostname, aggregated term vector>

Application Value

This analysis is very valuable for understanding website content characteristics:

• Can be used for website topic classification
• Helps with search engine optimization
• Content similarity comparison
• Competitor website content analysis
• Foundation data for content recommendation systems

Inverted Index

Working Principle

• Map function: Parses each document, outputs <word, document ID> key-value pairs
• Reduce function: Receives all document IDs for a given word, sorts them and outputs <word, document ID list> key-value pairs

Application Value

Inverted indexes are the fundamental data structure for modern search engines, used for:

1. Full-text search: Quickly find all documents containing query terms
2. Phrase search: Achieve precise phrase search through positional information
3. TF-IDF calculation: Provide term frequency statistics for information retrieval systems
4. Keyword highlighting: Help frontend display matched text fragments
5. Relevance ranking: Provide foundation data for search result ranking

MapReduce is particularly suitable for building inverted indexes because it can efficiently process large numbers of documents in parallel and naturally implement index merging in the Reduce phase.

Distributed Sort

Working Principle

• Map function: Extracts the key of each record, outputs <key, record> key-value pairs
• Reduce function: Directly outputs all received key-value pairs without modification

This seemingly simple example actually cleverly utilizes two core features of the MapReduce framework:

1. Partitioning mechanism: Ensures records with keys in the same range are sent to the same Reducer
2. Sorting property: Ensures keys received by Reducers are in order

MapReduce Framework's Special Contribution:

In distributed sorting, the MapReduce framework does most of the important work:

1. Custom Partitioner:

   // Example: Range partitioner
   func RangePartitioner(key string, numReducers int) int {
       // Determine which reducer to send to based on key range
       // This ensures global sorting
       if key < "D" {
           return 0
       } else if key < "N" {
           return 1
       } else {
           return 2
       }
   }
   

2. Sort Comparator:

   // Define natural sorting order for keys
   func KeyComparator(key1, key2 string) int {
       return strings.Compare(key1, key2)
   }
   

Application Value

Distributed sorting is a fundamental operation for many big data processing workflows:

1. Data preprocessing: Preparing large datasets for further analysis
2. Log analysis: Sorting massive log records by timestamp
3. Building indexes: Creating sorted indexes for databases or search engines
4. Merging sorted data: Combining multiple sorted datasets into one
5. TopN queries: Quickly finding top N records for a given metric

Overall Examples Analysis and Comparison

The examples demonstrate the versatility and adaptability of the MapReduce model:

1. Distributed Grep: Simplest application, basically only uses Map functionality, suitable for simple filtering operations
2. URL Access Frequency Count: Classic word count variant, demonstrates MapReduce advantages in statistical aggregation
3. Reverse Web-Link Graph: Shows how to use MapReduce to build complex relationship graphs and index structures
4. Term-Vector per Host: Combines text analysis and aggregation functions, suitable for advanced content analysis

Implementation

Multiple different implementations of the MapReduce interface are possible.

The right choice depends on the specific environment (meaning specific problems require specific analysis).

For example, one implementation might be suitable for small shared-memory machines, another for large NUMA multiprocessors, and yet another for even larger networked machine clusters.

Below is the computing environment widely used at Google:

This section describes an implementation targeted to the computing environment in wide use at Google: large clusters of commodity PCs connected together with switched Ethernet.

In our environment:

(1) Machines are typically dual-processor x86 processors running Linux, with 2-4 GB of memory per machine.

(2) Commodity networking hardware is used – typically either 100 megabits/second or 1 gigabit/second at the machine level, but averaging considerably less in overall bisection bandwidth.

(3) A cluster consists of hundreds or thousands of machines, and therefore machine failures are common.

(4) Storage is provided by inexpensive IDE disks attached directly to individual machines. A distributed file system developed in-house is used to manage the data stored on these disks. The file system uses replication to provide availability and reliability on top of unreliable hardware.

(5) Users submit jobs to a scheduling system. Each job consists of a set of tasks, and is mapped by the scheduler to a set of available machines within a cluster.

(1) Machines typically feature dual-processor x86 architecture, running Linux systems, with 2-4 GB of memory per machine.

(2) Commodity networking hardware is used — typically 100 megabits/second or 1 gigabit/second at the machine level, but the average overall bisection bandwidth is significantly lower.

(3) A cluster consists of hundreds or thousands of machines, so machine failures are common.

(4) Storage is provided by inexpensive IDE disks directly connected to individual machines. An internally developed distributed file system is used to manage data stored on these disks. The file system provides availability and reliability on top of unreliable hardware through data replication.

(5) Users submit jobs to a scheduling system. Each job consists of a set of tasks and is mapped by the scheduler to a set of available machines within the cluster.

Execution Process

The Map invocations are distributed across multiple machines by automatically partitioning the input data into a set of M splits. The input splits can be processed in parallel by different machines. Reduce invocations are distributed by partitioning the intermediate key space into R pieces using a partitioning function (e.g., hash(key) mod R). The number of partitions (R) and the partitioning function are specified by the user.

Following figure shows the overall flow of a MapReduce operation in our implementation. When the user program calls the MapReduce function, the following sequence of actions occurs (the numbered labels in Figure correspond to the numbers in the list below):

1. The MapReduce Library in the user program first splits the files into M pieces, each piece typically 16-64 MB in size;

2. The program copy on the Master is special, while other workers are assigned tasks by the master. There are typically M map tasks and R reduce tasks to assign.

   There are M map tasks and R reduce tasks to assign. The master picks idle workers and assigns each one a map task or a reduce task.

   Here, the word "idle" refers to idle workers - in Google's MapReduce architecture, the entire computation task is executed distributedly, including:

   1. Master (master node): A special program copy responsible for task scheduling and coordinating the entire computation process
   2. Workers (worker nodes): Other program copies responsible for executing actual computation tasks
   3. Idle workers: Worker nodes currently not executing any tasks and in a waiting state

   I hope this clarifies what idle workers are.

   So generally speaking, the entire MapReduce workflow related to idle worker nodes:

   1. When computation tasks begin, the system starts multiple program copies, with one as the master node and others as worker nodes
   2. The master node maintains the state of the entire cluster, including whether each worker node is idle
   3. When the master node detects that a worker node is "idle" (not executing tasks), it selects one from the pending M Map tasks or R Reduce tasks to assign to that worker node

3. A worker assigned a map task reads the contents of the corresponding input split, parses key/value pairs from the input data, and passes each pair to the user-defined map function.

   These intermediate key/value pairs produced by map are buffered in memory.

   A worker who is assigned a map task reads the contents of the corresponding input split. It parses key/value pairs out of the input data and passes each pair to the user-defined Map function. The intermediate key/value pairs produced by the Map function are buffered in memory.

   In fact, Hadoop does exactly this.

4. Buffered intermediate result pairs are periodically written to local disk, then partitioned into R regions by the partitioning function.

   These buffered pairs on local disk are passed back to the master, so the master can inform reduce workers of these pair locations.

   Periodically, the buffered pairs are written to local disk, partitioned into R regions by the partitioning function.

   The locations of these buffered pairs on the local disk are passed back to the master, who is responsible for forwarding these locations to the reduce workers.

5. When a reduce worker receives the buffered pair locations mentioned above, it uses RPC to read the corresponding partition data.

   When a reduce worker has read all the data, it sorts by key so that all data with the same key is sorted together.

   When a reduce worker is notified by the master about these locations, it uses remote procedure calls to read the buffered data from the local disks of the map workers.

   When a reduce worker has read all intermediate data, it sorts it by the intermediate keys so that all occurrences of the same key are grouped together.

   The sorting is needed because typically many different keys map to the same reduce task.

   If the amount of intermediate data is too large to fit in memory, an external sort is used.

   If the intermediate data is too large, external sorting programs are needed. This is where performance optimization points come in.

Steps 4 and 5 together are called shuffle

6. The Reduce Worker then iterates through this sorted intermediate data and passes this data along with its key-related data to the user's reduce function.

   The Reduce function's output is appended to the final output file.

   The reduce worker iterates over the sorted intermediate data and for each unique intermediate key encountered, it passes the key and the corresponding set of intermediate values to the user's Reduce function.

   The output of the Reduce function is appended to a final output file for this reduce partition.

7. When all map tasks and reduce tasks are completed, the master wakes up the user program.

   At this point, the user program receives a final computation result (MapReduce call).

   When all map tasks and reduce tasks have been completed, the master wakes up the user program. At this point, the MapReduce call in the user program returns back to the user code.

Master Data Structures

Understanding the key data structures maintained by the Master node in the MapReduce framework and its core responsibilities in task coordination.

The Master node is actually the "brain" of the entire MapReduce execution process, maintaining the following important data:

1. Task State Records: For each map and reduce task, the master node records its current state:

   • idle: pending tasks
   • in-progress: tasks assigned to workers but not yet completed
   • completed: finished tasks

2. Worker Machine Identity: For non-idle state tasks, the master node records the identity of the worker machine executing that task, used for tracking task execution and handling failures

3. Intermediate File Metadata: For completed map tasks, the master stores location and size information of intermediate result files produced by that task

Master as Information Transfer Channel

The Master node plays the role of an information transfer channel for intermediate result location information. When a map task completes, it informs the master node of which intermediate files were produced, along with their location and size information.

Additionally, Google's MapReduce implementation has job-level encapsulation, where each job contains a series of tasks, namely Map Tasks and Reduce Tasks.

To maintain metadata for a running job, it's necessary to save the state of all executing tasks and their machine IDs.

This information is crucial for reduce tasks, as reduce tasks need to know where to obtain the data they need to process.

Moreover, the Master also serves as an information channel from Map Task output to Reduce Task. The master node incrementally pushes this information to worker nodes executing reduce tasks.

When each Map Task ends, it notifies the Master of the location information of its output intermediate results, the Master then forwards this to the corresponding Reduce Task, and the Reduce Task fetches the corresponding size of data from the corresponding location.

Note: Since Map Task completion times are not uniform, this notification → forwarding → fetching process is incremental. It's reasonable to infer that the sorting of intermediate data on the reduce side should be a continuous merge process, unlikely to be global sorting after all data is in place. — from Muniao

The master keeps several data structures. For each map task and reduce task, it stores the state (idle, in-progress, or completed), and the identity of the worker machine (for non-idle tasks).

The master is the conduit through which the location of intermediate file regions is propagated from map tasks to reduce tasks. Therefore, for each completed map task, the master stores the locations and sizes of the R intermediate file regions produced by the map task. Updates to this location and size information are received as map tasks are completed. The information is pushed incrementally to workers that have in-progress reduce tasks.

Fault Tolerance

Distributed systems must gracefully handle various errors on distributed machines when processing large amounts of data. The paper roughly divides failures into three types:

1. Worker Failure
2. Master Failure
3. Semantics in the Presence of Failures

Worker Failure

The master periodically pings each worker node, and if there's no response, the master marks it as a failed node.

At this time, both completed and uncompleted map tasks are marked back to the initial idle state, then wait to be scheduled to other normal workers.

These map tasks are then re-executed and re-stored to local disk, and the master continues to report this information to Reduce workers.

If a Reducer worker has already processed one of the map tasks, it doesn't need to fetch processing data again from the master's provided information; if not processed, it continues to fetch.

On the Reducer worker side, when it discovers that the map worker processing a certain segment of map tasks has failed, for example, this Reduce program is R5, processing map tasks (41-51) on Worker-37, when R5 processes M47 and discovers a failure, its countermeasures are as follows:

• Transmission Interruption Handling: If R5's connection is interrupted while pulling data from Worker-37, it triggers exception handling

// Simplified pseudo-code
try {
    fetchMapOutput(worker37, mapTaskId47);
} catch (FetchFailureException e) {
    // Wait for Master notification of new data location
    waitForNotification(mapTaskId47);
    // Retry with new data location
    fetchMapOutput(worker51, mapTaskId47);
}

• Data Consistency: R5 discards incomplete data partially pulled from Worker-37

if (partialData && dataSource != currentSourceForTask) {
    discardPartialData();
    fetchFromNewSource();
}

• Notification Mechanism: Master notifies Reduce tasks through the following ways

1. Heartbeat responses include updated Map output location information
2. RPC calls notify state changes
3. Reduce tasks periodically poll Master for latest mapping information

Master Failure

Google's paper handles Master failure relatively simply: saving state through checkpoint mechanisms, but terminating the entire job when Master actually fails.

This design is based on two considerations:

1. Single Point Characteristic: There's typically only one Master node in the system, with relatively low failure probability
2. Simplified Design: Simple failure handling mechanisms reduce system complexity

However, in critical production environments, this simple handling approach obviously cannot meet high availability requirements.

With the development of distributed systems, more robust Master failure handling mechanisms should be seriously considered.

Semantics in the Presence of Failures

MapReduce provides a key promise: In deterministic operations, the results of distributed parallel execution are completely consistent with sequential execution.

This feature greatly simplifies the complexity of distributed programs.

How is this "computational result consistency" achieved?

Key Mechanism: Atomic Commits

MapReduce achieves result consistency through carefully designed atomic commit mechanisms:

1. Temporary file strategy
2. Task completion flow
3. Redundant execution handling

Core Role of Atomic Operations

Atomic operations are the foundation of the entire fault tolerance mechanism, mainly reflected in two levels:

1. Master data structure updates
2. File system atomic renaming - for example, atomic renaming operations when Reduce tasks complete; only one execution instance can successfully rename, meaning if there's a successfully named file, it indicates one instance has been successfully executed

Locality

A commonly used principle in computer science is called locality of reference (specifically spatial locality), which states that when a program executes sequentially, after accessing a block of data, it's highly likely to access data physically adjacent to that data next. This simple assertion is the foundation for all cache effectiveness, leading to the formation of a storage hierarchy system from slow to fast, cheap to expensive, large to small (hard disk → memory → cache → registers).

In distributed environments, this hierarchy system requires at least one more layer — network I/O. This is also the first sentence in the paper: "Network bandwidth is a relatively scarce resource in our computing environment."

In MapReduce systems, we also fully utilize input data locality. However, this time, instead of loading data over, we schedule the program to go there (Moving Computation is Cheaper than Moving Data). If input exists on GFS, it manifests as a series of logical blocks, each of which may have several (typically three) physical replicas. For each logical block of input, we can run Map Tasks on machines where one of its physical replicas resides (if it fails, try another replica), thereby minimizing network data transmission and reducing latency while saving bandwidth.

Network bandwidth is a relatively scarce resource in our computing environment. We conserve network bandwidth by taking advantage of the fact that the input data (managed by GFS) is stored on the local disks of the machines that make up our cluster. GFS divides each file into 64 MB blocks, and stores several copies of each block (typically 3 copies) on different machines. The MapReduce master takes the location information of the input files into account and attempts to schedule a map task on a machine that contains a replica of the corresponding input data. Failing that, it attempts to schedule a map task near a replica of that task's input data (e.g., on a worker machine that is on the same network switch as the machine containing the data). When running large MapReduce operations on a significant fraction of the workers in a cluster, most input data is read locally and consumes no network bandwidth.

Task Granularity

Deep analysis of the core design principles, influencing factors, and best practices of Task Granularity in the MapReduce framework.

The paper has an important viewpoint: "M and R should be much larger than the number of worker machines."

M and R Should Be Much Larger Than Worker Machine Count

1. Dynamic Load Balancing

   // Simplified load balancing scenario
   100 Map tasks, 10 machines
   - Machines 1-9: Each processes 10 tasks, uniform load
   - Machine 10: Slower hardware, only completes 5 tasks
   - Task allocator automatically redistributes remaining 5 tasks to machines that completed tasks
   

   When each machine processes multiple small tasks rather than single large tasks, fast machines can process more tasks while slow machines process fewer, naturally forming performance-based work distribution.

2. Accelerated Failure Recovery

   Assumption:
   - 2000 machines, each executing about 100 Map tasks
   - Single Worker-37 fails, has completed 92 Map tasks
   
   Impact:
   - Traditional design (1 large task per machine): Lose entire Worker-37 computation results
   - Fine-grained design: Only need to re-execute 92 small tasks, distributed to other 1999 machines
   - Recovery speed: About 1/20 of traditional method (average less than 1 additional task per machine)
   

   When a worker fails, its completed multiple small tasks can be quickly redistributed to other machines in the cluster for re-execution, significantly accelerating recovery speed.

Backup Tasks

This mechanism was designed by Google to solve the "straggler" problem.

One of the common causes that lengthens the total time taken for a MapReduce operation is a "straggler": a machine that takes an unusually long time to complete one of the last few map or reduce tasks in the computation.

An important optimization introduced to solve slow task problems.

Straggler Problem: Key Challenge in Distributed Systems

"Stragglers" refer to machines that complete tasks abnormally slowly, severely slowing down the completion time of entire MapReduce jobs. This problem is particularly prominent in large-scale distributed environments.

Main Causes

1. Hardware Issues:

   - Disk errors: Correctable errors reduce read speed from 30MB/s to 1MB/s
   - Network problems: Network card failures cause bandwidth reduction
   - CPU or memory failures: Significantly reduced processing capability
   

2. Resource Competition:

   - Multi-task scheduling conflicts: Other jobs occupy CPU, memory resources
   - I/O contention: Multiple processes compete for disk or network I/O
   - Memory pressure: Insufficient memory causes frequent page swapping
   

3. Software Issues:

   - Configuration errors: Such as Google's processor cache disabled bug (100x performance degradation)
   - GC pauses: Long pauses caused by garbage collection
   - System updates: Background services or updates consuming resources
   

Impact of Stragglers

In MapReduce jobs, job completion time is limited by the last completing task. When 99% of tasks complete quickly but a few tasks are abnormally slow, the entire job's completion time will be dominated by these slow tasks.

Google designed the backup task mechanism as a simple yet effective strategy, using moderate resource redundancy to reduce overall execution time.

Refinements

The paper provides several extensions beyond the basic Mapper and Reducer primitives that have become recognized as standard components: Partitioner, Combiner, and Reader/Writer.

Partitioning Function

The Partitioning Function in MapReduce is an important extension mechanism connecting the Map and Reduce phases.

Core Concept

The partitioning function is a key component in MapReduce that connects the Map and Reduce phases, determining which intermediate key-value pairs are sent to which Reduce task.

// Basic definition of partitioning function
type PartitionFunc func(key interface{}, numPartitions int) int

Core functions:

• Determines which Reduce task processes Map output intermediate key-value pairs, ultimately affecting output file organization
• Controls the number and content organization of final output files
• Affects data distribution and load balancing across the cluster

Default Hash Partitioning Mechanism

MapReduce provides a simple and efficient default partitioning strategy:

// Default hash partitioning implementation
public int getPartition(K key, V value, int numReduceTasks) {
    return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks;
}

Characteristics:

• Simple and efficient: Low computational overhead, suitable for most scenarios
• Relatively balanced: Uses hash function to evenly distribute keys across partitions
• Deterministic: Same keys always map to the same partition, ensuring correct aggregation

Data flow diagram:

    Map Output          Partition Function       Reduce Tasks
 [K1:V1, K2:V2]       hash(K) % R        ->    Reduce-0
 [K3:V3, K4:V4]                          ->    Reduce-1
 [K5:V5, K6:V6]                          ->    Reduce-2
       ...                                       ...

Custom Partitioning Function Use Cases

The paper points out that certain scenarios require specific partitioning logic, such as when processing URL data where you want all URLs from the same host to go into the same output file.

Combiner Function

Core Concept and Working Principle

Combiner is a key optimization component in the MapReduce framework that performs partial data aggregation on the Map side, reducing network transmission volume.

Workflow:
Map output → Combiner local aggregation → Network transmission → Reducer final aggregation

Applicable conditions:

• Map output intermediate keys have lots of duplicates
• Reduce function has commutativity and associativity (such as sum, max)

Performance Advantage Example

Using word count as an example:

Without Combiner:
Map1 output: <"hello",1>, <"world",1>, <"hello",1>, <"hello",1> // 4 records transmitted
Map2 output: <"hello",1>, <"hadoop",1>, <"hello",1> // 3 records transmitted
Total network transmission: 7 records

With Combiner:
Map1 output: <"hello",3>, <"world",1> // 2 records transmitted
Map2 output: <"hello",2>, <"hadoop",1> // 2 records transmitted
Total network transmission: 4 records (43% reduction)

For data following Zipf distribution (like word frequency), Combiner can significantly reduce network transmission and improve performance.

Difference from Reduce

• Reducer output writes to final result files
• Combiner output writes to intermediate files, then transmitted to Reducer

Combiner is an important optimization in the MapReduce framework for improving data processing efficiency through "pre-aggregation," significantly reducing data transmission and processing time, especially effective for aggregation operations.

Input and Output Types

Support for different input data formats:

1. TextInputFormat (default)
   - Each line as one record
   - Key: line offset (LongWritable)
   - Value: line content (Text)
   - Smart splitting: ensures splitting at line boundaries

2. KeyValueTextInputFormat
   - Split each line into key-value by delimiter (default Tab)
   - Suitable for: simple structured text data

3. SequenceFileInputFormat
   - Read binary sequence files (key-value pairs)
   - Support compression, efficient random access
   - Commonly used for passing data between MapReduce jobs

4. DBInputFormat
   - Read records from relational databases
   - Support SQL queries as data source

This demonstrates the flexibility and extensibility of the framework.

MapReduce's input/output interface design enables it to handle diverse data sources:

- HDFS files
- Local file systems
- S3, Azure Blob and other cloud storage
- HBase, MongoDB and other NoSQL databases
- Kafka streaming data

By implementing appropriate InputFormat/OutputFormat, developers can integrate MapReduce with almost any data source/target, demonstrating the framework's powerful extensibility and making it suitable for various big data processing scenarios.

Performance

The paper mentions two performance tests representing MapReduce framework's capability to handle two typical big data processing scenarios:

Test Scenario Analysis

1. Pattern Search Test: Search for specific patterns in about 1TB of data

   • Represents the "extract small amounts of valuable information from large datasets" computation pattern
   • Typical applications include log analysis, anomaly detection, specific record finding, etc.

2. Large Data Sorting Test: Sort about 1TB of data

   • Represents the "transform data from one representation to another" computation pattern
   • Typical applications include ETL processes, data preprocessing, data reorganization, etc.

Below, the entire MapReduce distributed framework will be analyzed from five angles:

1. Cluster Configuration
2. Grep
3. Sort
4. Effect of Backup Tasks
5. Machine Failures

Cluster Configuration Analysis

First, a cluster configuration is given:

// Cluster configuration overview
type ClusterConfig struct {
    Nodes           int     // About 1800 machines
    CpuPerNode      int     // 2 × 2GHz Intel Xeon per node (with hyperthreading)
    MemoryPerNode   string  // 4GB per node (actual usable 2.5-3GB)
    DisksPerNode    int     // 2 × 160GB IDE disks per node
    NetworkBandwidth string  // Gigabit Ethernet
    NetworkTopology string   // Two-level tree switching network
    RootBandwidth   string   // 100-200Gbps aggregate bandwidth
    Latency         string   // <1ms latency between nodes
}

Analysis:

• This is a balanced cluster design for both computation and I/O, particularly suitable for MapReduce's divide-and-conquer model
• Storage-wise, each node has over 300GB total storage space, providing sufficient local storage for TB-level data processing
• Network-wise, tree topology is simple but may form bottlenecks during shuffle phase
• Low inter-node latency (<1ms) is extremely beneficial for data transmission in the reduce phase
• Cluster scale (1800 nodes) enables effective parallelization when processing TB-level data

Grep

This is a typical "extract small amounts of information from massive data" scenario:

// Grep task configuration
type GrepJobConfig struct {
    InputSize       string  // About 1TB (10^10 100-byte records)
    Pattern         string  // Three-character pattern (matches 92,337 records)
    InputSplits     int     // M=15000 (about 64MB per block)
    ReduceTasks     int     // R=1 (single output file)
    PeakScanRate    string  // >30GB/s (with 1764 workers)
    TotalTime       int     // About 150 seconds (including 60 seconds startup overhead)
}

Performance Analysis:

1. Scalability Performance: From Figure 2, as worker count increases, scan rate linearly improves to 30GB/s, showing excellent horizontal scaling capability for map-intensive tasks
2. I/O Bound Characteristics: Grep is essentially I/O intensive work; the test achieving 30GB/s throughput approaches the theoretical upper limit of total disk I/O for 1764 nodes
3. Optimization Opportunities: About 60 seconds startup overhead (40% of total time) shows an optimization point - GFS metadata operations and task distribution can be further optimized
4. R=1 Design: Single reduce design suits this "filtering" scenario, but also means final result collection could become a bottleneck (not manifested in this example due to small data volume)

Sort

Comprehensive test of the entire MapReduce framework capability:

// Sort task characteristic analysis
type SortJobAnalysis struct {
    InputSize       string  // About 1TB (10^10 100-byte records)
    InputRate       string  // Peak 13GB/s (lower than Grep due to writing intermediate data)
    ShufflePattern  string  // Two-phase pattern, related to reduce task batching
    OutputRate      string  // 2-4GB/s (dual replica writes, actual physical writes 4-8GB/s)
    MapTasks        int     // M=15000 (about 64MB per block)
    ReduceTasks     int     // R=4000 (partitioning strategy leverages key distribution knowledge)
    TotalTime       int     // 891 seconds (close to TeraSort benchmark 1057 seconds)
}

Technical Analysis:

1. Data Pipeline: Test clearly shows MapReduce three-phase pipeline - map phase (0-200 seconds), shuffle phase (200-600 seconds), and reduce phase (600-850 seconds)
2. Resource Bottleneck Shifting:
   • 0-200 seconds: Bottleneck in disk I/O and CPU (parsing data)
   • 200-600 seconds: Bottleneck shifts to network bandwidth (shuffle)
   • 600-850 seconds: Bottleneck in sorting computation and output disk I/O
3. Locality Optimization Effect: Input rate (13GB/s) higher than shuffle rate mainly due to data locality optimization, with most reads going through local disk rather than network
4. Replication Overhead: Output rate (2-4GB/s) relatively low mainly due to GFS dual replica strategy, actual physical writes are twice this rate

Effect of Backup Tasks

Figure 3 also shows:

// Backup task impact analysis
type BackupTaskAnalysis struct {
    WithBackup      int     // Normal execution, total time 891 seconds
    WithoutBackup   int     // 1283 seconds, 44% increase
    StragglerDelay  int     // Last 5 reduce tasks took additional 300 seconds
    EfficiencyGain  string  // Backup task mechanism improves performance by 44%
}

Professional Interpretation:

1. Severity of Straggler Problem: Data clearly demonstrates the severity of the "straggler problem" in distributed systems - just 5 slow tasks increased total time by 44%
2. Root Cause Analysis: Stragglers typically stem from:
   • Hardware anomalies (like disk performance degradation, memory errors)
   • Resource competition (like interference from other processes)
   • Data skew (some reduce tasks process significantly more data than others)
3. Cloud-Native Environment Significance: In shared resource cloud environments, straggler problems are more prevalent; backup task mechanisms are key to ensuring performance predictability

Machine Failures

Figure 3 also shows:

// Fault tolerance capability analysis
type FaultToleranceAnalysis struct {
    NodesKilled     int     // 200 nodes (about 11.5% of nodes)
    RecoveryPattern string  // Brief negative input rate, then quick recovery
    TotalTime       int     // 933 seconds, only 5% increase
    KeyMechanism    string  // Automatically detect failures and re-execute tasks
}

Analysis:

1. Failure Impact Visualization: Negative input rate in the graph intuitively shows how node failures cause loss of completed work and re-execution requirements
2. Quick Recovery Principle:
   • Task state tracking: master node continuously tracks each task's state
   • Heartbeat detection: detect worker failures through periodic heartbeats
   • Task rescheduling: reassign failed node tasks to healthy nodes
   • Redundant execution: key is MapReduce design allowing any node to handle any task
3. Comparison with Traditional Systems: Traditional MPP databases typically fail completely or have 50%+ performance degradation with 11% node failure; MapReduce's 5% performance loss highlights its excellent fault tolerance capability

Experience

This content is excerpted from Jeff Dean and Sanjay Ghemawat's MapReduce paper, detailing MapReduce's early development history and application within Google.

Technical Development History

The first version of the MapReduce library was developed in February 2003, with major enhancements in August of the same year, including:

• Locality optimization
• Dynamic load balancing of task execution across worker nodes
• Other performance optimizations

Wide Application Scope

MapReduce gained widespread application within Google, covering multiple domains:

1. Large-scale machine learning problems
2. Clustering problems for Google News and Froogle (early Google Shopping) products
3. Popular query report data extraction (such as Google Zeitgeist)
4. Attribute extraction from large-scale web corpora (such as geographical locations for localized search)
5. Large-scale graph computations

Explosive Growth

From the graph, MapReduce usage within Google showed exponential growth:

• Early 2003: Near 0 instances
• End of September 2004: Nearly 900 instances

This rapid growth indicates MapReduce gained extremely high recognition and application value within Google.

Success Factor Analysis

Key factors for MapReduce success:

1. Simplified Distributed Computing: Enables developers to write simple programs that run efficiently on thousands of machines
2. Accelerated Development Cycles: Significantly shortened development and prototyping cycles
3. Lowered Technical Barriers: Allows programmers without distributed/parallel systems experience to easily leverage large-scale computing resources

Scale and Efficiency Analysis (August 2004 Data)

From table data, the following insights emerge:

1. Wide Usage: Executed 29,423 MapReduce jobs in a single month
2. High Processing Efficiency: Average job completion time was 634 seconds (about 10.5 minutes)
3. Large Computation Scale:
   • Used equivalent of 79,186 machine-days of computation time
   • Processed 3,288 TB of input data
   • Generated 758 TB of intermediate data
   • Output 193 TB of results
4. Task Distribution Characteristics:
   • Each job used average of 157 worker machines
   • Average 1.2 worker node failures per job (indicating good fault tolerance)
   • Average 3,351 map tasks and 55 reduce tasks per job
5. Code Reusability:
   • 395 unique map implementations
   • 269 unique reduce implementations
   • 426 unique map/reduce combinations

Conclusions

The paper's conclusions explain why this paper is so famous:

Three Key Success Factors of MapReduce

1. Simple and Easy-to-Use Programming Model

MapReduce's primary success factor is its concise programming interface:

// Users only need to define these two functions, without worrying about distributed system complexity
func Map(key, value string) []KeyValue { /* User-defined mapping logic */ }
func Reduce(key string, values []string) string { /* User-defined reduction logic */ }

This design is extremely developer-friendly because it:

• Hides the complex details of parallelization
• Automatically handles fault tolerance mechanisms
• Built-in locality optimization
• Provides transparent load balancing

This enables even programmers without distributed systems experience to easily write efficient distributed programs.

2. Powerful Expressiveness

The MapReduce model can easily express various types of computational problems, widely applied within Google for:

• Web search service data generation
• Large-scale sorting
• Data mining
• Machine learning
• Many other systems

This versatility makes MapReduce a fundamental computing framework within Google.

3. Excellent Scalability

MapReduce implementation can scale to large clusters containing thousands of machines:

// Pseudo-code: MapReduce scheduling process
func Schedule(input []string, mappers int, reducers int) Result {
    // Automatically handles:
    // 1. Task allocation and parallelization
    // 2. Machine failure detection and recovery
    // 3. Data locality optimization
    // 4. Intermediate result management
}

This enables efficient handling of large-scale computational problems encountered by Google, laying the foundation for big data processing.

Three Key Insights from Research Team

1. Value of Restricted Programming Models

Research shows that by consciously restricting the programming model, enormous system advantages can be gained:

• Easy parallelization and distributed computing
• Natural implementation of fault tolerance mechanisms
• Reduced development and maintenance costs

This "less is more" philosophy contrasts sharply with other systems attempting to provide completely general parallel programming environments.

2. Network Bandwidth is a Scarce Resource

The research team found network bandwidth is a precious resource in distributed systems, so many optimizations target reducing network transmission:

• Locality optimization: Prioritize reading data from local disks, reducing cross-network data transmission
• Local intermediate data storage: Write intermediate results to local disks rather than distributed storage, saving network bandwidth

These designs are particularly important in large-scale clusters, where data transmission can become system bottlenecks.

3. Importance of Redundant Execution

Redundant execution is a key innovation in MapReduce, used for:

• Reducing impact of slow machines (stragglers)
• Gracefully handling machine failures
• Preventing data loss

// Pseudo-code: Redundant task scheduling in MapReduce
func scheduleBackupTasks(slowTasks []Task) {
    for _, task := range slowTasks {
        if time.Now() - task.StartTime > slowThreshold {
            // Launch backup copy of same task on another machine
            launchDuplicateTask(task)
        }
    }
}

This mechanism significantly improves reliability and performance consistency of large distributed systems.

Technical Legacy of MapReduce

The paper's conclusions reveal that MapReduce is not just a technical innovation, but a new paradigm for large-scale data processing:

1. Architectural Influence: MapReduce design philosophy influenced later big data processing frameworks like Hadoop, Spark
2. Programming Model Innovation: Proved that simplified programming models can solve complex distributed computing problems
3. Engineering Practice Revolution: Changed methods for building large-scale data processing systems, from expert systems to general frameworks
4. Commercial Value Creation: Laid foundation for later big data ecosystems, creating enormous commercial value

In summary, MapReduce successfully solved core challenges of large-scale distributed data processing through simple yet powerful abstractions, making massive data processing accessible. This is why it achieved tremendous success both within Google and across the industry.

Overall Architectural Design Principles

In high-concurrency API management scenarios, gateway architecture must adhere to the following core design principles:

1. Layered Architecture Design

   Traffic Ingress Layer → Processing Layer → Routing Layer → Backend Service Layer
   

   Adopt a strictly layered architecture with separation of concerns at each layer, making the system more scalable horizontally and optimizable vertically.

2. Stateless Design

   Design stateless gateway nodes to ensure any gateway instance can handle any request - this is fundamental to supporting high concurrency. Session state and user information should be stored in distributed caches or dedicated state storage systems.

High-Performance Technology Stack Selection

1. Core Technology Selection

   1. Data Plane: Based on high-performance proxies like Envoy, NGINX, or custom components built with Go/Rust
   2. Control Plane: Employ efficient configuration management and service discovery mechanisms

2. Asynchronous I/O Model

   1. Adopt non-blocking I/O models (such as Go's goroutine+channel, Rust's tokio, Node.js event loop)
   2. Avoid traditional thread pool models to reduce context switching overhead

Multi-Level Caching Architecture

1. Global Distributed Cache Layer

   Client → CDN → Edge Cache → API Gateway Local Cache → Service Cache
   

2. Multi-Dimensional Caching Strategy

   1. Routing Information Cache: Local high-speed cache with periodic updates
   2. Authentication Information Cache: Distributed token validation result caching
   3. Response Data Cache: Intelligent caching strategy based on content characteristics

Dynamic Scaling Design

1. Flexible Deployment Architecture

   Multi-Region → Multi-AZ → Multi-Cluster → Multi-Instance
   

2. Elastic Scaling Strategy

   1. Predictive Scaling: Predict scaling needs based on historical traffic patterns
   2. Reactive Scaling: Trigger scaling based on real-time metrics (CPU, memory, request queue depth)
   3. Graceful Scale-Down: Ensure graceful connection closure and request completion processing

Efficient Traffic Control Mechanisms

1. Multi-Level Rate Limiting Design

   flowchart LR
       Client --> GlobalLimit["Global Rate Limiting Layer"]
       GlobalLimit --> ServiceLimit["Service-Level Rate Limiting"]
       ServiceLimit --> APILimit["API-Level Rate Limiting"]
       APILimit --> UserLimit["User-Level Rate Limiting"]
       UserLimit --> Backend["Backend Services"]
   

2. Adaptive Flow Control Algorithms

   1. Token Bucket + Leaky Bucket Combination: Balance burst traffic and steady traffic
   2. Priority-Based Differential Processing: Prioritize core API protection
   3. Adaptive Rate Limiting: Dynamically adjust rate limiting thresholds based on backend service health

Gateway Cluster High Availability Design

1. Multi-Region Deployment Architecture

   1. Geographic-Level Redundancy: Cross-region deployment ensures regional-level fault isolation
   2. Proximity Access: Intelligent DNS or global load balancing for traffic proximity routing

2. Fault Isolation Strategy

   Client Grouping → Gateway Instance Grouping → Backend Service Grouping
   

   1. Bulkhead Pattern: Isolate client requests to different gateway instance groups
   2. Circuit Breaker Mechanism: Intelligent circuit breaker design based on multi-dimensional metrics like error rates and latency
   3. Degradation Strategy: Define clear service degradation paths and fallback mechanisms

Request Processing Optimization

1. Request Processing Pipeline

   Request Reception → Authentication & Authorization → Request Transformation → 
   Routing Decision → Load Balancing → Backend Invocation → Response Processing
   

2. Performance Optimization Techniques

   1. Batch Processing: Merge fragmented requests to reduce network round trips
   2. Request Collapsing: Merge concurrent requests for the same resource
   3. Parallel Processing: Parallelize cross-service request processing
   4. Streaming Response Processing: Stream transmission for large responses
   5. Zero-Copy Technology: Reduce data copying stages

Efficient Communication Protocols

1. Protocol Support and Optimization

   1. HTTP/2 Multiplexing: Reduce connection establishment overhead
   2. gRPC Support: Efficient binary transmission and stream processing
   3. WebSocket Optimization: Long connection management and heartbeat mechanisms

2. Connection Pool Management

   1. Dynamically adjustable backend connection pools
   2. Long connection reuse and keep-alive strategies
   3. Connection warm-up mechanisms to avoid cold start latency

End-to-End Observability

1. Multi-Dimensional Monitoring System

   Infrastructure Metrics → Gateway Performance Metrics → API Call Metrics → Business Metrics
   

2. Real-Time Monitoring and Alerting

   1. Health Checks: Combination of active and passive health detection
   2. Performance Analysis: Key metrics like request latency distribution and queue depth
   3. Anomaly Detection: Machine learning-based abnormal behavior identification

Dynamic Configuration Update Mechanism

1. Dynamic Configuration Architecture

   1. Distributed configuration center + local cache
   2. Configuration change event notification mechanism
   3. Incremental configuration updates to reduce resource consumption

2. Canary Release Capabilities

   1. Canary deployment for configuration changes
   2. Smooth traffic migration switching
   3. Emergency rollback mechanisms

While browsing through technical articles, I came across this insightful piece on microservices architecture, which resonated with our company's actual business scenarios.

Challenges at Scale

The article discusses how client applications access microservices in a distributed architecture. The main challenges include:

1. API Granularity Mismatch: Microservices typically expose fine-grained APIs, while clients need aggregated data, forcing multiple service interactions
2. Client Requirement Diversity: Different clients (desktop browsers, mobile apps, third-party applications) require different data structures and payloads
3. Network Performance Variations: Mobile networks are typically slower with higher latency compared to internal networks, impacting user experience
4. Dynamic Service Instance Changes: Service instances and their locations (host + port) change dynamically
5. Protocol Heterogeneity: Backend services may use different protocols, some of which are not web-friendly

I'll skip the detailed explanations here - feel free to reach out for deeper technical discussions.

Solution Architecture

The author proposes implementing the API Gateway pattern as the solution:

1. Unified Entry Point: Design an API gateway as the single entry point for all clients
2. Request Processing Strategies:
   • Simple proxy/routing to corresponding services
   • Fan-out requests to multiple services with result aggregation
3. Client-Specific APIs: Provide customized APIs for different clients rather than one-size-fits-all solutions
4. Cross-Cutting Concerns: Implement authentication, authorization, SSL termination, caching, and other shared functionalities

The article also introduces a pattern variant: Backends for Frontends (BFF), which involves designing dedicated API gateways for each client type (web applications, mobile apps, third-party applications).

Personal Analysis

Advantages

Centralized Management: All API traffic flows through a single entry point, enabling unified monitoring, tracing, and governance

A financial institution reduced security incident response time from hours to minutes 
after implementation, as all suspicious traffic could be quickly identified at the gateway layer.

Unified Cross-Cutting Concerns: Authentication, authorization, rate limiting, and other functionalities need to be implemented only once at the gateway layer

Protocol Translation & Aggregation: Convert internal protocols to unified external interfaces, reducing client request counts. An e-commerce platform reduced mobile app loading time by 68% by aggregating 7 independent calls into a single API

Decoupling Implementation: Clients are decoupled from internal microservice structure, allowing services to evolve independently without affecting clients

Traffic Control: Provides intelligent routing, load balancing, and circuit breaker capabilities, enhancing system resilience

These benefits are well-established in the industry.

Disadvantages

Potential Single Point of Failure: All traffic depends on a single component, which could become a system bottleneck if not properly designed - though this depends on the distributed architecture team's capabilities

Increased Latency: Additional network hops may increase response times (typically 5-10ms)

Complexity Escalation: As scale increases, the gateway may become bloated and difficult to maintain

A large enterprise's API gateway accumulated over 200 custom processing rules, eventually reaching a point where no one dared to modify the codebase.

Team Collaboration Challenges: Gateway changes require cross-team coordination, potentially creating development bottlenecks

Applicable Scenarios

1. Small to Medium Microservices Architecture: Moderate service count (10-50 services) with limited client types
2. High-Security Systems: Financial, healthcare, and other domains requiring unified security policies
3. Mixed Protocol Environments: Internal services using different protocols (REST, gRPC, AMQP, etc.)
4. Centralized API Governance: Enterprise applications requiring centralized API governance strategies

Core Functionality & Implementation Effects

The author outlines three core functions provided by API gateways:

1. Reverse Proxy/Gateway Routing: Using Layer 7 routing to redirect HTTP requests
2. Request Aggregation: Aggregating multiple internal microservice requests into single client requests
3. Cross-Cutting Concerns & Gateway Offloading: Centralized implementation of common functionalities

Benefits

1. Client-Microservice Decoupling: Isolates clients from microservice architecture details
2. Optimal API Provision: Provides customized APIs for various client types
3. Reduced Request Count: Single requests retrieve multi-service data, reducing network overhead
4. Simplified Client Logic: Complex call logic migrates from client to API gateway
5. Protocol Translation: Converts public web-friendly API protocols to internal protocols

Drawbacks

1. Increased Complexity: Introduces a new component requiring development, deployment, and management
2. Increased Response Time: Additional network hops may increase latency (minimal impact for most applications)
3. Single Point of Failure Risk: A single API gateway may become a bottleneck or single point of failure

These align with my initial assessment.

Production Deployment Strategies

┌───────────┐  ┌───────────┐  ┌─────────────┐
│  Web BFF  │  │ Mobile BFF│  │ Third-party │
│           │  │           │  │   BFF       │
└─────┬─────┘  └─────┬─────┘  └─────┬───────┘
      │              │              │
      ▼              ▼              ▼
┌─────────────────────────────────────┐
│        Core API Gateway             │
│ (Auth, Monitoring, Rate Limiting,   │
│      Basic Routing)                 │
└─────────────────────────────────────┘
      │              │              │
      ▼              ▼              ▼
┌─────────┐     ┌───────────┐     ┌───────────┐
│Service  │     │Service    │     │Service    │
│Cluster 1│     │Cluster 2  │     │Cluster 3  │
└─────────┘     └───────────┘     └───────────┘

Scale Considerations

1. Small Applications (5-15 microservices):
   • Recommendation: Single API gateway
   • Rationale: Simple and efficient, avoiding over-engineering
2. Medium Applications (15-50 microservices):
   • Recommendation: Basic API gateway + key BFFs
   • Rationale: Balances complexity and client optimization needs
3. Large Applications (50+ microservices):
   • Recommendation: Complete three-tier architecture (Edge Gateway + BFF Layer + Internal Gateway)
   • Rationale: Supports organizational scaling and complex business domains

Decision Factors

When selecting API gateway strategies, consider:

1. Organizational Structure: Conway's Law suggests system design often reflects organizational structure
2. Scaling Expectations: Consider business growth and diversification over 3-5 years
3. Operational Capabilities: Assess team capacity for managing multiple gateways
4. Consistency Requirements: Business requirements for API behavior consistency
5. Performance Budget: Evaluate latency impact of additional network hops

Regardless of the chosen pattern, maintain lightweight gateway layers, avoid excessive business logic drift leading to "smart pipes" anti-patterns, and establish robust monitoring systems to ensure gateway layer performance and stability.

2025-03-07 Update: Recently, Xiaohongshu's gateway practices follow many of the principles below. —— 《Xiaohongshu Launches Self-Developed Rust High-Performance Layer 7 Gateway ROFF》

This article is divided into two parts:

1. Performance Bottleneck Analysis Methods
2. Hands-on Analysis of APISIX, Kong, and Nginx

Performance Bottleneck Analysis Methods

Performance Bottleneck Analysis Methods (The Real Deal)

This section covers the actual methods.

Monitoring Metrics Analysis

• Latency Metrics: Monitor request latency and integration latency
• Error Rate Metrics: Monitor 4XX and 5XX error rates to identify system failure points
• Throughput Metrics: Request count and requests per second (RPS)
• Cache Hit Metrics: Monitor CacheHitCount and CacheMissCount ratios

System Resource Monitoring

• CPU utilization analysis
• Memory consumption analysis
• Network I/O monitoring
• Disk I/O monitoring (especially for logs and cache)

Request Flow Analysis

• Request processing chain tracing
• Hot path identification
• Serial processing bottleneck discovery

Infrastructure-Level Optimization

Hardware Resource Optimization

• Increase CPU cores and memory capacity
• Use SSD storage to improve I/O speed
• Deploy high-performance network cards to reduce network latency
• Properly evaluate and adjust container resource limits (in Kubernetes environments)

Network Optimization

• Optimize network topology structure
• Reduce network hops
• Use CDN to accelerate static content
• Implement DNS optimization
• Deploy gateways and backend services in the same network zone

Operating System Tuning

• Adjust TCP/IP stack parameters (such as increasing connection queues)
• Optimize file descriptor limits
• Adjust kernel parameters (such as somaxconn, tcpfintimeout)
• Optimize client thread count in Linux environments

Gateway Configuration-Level Optimization

Connection Pool Optimization

• Configure appropriate database connection pool sizes (recommended to match expected client count) —— Oracle Database Administrator Essentials
• Adjust backend service connection pool parameters
• Set connection timeout and retry strategies
• Implement connection keep-alive mechanisms

HTTP Optimization

• Enable HTTP keep-alive to reuse TCP connections —— Oracle Performance Practices
• Configure appropriate chunked encoding strategies
• Adjust HTTP header size limits
• Set reasonable request/response timeout values

Caching Strategy Optimization

• Implement response caching to reduce backend calls —— Amazon API Optimization Practices
• Configure cache key strategies and TTL (Time To Live)
• Implement multi-level caching strategies
• Optimize cache validation and invalidation strategies

Request Processing Optimization

Concurrent Processing Optimization

• Increase worker thread count
• Implement asynchronous processing patterns
• Optimize thread pool configuration
• Use event-driven architecture for high-concurrency request handling

Load Balancing Strategies

• Implement intelligent load balancing algorithms (weighted round-robin, least connections, etc.)
• Configure dynamic load balancing strategies
• Implement service health checks and automatic failover
• Adjust weights based on backend service capacity

Routing Optimization

• Optimize routing lookup algorithms
• Implement routing caching
• Configure routing warm-up strategies
• Implement intelligent routing based on traffic characteristics

Data Processing Optimization

Message Processing Optimization

• Configure "spill to disk" strategies for large messages (e.g., set >4MB messages to write to disk) 3
• Implement request/response compression 1
• Optimize serialization/deserialization processes
• Reduce unnecessary data transformations

Protocol Optimization

• Use HTTP/2 to reduce latency and improve parallel processing capability
• Use WebSocket and other long-connection protocols in appropriate scenarios
• Consider using gRPC to improve microservice communication efficiency
• Implement protocol upgrade strategies

Monitoring and Logging Optimization

Log Optimization

• Reduce unnecessary trace information (set production environment to ERROR or FATAL level) 3
• Disable or reduce access log recording 3
• Disable transaction logs to reduce disk I/O burden 3
• Implement asynchronous log writing strategies

Monitoring Optimization

• Disable real-time monitoring to reduce overhead 3
• Disable or reduce traffic monitoring 3
• Configure reasonable monitoring sampling rates
• Implement intelligent alerting thresholds to avoid excessive monitoring system load

Security Performance Optimization

Authentication Optimization

• Cache authentication results
• Use lightweight token validation
• Implement tiered authentication strategies
• Optimize JWT processing workflows

SSL/TLS Optimization

• Use session reuse to reduce handshake overhead
• Configure OCSP stapling
• Implement TLS connection pools
• Select efficient cipher suites

Advanced Optimization Strategies

Circuit Breaking and Rate Limiting

• Implement request rate limiting to protect backend systems
• Configure circuit breakers to prevent system overload
• Implement backoff algorithms for retry handling
• Traffic shaping to optimize request distribution

Service Mesh Integration

• Integrate with service meshes like Istio/Envoy
• Delegate some gateway functions to sidecar proxies
• Implement collaborative strategies between gateways and service meshes
• Leverage traffic management capabilities provided by service meshes

Architectural Optimization

• Consider multi-tier gateway architecture (edge gateway + internal gateway)
• Implement edge computing to reduce latency
• Domain-driven API gateway design
• Gateway sharding based on traffic characteristics

By systematically analyzing performance bottlenecks in the above aspects and applying corresponding optimization techniques, gateway performance, throughput, and reliability can be significantly improved. The key is to select the most suitable combination of optimization strategies based on actual system characteristics and business requirements, and maintain high system performance through continuous monitoring and tuning.

Hands-on Analysis

Nginx Gateway

Main Performance Bottlenecks

1. Connection Handling Capacity Bottlenecks

   • Default worker process configuration may not match server CPU cores
   • Connection pool size limits cause connection rejection under high concurrency
   • File descriptor limits cause "too many open files" errors

2. Configuration Complexity Bottlenecks

   • Static configuration files require manual modification and reloading
   • Large-scale routing rules make configuration maintenance difficult
   • Configuration changes require reloading, potentially causing request interruptions

3. SSL Processing Bottlenecks

   • High SSL handshake overhead, CPU usage spikes under high concurrency
   • Inefficient key exchange algorithms
   • Improper session cache configuration causes repeated handshakes

Optimization Methods

1. Worker Process and Connection Optimization

   • Set worker_processes to match CPU core count
   • Increase worker_connections value (typically set to 4096 or higher)
   • Use worker_cpu_affinity to bind worker processes to specific CPU cores
   • Adjust system file descriptor limits (ulimit -n)

2. Event Processing Optimization

   • Enable multi_accept and accept_mutex
   • Use epoll event handling model (on Linux systems)
   • Adjust worker_aio_requests to improve async I/O performance

3. HTTP Optimization

   • Configure keepalive_timeout and keepalive_requests parameters
   • Enable sendfile, tcp_nopush, and tcp_nodelay options
   • Implement gzip compression to reduce transmitted data
   • Set client_body_buffer_size and client_max_body_size to limit request size

4. SSL Performance Optimization

   • Enable ssl_session_cache shared to improve session reuse rate
   • Configure OCSP stapling to reduce handshake latency
   • Use ECC certificates to reduce computational overhead
   • Prioritize high-performance cipher suites (like AES-GCM)

Kong Gateway

A reliable and well-established gateway.

Main Performance Bottlenecks

1. Database Dependency Bottlenecks

   • PostgreSQL/Cassandra database queries become performance bottlenecks
   • Database pressure increases during configuration changes
   • Database consistency challenges in distributed deployments

2. Lua Script Processing Bottlenecks

   • Overly long plugin execution chains increase request latency 1
   • Improper Lua VM memory usage causes performance degradation
   • JIT compilation limitations affect dynamic script performance

3. JWT Validation Bottlenecks

   • High JWT validation processing overhead, noticeable in high-percentile latency 2
   • Kong's 99.99 percentile latency can reach 3 times that of NGINX 2

Optimization Methods

1. Database Optimization

   • Use DB-less mode to reduce database dependencies
   • Increase database connection pool size (pg_pool parameter in kong.conf)
   • Implement database read-write separation (master-slave architecture)
   • Consider using declarative configuration instead of database storage

2. Plugin Chain Optimization

   • Enable only necessary plugins to reduce processing chain length
   • Adjust plugin execution order (lightweight, high-frequency plugins first)
   • Configure independent caches for plugins (like Redis cache for rate-limiting plugin)
   • Monitor and optimize long-running plugins

3. Cache Optimization

   • Configure lua_shared_dict cache size
   • Adjust plugin-level cache TTL
   • Use external Redis cache to improve hit rates
   • Configure entity cache to reduce database queries

4. Connection Pool Optimization

   • Adjust upstream_keepalive parameter (typically 100-200)
   • Increase nginxupstreamkeepalive_timeout value
   • Set reasonable nginxupstreamkeepalive_requests value
   • Increase nginxhttpclientbodybuffer_size for large request bodies

APISIX Gateway

Main Performance Bottlenecks

1. etcd Dependency Bottlenecks

   • etcd cluster stability affects gateway configuration propagation
   • Frequent configuration changes increase etcd pressure
   • etcd read-write latency affects dynamic routing updates

2. Route Matching Bottlenecks

   • Large numbers of fine-grained routes increase matching latency
   • Complex regex routes reduce matching efficiency
   • Untimely route cache updates cause routing errors

Optimization Methods

1. etcd Optimization

   • Build highly available etcd clusters
   • Optimize etcd configuration (like setting reasonable heartbeat intervals)
   • Implement etcd sharding to reduce single-node pressure
   • Increase config_center.timeout parameter value (default 30 seconds)

2. Route Optimization

   • Use prefix matching instead of full regex
   • Increase route cache TTL
   • Reduce route rule complexity, split overly complex rules
   • Use domain or hostname pre-filtering

Envoy Gateway

Main Performance Bottlenecks

1. xDS Configuration Update Bottlenecks

   • Dynamic configuration updates cause resource reallocation overhead
   • Control Plane communication latency affects configuration delivery
   • Large numbers of listener and cluster configurations cause high memory usage

2. Filter Chain Processing Bottlenecks

   • Long HTTP filter chains increase processing latency
   • Complex filter logic causes high CPU usage
   • Lua filters are less efficient than native filters

Optimization Methods

1. xDS Configuration Optimization

   • Implement incremental xDS to reduce configuration update overhead
   • Optimize Control Plane communication (use gRPC streams instead of polling)
   • Set reasonable configuration cache TTL
   • Use Aggregated Discovery Service (ADS) to ensure configuration consistency

2. Filter Optimization

   • Reduce filter chain length, keep only necessary filters
   • Prioritize native C++ filters over Lua or WASM
   • Adjust filter execution order (high-frequency filters first)
   • Enable statistical monitoring for critical filters

Performance Comparison and Selection Recommendations for Different Gateways

Performance Comparison

• In standard API call tests, NGINX API Management Module performance can be 2x better than Kong Data Support
• In terms of latency, NGINX adds 20-30% lower latency than Kong Data Support
• In CPU efficiency, NGINX is about 40% more efficient than Kong Data Support
• In JWT validation scenarios, NGINX can handle 2x more API calls than Kong Data Support

Selection Recommendations

1. NGINX Suitable Scenarios:

   • Stable API gateway needs with static routing configuration
   • Scenarios prioritizing performance and low latency
   • Lightweight gateway needs in resource-constrained environments
   • Primarily providing reverse proxy and load balancing functions

2. Kong Suitable Scenarios:

   • Need rich API management features (authentication, rate limiting, transformation, etc.)
   • Teams pursuing development convenience (RESTful API configuration)
   • Microservice architectures with dynamic routing needs
   • Can accept some performance loss for feature richness

3. APISIX Suitable Scenarios:

   • Teams pursuing balance between dynamic routing and high performance
   • Cloud-native architectures with service discovery integration needs
   • Scenarios requiring fine-grained traffic control

4. Envoy Suitable Scenarios:

   • Kubernetes/Istio service mesh infrastructure
   • Modern cloud architectures requiring advanced observability
   • DevOps teams pursuing programmability and extensibility

By understanding the performance bottleneck characteristics and optimization methods of different gateways, you can choose the most suitable gateway type based on your business characteristics and technology stack, and implement targeted performance optimization to achieve the best balance of gateway performance and functionality.

Summary

Gateway performance optimization is systematic work that requires analysis and optimization from multiple levels. Through the analysis methods, optimization techniques, and code examples provided above, you can implement targeted optimization for different performance bottlenecks. The key points are:

1. Establish comprehensive performance monitoring systems to detect performance bottlenecks promptly 1
2. Adopt multi-level caching strategies to reduce redundant calculations and network requests
3. Optimize connection pool management to improve connection reuse efficiency
4. Implement efficient load balancing algorithms for intelligent request distribution
5. Use regular benchmark testing to continuously evaluate and optimize performance

For gateways in cloud-native environments, you can also consider leveraging Kubernetes' auto-scaling capabilities to dynamically adjust gateway instance counts in response to traffic changes.

Overall Architecture Overview

Thanks to Zhang Yanfei for the diagram.

After you basically understand what network card drivers, hardware interrupts, software interrupts, and ksoftirqd threads are, you can provide a kernel packet reception path diagram as shown above.

The general process is as follows:

1. When the network card receives data, it writes the received frames to memory using DMA, then sends an interrupt to the CPU to notify that data has arrived.
2. When the CPU receives the interrupt request, it calls the interrupt handler registered by the network device driver.
3. The network card's interrupt handler doesn't do much work - it issues a software interrupt request and quickly releases CPU resources.
4. The ksoftirqd kernel thread detects the software interrupt request, calls poll to start polling for packet reception, and forwards received packets to various protocol stack layers for processing. For TCP packets, they are placed in the user socket's receive queue.

Foundation Work Before Everything Else

Before Linux drivers, kernel protocol stacks, and other modules can receive network card data packets, extensive preparation work must be done:

1. Pre-create ksoftirqd kernel threads;
2. Register processing functions for various protocols;
3. Pre-initialize the network device subsystem;
4. Start up the network card.

Initialization Work

Creating ksoftirqd Kernel Threads

All Linux software interrupts are handled in dedicated kernel threads (ksoftirqd), so it's essential to understand how these threads are initialized.

First, there isn't just one thread, but N threads, where N equals the number of cores on your machine.

During system initialization, the smpboot_register_percpu_thread function is called in kernel/smpboot.c, which further executes spawn_ksoftirqd (located in kernel/softirq.c) to create softirqd threads, as shown below:

Related code:

static struct smp_hotplug_thread softirq_threads = {
    .store            = &ksoftirqd,
    .thread_should_run    = ksoftirqd_should_run,
    .thread_fn        = run_ksoftirqd,
    .thread_comm        = "ksoftirqd/%u",
};

static __init int spawn_ksoftirqd(void)
{
    cpuhp_setup_state_nocalls(CPUHP_SOFTIRQ_DEAD, "softirq:dead", NULL,
                  takeover_tasklets);
    BUG_ON(smpboot_register_percpu_thread(&softirq_threads));

    return 0;
}
early_initcall(spawn_ksoftirqd);

After ksoftirqd is created, it enters its thread loop functions ksoftirqdshouldrun and run_ksoftirqd, then checks if there are any software interrupts to process.

Software interrupts include not only network software interrupts but other types as well. The Linux kernel defines all software interrupt types in interrupt.h:

// file: include/linux/interrupt.h
enum
{
    HI_SOFTIRQ=0,
    TIMER_SOFTIRQ,
    NET_TX_SOFTIRQ,
    NET_RX_SOFTIRQ,
    BLOCK_SOFTIRQ,
    IRQ_POLL_SOFTIRQ,
    TASKLET_SOFTIRQ,
    SCHED_SOFTIRQ,
    HRTIMER_SOFTIRQ,
    RCU_SOFTIRQ,    /* Preferable RCU should always be the last softirq */

    NR_SOFTIRQS
};

Network Subsystem Initialization

During network subsystem initialization, softnetdata is initialized for each CPU, and processing functions are registered for RXSOFTIRQ and TX_SOFTIRQ, as shown in Figure 2.4.

The Linux kernel initializes various subsystems by calling subsys_initcall.

Key Point!!! The network subsystem initialization mentioned here executes the netdevinit function.

This is the netdevinit function in subsysinitcall(netdev_init). Code as follows:

static int __init net_dev_init(void)
{
    ......

    /*
     *    Initialise the packet receive queues.
     */

  /*
   * Allocate a softnet_data structure for each CPU. The poll_list in this structure
   * is used to wait for driver programs to register their poll functions, which can
   * be seen later when network card drivers initialize.
   */
        for_each_possible_cpu(i) {
        struct softnet_data *sd = &per_cpu(softnet_data, i);

        memset(sd, 0, sizeof(*sd));
        skb_queue_head_init(&sd->input_pkt_queue);
        skb_queue_head_init(&sd->process_queue);
        sd->completion_queue = NULL;
        INIT_LIST_HEAD(&sd->poll_list);
    ......
  }
  ......
    /*
     * open_softirq registers a processing function for each type of software interrupt.
     * NET_TX_SOFTIRQ processing function is net_tx_action;
     * NET_RX_SOFTIRQ processing function is net_rx_action;
     */
  open_softirq(NET_TX_SOFTIRQ, net_tx_action);
    open_softirq(NET_RX_SOFTIRQ, net_rx_action);
}

subsys_initcall(net_dev_init);

Following opensoftirq further reveals that this registration method is recorded in the softirqvec variable. Later, when the softirqd thread receives software interrupts, it will use this variable to find the corresponding processing function for each type of software interrupt.

void open_softirq(int nr, void (*action)(struct softirq_action *))
{
    softirq_vec[nr].action = action;
}

Protocol Stack Registration

Network Card Driver Initialization

Each driver uses module_init to register an initialization function with the kernel. When the driver is loaded, the kernel calls this function.

After completion, the Linux kernel knows the driver's relevant information, such as the igb network card driver's igbdrivername and igb_probe function address.

When a network card device is identified, the kernel calls its driver's probe method. (Continuing with the igb network card driver example), the igbdriver's probe method is igbprobe.

The purpose of the igb_probe method is to get the device into a ready state as quickly as possible.

Additionally, there's a critical step: registering the poll function required by the NAPI mechanism, which for the igb network card driver is igb_poll.

After Initialization Completion

Starting the Network Card

After all the above initialization is complete, the network card can be started. The general startup sequence is similar, as shown below:

igb_open code:

static int __igb_open(struct net_device *netdev, bool resuming)
{
  // Allocate transmit descriptor arrays
  err = igb_setup_all_tx_resources(adapter);
  // Allocate receive descriptor arrays
    err = igb_setup_all_rx_resources(adapter);

  // Register interrupt handler
  err = igb_request_irq(adapter);
    if (err)
        goto err_req_irq;

  // Enable NAPI
      for (i = 0; i < adapter->num_q_vectors; i++)
        napi_enable(&(adapter->q_vector[i]->napi));
    ......
}

The igbopen function calls igbsetupalltxresources and igbsetupallrxresources. In the igbsetupallrx_resources step, RingBuffer is allocated and the mapping relationship between memory and Rx queues is established.

static int igb_setup_all_rx_resources(struct igb_adapter *adapter)
{
    ......

    for (i = 0; i < adapter->num_rx_queues; i++) {
        err = igb_setup_rx_resources(adapter->rx_ring[i]);
        ...
    }

    return err;
}

Using a for loop with the igbsetuprxresources function, several queues are created. The igbsetuprxresources function:

int igb_setup_rx_resources(struct igb_ring *rx_ring)
{
    struct device *dev = rx_ring->dev;
    int size;

    // 1. Allocate igb_rx_buffer array memory
    size = sizeof(struct igb_rx_buffer) * rx_ring->count;

    rx_ring->rx_buffer_info = vzalloc(size);
    if (!rx_ring->rx_buffer_info)
        goto err;

    /* Round up to nearest 4K */
    // 2. Allocate e1000_adv_rx_desc DMA array memory
    rx_ring->size = rx_ring->count * sizeof(union e1000_adv_rx_desc);
    rx_ring->size = ALIGN(rx_ring->size, 4096);

    rx_ring->desc = dma_alloc_coherent(dev, rx_ring->size,
                       &rx_ring->dma, GFP_KERNEL);
    if (!rx_ring->desc)
        goto err;

    // 3. Initialize queue members
    rx_ring->next_to_alloc = 0;
    rx_ring->next_to_clean = 0;
    rx_ring->next_to_use = 0;

    return 0;

err:
    vfree(rx_ring->rx_buffer_info);
    rx_ring->rx_buffer_info = NULL;
    dev_err(dev, "Unable to allocate memory for the Rx descriptor ring\n");
    return -ENOMEM;
}

From the above source code, you can see that internally, a RingBuffer doesn't have just one circular queue array, but two:

1. igbrxbuffer array: This array is used by the kernel, allocated via vzalloc;
2. e1000advrxdesc array: This array is used by network card hardware, allocated via dmaalloc_coherent.

Then there's the final step of interrupt function registration, which can be seen in igbrequestirq.

OK, that's all the preparation work! Next comes receiving data packets.

Starting to Receive Data Packets

This section includes hardware interrupt processing.

Hardware Interrupt Processing

First, when data frames arrive at the network card from the network cable, the first stop is the network card's receive queue. The network card searches for available memory locations in its allocated RingBuffer, and when found, the DMA engine will DMA the data to the memory previously associated with the network card. At this point, the CPU is unaware.

When the DMA operation completes, the network card sends a hardware interrupt to the CPU, notifying it that data has arrived. The hardware interrupt processing is as follows:

In the "Starting the Network Card" section mentioned earlier, the network card's hardware interrupt registered processing function is igbmsixring.

// file: drivers/net/ethernet/intel/igb/igb_main.c
static irqreturn_t igb_msix_ring(int irq, void *data)
{
    struct igb_q_vector *q_vector = data;

    /* Write the ITR value calculated from the previous interrupt. */
    igb_write_itr(q_vector);

    napi_schedule(&q_vector->napi);

    return IRQ_HANDLED;
}

The igbwriteitr only records hardware interrupt frequency. Following the napischedule call, you'll discover that Linux only completes simple necessary work in hardware interrupts, leaving most processing to software interrupts. From the above code, you can see that the hardware interrupt processing is really very short - it just records a register, modifies the CPU's polllist, and then issues a software interrupt. That's it, the hardware interrupt work is complete.

ksoftirqd Kernel Thread Processing Software Interrupts

Network packet reception processing mainly occurs in the ksoftirqd kernel thread, where all software interrupts are processed, as shown below:

Network Protocol Stack Processing

The netifreceiveskb function processes packets according to their protocol. For UDP packets, packets are sent sequentially to protocol processing functions like iprcv, udprcv, etc., as shown below:

IP Layer Processing

Linux IP layer operations are in the code file net/ipv4/ip_input.c.

Summary

The network module is the most complex module in the Linux kernel. The entire process involves interactions between many kernel components, such as network card drivers, protocol stacks, kernel ksoftirqd threads, etc. It looks complex, but the overall picture is actually quite clear. Simple summary as follows.

After a user executes a recvfrom call, the user process enters kernel mode through the system call. If the receive queue has no data, the process enters sleep state and is suspended by the operating system. This part is relatively simple. Next comes the work between various Linux kernel components.

First, before starting packet reception, Linux must do extensive preparation work:

• Create ksoftirqd kernel threads, set up their thread functions, and rely on them to handle software interrupts later;
• Protocol stack registration: Linux implements many protocols like ARP, ICMP, IP, UDP, and TCP. Each protocol registers its processing function, making it convenient to quickly find the corresponding processing function when packets arrive;
• Network card driver initialization: Each driver has an initialization function that the kernel will initialize. During this initialization process, prepare DMA and tell the kernel the NAPI poll function address;
• Start the network card: Allocate RX and TX queues, register interrupt corresponding processing functions.

After preparation work is complete, data arrives. The first to greet it is the network card:

• The network card DMAs data frames to memory's RingBuffer, then sends an interrupt notification to the CPU;
• CPU responds to the interrupt request, calls the interrupt processing function registered when the network card started;
• The interrupt processing function only issues a software interrupt request and does nothing else;
• Kernel thread ksoftirqd discovers a software interrupt request has arrived, first disables hardware interrupts;
• ksoftirqd thread starts calling the driver's poll function to receive packets;
• The poll function sends received packets to the protocol stack's registered ip_rcv function;
• The iprcv function sends packets to the udprcv function (for TCP packets, sent to tcprcvv4).

Some Conclusions

Question 1: What exactly is RingBuffer, and why does RingBuffer drop packets?

RingBuffer is a special area in memory, a circular queue array. In fact, this data structure includes igbrxbuffer circular queue arrays, e1000advrx_desc circular queue arrays, and numerous skbs.

If RingBuffer represents pointer arrays, they are pre-allocated. If they are skbs, they are dynamically allocated during the packet reception process.

Question 2: What are software interrupts and hardware interrupts respectively?

Key flow of data packet reception in Linux network stack:

1. Hardware Stage: Network card places received data packets into RingBuffer
2. Hardware Interrupt Trigger: Network card generates hardware interrupt to notify CPU
3. Hardware Interrupt Processing: Add network card device to poll_list doubly linked list in softnet_data structure
4. Software Interrupt Trigger: Trigger NET_RX_SOFTIRQ software interrupt
5. Software Interrupt Processing: Traverse poll_list, execute network card driver's poll function to collect network packets
6. Protocol Stack Processing: Forward data packets to protocol processing functions like ip_rcv, udp_rcv, tcp_rcv_v4

This describes the Linux NAPI (New API) mechanism, an efficient method for handling network data packets.

Actual Application of RingBuffer in Network Stack

Network Card RX/TX Ring Buffers

/* Simplified network card RX Ring structure */
struct e1000_rx_desc {
    __le64 buffer_addr;    /* Data buffer address */
    __le16 length;         /* Packet length */
    __le16 checksum;       /* Checksum */
    __u8  status;          /* Descriptor status */
    __u8  errors;          /* Error code */
    __le16 special;
};

In practice, an Intel network card's RX Ring might contain 256 such descriptors, forming a circular structure.

Actual Example of Hardware and Software Interrupt Cooperation

In Intel 82599 network card driver:

/* Hardware interrupt handler */
static irqreturn_t ixgbe_msix_lsc(int irq, void *data)
{
    struct net_device *netdev = data;

    /* Disable network card interrupt */
    ixgbe_disable_interrupt();

    /* Add device to poll_list */
    napi_schedule(&adapter->q_vector[vector]->napi);

    return IRQ_HANDLED;
}

/* NAPI poll function */
static int ixgbe_poll(struct napi_struct *napi, int budget)
{
    struct ixgbe_q_vector *q_vector = container_of(napi, struct ixgbe_q_vector, napi);
    struct ixgbe_adapter *adapter = q_vector->adapter;
    int work_done = 0;

    /* Batch receive packets from RingBuffer, process at most budget packets */
    work_done = ixgbe_clean_rx_irq(q_vector, budget);

    /* If work is incomplete, stay in poll_list */
    if (work_done < budget) {
        napi_complete(napi);
        ixgbe_enable_interrupt();
    }

    return work_done;
}

Check Out My Results

My requirement: I want to read Linux source code, manage my Linux source code reading with git, and save my source code reading with my self-built GitLab. Then I discovered that simply using git push for a 2GB Linux source code file would cause problems.

So I wrote a script to solve this problem.

The script content is as follows:

#!/bin/bash

# Configuration parameters (modify as needed)
REMOTE_NAME="origin"              # Remote repository name
BRANCH_NAME="main"                # Target branch name
REMOTE_URL="git@gitlab.cheverjohn.me:CheverJohn/linux.git"  # Remote repository URL
BATCH_SIZE=500                    # Number of files per batch
COMMIT_MESSAGE="Batch file upload"      # Commit message

# Color configuration
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No color

# Check if we're inside a git repository
if ! git rev-parse --is-inside-work-tree > /dev/null 2>&1; then
    echo -e "${YELLOW}Current directory is not a git repository, initializing...${NC}"
    git init
    echo -e "${GREEN}Git repository initialized${NC}"
else
    echo -e "${GREEN}Git repository already exists${NC}"
fi

# Check if remote repository is configured
if ! git remote | grep -q "$REMOTE_NAME"; then
    echo -e "${YELLOW}Adding remote repository $REMOTE_NAME: $REMOTE_URL${NC}"
    git remote add $REMOTE_NAME $REMOTE_URL
    echo -e "${GREEN}Remote repository added${NC}"
else
    CURRENT_URL=$(git remote get-url $REMOTE_NAME 2>/dev/null || echo "")
    if [ "$CURRENT_URL" != "$REMOTE_URL" ]; then
        echo -e "${YELLOW}Updating remote repository URL: $REMOTE_URL${NC}"
        git remote set-url $REMOTE_NAME $REMOTE_URL
        echo -e "${GREEN}Remote repository URL updated${NC}"
    else
        echo -e "${GREEN}Remote repository correctly configured${NC}"
    fi
fi

# Ensure main branch exists
if ! git show-ref --quiet refs/heads/$BRANCH_NAME; then
    echo -e "${YELLOW}Creating branch $BRANCH_NAME...${NC}"
    git checkout -b $BRANCH_NAME
    echo -e "${GREEN}Branch $BRANCH_NAME created${NC}"
else
    echo -e "${GREEN}Branch $BRANCH_NAME already exists${NC}"
    git checkout $BRANCH_NAME
fi

# Get all untracked and modified files
echo -e "${BLUE}Getting list of files to upload...${NC}"
FILES=($(git ls-files --others --exclude-standard) $(git diff --name-only))
TOTAL_FILES=${#FILES[@]}

if [ $TOTAL_FILES -eq 0 ]; then
    echo -e "${YELLOW}No files found to add, trying to add all files...${NC}"
    git add -A
    FILES=($(git diff --name-only --cached))
    TOTAL_FILES=${#FILES[@]}

    if [ $TOTAL_FILES -eq 0 ]; then
        echo -e "${RED}Error: No files found to push${NC}"
        exit 1
    fi
else
    # Add all files to staging area
    echo -e "${BLUE}Adding all files to staging area...${NC}"
    git add -A
fi

echo -e "${GREEN}Found $TOTAL_FILES files to upload${NC}"

# Calculate number of batches
BATCH_COUNT=$(( ($TOTAL_FILES + $BATCH_SIZE - 1) / $BATCH_SIZE ))
echo -e "${GREEN}Will upload in $BATCH_COUNT batches${NC}"

# First commit all files
echo -e "${BLUE}Committing all files...${NC}"
git commit -m "$COMMIT_MESSAGE"

# Push in batches
echo -e "${BLUE}Starting batch push...${NC}"
git push -u $REMOTE_NAME $BRANCH_NAME

if [ $? -eq 0 ]; then
    echo -e "${GREEN}All files successfully pushed to remote repository${NC}"
else
    echo -e "${YELLOW}Regular push failed, trying batch push method...${NC}"
    # Using git batch push method (referenced from search results examples)
    # Based on git_batch_push.sh concept, but simplified implementation

    # Use git rev-list to get all commits
    COMMITS=($(git rev-list --reverse HEAD))
    TOTAL_COMMITS=${#COMMITS[@]}

    if [ $TOTAL_COMMITS -eq 0 ]; then
        echo -e "${RED}Error: No commits found${NC}"
        exit 1
    fi

    echo -e "${GREEN}Found $TOTAL_COMMITS commits, will push in batches${NC}"

    # Calculate number of batches (500 commits per batch)
    COMMIT_BATCH_SIZE=500
    COMMIT_BATCH_COUNT=$(( ($TOTAL_COMMITS + $COMMIT_BATCH_SIZE - 1) / $COMMIT_BATCH_SIZE ))

    echo -e "${GREEN}Will push commits in $COMMIT_BATCH_COUNT batches${NC}"

    # Push commits in batches
    for ((i=0; i<$COMMIT_BATCH_COUNT; i++)); do
        START=$(($i * $COMMIT_BATCH_SIZE))
        END=$((($i + 1) * $COMMIT_BATCH_SIZE))

        if [ $END -gt $TOTAL_COMMITS ]; then
            END=$TOTAL_COMMITS
        fi

        BATCH_END_INDEX=$((END - 1))
        TARGET_COMMIT=${COMMITS[$BATCH_END_INDEX]}

        echo -e "${BLUE}Pushing batch $((i+1))/${COMMIT_BATCH_COUNT} (commits $((START+1))-$END)...${NC}"

        if git push $REMOTE_NAME $TARGET_COMMIT:refs/heads/$BRANCH_NAME; then
            echo -e "${GREEN}Batch $((i+1)) successfully pushed${NC}"
        else
            echo -e "${RED}Batch $((i+1)) push failed${NC}"
            echo -e "${YELLOW}Trying alternative push method...${NC}"

            # If above method fails, try another batch push method
            if [ $i -eq 0 ]; then
                # First batch, create new branch
                git push $REMOTE_NAME $BRANCH_NAME
            else
                # Get the end commit of previous batch
                PREV_END=$(($START - 1))
                PREV_COMMIT=${COMMITS[$PREV_END]}
                CURR_COMMIT=${COMMITS[$BATCH_END_INDEX]}

                echo -e "${BLUE}Using range push $PREV_COMMIT..$CURR_COMMIT${NC}"
                git push $REMOTE_NAME $PREV_COMMIT:refs/heads/$BRANCH_NAME $CURR_COMMIT:refs/heads/$BRANCH_NAME
            fi
        fi

        echo
    done
fi

echo -e "${GREEN}Batch upload process completed!${NC}"

The usage method is simple.

chmod +x git_batch_push.sh

Then run the script in the current folder.

Final Results

This article focuses on practical implementation with a brief introduction to the underlying principles.

Throughout this article, I'll first cover the basic principles, then walk through three main sections: how to create GPG keys, how to manage your keys (key management commands), and hands-on practice: using self-created GPG keys to decrypt transmitted encrypted files.

Principle Overview

The principle behind using GPG for encrypting and decrypting documents is straightforward, as described in the documentation:

The procedure for encrypting and decrypting documents is straightforward with this mental model. If you want to encrypt a message to Alice, you encrypt it using Alice's public key, and she decrypts it with her private key. If Alice wants to send you a message, she encrypts it using your public key, and you decrypt it with your key.

The entire process is: I encrypt the file I want to send you with my private key. Then you use my public key to decrypt the file I want to send you, which has been encrypted by me.

This is the basic process.

This article provides a quick overview of how GPG encrypts document content.

This article is divided into two parts, starting with creating a usable GPG key.

How to Create GPG Keys

[Se] gpg --list-keys
gpg: checking the trustdb
gpg: no ultimately trusted keys found

First, let's list whether my machine currently has any GPG keys. As we can see, there are none. This mainly refers to whether there are GPG public keys. Let's also check for private keys.

[Se] gpg --list-secret-keys
[Se]

Nothing there either.

One Command to Rule Them All

Let's start creating.

The detailed steps begin with gpg --full-generate-key, as follows:

[Se] gpg --full-generate-key
gpg (GnuPG) 2.4.0; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
   (9) ECC (sign and encrypt) *default*
  (10) ECC (sign only)
  (14) Existing key from card
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (3072) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y

GnuPG needs to construct a user ID to identify your key.

Real name: Chenwei Jiang
Email address: cheverjonathan@gmail.com
Comment: Used for learning
You selected this USER-ID:
    "Chenwei Jiang (Used for learning) <cheverjonathan@gmail.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

gpg: revocation certificate stored as '/home/cheverjohn/.gnupg/openpgp-revocs.d/D1DED33E8FE9CA4B315A488968CC99424BF9EF81.rev'
public and secret key created and signed.

pub   rsa4096 2023-09-19 [SC]
      D1DED33E8FE9CA4B315A488968CC99424BF9EF81
uid                      Chenwei Jiang (Used for learning) <cheverjonathan@gmail.com>
sub   rsa4096 2023-09-19 [E]

Detailed Stage-by-Stage Command Explanation

Let me break down what happens at each stage.

Stage 1: Select Encryption and Signing Algorithm

In the first stage, you need to select the encryption algorithm. I chose option 1, which means both encryption and signing use the RSA algorithm.

[Se] gpg --full-generate-key
gpg (GnuPG) 2.4.0; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
   (9) ECC (sign and encrypt) *default*
  (10) ECC (sign only)
  (14) Existing key from card
Your selection? 1

Oh, and there's also a copyright notice.

Stage 2: Select Key Length

Longer keys are more secure. I chose 4096 here.

RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (3072) 4096
Requested keysize is 4096 bits

Stage 3: Set Key Validity Period

Set the validity period.

Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y

For this demonstration, I configured Key does not expire at all.

Stage 4: Personal Information

This section will ultimately be used to generate your user ID.

GnuPG needs to construct a user ID to identify your key.

Real name: Chenwei Jiang
Email address: cheverjonathan@gmail.com
Comment: Used for learning
You selected this USER-ID:
    "Chenwei Jiang (Used for learning) <cheverjonathan@gmail.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O

Here I set my Real name to "Chenwei Jiang", my Email address to "cheverjonathan@gmail.com", and added a comment. These are some basic information I commonly use, primarily for demonstration purposes.

The result is a USER-ID: "Chenwei Jiang (Used for learning) cheverjonathan@gmail.com".

How to Manage Your Keys (Key Management Commands)

This section covers how to manage multiple keys on a host machine.

Listing Keys

There are two types of key listings: public keys and private keys.

[Se] gpg --list-keys
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
/home/cheverjohn/.gnupg/pubring.kbx
-----------------------------------
pub   rsa4096 2023-09-19 [SC]
      D1DED33E8FE9CA4B315A488968CC99424BF9EF81
uid           [ultimate] Chenwei Jiang (Used for learning) <cheverjonathan@gmail.com>
sub   rsa4096 2023-09-19 [E]

The above command shows the details.

The key section:

/home/cheverjohn/.gnupg/pubring.kbx
-----------------------------------
pub   rsa4096 2023-09-19 [SC]
      D1DED33E8FE9CA4B315A488968CC99424BF9EF81
uid           [ultimate] Chenwei Jiang (Used for learning) <cheverjonathan@gmail.com>
sub   rsa4096 2023-09-19 [E]

The first line shows the public key filename.

Exporting Keys

The public key file is located at ~/.gnupg/pubring.kbx and stored in binary format. Using the armor parameter converts it to ASCII display.

Using commands to export as public-key.txt and private-key.txt:

[gpg] pwd
/home/cheverjohn/Se/gpg
[gpg] ls
[gpg] ls -la
total 0
drwxr-xr-x. 1 cheverjohn cheverjohn  0 Sep 19 23:36 .
drwxr-xr-x. 1 cheverjohn cheverjohn 34 Sep 19 23:36 ..
[gpg] gpg --armor --output public-key.txt --export 'Chenwei Jiang (used for learning) <cheverjonathan@gmail.com>'
[gpg] ls
public-key.txt
[gpg] cat public-key.txt
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBGUJt+QBEADAzoLPAdK8GfJ/5Ouxh2rOrMsClMmoOznMm2GOBcqSaQsdmP4G
................................................................
oM3YFFujtMxK/cQ/KkbmwAtlMkWx5x8RT/dJ
=NMtR
-----END PGP PUBLIC KEY BLOCK-----
[gpg]

As shown above, this demonstrates the steps to export public-key.txt. The detailed command is:

gpg --armor --output public-key.txt --export 'Chenwei Jiang (used for learning) <cheverjonathan@gmail.com>'

Below are the steps to export private-key.txt:

[gpg] ls
public-key.txt
[gpg] gpg --armor --output private-key.txt --export-secret-keys
[gpg] ls
private-key.txt  public-key.txt
[gpg]

If you previously set a password, this operation requires the password. The command requiring password is:

gpg --armor --output private-key.txt --export-secret-keys

Uploading Public Keys

Public key servers are network servers specifically for storing user public keys. The --send-keys subcommand can achieve this.

Public key servers have no verification mechanism - anyone can upload public keys in your name, so there's no guarantee of reliability for public keys on servers.

Generally, we publish a public key fingerprint on our own website for others to verify that the downloaded public key is authentic. The --fingerprint subcommand can generate a public key fingerprint.

Hands-On Practice / Practice Makes Perfect

In this section, I'll demonstrate exporting public keys, encrypting with private keys, and decrypting with public keys on another device.

I'll show how to encrypt a file containing "hello world" and then decrypt it remotely.

The steps are divided into encryption and decryption.

Detailed steps:

1. Create file with the following commands:

   [gpg] touch demo.txt
   echo "hello world" > demo.txt
   [gpg] ls
   demo.txt  private-key.txt  public-key.txt
   [gpg] cat demo.txt
   hello world
   [gpg]
   

   The file creation commands are:

   touch demo.txt
   echo "hello world" > demo.txt
   

   Now we have a demo.txt file containing "hello world".

2. Encrypt the file:

   [gpg] ls
   demo.txt  private-key.txt  public-key.txt
   [gpg] gpg --recipient 'cheverjonathan@gmail.com' --output demo.gpg --encrypt demo.txt
   [gpg] ls
   demo.gpg  demo.txt  private-key.txt  public-key.txt
   [gpg] cat demo.gpg
   �
   ..............
   �&5p�.�)I�槹iQ%
   [gpg]
   

   Encryption command:

   gpg --recipient 'cheverjonathan@gmail.com' --output demo.gpg --encrypt demo.txt
   

   You can see that demo.gpg is the encrypted file.

3. Decrypt the file:

   [gpg] gpg --output demo --decrypt demo.gpg
   gpg: encrypted with rsa4096 key, ID 13C117D5FEC4F051, created 2023-09-19
         "Chenwei Jiang (Used for learning) <cheverjonathan@gmail.com>"
   [gpg] ls
   demo  demo.gpg  demo.txt  private-key.txt  public-key.txt
   [gpg] cat demo
   hello world
   [gpg]
   

   Decryption command:

   gpg --output demo --decrypt demo.gpg
   

   After entering this command, you need to input the password set earlier, then you can get the file.

   The above demonstrates the process of encrypting files using local keys and decrypting them locally.

   Next, let's begin...

   Obtaining Files on Other Computers, Encrypting Them, Then Having the Original Host Decrypt Using Private Keys

   First, we transfer public-key.txt to another host. You can use GitHub's functionality to upload/download files. For convenience, I'll upload the following file tree from the above process:

   cheverjohn@Dell-G33579 git:(doc/test-gpg*)% tree ~/workspace/Opensource/github.com/Chever-John/cheverjohn.me/docs/wait-for-publish/gpg
   .
   ├── demo
   ├── demo.gpg
   ├── demo.txt
   ├── private-key.txt
   └── public-key.txt
   

   After removing private-key.txt, I'll upload the files to GitHub.

   You can see I've downloaded the public-key.txt file on another host, as shown in the file tree:

   cheverjohn:wait-for-publish/ git:(doc/test-gpg*)$ tree gpg                                                                   [0:54:06]
   gpg
   ├── demo
   ├── demo.gpg
   ├── demo.txt
   └── public-key.txt
   
   1 directory, 4 files
   

   Next, I need to use this public-key.txt to encrypt a file, then transfer this file back to the host with the private key for decryption.

   First, we need to import this public-key.txt locally:

   gpg --import public-key.txt 
   

   Command execution result:

   cheverjohn:gpg/ git:(doc/test-gpg*)$ gpg --import public-key.txt                                                             [1:20:11]
   gpg: key 3BE465D20064251C: public key "Chenwei Jiang (Learning second) <cheverjonathan@gmail.com>" imported
   gpg: Total number processed: 1
   gpg:               imported: 1
   

   Now we need to use this public key to encrypt text.

   First, let's create a file hello-no-encrypt.txt:

   cheverjohn:gpg/ git:(doc/test-gpg*)$ touch hello-no-encrypt.txt                                                              [1:16:07]
   cheverjohn:gpg/ git:(doc/test-gpg*)$ nvim hello-no-encrypt.txt                                                               [1:16:15]
   cheverjohn:gpg/ git:(doc/test-gpg*)$ cat hello-no-encrypt.txt                                                                [1:16:24]
   Hello Sasa!
   cheverjohn:gpg/ git:(doc/test-gpg*)$ tree                                                                                    [1:16:28]
   .
   ├── demo
   ├── demo.gpg
   ├── demo.txt
   ├── hello-no-encrypt.txt
   └── public-key.txt
   
   1 directory, 5 files
   

   We need to encrypt this file and then send it to the host with the private key for decryption.

   The entire encryption process:

   cheverjohn:gpg/ git:(doc/test-gpg*)$ gpg --output hello-encrypted.gpg --encrypt --recipient 'Chenwei Jiang (Learning second) <cheverjonathan@gmail.com>' hello-no-encrypt.txt
   gpg: 617C5542800731C1: There is no assurance this key belongs to the named user
   
   sub  rsa4096/617C5542800731C1 2023-09-21 Chenwei Jiang (Learning second) <cheverjonathan@gmail.com>
    Primary key fingerprint: 5588 D37D AF51 50FD 9186  47E7 3BE4 65D2 0064 251C
         Subkey fingerprint: A038 0DA1 D481 62C7 517C  D6C6 617C 5542 8007 31C1
   
   It is NOT certain that the key belongs to the person named
   in the user ID.  If you *really* know what you are doing,
   you may answer the next question with yes.
   
   Use this key anyway? (y/N) y
   

   Encryption command:

   gpg --output hello-encrypted.gpg --encrypt --recipient 'Chenwei Jiang (Learning second) <cheverjonathan@gmail.com>' hello-no-encrypt.txt
   

   Then we need to upload the encrypted file hello-encrypted.gpg to the machine with the private key.

   Again, I'll use GitHub as the file transfer tool.

   You can see I've obtained the file on the host machine (the machine with the keys) and started decryption:

   [gpg] tree                                                                                                         git:(doc/test-gpg)
   .
   ├── demo
   ├── demo.gpg
   ├── demo.txt
   ├── hello-encrypted.gpg
   ├── hello-no-encrypt.txt
   └── public-key.txt
   
   1 directory, 6 files
   

   Next, we need to decrypt hello-encrypted.gpg:

   [gpg] gpg --output hello-decrypted.txt --decrypt hello-encrypted.gpg                                               git:(doc/test-gpg)
   gpg: encrypted with rsa4096 key, ID 617C5542800731C1, created 2023-09-21
         "Chenwei Jiang (Learning second) <cheverjonathan@gmail.com>"
   [gpg] ls                                                                                                        git:(doc/test-gpg*)  ✱
   demo  demo.gpg  demo.txt  hello-decrypted.txt  hello-encrypted.gpg  hello-no-encrypt.txt  public-key.txt
   [gpg] cat hello-decrypted.txt                                                                                   git:(doc/test-gpg*)  ✱
   Hello Sasa!
   

   Decryption command:

   gpg --output hello-decrypted.txt --decrypt hello-encrypted.gpg
   

Related Articles

• gnupg.org Official Manual

Hello!

I am Chever John, a software engineer.

Just as title says.

Background Description

While wandering around the community, I found a user confused about "how to use multilingual plugins in APISIX Ingress environment". I happen to be a user of go-plugin-runner and have a little knowledge of the APISIX Ingress project, so this document was born.

Proposal Description

Based on version 0.3 of the go-plugin-runner plugin and version 1.4.0 of APISIX Ingress, this article goes through building the cluster, building the image, customizing the helm chart package, and finally, deploying the resources. It is guaranteed that the final result can be derived in full based on this documentation.

go-plugin-runner: 0.3
APISIX Ingress: 1.4.0

kind: kind v0.12.0 go1.17.8 linux/amd64
kubectl version: Client Version: v1.23.5/Server Version: v1.23.4
golang: go1.18 linux/amd64

Begin

Build a cluster environment

Select kind to build a local cluster environment. The command is as follows:

cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
EOF

Build the go-plugin-runner executable

If you have finished writing the plugin, you can start compiling the executable to run with APISIX.

This article recommends two packaging build options.

1. put the packaging process into the Dockerfile and finish the compilation process when you build the docker image later.
2. You can also follow the scheme used in this document, building the executable first and then copying the packaged executable to the image.

How you choose the option should depend on your local hardware considerations. The reason for selecting the second option here is that I want to rely on my powerful local hardware to increase the building speed and speed up the process.

Go to the go-plugin-runner directory

Choose a folder address /home/chever/api7/cloud_native/tasks/plugin-runner and place our apisix-go-plugin-runner project in this folder.

After successful placement, the file tree is shown below:

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner$ tree -L 1
.
└── apisix-go-plugin-runner

1 directory, 0 files

Then you need to go to the apisix-go-plugin-runner/cmd/go-runner/plugins directory and write the plugins you need in that directory. This article will use the default plugin say for demonstration purposes.

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner/apisix-go-plugin-runner$ tree cmd
cmd
└── go-runner
    ├── main.go
    ├── main_test.go
    ├── plugins
    │   ├── fault_injection.go
    │   ├── fault_injection_test.go
    │   ├── limit_req.go
    │   ├── limit_req_test.go
    │   ├── say.go
    │   └── say_test.go
    └── version.go
    
2 directories, 10 files

After writing the plugins, start compiling the executable formally, and note here that you should build static executables, not dynamic ones.

The package compile command is as follows.

CGO_ENABLED=0 go build -a -ldflags '-extldflags "-static"' .

This successfully packages a statically compiled go-runner executable.

In the apisix-go-plugin-runner/cmd/go-runner/ directory, you can see that the current file tree looks like this:

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner/apisix-go-plugin-runner/cmd/go-runner$ tree -L 1
.
├── go-runner
├── main.go
├── main_test.go
├── plugins
└── version.go

1 directory, 4 files

Please remember the path apisix-go-plugin-runner/cmd/go-runner/go-runner, we will use it later.

Build Docker Image

The image is built here in preparation for installing APISIX later using helm.

Write Dockerfile

Return to the path /home/chever/api7/cloud_native/tasks/plugin-runner and create a Dockerfile in that directory, a demonstration of which is given here.

ARG ENABLE_PROXY=false

# Build Apache APISIX
FROM api7/apisix-base:1.19.9.1.5

ADD ./apisix-go-plugin-runner /usr/local/apisix-go-plugin-runner

ARG APISIX_VERSION=2.13.1
LABEL apisix_version="${APISIX_VERSION}"

ARG ENABLE_PROXY
RUN set -x \
    && (test "${ENABLE_PROXY}" != "true" || /bin/sed -i 's,http://dl-cdn.alpinelinux.org,https://mirrors.aliyun.com,g' /etc/apk/repositories) \
    && apk add --no-cache --virtual .builddeps \
        build-base \
        automake \
        autoconf \
        make \
        libtool \
        pkgconfig \
        cmake \
        unzip \
        curl \
        openssl \
        git \
        openldap-dev \
    && luarocks install https://github.com/apache/apisix/raw/master/rockspec/apisix-${APISIX_VERSION}-0.rockspec --tree=/usr/local/apisix/deps PCRE_DIR=/usr/local/openresty/pcre \
    && cp -v /usr/local/apisix/deps/lib/luarocks/rocks-5.1/apisix/${APISIX_VERSION}-0/bin/apisix /usr/bin/ \
    && (function ver_lt { [ "$1" = "$2" ] && return 1 || [ "$1" = "`echo -e "$1\n$2" | sort -V | head -n1`" ]; };  if [ "$APISIX_VERSION" = "master" ] || ver_lt 2.2.0 $APISIX_VERSION; then echo 'use shell ';else bin='#! /usr/local/openresty/luajit/bin/luajit\npackage.path = "/usr/local/apisix/?.lua;" .. package.path'; sed -i "1s@.*@$bin@" /usr/bin/apisix ; fi;) \
    && mv /usr/local/apisix/deps/share/lua/5.1/apisix /usr/local/apisix \
    && apk del .builddeps \
    && apk add --no-cache \
        bash \
        curl \
        libstdc++ \
        openldap \
        tzdata \
    # forward request and error logs to docker log collector
    && ln -sf /dev/stdout /usr/local/apisix/logs/access.log \
    && ln -sf /dev/stderr /usr/local/apisix/logs/error.log

WORKDIR /usr/local/apisix

ENV PATH=$PATH:/usr/local/openresty/luajit/bin:/usr/local/openresty/nginx/sbin:/usr/local/openresty/bin

EXPOSE 9080 9443

CMD ["sh", "-c", "/usr/bin/apisix init && /usr/bin/apisix init_etcd && /usr/local/openresty/bin/openresty -p /usr/local/apisix -g 'daemon off;'"]

STOPSIGNAL SIGQUIT

这份 Dockerfile 配置文档，来源于这个链接。我做的唯一修改如下：

ARG ENABLE_PROXY=false

# Build Apache APISIX
FROM api7/apisix-base:1.19.9.1.5

ADD ./apisix-go-plugin-runner /usr/local/apisix-go-plugin-runner

ARG APISIX_VERSION=2.13.1
LABEL apisix_version="${APISIX_VERSION}"

ARG ENABLE_PROXY

Package all the /apisix-go-plugin-runner files in the /home/chever/api7/cloud_native/tasks/plugin-runner directory into a Docker image. Note down the location of the executable apisix-go-plugin-runner/cmd/go-runner/go-runner and the location of the /usr/local/apisix-go-plugin-runner directory in the Dockerfile above to get the final location of the executable in the Docker image is located as follows.

/usr/local/apisix-go-plugin-runner/cmd/go-runner/go-runner

Please make a note of this address. We will use it in the rest of the configuration.

Begin to build Docker Image

Start building a Docker image based on the Dockerfile. The command is executed in the /home/chever/api7/cloud_native/tasks/plugin-runner directory. The command is as follows:

docker build -t apisix/forrunner:0.1 .

Command Explanation: Build an image with the name apisix/forrunner and mark it as version 0.1.

Load the image to the cluster environment

kind  load docker-image apisix/forrunner:0.1 

Load the image into the kind cluster environment to pull the custom local image for installation during the helm installation.

Install APISIX Ingress

Customize the helm chart

This section focuses on modifying the values.yaml file in the official helm package so that it can install locally packaged images and run the go-plugin-runner executable properly.

Fetch Official helm chart

First, fetch the latest apisix helm chart package with the following command:

helm fetch apisix/apisix

The file tree is as follows:

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner$ tree -L 1
.
├── apisix-0.9.1.tgz
└── apisix-go-plugin-runner

1 directory, 1 file

Unzip

Unzip the apisix-0.9.1.tgz file and prepare to rewrite the configuration. The unzip command is as follows.

tar zxvf apisix-0.9.1.tgz

The file tree is as follows:

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner$ tree -L 1
.
├── apisix
├── apisix-0.9.1.tgz
└── apisix-go-plugin-runner

2 directories, 1 file

Change values.yaml

Go to the apisix folder and modify the values.yaml file. The two changes are as follows:

image:
  repository: apisix/forrunner
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: 0.1

The first change sets the image for the helm installation to be a locally packaged image of your own.

extPlugin:
  enabled: true
  cmd: ["/usr/local/apisix-go-plugin-runner/cmd/go-runner/go-runner", "run"]

The second change sets the position of go-runner in the container after running the container.

Compress the modified helm chart

Once configured, compress the apisix file. The compression command is as follows:

tar zcvf apisix.tgz apisix/

The compressed file is obtained, at which point the file tree is as follows:

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner$ tree -L 1
.
├── apisix
├── apisix-0.9.1.tgz
├── apisix-go-plugin-runner
└── apisix.tgz

2 directories, 2 files

Execute the helm install command

Create namespace

Before installation, create namespaces with the following command:

kubectl create ns ingress-apisix

Then install APISIX using helm with the following command:

helm install apisix ./apisix.tgz --set gateway.type=NodePort --set ingress-controller.enabled=true --namespace ingress-apisix --set ingress-controller.config.apisix.serviceNamespace=ingress-apisix

Create httpbin service and ApisixRoute resources

Create an httpbin backend resource to run with the deployed ApisixRoute resource to test that the functionality is working correctly.

Create httpbin service

Create an httpbin service with the following command:

kubectl run httpbin --image kennethreitz/httpbin --port 80

Expose the port with the following command:

kubectl expose pod httpbin --port 80

Create ApisixRoute Resource

Create the go-plugin-runner-route.yaml file to enable the ApisixRoute resource, with the following configuration file:

apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
  name: plugin-runner-demo
spec:
  http:
  - name: rule1
    match:
      hosts:
      - local.httpbin.org
      paths:
      - /get
    backends:
    - serviceName: httpbin
      servicePort: 80
    plugins:
    - name: ext-plugin-pre-req
      enable: true
      config:
        conf:
        - name: "say"
          value: "{\"body\": \"hello\"}"

The create resource command is as follows:

kubectl apply -f go-plugin-runner-route.yaml

Test

The command is as follows to test if the plugin written in Golang is working correctly:

kubectl exec -it -n ${namespace of Apache APISIX} ${Pod name of Apache APISIX} -- curl http://127.0.0.1:9080/get -H 'Host: local.httpbin.org'

Here I derived from the kubectl get pods --all-namespaces command that the ${namespace of Apache APISIX} and ${Pod name of Apache APISIX} parameters here are ingress-apisix and apisix- 55d476c64-s5lzw, execute the command as follows:

kubectl exec -it -n ingress-apisix apisix-55d476c64-s5lzw -- curl http://127.0.0.1:9080/get -H 'Host: local.httpbin.org'

The expected response obtained is:

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner$ kubectl exec -it -n ingress-apisix apisix-55d476c64-s5lzw -- curl http://127.0.0.1:9080/get -H 'Host: local.httpbin.org'
Defaulted container "apisix" out of: apisix, wait-etcd (init)
hello