TL;DR

• Symptom: https://traefik.example.local/dashboard/ shows Not secure in Chrome/Safari; Firefox works fine.
• Root cause: macOS Trust validator rejects wildcard cert *.example.local for subdomain traefik.example.local (OpenSSL accepts it, Apple Trust does not).
• Fix: Issue explicit hostname certificate for traefik.example.local instead of betting on wildcard.
• In one sentence: Validator rules determine everything, not "the cert is broken."

This was not "the certificate is broken." It was "the certificate is validated by a different engine." OpenSSL said OK. Apple Trust said no. Chrome/Safari follow Apple Trust.

This post has two parts: TLS basics + the real troubleshooting chain.

TLS basics (only the essentials)

Let’s keep the core pieces:

• Certificate chain: leaf → intermediate (optional) → root CA. Browsers trust roots, not leaves.
• SAN (Subject Alternative Name): hostname matching is done here; CN is legacy.
• Trust store: macOS uses System Keychain; Firefox has its own store; Chrome/Safari use system trust.
• Validation: the same cert can be accepted by one validator and rejected by another.

One sentence: “Valid” does not mean “accepted by your current validator.”

Problem Symptoms and Impact Scope

Accessing https://traefik.example.local/dashboard/#/ showed Not secure in Chrome/Safari.

Impact scope:

• ❌ Chrome (relies on macOS System Keychain)
• ❌ Safari (relies on macOS System Keychain)
• ✅ Firefox (uses its own cert store, unaffected)

Reproduction prerequisites:

• macOS version: Sonoma 14.x+ (other versions may behave differently)
• CA already installed in macOS Keychain and set to "Always Trust"
• Certificate issued by mkcert (version 1.4.4)
• Service running on 10.0.0.100

I needed three answers:

1. Does DNS point to the right server?
2. Is the cert chain complete and SAN correct?
3. Does the system trust engine accept the hostname?

Hypothesis Checklist (in elimination order)

Before troubleshooting, list possible root causes:

• H1: CA not properly installed in system trust store

  • Validation: Check macOS Keychain, confirm CA cert exists and is "Always Trust"
  • Expected: If CA missing, all browsers would fail (but Firefox works, so low probability)

• H2: Incomplete certificate chain (missing intermediate cert)

  • Validation: openssl s_client -showcerts to check chain depth
  • Expected: Self-signed CA usually has only two levels (leaf + root), mkcert default is complete

• H3: SAN does not include target domain

  • Validation: openssl x509 -text to check Subject Alternative Name field
  • Expected: If SAN missing or mismatched, OpenSSL would also fail (but curl passed, so low probability)

• H4: Validator rule differences (OpenSSL vs Apple Trust)

  • Validation: Use security verify-cert to simulate macOS system validation
  • Expected: This is the most likely root cause — different validators have different hostname matching rules for wildcards

Validate each hypothesis in order.

Validation Flow Visualization

This diagram shows the 4-layer validation progression (no execution, conceptual only):

#!/usr/bin/env bash
# Validation flow: 4-layer check to identify which layer rejects

cat <<'EOF'

┌─────────────────────────────────────────────────────────┐
│  Layer 1: DNS Resolution                                │
│  dig traefik.example.local +short                       │
│  ✅ Result: 10.0.0.100                                   │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│  Layer 2: Certificate Content (SAN)                     │
│  openssl s_client + grep "Subject Alternative Name"     │
│  ✅ Result: DNS:*.example.local, DNS:example.local      │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│  Layer 3: OpenSSL Validator (Linux/curl)                │
│  curl -Iv https://traefik.example.local/                │
│  ✅ Result: SSL certificate verify ok                   │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│  Layer 4: Apple Trust Validator (macOS)                 │
│  security verify-cert -n traefik.example.local          │
│  ❌ Result: Host name mismatch                          │
└─────────────────────────────────────────────────────────┘

Key finding: Layer 3 ✅ but Layer 4 ❌
→ Different validators have different hostname matching rules for wildcards
→ Chrome/Safari rely on Apple Trust (Layer 4)

EOF

The troubleshooting chain (in order)

1) DNS check

dig traefik.example.local +short

Expected output:

10.0.0.100

Observation: DNS points correctly, network layer is fine.

Conclusion: Network connectivity ruled out, proceed to certificate layer.

2) Inspect the cert chain and SAN

openssl s_client -connect traefik.example.local:443 -servername traefik.example.local -showcerts 2>/dev/null | openssl x509 -text -noout | grep -A2 "Subject Alternative Name"

Expected output (key part):

X509v3 Subject Alternative Name:
    DNS:*.example.local, DNS:example.local

Observation: SAN contains wildcard *.example.local, which should theoretically match traefik.example.local per RFC 6125.

Conclusion: Certificate fields are correct, H3 (SAN missing) eliminated.

3) OpenSSL validation (passes)

curl -Iv https://traefik.example.local/ 2>&1 | grep -E "SSL certificate verify|subject:"

Expected output:

* SSL certificate verify ok.
* subject: CN=*.example.local

Observation: OpenSSL validator considers cert valid, hostname matching passed.

Conclusion: OpenSSL finds no issues, H2 (incomplete chain) eliminated.

4) macOS system validation (fails)

# Export cert to temp file first (if not already done)
echo | openssl s_client -connect traefik.example.local:443 -servername traefik.example.local 2>/dev/null | \
  openssl x509 > /tmp/traefik-leaf.pem

# macOS system-level validation
security verify-cert -c /tmp/traefik-leaf.pem -p ssl -n traefik.example.local -L -v

Expected output (key error):

...certificate verification failed
...Host name mismatch

Observation: Apple Trust explicitly rejects this wildcard match for traefik.example.local.

Conclusion: Apple Trust considers this SAN unable to match the target domain. This is the pivotal point: OpenSSL OK, Apple Trust not OK. Chrome/Safari rely on Apple Trust. H4 (validator rule differences) confirmed.

Root cause

Not a missing CA, not a broken chain. The real cause:

macOS hostname validation did not accept the wildcard cert for traefik.example.local.

This is not theory; it is the system validator's explicit result.

Minimal, reliable fix

Stop relying on the wildcard. Issue an explicit hostname certificate.

mkcert -cert-file traefik.example.local.pem -key-file traefik.example.local-key.pem "traefik.example.local"

If you want to keep wildcard + apex too:

mkcert -cert-file traefik.example.local.pem -key-file traefik.example.local-key.pem \
  "traefik.example.local" "*.example.local" "example.local"

Then replace the Traefik TLS cert and restart the service.

A reusable checklist

• DNS resolution is correct (dig)
• SAN includes the target hostname (openssl x509 -text)
• OpenSSL validation (curl -Iv)
• macOS validation (security verify-cert)
• Browser trust store alignment

Retrospective and Prevention

One-Sentence Lesson

The most valuable evidence: The Host name mismatch output from security verify-cert — it directly proved the issue was not "CA trust" but "hostname validation rules."

If you remember one thing: Don't judge cert correctness by "it opens on my machine." Different validators (OpenSSL / Apple Trust / Firefox NSS) can reach different conclusions on the same cert.

Prevention Checklist (for next time you issue certs)

For internal self-signed CA (mkcert/self-built CA):

• Prefer explicit hostnames over betting on wildcard cross-platform compatibility
• If you must use wildcards, pre-validate with security verify-cert on macOS
• Keep issuance command records (with complete SAN list) for easy re-issuance

For public certs (Let's Encrypt, etc.):

• Request both example.com and *.example.com if you need to cover apex and subdomains
• Validate on at least 3 platforms: Linux (OpenSSL) / macOS (Apple Trust) / Windows (Schannel)

Validation template (reusable script):

#!/usr/bin/env bash
# Quick validation of cert behavior across validators

DOMAIN="traefik.example.local"

echo "==> OpenSSL validation"
curl -Iv https://${DOMAIN}/ 2>&1 | grep -E "SSL certificate verify|subject:"

echo "==> macOS validation (must run on macOS)"
# Export cert first
echo | openssl s_client -connect ${DOMAIN}:443 -servername ${DOMAIN} 2>/dev/null | \
  openssl x509 > /tmp/cert-to-verify.pem
# System validation
security verify-cert -c /tmp/cert-to-verify.pem -p ssl -n ${DOMAIN} -L -v

echo "==> Check SAN"
openssl s_client -connect ${DOMAIN}:443 -servername ${DOMAIN} </dev/null 2>/dev/null | \
  openssl x509 -text -noout | grep -A2 "Subject Alternative Name"

Final Conclusion

"I already installed the CA" is not the end of the story. What matters is which validator you are actually using.

This time the fix is simple and clean: issue an explicit cert for the host and the warning disappears.

This post summarizes what worked (and what hurt) when I deployed multiple self-hosted services in this repo (e.g. dockhand, vaultwarden, Checkmate, Miniflux). It focuses on repeatable defaults, pitfalls, and a practical debugging workflow.

Goals and non-negotiables

• Domain-only access: Do not expose host ports for business services. Avoid IP:port bypassing auth or leaking admin panels.
• Single entrypoint: Only open 80/443 externally. Let Traefik handle reverse proxying, TLS, and security headers.
• Shared middleware: Run shared MongoDB / Redis / PostgreSQL once and reuse them across apps to reduce waste.
• Never break userspace: Any change must not break existing online services (routing/port/network/auth conflicts).

Repo layout and my deployment defaults

• One app per directory: e.g. dockhand/, vaultwarden/, Checkmate/, v2/ (Miniflux).
• Each app has a runnable docker-compose.yml: avoid scattered upstream examples under contrib/.
• Shared middleware is a separate project: middlewares/ runs postgres / mongodb / redis.
  • Shared middleware joins internal networks only and exposes no public ports.
  • Apps join shared networks via external networks.
• One Traefik network: all services that Traefik should reach join an external network like traefik_proxy.
• Prefer Docker labels (docker provider): use file provider only for a few non-container targets or special cases.

Standard deployment playbook (the core tricks)

1) Do not publish host ports

For public-facing services:

• Use expose: (container-network visible only) instead of ports: (host visible).
• Route traffic via Traefik using labels.

This immediately reduces:

• Port conflicts (e.g. host 3000 already in use).
• IP:port bypass risks.

2) Traefik labels (domain routing + TLS + security headers)

Typical label set (example):

• traefik.enable=true
• traefik.http.routers.<name>.rule=Host(`example.com`)
• traefik.http.routers.<name>.entrypoints=websecure
• traefik.http.routers.<name>.tls.certresolver=letsencrypt
• traefik.http.routers.<name>.middlewares=security-headers@file
• traefik.http.services.<name>.loadbalancer.server.port=<container_port>

3) Multi-network containers must pin the Traefik network

When a container joins both:

• traefik_proxy (for Traefik)
• infra_backend (for databases/caches)

Traefik may pick the wrong container IP (from a network Traefik cannot reach), resulting in 504 Gateway Timeout.

Fix: add this label to the service:

• traefik.docker.network=traefik_proxy

This was the typical cause for Checkmate: adding it removed the 504 immediately.

Shared middleware (middlewares/)

Design principles

• High cohesion, low coupling: middleware provides network services only.
• Least privilege: no public ports by default; internal networks only.
• Centralized persistence: volumes managed in middlewares/ for easier backup/migration.

Practical notes

• Use a stable shared network name (e.g. infra_backend), and let apps join with external: true.
• Use service names in connection strings (e.g. postgres / mongodb).
• Prefer separate DB/schema/user per app to avoid data pollution.

App-specific pitfalls I hit this time

Dockhand (high-risk admin surface)

• Do not publish host ports, domain + Traefik only.
• Always add auth: protect it with Traefik basicAuth.
• Keep it simple; complexity is how incidents are born.

Vaultwarden (Bitwarden-compatible)

This is where it is easy to “be clever” and break clients:

• Do not put BasicAuth on the entire site: official clients and browser extensions will fail.
• Correct approach: protect /admin only.
  • Client paths like /, /api, /identity must not be blocked by BasicAuth.
• If “the UI loads but login fails”, check:
  • whether Cloudflare is challenging API requests (see below)
  • whether /api is treated as ServerConfig (may require rewriting to /api/config in Traefik)

Checkmate (multi-network + MongoDB)

• Sharing MongoDB + multiple networks often triggers the Traefik wrong-network issue:
  • symptoms: 504
  • fix: traefik.docker.network=traefik_proxy

Miniflux (rss reader)

• Miniflux requires PostgreSQL. No MongoDB/Redis is needed (unless you add RSSHub/Browserless, etc.).
• HEAD / returning 405 can be normal as long as GET works; do not misdiagnose it as downtime.

Cloudflare pitfall: clients/extensions cannot log in

I hit the same issue twice:

• Vaultwarden browser extension: Cloudflare challenge → 403
• Reeder (Google Reader API): Cloudflare challenge → 403

Common signals:

• response headers contain server: cloudflare + cf-mitigated: challenge
• requests never reach Traefik/app logs

Mitigation options (recommended order):

• A: set the subdomain to DNS-only (grey cloud): the most robust, no client-side hacks needed.
• B: keep proxy (orange cloud), but skip challenge for this host/path:
  • at minimum allow /api, /identity, /reader/api/0, /accounts/ClientLogin, etc.

Key takeaway:

• “It works in a browser” does not mean “it works in a client”. Browsers can pass challenges; clients usually cannot.

Debug playbook (fast to slow)

1) Decide whether the request reaches your server

• Cloudflare 403 challenge: request never reaches your server. Fix Cloudflare/WAF first.
• Traefik 404/504: request reaches Traefik, but routing/network/backend is wrong.
• Backend 4xx/5xx: request reaches the app. Inspect app logs/config.

2) Traefik-side debugging

• Check access logs: which router/service handled the request and what status code it returned.
• Multi-network containers: suspect missing traefik.docker.network first.
• Same host defined twice: suspect file-provider vs docker-provider conflicts (delete legacy file routes).

3) App-side debugging

• docker logs <container>: verify startup, listening port, DB connectivity.
• Verify you did not accidentally use ports: and expose/conflict host ports.

Final security checklist (before going public)

• No business ports published on the host (only keep 80/443)
• Admin surfaces must be authenticated (dockhand, vaultwarden /admin, etc.)
• Databases/caches are never exposed to the public internet
• No Traefik routing conflicts (avoid defining the same host in both file-provider and docker-provider)
• Cloudflare must not challenge API paths (clients/extensions/third-party integrations will break)

This is a troubleshooting log template. Facts first, evidence second, conclusions last.

TL;DR

• Symptoms: The DNS service becomes unresponsive; the whole machine stutters/freezes; sometimes it looks like storage I/O issues or a “disk drop”.
• Pattern: Random, usually after 1+ day of continuous running.
• Recovery: Power off and let it cool down for a while, then it works again.
• Current conclusion (probabilistic): ____ (e.g., thermals / power / storage-controller reset).

Impact

• Direct impact: ____ (e.g., LAN-wide DNS timeouts).
• Indirect impact: ____ (services that depend on name resolution).
• Time window: ____ (start time / duration / auto-recovery?).

Environment (reproducible)

Hardware

• CPU: Intel® N100 (4C/4T, 6MB cache, up to 3.4GHz)
• GPU: Intel® UHD Graphics
• RAM: 8GB LPDDR5 (single channel, onboard, 4800MHz)
• Storage: UFS 2.1 256G
• Wireless: Intel AX200/201 (Wi‑Fi 6 / Bluetooth 5.2)
• Ethernet: 2.5G RJ45 ×1 (PoE IEEE 802.3at)
• Ports: USB 3.2 Gen2 Type‑A ×2; USB 3.2 Gen2 Type‑C ×1 (Alt DP/PD); HDMI ×1
• Power: 65W USB‑C Power Delivery adapter

Software

• Actual OS: Linux (current kernel: 6.8.12-5-pve; factory OS: Windows 11 Home)
• Kernel (if Linux): 6.8.12-5-pve
• Bootloader (GRUB): grub2 2.06-13+pmx2
• DNS stack: ____ (dnsmasq / unbound / AdGuard Home / Pi‑hole / other)
• Deployment: ____ (systemd / Docker / other)
• Other services: ____
• Logging policy: ____ (write frequency / rotation / same disk?).

Problem statement (facts only)

• First seen: ____
• Frequency: about every ____ hours/days (random).
• Observable symptoms during failure:
  • DNS: ____ (timeouts / refusal / high latency)
  • System: ____ (SSH/RDP reachable? CPU/memory spike?)
  • Storage: ____ (mount disappears / read-only remount / I/O error / device reset)
  • Network: ____ (packet loss? NIC reset?)
• Recovery: ____ (power off for ____ minutes; does reboot also work: ____)
• Load correlation: ____ (QPS / logging / other tasks).

Immediate mitigation (timeline)

• T+0: ____ (e.g., switch upstream DNS)
• T+5m: ____ (stop DNS or high-IO workloads)
• T+10m: ____ (collect logs and metrics snapshots)
• Result: ____

Evidence model (align everything by timestamp)

• One timeline: failure start time, temperature peak, I/O errors, service timeouts.
• One core question: is DNS dead, or is the storage/system dead and DNS is just the first visible symptom?

Hypotheses (prioritized)

• H1: Thermals / overheating → throttling or protection → instability
  • Evidence: ____
• H2: Power instability (PD/PoE/adapter) → controller reset
  • Evidence: ____
• H3: Storage media/controller issue → I/O stall or device drop
  • Evidence: ____
• H4: Software resource exhaustion (FD/memory/log amplification/IO saturation)
  • Evidence: ____
• H5: Kernel/driver bug (power management, device PM policy)
  • Evidence: ____

Data to capture (during failure)

• Temps/fans: ____ (sensors, /sys/class/thermal/, Windows tools)
• System logs: ____ (journalctl, Windows Event Viewer)
• Storage logs: ____ (I/O error, timeout, reset, read-only remount)
• Resource metrics: ____ (CPU/load/iowait/memory/disk throughput)
• DNS metrics: ____ (QPS, cache hit ratio, upstream latency, error rate)
• Client checks: ____ (dig/nslookup outputs)

Key evidence (3–5 snippets)

[time] ____

[time] ____

[time] ____

Investigation (Attempt → Observation → Conclusion)

Attempt A: ____

• Change: ____
• Observation: ____
• Conclusion: ____

Attempt B: ____

• Change: ____
• Observation: ____
• Conclusion: ____

Attempt C: ____

• Change: ____
• Observation: ____
• Conclusion: ____

Final conclusion (probability, not absolute claims)

• Most likely root cause: ____
• Secondary factors: ____
• Ruled out: ____ (with evidence)

Fix plan (short-term vs long-term)

Short-term mitigation

• ____ (e.g., improve cooling)
• ____ (e.g., cap power / disable aggressive power saving)
• ____ (e.g., reduce disk writes / configure log rotation)

Long-term fix

• ____ (e.g., replace power adapter / replace storage / BIOS update)
• ____ (e.g., airflow redesign / thermal pads)
• ____ (e.g., kernel/OS change)

Validation

• Uptime: ____ days without recurrence
• Temperature: CPU ≤ ____°C; storage ≤ ____°C
• Logs: no I/O errors / resets / read-only remount

Retrospective

• Biggest learning: ____
• What I will do earlier next time: ____ (monitoring, alerting, auto log capture)
• Open questions: ____

Attempt plan: Monitor 192.168.22.18 (Minisforum) via a monitoring LXC (192.168.22.19) on PVE 192.168.22.12

Goal: turn “it feels like overheating / disk drop / DNS died” into an evidence-aligned timeline: temps + kernel/storage errors + real DNS query success/latency + host reachability.

Note: never put passwords into configs or the repo. Any credentials should live in root-only local files on the monitoring host.

Extra pitfall: port 22 is reachable, but ssh still hangs / drops (more common across subnets)

This one is misleading: nc says 22/tcp open, so people assume “network is fine”. But ssh can still stall during handshake and eventually time out (or die with Broken pipe).

Topology (what happened in this incident)

• Client subnet: 192.168.11.0/24
• Target host: 192.168.22.12 (PVE)
• Same-subnet control point: 192.168.22.18 (SSH from here to .12 is stable)
• Correlated signal: DNS server (e.g. 192.168.22.53:53) experienced timeouts — reverse lookups can amplify SSH handshake latency.

Symptoms (typical)

• nc -vz -G 2 192.168.22.12 22 reports succeeded (TCP handshake is OK)
• ssh root@192.168.22.12 hangs and later shows:
  • ssh_dispatch_run_fatal: Connection to 192.168.22.12 port 22: Operation timed out
  • or Read from remote host 192.168.22.12: Operation timed out / client_loop: send disconnect: Broken pipe

Quick triage (split “port is open” into smaller truths)

1. Check whether the SSH banner is returned immediately (protocol only, no auth):

nc -v 192.168.22.12 22

You should quickly see SSH-2.0-OpenSSH_.... If the TCP connect succeeds but the banner is slow/missing, the issue is likely in sshd/system path (DNS reverse lookup stalls, high load, policy/ACL causing unstable data path), not “port blocked”.

2. Capture where SSH stalls (useful evidence):

ssh -vvv -o ConnectTimeout=5 -o ServerAliveInterval=5 -o ServerAliveCountMax=1 root@192.168.22.12

Minimal fix (eliminate DNS/GSSAPI special cases first)

If you can SSH from a same-subnet host (e.g. .22.18) into .22.12, prefer the server-side fix:

1. On .22.12, edit /etc/ssh/sshd_config and make sure you have:

• UseDNS no
• GSSAPIAuthentication no

2. Validate and restart:

sshd -t && systemctl restart ssh
systemctl status ssh --no-pager

Practical rationale: when DNS/reverse lookup is flaky, UseDNS yes can block parts of the handshake/audit path. Disabling it removes an entire class of “it depends on DNS today” failures.

If it is still flaky: continue with the “cross-subnet triad”

• Return path routing (the target must know how to route back to 192.168.11.0/24):

ip route get 192.168.11.12

• Firewall/ACL/state tracking: allow not only 22/tcp inbound but also the correct return direction and conntrack behavior (especially with gateways/NAT/policy routing).
• MTU/fragmentation: tunnels/PPPoE/VLAN paths can yield “SYN works but data is unstable”; validate with DF-sized pings on Linux and temporarily lower MSS/MTU to confirm.

Architecture (minimal, but sufficient)

• 192.168.22.19 (monitoring LXC on PVE 192.168.22.12): Prometheus + Grafana + blackbox_exporter (optional Pushgateway)
• 192.168.22.18 (target host): nodeexporter (host metrics) + (optional) smartctlexporter + custom textfile metrics (kernel/I/O error counter)

Data model:

• Time series (Prometheus): scrape every 15s, keep at least 48h around failures.
• Active probing (blackbox_exporter): probe .18/.54 from .19 to separate “host down” from “service down”.
• Alerting: pin the incident timestamp immediately.

Attempt D: Bring up the monitoring stack (Prometheus + Grafana + blackbox_exporter)

D0. Pre-checks (network & ports)

From 192.168.22.19, verify reachability to 192.168.22.18:

ping 192.168.22.18
Test-NetConnection 192.168.22.18 -Port 22

Ports used (defaults):

• .18:9100 node_exporter
• .18:9633 smartctl_exporter (optional)
• .19:9090 Prometheus
• .19:3000 Grafana
• .19:9115 blackbox_exporter
• .19:9091 Pushgateway (optional)

D1. Deploy node_exporter on 192.168.22.18 (systemd, stable)

Run on .18:

export VER="1.7.0"
curl -fsSL -o /tmp/node_exporter.tar.gz "https://github.com/prometheus/node_exporter/releases/download/v${VER}/node_exporter-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/node_exporter.tar.gz
install -m 0755 /tmp/node_exporter-${VER}.linux-amd64/node_exporter /usr/local/bin/node_exporter

useradd --system --no-create-home --shell /usr/sbin/nologin nodeexp || true
mkdir -p /var/lib/node_exporter/textfile_collector
chown -R nodeexp:nodeexp /var/lib/node_exporter

cat >/etc/systemd/system/node_exporter.service <<'EOF'
[Unit]
Description=Prometheus Node Exporter
After=network-online.target
Wants=network-online.target

[Service]
User=nodeexp
Group=nodeexp
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=:9100 \
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now node_exporter
systemctl status node_exporter --no-pager

Verify (from .12 or any LAN host):

curl -fsS http://192.168.22.18:9100/metrics | head

D2. (Recommended) Add a kernel/storage error counter as a custom metric

The keywords that matter: I/O error|timeout|reset|remount read-only|EXT4-fs error|BTRFS error|blk_update_request

Use node_exporter textfile collector and a timer:

cat >/usr/local/bin/collect_kernel_error_metrics.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

OUT="/var/lib/node_exporter/textfile_collector/kernel_errors.prom"
TMP="$(mktemp)"

COUNT="$(journalctl -k --since "10 min ago" --no-pager 2>/dev/null | \
  grep -Eic 'I/O error|timeout|reset|remount read-only|EXT4-fs error|BTRFS error|blk_update_request|nvme|ufs' || true)"

NOW="$(date +%s)"

{
  echo "# HELP kernel_error_events_10m Number of kernel error-like events in last 10 minutes"
  echo "# TYPE kernel_error_events_10m gauge"
  echo "kernel_error_events_10m ${COUNT}"
  echo "# HELP kernel_error_scrape_time_seconds Last collect time"
  echo "# TYPE kernel_error_scrape_time_seconds gauge"
  echo "kernel_error_scrape_time_seconds ${NOW}"
} >"${TMP}"

mv "${TMP}" "${OUT}"
EOF

chmod +x /usr/local/bin/collect_kernel_error_metrics.sh

cat >/etc/systemd/system/kernel-error-metrics.service <<'EOF'
[Unit]
Description=Collect kernel error metrics for node_exporter textfile collector

[Service]
Type=oneshot
ExecStart=/usr/local/bin/collect_kernel_error_metrics.sh
EOF

cat >/etc/systemd/system/kernel-error-metrics.timer <<'EOF'
[Unit]
Description=Run kernel error metrics collector every 60s

[Timer]
OnBootSec=30s
OnUnitActiveSec=60s
AccuracySec=1s

[Install]
WantedBy=timers.target
EOF

systemctl daemon-reload
systemctl enable --now kernel-error-metrics.timer

Verify:

curl -fsS http://192.168.22.18:9100/metrics | grep -E '^kernel_error_events_10m|^kernel_error_scrape_time_seconds' || true

D3. (Optional) smartctl_exporter for disk health

If the storage device does not expose SMART/health, this may not work. Still worth trying.

apt-get update
apt-get install -y smartmontools

export VER="0.12.0"
curl -fsSL -o /tmp/smartctl_exporter.tar.gz "https://github.com/prometheus-community/smartctl_exporter/releases/download/v${VER}/smartctl_exporter-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/smartctl_exporter.tar.gz
install -m 0755 /tmp/smartctl_exporter-${VER}.linux-amd64/smartctl_exporter /usr/local/bin/smartctl_exporter

useradd --system --no-create-home --shell /usr/sbin/nologin smartctl-exp || true

cat >/etc/systemd/system/smartctl_exporter.service <<'EOF'
[Unit]
Description=Prometheus Smartctl Exporter
After=network-online.target
Wants=network-online.target

[Service]
User=smartctl-exp
Group=smartctl-exp
ExecStart=/usr/local/bin/smartctl_exporter --web.listen-address=:9633
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now smartctl_exporter

D4. Deploy Prometheus + Grafana + blackbox_exporter on 192.168.22.12 (PVE) via LXC (recommended)

Since 192.168.22.12 is also a PVE host, avoid piling Docker onto the host OS. The simplest operationally is: one Debian LXC container running the monitoring stack with systemd.

D4.1 Create a Debian LXC container on PVE (.12)

You can do it in the PVE Web UI, or via CLI like this (replace <CTID>, <GW>; LXC IP is pinned to 192.168.22.19):

# Run on the PVE host 192.168.22.12
pct create <CTID> local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname monitoring \
  --cores 2 --memory 2048 --swap 512 \
  --rootfs local-lvm:8 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.22.19/24,gw=<GW> \
  --unprivileged 1

pct set <CTID> -features keyctl=1,nesting=1
pct start <CTID>

D4.2 Install Prometheus in the container (systemd)

Enter the container:

pct exec <CTID> -- bash

Inside the container:

apt-get update
apt-get install -y ca-certificates curl tar

useradd --system --no-create-home --shell /usr/sbin/nologin prometheus || true
mkdir -p /etc/prometheus /var/lib/prometheus
chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

export VER="2.54.1"
curl -fsSL -o /tmp/prometheus.tar.gz "https://github.com/prometheus/prometheus/releases/download/v${VER}/prometheus-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/prometheus.tar.gz
install -m 0755 /tmp/prometheus-${VER}.linux-amd64/prometheus /usr/local/bin/prometheus
install -m 0755 /tmp/prometheus-${VER}.linux-amd64/promtool /usr/local/bin/promtool
cp -r /tmp/prometheus-${VER}.linux-amd64/consoles /etc/prometheus/
cp -r /tmp/prometheus-${VER}.linux-amd64/console_libraries /etc/prometheus/
chown -R prometheus:prometheus /etc/prometheus

Create /etc/prometheus/prometheus.yml (scrape .18 + active probes; DNS is the AdGuardDNS LXC at .54):

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "minisforum-node"
    static_configs:
      - targets: ["192.168.22.18:9100"]

  - job_name: "minisforum-smartctl"
    static_configs:
      - targets: ["192.168.22.18:9633"]

  - job_name: "blackbox-icmp"
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
          - 192.168.22.18
          - 192.168.22.54
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

  - job_name: "blackbox-tcp"
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
          - 192.168.22.18:22
          - 192.168.22.18:9100
          - 192.168.22.54:53
          - 192.168.22.54:3000
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

Create the systemd service:

cat >/etc/systemd/system/prometheus.service <<'EOF'
[Unit]
Description=Prometheus
After=network-online.target
Wants=network-online.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now prometheus

D4.3 Install blackbox_exporter in the container (systemd)

useradd --system --no-create-home --shell /usr/sbin/nologin blackbox || true
mkdir -p /etc/blackbox_exporter
chown -R blackbox:blackbox /etc/blackbox_exporter

export VER="0.25.0"
curl -fsSL -o /tmp/blackbox.tar.gz "https://github.com/prometheus/blackbox_exporter/releases/download/v${VER}/blackbox_exporter-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/blackbox.tar.gz
install -m 0755 /tmp/blackbox_exporter-${VER}.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter

cat >/etc/blackbox_exporter/blackbox.yml <<'EOF'
modules:
  icmp:
    prober: icmp
    timeout: 5s

  tcp_connect:
    prober: tcp
    timeout: 5s
EOF

chown -R blackbox:blackbox /etc/blackbox_exporter

cat >/etc/systemd/system/blackbox_exporter.service <<'EOF'
[Unit]
Description=Prometheus Blackbox Exporter
After=network-online.target
Wants=network-online.target

[Service]
User=blackbox
Group=blackbox
ExecStart=/usr/local/bin/blackbox_exporter \
  --config.file=/etc/blackbox_exporter/blackbox.yml \
  --web.listen-address=:9115
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now blackbox_exporter

D4.4 Install Grafana in the container (systemd)

Do not commit any Grafana admin password. Install it, then set the password manually.

apt-get install -y apt-transport-https software-properties-common wget gpg
mkdir -p /etc/apt/keyrings
wget -qO- https://apt.grafana.com/gpg.key | gpg --dearmor >/etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" >/etc/apt/sources.list.d/grafana.list
apt-get update
apt-get install -y grafana
systemctl enable --now grafana-server

Reset admin password (run in the container):

grafana-cli admin reset-admin-password

Attempt E: Deploy AdGuardDNS on 192.168.22.18 via LXC (fixed IP: 192.168.22.54) + real DNS query probing

Goal: isolate DNS service from the host and pin the DNS IP to 192.168.22.54 so monitoring targets never drift.

E0. Create the AdGuardDNS LXC container on PVE (.18) with IP 192.168.22.54

Example (replace <CTID>, <GW>):

# Run on the PVE host 192.168.22.18
pct create <CTID> local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname adguarddns \
  --cores 1 --memory 1024 --swap 256 \
  --rootfs local-lvm:4 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.22.54/24,gw=<GW> \
  --unprivileged 1

pct set <CTID> -features keyctl=1,nesting=1
pct start <CTID>

E1. Install AdGuard Home inside the container (as AdGuardDNS)

Enter the container:

pct exec <CTID> -- bash

Inside the container:

apt-get update
apt-get install -y ca-certificates curl tar

export VER="0.107.57"
curl -fsSL -o /tmp/adguardhome.tar.gz "https://github.com/AdguardTeam/AdGuardHome/releases/download/v${VER}/AdGuardHome_linux_amd64.tar.gz"
tar -C /opt -xzf /tmp/adguardhome.tar.gz

/opt/AdGuardHome/AdGuardHome -s install
systemctl status AdGuardHome --no-pager

First-time setup:

• Admin UI: http://192.168.22.54:3000
• DNS: 192.168.22.54:53 (UDP/TCP)

E1. Real DNS query probe using dig + Pushgateway (credential-free)

Use .12 to run dig periodically and push two metrics: success and latency.

1. Optionally install Pushgateway on the monitoring side (the .12 monitoring LXC) via systemd:

apt-get update
apt-get install -y ca-certificates curl tar

useradd --system --no-create-home --shell /usr/sbin/nologin pushgateway || true
mkdir -p /etc/pushgateway /var/lib/pushgateway
chown -R pushgateway:pushgateway /etc/pushgateway /var/lib/pushgateway

export VER="1.10.0"
curl -fsSL -o /tmp/pushgateway.tar.gz "https://github.com/prometheus/pushgateway/releases/download/v${VER}/pushgateway-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/pushgateway.tar.gz
install -m 0755 /tmp/pushgateway-${VER}.linux-amd64/pushgateway /usr/local/bin/pushgateway

cat >/etc/systemd/system/pushgateway.service <<'EOF'
[Unit]
Description=Prometheus Pushgateway
After=network-online.target
Wants=network-online.target

[Service]
User=pushgateway
Group=pushgateway
ExecStart=/usr/local/bin/pushgateway --web.listen-address=:9091
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now pushgateway
systemctl status pushgateway --no-pager

2. Add a Prometheus scrape job (use 127.0.0.1 since Pushgateway runs on the same container):

- job_name: "pushgateway"
  honor_labels: true
  static_configs:
    - targets: ["127.0.0.1:9091"]

3. Create a probe script (DNS IP is pinned to 192.168.22.54):

cat >/opt/monitoring/dns_probe_push.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

DNS_IP="192.168.22.54"
NAME="www.baidu.com"
PUSH_URL="http://127.0.0.1:9091/metrics/job/dns_probe/instance/${DNS_IP}"

OUT="$(dig +tries=1 +time=2 +stats @"${DNS_IP}" "${NAME}" A 2>/dev/null || true)"

if echo "${OUT}" | grep -qE '^;; ANSWER SECTION:'; then
  SUCCESS=1
else
  SUCCESS=0
fi

LAT_MS="$(echo "${OUT}" | awk -F': ' '/^;; Query time:/{print $2}' | awk '{print $1}' || true)"
LAT_MS="${LAT_MS:-0}"

cat <<METRICS | curl -fsS --data-binary @- "${PUSH_URL}" >/dev/null
# TYPE dns_probe_success gauge
dns_probe_success ${SUCCESS}
# TYPE dns_probe_latency_ms gauge
dns_probe_latency_ms ${LAT_MS}
METRICS
EOF

chmod +x /opt/monitoring/dns_probe_push.sh

4. Schedule it (during investigation, 15s is useful):

crontab -e
*/1 * * * * /opt/monitoring/dns_probe_push.sh
*/1 * * * * sleep 15; /opt/monitoring/dns_probe_push.sh
*/1 * * * * sleep 30; /opt/monitoring/dns_probe_push.sh
*/1 * * * * sleep 45; /opt/monitoring/dns_probe_push.sh

E2. (Optional) AdGuard Home internal stats (requires admin API creds; never commit them)

If you need AdGuard-specific stats (blocked/allowed/qps/upstream latency), run an exporter on .12 that talks to AdGuard’s admin API. Store credentials in a root-only local file (e.g. /opt/monitoring/secrets/adguard.env, chmod 600) and keep it out of Git.

This is a practical troubleshooting post. Facts first, evidence second, conclusions last.

TL;DR

• Symptom: Cross-VLAN ping times out. The target host shows echo reply sent, yet the client never receives it.
• Root cause #1 (decisive): PVE intentionally kept OpenWrt (192.168.22.13) as the default gateway for QoS/traffic handling, but had no explicit routes for other internal VLAN subnets, so return traffic to internal networks followed the default route into OpenWrt and never traversed the primary L3 gateway (classic asymmetric path).
• Root cause #2 (amplifier): OpenWrt had masq=1 on the LAN zone, applying SNAT to internal east-west traffic and making paths/sessions unpredictable.
• Fix:
  • Keep PVE default gateway = OpenWrt (192.168.22.13) for QoS, but add explicit routes for internal subnets (e.g., 192.168.11.0/24, 192.168.183.0/24) via the primary gateway (192.168.22.1)
  • Disable masquerade on OpenWrt lan zone
• Prevention: Route internally, NAT only on real WAN egress; if a side gateway must be the default route (QoS/proxy), explicitly split internal routing vs Internet default; use tcpdump + ip route get as a standard SOP.

Background (Topology and Roles)

Multi-VLAN home network:

• Cloud Gateway Fiber (primary L3 gateway)
  • Inter-VLAN routing
  • Example subnets:
    • VLAN11: 192.168.11.0/24 (GW 192.168.11.1)
    • VLAN22 (PVE/Lab): 192.168.22.0/24 (GW 192.168.22.1)
    • VLAN183: 192.168.183.0/24 (GW 192.168.183.1)
• OpenWrt (secondary / side gateway)
  • 192.168.22.13/24 on br-lan
  • fw3 + iptables, plus proxy tooling (OpenClash/Passwall)
• PVE node: 192.168.22.12/24 on vmbr0
• Test hosts:
  • 192.168.183.235 (VLAN183)
  • 192.168.11.29 (VLAN11)

The intended design is simple:

• Inter-VLAN traffic should be routed by the primary gateway.
• NAT should only happen on a true WAN egress, not on internal VLAN-to-VLAN paths.

ASCII topology (printable via shell)

This script does not change any config. It prints the topology and the two key paths (bad vs fixed) so readers can visualize the data flow:

#!/usr/bin/env bash
set -euo pipefail

# Print the topology and the two paths (bad vs fixed).
cat <<'EOF'

                    +------------------------------+
                    |   Cloud Gateway Fiber (L3)   |
                    |   VLAN11: 192.168.11.1       |
                    |   VLAN22: 192.168.22.1       |
                    |   VLAN183: 192.168.183.1     |
                    +---------------+--------------+
                                    |
                                  VLAN22
                                    |
                   +----------------+----------------+
                   |                                 |
        +----------v-----------+          +----------v-----------+
        |  OpenWrt (side GW)   |          |        PVE           |
        |  192.168.22.13       |          |  192.168.22.12       |
        |  QoS / traffic mgmt  |          |  vmbr0               |
        +----------------------+          +----------------------+

Clients:
  VLAN183 host: 192.168.183.235
  VLAN11  host: 192.168.11.29

Problem #1 (before):
  192.168.183.235 -> Fiber -> PVE
  PVE reply -> (default gw) OpenWrt -> [NAT/proxy/unknown] -> ???   (Fiber never sees the reply)

Fix idea:
  Keep PVE default gw = OpenWrt (for QoS),
  but route internal subnets via Fiber.

Problem #2 (amplifier):
  OpenWrt LAN masquerade (SNAT) rewrites east-west traffic,
  making internal routing/session behavior unpredictable.

EOF

Incidents and Impact

Incident #1: VLAN183 could not ping PVE

• Client 192.168.183.235 → ping 192.168.22.12 timed out
• On PVE, captures showed:
  • ICMP echo request arriving
  • ICMP echo reply being generated
• On the primary gateway (Fiber), captures on br22/br0 showed requests only, no replies

Meaning: the reply never traversed the primary gateway.

Incident #2: After fixing #1, VLAN11 still could not ping PVE

• 192.168.11.29 could not ping 192.168.22.12
• But it could ping OpenWrt 192.168.22.13

This pointed to OpenWrt NAT/firewall behavior affecting internal reachability.

Investigation (Evidence-driven)

1) Confirm the primary gateway is fine

On Fiber:

• It could ping both 192.168.183.235 and 192.168.22.12.
• ip route showed both subnets as directly connected.

Conclusion: core L3 routing on Fiber was not broken.

2) Hop-by-hop capture: find where the reply disappears

• On PVE: request/reply both exist → PVE is replying.
• On Fiber (br22/br0): request only → reply never reached Fiber.

Conclusion: the return path is not going through the primary gateway.

3) Let ip route get tell the truth

On PVE:

• ip route showed: default via 192.168.22.13 dev vmbr0
• ip route get 192.168.183.235 showed: via 192.168.22.13

Root cause #1: PVE had no explicit route to internal VLAN subnets, so return traffic to 192.168.183.0/24 naturally followed the default route into OpenWrt. Once that happens, any NAT/proxy/misroute on OpenWrt can prevent the primary gateway from ever seeing the reply.

That creates asymmetric routing:

• Forward: 183 → Fiber → 22 → PVE (OK)
• Return: PVE → OpenWrt → (NAT/proxy/unknown) (uncontrolled)

4) Validate OpenWrt firewall/NAT configuration

After confirming OpenWrt uses fw3 + iptables (no nftables), the key finding was:

• lan zone had masq='1'

Root cause #2: OpenWrt applied SNAT to internal traffic, rewriting sources to 192.168.22.13 and making internal routing/session behavior unstable, especially with proxy redirection.

Fix (Minimal changes)

Fix #1: Keep default gateway = OpenWrt, but route internal subnets via Fiber

Constraint: PVE default route must remain 192.168.22.13 (OpenWrt) for QoS/traffic handling. In that case, do not touch the default route. Instead, explicitly route internal VLAN subnets via the primary L3 gateway (192.168.22.1).

Temporary test (examples for VLAN11/VLAN183; add others as needed):

# Keep default via OpenWrt for QoS / traffic management
# default via 192.168.22.13

# Route internal VLANs via the primary L3 gateway (Fiber)
ip route replace 192.168.183.0/24 via 192.168.22.1 dev vmbr0
ip route replace 192.168.11.0/24  via 192.168.22.1 dev vmbr0

Verification tips:

• ip route get 192.168.183.235 should show via 192.168.22.1
• ICMP replies should be visible on Fiber interfaces (e.g., br22/br0)

Persist it in Proxmox networking (commonly /etc/network/interfaces for vmbr0) with post-up:

post-up ip route replace 192.168.183.0/24 via 192.168.22.1 dev vmbr0
post-up ip route replace 192.168.11.0/24  via 192.168.22.1 dev vmbr0
post-down ip route del 192.168.183.0/24 via 192.168.22.1 dev vmbr0 || true
post-down ip route del 192.168.11.0/24  via 192.168.22.1 dev vmbr0 || true

Verification:

• Fiber sees ICMP replies on br22/br0.
• 192.168.183.235 can ping 192.168.22.12 again.

Fix #2: Disable masquerade on OpenWrt LAN zone

uci set firewall.@zone[0].masq='0'
uci commit firewall
/etc/init.d/firewall restart

Verification:

• 192.168.11.29 can ping 192.168.22.12.
• Internal traffic keeps real source IPs, making debugging and ACLs sane.

Prevention

• Route internally; NAT only on real WAN
  • lan/internal zones: masq=0
  • wan egress: masq=1
• If a side gateway must be the default route (QoS/proxy), split routing explicitly
  • Default route can stay on OpenWrt, but internal subnets must have explicit routes via the primary L3 gateway (or use policy routing by destination).
  • Keep east-west traffic on predictable L3 routing. Do not let it fall into NAT/proxy black boxes.
• Standard SOP (Standard Operating Procedure)
  • Meaning: for “cross-VLAN ping fails”, do not guess. Follow the same evidence-driven checklist every time.
  • Step 1: packet captures answer 3 questions: did the request arrive, did the target reply, and at which hop did the reply disappear?
    • On PVE: tcpdump -ni vmbr0 icmp
    • On the primary gateway (both VLAN interfaces): tcpdump -ni br22 icmp and tcpdump -ni br0 icmp
  • Step 2: verify routing decisions (no imagination)
    • On PVE: ip route + ip route get 192.168.183.235
    • On OpenWrt/Fiber: ip route get 192.168.183.235 (confirm the actual next hop)
  • Step 3: simplify the data path first
    • Restore pure routing with real source IPs and predictable return paths before layering QoS/proxy/policy routing.
• Make proxy behavior explicit
  • Exclude internal subnets from transparent proxy/tunnel redirection.

One-line lesson

If ICMP looks like “ghost packets”, you most likely built a black box with the wrong default gateway and the wrong NAT — not a broken protocol stack.

As AI assistants become increasingly integrated into our daily workflows, understanding how to properly leverage tools like Claude can significantly enhance your productivity and decision-making. This comprehensive guide will help you maximize the value you get from Claude while maintaining good practices and realistic expectations.

Understanding Claude's Core Strengths

Before diving into specific use cases, it's crucial to understand what Claude excels at:

1. Text Analysis and Processing

Claude can quickly analyze, summarize, and extract insights from large volumes of text. Whether you're dealing with research papers, meeting notes, or technical documentation, Claude can help you identify key points and patterns.

2. Creative and Technical Writing

From drafting emails to writing code, Claude can assist with various forms of content creation while maintaining your personal voice and style preferences.

3. Problem-Solving and Research

Claude can help break down complex problems into manageable components and provide structured approaches to finding solutions.

4. Learning and Explanation

Need something explained? Claude can adapt explanations to your level of expertise and learning style.

Essential Best Practices

Start with Clear, Specific Prompts

Good: "Help me write a professional email declining a meeting request for Thursday afternoon, suggesting alternative times next week, and maintaining a collaborative tone."

Poor: "Write an email for me."

The more context and specificity you provide, the better Claude can tailor its response to your needs.

Iterate and Refine

Don't expect perfect results on the first try. Use follow-up prompts to:

• Ask for clarification or expansion on specific points
• Request different approaches or styles
• Provide feedback on what works and what doesn't

Verify Important Information

While Claude is knowledgeable, always verify critical facts, especially for:

• Medical or legal advice
• Financial decisions
• Current events or rapidly changing information
• Technical specifications or requirements

Practical Use Cases for Individuals

Professional Development

Email Communication:

• Draft professional emails with appropriate tone
• Summarize long email threads
• Translate complex technical concepts for non-technical audiences

Document Creation:

• Create structured reports and presentations
• Write job application materials
• Develop project plans and timelines

Learning and Skill Development:

• Get explanations of complex topics
• Practice interview questions
• Learn new programming languages or frameworks

Personal Productivity

Research and Planning:

• Compare options for major purchases
• Plan travel itineraries
• Research topics of personal interest

Creative Projects:

• Brainstorm ideas for hobbies or side projects
• Get feedback on creative writing
• Plan events or celebrations

Daily Life Management:

• Create meal plans and shopping lists
• Organize and prioritize tasks
• Draft important personal correspondence

Advanced Techniques

Context Management

Provide relevant background information at the start of conversations:

• Your role and expertise level
• The specific context or constraints you're working within
• Your goals and success criteria

Chain of Thought Prompting

For complex problems, ask Claude to "think step by step" or "explain your reasoning." This often leads to more thorough and accurate responses.

Role-Playing Scenarios

Ask Claude to take on specific roles or perspectives:

• "Act as a senior software engineer reviewing my code"
• "Respond as someone unfamiliar with this topic would"
• "Take the perspective of a potential customer"

Template Creation

Develop reusable prompts for recurring tasks:

• Meeting summary templates
• Email response frameworks
• Problem-solving methodologies

What Claude Cannot Do

Understanding limitations is crucial for effective use:

Real-Time Information

Claude's training has a knowledge cutoff and cannot access current web information or real-time data.

Personal Data Access

Claude cannot access your personal files, emails, or private information unless you explicitly share it in the conversation.

Execute Actions

Claude cannot directly perform actions in external systems, send emails, or make purchases on your behalf.

Replace Human Judgment

While Claude can provide insights and suggestions, important decisions should always involve human judgment and expertise.

Maintaining Privacy and Security

Information Sharing

• Avoid sharing sensitive personal information (SSNs, passwords, financial details)
• Be cautious with proprietary or confidential business information
• Consider anonymizing data when seeking help with specific scenarios

Data Retention

• Understand the platform's data retention policies
• Don't rely on Claude to remember information between separate conversations
• Keep your own records of important insights or decisions

Building an Effective Workflow

1. Define Your Use Cases

Identify specific, recurring tasks where Claude can add value to your routine.

2. Develop Standard Prompts

Create and refine prompt templates for your most common use cases.

3. Set Realistic Expectations

Understand that Claude is a tool to augment your capabilities, not replace your thinking and judgment.

4. Measure Impact

Track how Claude usage affects your productivity and the quality of your work.

5. Stay Updated

AI capabilities evolve rapidly. Stay informed about new features and best practices.

Common Pitfalls to Avoid

Over-Reliance

Don't become dependent on Claude for basic thinking or decision-making. Use it to enhance your capabilities, not replace them.

Prompt Laziness

Avoid vague or minimal prompts. Investing time in clear communication yields better results.

Ignoring Context

Remember to provide sufficient context for complex or specialized topics.

Assuming Perfection

Always review and validate Claude's outputs, especially for important tasks.

Conclusion

Claude can be a powerful ally in both professional and personal contexts when used thoughtfully and strategically. The key to success lies in understanding its strengths and limitations, developing clear communication patterns, and maintaining appropriate skepticism and verification practices.

Start with simple use cases, build your skills gradually, and always remember that Claude is most effective when it augments your human intelligence rather than attempting to replace it. With practice and the right approach, you'll find Claude becomes an invaluable tool in your personal and professional toolkit.

Remember: the goal isn't to use Claude for everything, but to use it for the right things in the right way.

AI has been moving at a breakneck pace. Many people are getting hooked on barking orders at AI and then discovering piles of error‑ridden code it produced (and in truth, you may not fully understand the code the AI wrote either).

If you do understand AI, sure—you know how to use prompt engineering. Still, I want to reinforce the fundamentals of software engineering.

Object‑oriented programming, design principles, design patterns, coding conventions, and refactoring.

Programming paradigms/styles

The mainstream programming paradigms/styles are:

• Procedural programming
• Object‑oriented programming (OOP)
• Functional programming

Among them, OOP remains the most widely adopted paradigm. Most popular programming languages today are object‑oriented or provide strong OO support.

Design principles

Definition: distilled practical experience for code design.

These principles often sound abstract and are described vaguely.

The hard part: understand the original intent, the specific problems they solve, and the scenarios where they apply.

Common principles include:

• SOLID principles

  You might wonder why SOLID uniquely comes with five additional “sub‑principles.”

  I had the same question, so I looked it up.

  SOLID is called the cornerstone of object‑oriented design because it offers a concrete, actionable toolkit—rather than vague ideas—for solving real problems in software development.

  Each sub‑principle targets a different layer of design pitfalls. It stands out precisely because it is not a single fuzzy rule, but a collection of five specific, operable design principles.

  • Single Responsibility Principle (SRP)
  • Open/Closed Principle (OCP)
  • Liskov Substitution Principle (LSP)
  • Interface Segregation Principle (ISP)
  • Dependency Inversion Principle (DIP)

• KISS principle (Keep It Simple, Stupid)

• DRY principle (Don’t Repeat Yourself)

• YAGNI principle (You Aren’t Gonna Need It)

• Law of Demeter (LOD)

Design patterns

Definition: recurring solutions or design approaches distilled from frequently encountered problems in software development.

Most design patterns are fundamentally about extensibility.

The hard part: knowing which problem each pattern addresses, recognizing typical application scenarios, and—importantly—avoiding overuse.

What’s included: there are 23 classic design patterns.

As languages evolve:

• Some patterns (e.g., Singleton) have aged poorly and even become anti‑patterns.
• Some are now built into languages (e.g., Iterator).
• New patterns keep emerging (e.g., Monostate).

The 23 classic patterns are grouped into three categories: creational, structural, and behavioral.

Creational

Common: Singleton, Factory (Factory Method and Abstract Factory), Builder.

Less common: Prototype.

Structural

Common: Proxy, Bridge, Decorator, Adapter.

Less common: Facade, Composite, Flyweight.

Behavioral

Common: Observer, Template Method, Strategy, Chain of Responsibility, Iterator, State.

Less common: Visitor, Memento, Command, Interpreter, Mediator.

Coding conventions

Definition: primarily concerned with readability. Compared with design principles and design patterns, conventions are more concrete and closer to code‑level details. Even if you’re unfamiliar with design principles or patterns, at minimum you should master basic coding conventions.

For example: how to name variables, classes, and functions; how to write comments; keep functions short; avoid too many parameters; and so on.

There are plenty of classic books to learn from—“Refactoring,” “Code Complete,” and “Clean Code,” to name a few.

Each convention is straightforward and specific. Just remember them and follow them. Unlike design principles, they require less personal interpretation.

Refactoring

As long as a project is alive and people keep working on it, the software will evolve. New features will inevitably push older code to be refactored. Refactoring is the practical way to keep code quality from decaying to an unrecoverable state.

The toolkit for refactoring includes everything mentioned earlier: paradigms, design principles, design patterns, and coding conventions.

While design patterns can improve extensibility, overuse or misuse increases complexity and harms readability.

In early development, unless absolutely necessary, don’t over‑design or apply complex patterns.

Instead, when code shows concrete issues, refactor to address those issues using the appropriate principles and patterns.

This effectively prevents premature over‑engineering.

Must‑know points

As follows:

• The purpose (why), target (what), timing (when), and methods (how) of refactoring
• Techniques for safe refactoring: unit tests and code testability
• Two scales of refactoring: large‑scale/high‑level and small‑scale/low‑level

How the five pieces fit together

The relationship among OOP, design principles, design patterns, coding conventions, and refactoring is as follows:

• OOP, with its rich features (encapsulation, abstraction, inheritance, polymorphism), enables sophisticated design ideas and is the foundation for implementing many principles and patterns.
• Design principles guide design decisions and help determine whether a particular pattern should be applied in a given scenario. For example, the Open/Closed Principle underpins patterns such as Strategy and Template Method.
• Design patterns are recurring solutions to common design problems in software development. Their main goal is to improve extensibility. In terms of abstraction, principles are more abstract; patterns are more concrete and executable.
• Coding conventions primarily address readability. Compared with principles and patterns, conventions are more specific, detail‑oriented, and directly actionable. Continuous small‑scale refactoring relies heavily on conventions as its theoretical backbone.

Ultimately, this article is about one thing: writing high‑quality code. Once you trace it back to that, how to do many things—and how to implement them in code—becomes clear.

mindmap
  root((Writing High-Quality Code))
    Object-Oriented
      Encapsulation, Abstraction, Inheritance, Polymorphism
      OOP vs Procedural Programming
      OO Analysis, Design, Programming
      Interface vs Abstract Class
      Program to an Interface, Not an Implementation
      Favor Composition Over Inheritance
      Anemic vs Rich Domain Model
    Design Principles
      SOLID Principles
        SRP Single Responsibility
        OCP Open/Closed
        LSP Liskov Substitution
        ISP Interface Segregation
        DIP Dependency Inversion
      Others
        DRY Principle
        KISS Principle
        YAGNI Principle
        Law of Demeter (LOD)
    Coding Conventions
      20 Quick Rules to Improve Code Quality
    Refactoring
      Purpose, Target, Timing, Methods
      Unit Tests and Testability
      Large-Scale (High-Level)
      Small-Scale (Low-Level)
    Design Patterns
      Creational
        Common
          Singleton
          Factory (Factory Method & Abstract Factory)
          Builder
        Less Common
          Prototype
      Structural
        Common
          Proxy
          Bridge
          Decorator
          Adapter
        Less Common
          Facade
          Composite
          Flyweight
      Behavioral
        Common
          Observer
          Template Method
          Strategy
          Chain of Responsibility
          Iterator
          State
        Less Common
          Visitor
          Memento
          Command
          Interpreter
          Mediator

mindmap
  root((Writing High-Quality Code))
    Object-Oriented
      Encapsulation, Abstraction, Inheritance, Polymorphism
      OOP vs Procedural Programming
      OO Analysis, Design, Programming
      Interface vs Abstract Class
      Program to an Interface, Not an Implementation
      Favor Composition Over Inheritance
      Anemic vs Rich Domain Model
    Design Principles
      SOLID Principles
        SRP Single Responsibility
        OCP Open/Closed
        LSP Liskov Substitution
        ISP Interface Segregation
        DIP Dependency Inversion
      Others
        DRY Principle
        KISS Principle
        YAGNI Principle
        Law of Demeter (LOD)
    Coding Conventions
      20 Quick Rules to Improve Code Quality
    Refactoring
      Purpose, Target, Timing, Methods
      Unit Tests and Testability
      Large-Scale (High-Level)
      Small-Scale (Low-Level)
    Design Patterns
      Creational
        Common
          Singleton
          Factory (Factory Method & Abstract Factory)
          Builder
        Less Common
          Prototype
      Structural
        Common
          Proxy
          Bridge
          Decorator
          Adapter
        Less Common
          Facade
          Composite
          Flyweight
      Behavioral
        Common
          Observer
          Template Method
          Strategy
          Chain of Responsibility
          Iterator
          State
        Less Common
          Visitor
          Memento
          Command
          Interpreter
          Mediator

PS: A bunch of keywords for further exploration:

System architecture principles High cohesion and low coupling Law of Demeter (principle of least knowledge) Composition over inheritance Separation of concerns Design by contract Cloud‑native design principles Resilience design principles Observability principles Immutable infrastructure Service autonomy Declarative configuration Security by design Defense in depth Principle of least privilege Secure by default Separation of duties Fail‑safe

When legacy systems are initially created and coded, the final system performance and maintainability depend entirely on the developer's expertise.

When you inherit a system with the following issues, this article might help you:

1. No monitoring
2. No alerting
3. No business success rate metrics
4. Frequent optimistic locking failures
5. Black box system response time (yes, your business users only vaguely tell you the system is slow, but don't know specifically where it's slow)
6. Complex log troubleshooting

Yes, these are the headache-inducing problems I've encountered over the past month.

Now that the month is up, I'm starting to take action and perform a series of performance optimizations on this system.

First, you need to understand the overall situation of the system. For optimization, you should at least know what you're optimizing. Business users should also know where exactly their system is slow.

So what you need to do is establish system baselines.

Establishing System Baselines

System baselines are quantitative descriptions of key performance indicators when the system is running normally.

They are the starting point for system performance optimization and the standard reference for measuring optimization effectiveness. Simply put, system baselines are your system's "health checkup report," recording system performance under various conditions. (1, 2)

System Baselines Include the Following

A complete system baseline should include the following aspects:

Performance Metrics

• Response time (average, P95, P99)
• Throughput (QPS/TPS)
• Latency
• Concurrent processing capability

Resource Usage

• CPU utilization
• Memory usage and allocation
• Disk I/O
• Network I/O
• Connection pool usage status

Business Metrics

• Business success rate
• Error rate
• Key business process completion time

System Stability Metrics

• Mean Time Between Failures (MTBF)
• Mean Time To Recovery (MTTR)
• Lock contention situations (such as the optimistic lock failure rate you mentioned)

Taking this opportunity during the May Day holiday to thoroughly analyze one of the most famous papers: "MapReduce: Simplified Data Processing on Large Clusters."

Of course, this article is deeply influenced by the article from "Muniao's Notes," so my article is purely a combination of two articles plus my own thoughts.

I've always heard senior colleagues talk about how powerful MapReduce distributed computing is, so today's the opportunity to dive in!

Introduction

MapReduce in the paper is actually a concept. But in another context, it can also be a programming model, as well as a distributed system implementation supporting that model. I found an article that explains this concept better:

MapReduce is a concept proposed in Google's 2004 paper (Google internally wrote the first version starting in 2003).

In Google's context, MapReduce is both a programming model and a distributed system implementation supporting that model. Its introduction enables developers without distributed systems background to leverage large-scale clusters for high-throughput processing of massive data with relative ease.

This article also has a sentence explaining the problem-solving approach for applying this technology: Finding pain points in requirements (such as how to maintain, update, and rank massive indexes), performing high-level abstractions of key processing flows (sharding Map, on-demand Reduce), for efficient system implementation (so-called tailored solutions).

Among these, finding an appropriate computational abstraction is the most difficult part, requiring both intuitive understanding of requirements and extremely high computer science literacy.

The quote above is from the referenced article from "Muniao's Notes."

Returning to the paper, we can see that on the first page, Google's experts clearly explained what this is.

As a reaction to this complexity, we designed a new abstraction that allows us to express the simple computations we were trying to perform but hides the messy details of parallelization, fault-tolerance, data distribution and load balancing in a library.

This means we abstracted something to express a computational method that can hide many conceptual details (parallelization, fault tolerance, data distribution, and load balancing).

This abstraction is inspired by the map and reduce primitives present in Lisp and many other functional languages.

We realized that most of our computations involved applying a map operation to each logical "record" in our input in order to compute a set of intermediate key/value pairs, and then applying a reduce operation to all the values that shared the same key, in order to combine the derived data appropriately.

Most of our computations involve applying a map operation to each logical record in the input to compute a set of intermediate key/value pairs, then applying a reduce operation to all values sharing the same key to appropriately combine the derived data.

Our use of a functional model with user-specified map and reduce operations allows us to parallelize large computations easily and to use re-execution as the primary mechanism for fault tolerance.

We use a functional model with user-specified map and reduce operations, enabling easy parallelization of large computations.

I find the last sentence very interesting:

use re-execution as the primary mechanism for fault tolerance.

Using "re-execution" as the primary mechanism for fault tolerance.

The paper's Abstract outlines the content:

1. Section 1 is the Introduction above
2. Section 2 describes the basic programming model and gives several examples, Programming Model
3. Section 3 describes an implementation of the MapReduce interface tailored towards our cluster-based computing environment, Implementation
   • Implementation of MapReduce interface customized for cluster computing environments
4. Section 4 describes several refinements of the programming model that we have found useful - Several programming model improvements
5. Section 5 has performance measurements of our implementation for a variety of tasks - Performance measurements of implementing various tasks
6. Section 6 explores the use of MapReduce within Google including our experiences in using it as the basis for a rewrite of our production indexing system - MapReduce applications within Google
7. Section 7 discusses related and future work

Programming Model

The Map key is a normal key, and the value can be thought of as a string array.

MapReduce, simply put, consists of two functions: the map function and the reduce function.

The Map function receives an input pair and generates a set of intermediate key/value pairs, then the MapReduce library combines all intermediate values associated with the same key.

Example

Below is pseudo-code, grandfather-level, from the original paper:

map(String key, String value):
    // key: document name
    // value: document contents
    for each word w in value:
      EmitIntermediate(w, "1");

reduce(String key, Iterator values):
    // key: a word
    // values: a list of counts
    int result = 0;
    for each v in values:
      result += ParseInt(v);
    Emit(AsString(result));

This is a classic MapReduce word count implementation, one of the most common examples in the MapReduce programming model.

• key: document name
• value: complete document content (text string)

EmitIntermediate represents the function provided by the MapReduce framework for outputting intermediate key/value pairs. Each call to this function produces a key-value pair: (word, "1"), indicating that the word appeared once.

For example, processing document content "hello world hello":

• First word "hello" → EmitIntermediate("hello", "1")
• Second word "world" → EmitIntermediate("world", "1")
• Third word "hello" → EmitIntermediate("hello", "1")

Map phase intermediate results:

("hello", "1")
("world", "1")
("hello", "1")

Shuffle Stage (Automatically Completed by Framework)

Between Map and Reduce, the MapReduce framework automatically performs the Shuffle operation:

1. Collect all mapper outputs
2. Sort by key (word)
3. Group all values with the same key together

So the above example after Shuffle:

("hello", ["1", "1"])
("world", ["1"])

Reduce Function

After Shuffle comes the Reduce function's work. The pseudo-code above essentially performs accumulation, which needs no further explanation.

Complete Execution Flow

Here's a larger example showing the entire MapReduce execution flow:

Assume three documents:

• document1.txt: "hello world"
• document2.txt: "hello mapreduce"
• document3.txt: "mapreduce world example"

Map Phase (Parallel Execution)

Mapper 1 processes document1.txt:

EmitIntermediate("hello", "1")
EmitIntermediate("world", "1")

Mapper 2 processes document2.txt:

EmitIntermediate("hello", "1")
EmitIntermediate("mapreduce", "1")

Mapper 3 processes document3.txt:

EmitIntermediate("mapreduce", "1")
EmitIntermediate("world", "1")
EmitIntermediate("example", "1")

Shuffle Phase (Automatically Completed by Framework):

("hello", ["1", "1"])
("world", ["1", "1"])
("mapreduce", ["1", "1"])
("example", ["1"])

Reduce Phase (Parallel Execution):

Reducer processes "hello":

result = 0
result += 1 = 1
result += 1 = 2
Emit("2")  # Output ("hello", "2")

Similarly for other words...

Final Results

("hello", "2")
("world", "2")
("mapreduce", "2")
("example", "1")

MapReduce Framework's Role

In this process, the MapReduce framework is responsible for:

1. Splitting input data into multiple shards, assigning them to different Mappers
2. Executing multiple Map tasks in parallel
3. Performing Shuffle operations, reorganizing and sorting intermediate results
4. Executing multiple Reduce tasks in parallel
5. Collecting and integrating Reduce outputs
6. Handling task failures and retries
7. Optimizing data locality, processing data on nodes where it resides when possible

This pattern allows developers to focus on business logic (Map and Reduce functions) without worrying about complex issues like parallelization, distributed computing, and fault tolerance.

More Examples

Here are more examples:

Distributed Grep

Working Principle

In this example:

• Map function checks each line of input text, emitting the line if it matches a specified pattern
• Reduce function is a simple identity function that directly copies intermediate results to output

Application Value

This pattern is very suitable for quickly finding text lines with specific patterns in large-scale distributed file systems. It fully utilizes MapReduce's parallel processing capabilities and is extremely efficient when searching for specific error information in TB or even PB-level log files.

Count of URL Access Frequency

Working Principle

• Map function: Processes web request logs, emitting <URL, 1> key-value pairs for each URL
• Reduce function: Sums all counts for the same URL, outputting <URL, total count>

Application Value

This is a fundamental operation in web analytics, crucial for understanding website traffic distribution, identifying popular content, and detecting abnormal access patterns. In large websites, daily log data can reach several TB, and MapReduce can effectively handle this scale of data.

Reverse Web-Link Graph

Working Principle

• Map function: Analyzes web content, outputting <target URL, source URL> for each discovered link
• Reduce function: Collects all source URLs for a target URL, outputting <target URL, source URL list>

Application Value

Reverse link graphs are one of the core data structures for modern search engines, used in the following scenarios:

• Foundation data for PageRank and other web importance algorithms
• Analyzing citation relationships between websites
• Discovering influential content creators
• Providing backlink analysis tools for webmasters

Building a complete web reverse link graph is a computation-intensive task, and the MapReduce model is very suitable for this naturally parallelizable problem.

Term-Vector per Host

Working Principle

• Map function: Analyzes document content, extracts hostname from URL, outputs <hostname, document term vector>
• Reduce function: Merges all term vectors for the same host, filters low-frequency terms, outputs <hostname, aggregated term vector>

Application Value

This analysis is very valuable for understanding website content characteristics:

• Can be used for website topic classification
• Helps with search engine optimization
• Content similarity comparison
• Competitor website content analysis
• Foundation data for content recommendation systems

Inverted Index

Working Principle

• Map function: Parses each document, outputs <word, document ID> key-value pairs
• Reduce function: Receives all document IDs for a given word, sorts them and outputs <word, document ID list> key-value pairs

Application Value

Inverted indexes are the fundamental data structure for modern search engines, used for:

1. Full-text search: Quickly find all documents containing query terms
2. Phrase search: Achieve precise phrase search through positional information
3. TF-IDF calculation: Provide term frequency statistics for information retrieval systems
4. Keyword highlighting: Help frontend display matched text fragments
5. Relevance ranking: Provide foundation data for search result ranking

MapReduce is particularly suitable for building inverted indexes because it can efficiently process large numbers of documents in parallel and naturally implement index merging in the Reduce phase.

Distributed Sort

Working Principle

• Map function: Extracts the key of each record, outputs <key, record> key-value pairs
• Reduce function: Directly outputs all received key-value pairs without modification

This seemingly simple example actually cleverly utilizes two core features of the MapReduce framework:

1. Partitioning mechanism: Ensures records with keys in the same range are sent to the same Reducer
2. Sorting property: Ensures keys received by Reducers are in order

MapReduce Framework's Special Contribution:

In distributed sorting, the MapReduce framework does most of the important work:

1. Custom Partitioner:

   // Example: Range partitioner
   func RangePartitioner(key string, numReducers int) int {
       // Determine which reducer to send to based on key range
       // This ensures global sorting
       if key < "D" {
           return 0
       } else if key < "N" {
           return 1
       } else {
           return 2
       }
   }
   

2. Sort Comparator:

   // Define natural sorting order for keys
   func KeyComparator(key1, key2 string) int {
       return strings.Compare(key1, key2)
   }
   

Application Value

Distributed sorting is a fundamental operation for many big data processing workflows:

1. Data preprocessing: Preparing large datasets for further analysis
2. Log analysis: Sorting massive log records by timestamp
3. Building indexes: Creating sorted indexes for databases or search engines
4. Merging sorted data: Combining multiple sorted datasets into one
5. TopN queries: Quickly finding top N records for a given metric

Overall Examples Analysis and Comparison

The examples demonstrate the versatility and adaptability of the MapReduce model:

1. Distributed Grep: Simplest application, basically only uses Map functionality, suitable for simple filtering operations
2. URL Access Frequency Count: Classic word count variant, demonstrates MapReduce advantages in statistical aggregation
3. Reverse Web-Link Graph: Shows how to use MapReduce to build complex relationship graphs and index structures
4. Term-Vector per Host: Combines text analysis and aggregation functions, suitable for advanced content analysis

Implementation

Multiple different implementations of the MapReduce interface are possible.

The right choice depends on the specific environment (meaning specific problems require specific analysis).

For example, one implementation might be suitable for small shared-memory machines, another for large NUMA multiprocessors, and yet another for even larger networked machine clusters.

Below is the computing environment widely used at Google:

This section describes an implementation targeted to the computing environment in wide use at Google: large clusters of commodity PCs connected together with switched Ethernet.

In our environment:

(1) Machines are typically dual-processor x86 processors running Linux, with 2-4 GB of memory per machine.

(2) Commodity networking hardware is used – typically either 100 megabits/second or 1 gigabit/second at the machine level, but averaging considerably less in overall bisection bandwidth.

(3) A cluster consists of hundreds or thousands of machines, and therefore machine failures are common.

(4) Storage is provided by inexpensive IDE disks attached directly to individual machines. A distributed file system developed in-house is used to manage the data stored on these disks. The file system uses replication to provide availability and reliability on top of unreliable hardware.

(5) Users submit jobs to a scheduling system. Each job consists of a set of tasks, and is mapped by the scheduler to a set of available machines within a cluster.

(1) Machines typically feature dual-processor x86 architecture, running Linux systems, with 2-4 GB of memory per machine.

(2) Commodity networking hardware is used — typically 100 megabits/second or 1 gigabit/second at the machine level, but the average overall bisection bandwidth is significantly lower.

(3) A cluster consists of hundreds or thousands of machines, so machine failures are common.

(4) Storage is provided by inexpensive IDE disks directly connected to individual machines. An internally developed distributed file system is used to manage data stored on these disks. The file system provides availability and reliability on top of unreliable hardware through data replication.

(5) Users submit jobs to a scheduling system. Each job consists of a set of tasks and is mapped by the scheduler to a set of available machines within the cluster.

Execution Process

The Map invocations are distributed across multiple machines by automatically partitioning the input data into a set of M splits. The input splits can be processed in parallel by different machines. Reduce invocations are distributed by partitioning the intermediate key space into R pieces using a partitioning function (e.g., hash(key) mod R). The number of partitions (R) and the partitioning function are specified by the user.

Following figure shows the overall flow of a MapReduce operation in our implementation. When the user program calls the MapReduce function, the following sequence of actions occurs (the numbered labels in Figure correspond to the numbers in the list below):

1. The MapReduce Library in the user program first splits the files into M pieces, each piece typically 16-64 MB in size;

2. The program copy on the Master is special, while other workers are assigned tasks by the master. There are typically M map tasks and R reduce tasks to assign.

   There are M map tasks and R reduce tasks to assign. The master picks idle workers and assigns each one a map task or a reduce task.

   Here, the word "idle" refers to idle workers - in Google's MapReduce architecture, the entire computation task is executed distributedly, including:

   1. Master (master node): A special program copy responsible for task scheduling and coordinating the entire computation process
   2. Workers (worker nodes): Other program copies responsible for executing actual computation tasks
   3. Idle workers: Worker nodes currently not executing any tasks and in a waiting state

   I hope this clarifies what idle workers are.

   So generally speaking, the entire MapReduce workflow related to idle worker nodes:

   1. When computation tasks begin, the system starts multiple program copies, with one as the master node and others as worker nodes
   2. The master node maintains the state of the entire cluster, including whether each worker node is idle
   3. When the master node detects that a worker node is "idle" (not executing tasks), it selects one from the pending M Map tasks or R Reduce tasks to assign to that worker node

3. A worker assigned a map task reads the contents of the corresponding input split, parses key/value pairs from the input data, and passes each pair to the user-defined map function.

   These intermediate key/value pairs produced by map are buffered in memory.

   A worker who is assigned a map task reads the contents of the corresponding input split. It parses key/value pairs out of the input data and passes each pair to the user-defined Map function. The intermediate key/value pairs produced by the Map function are buffered in memory.

   In fact, Hadoop does exactly this.

4. Buffered intermediate result pairs are periodically written to local disk, then partitioned into R regions by the partitioning function.

   These buffered pairs on local disk are passed back to the master, so the master can inform reduce workers of these pair locations.

   Periodically, the buffered pairs are written to local disk, partitioned into R regions by the partitioning function.

   The locations of these buffered pairs on the local disk are passed back to the master, who is responsible for forwarding these locations to the reduce workers.

5. When a reduce worker receives the buffered pair locations mentioned above, it uses RPC to read the corresponding partition data.

   When a reduce worker has read all the data, it sorts by key so that all data with the same key is sorted together.

   When a reduce worker is notified by the master about these locations, it uses remote procedure calls to read the buffered data from the local disks of the map workers.

   When a reduce worker has read all intermediate data, it sorts it by the intermediate keys so that all occurrences of the same key are grouped together.

   The sorting is needed because typically many different keys map to the same reduce task.

   If the amount of intermediate data is too large to fit in memory, an external sort is used.

   If the intermediate data is too large, external sorting programs are needed. This is where performance optimization points come in.

Steps 4 and 5 together are called shuffle

6. The Reduce Worker then iterates through this sorted intermediate data and passes this data along with its key-related data to the user's reduce function.

   The Reduce function's output is appended to the final output file.

   The reduce worker iterates over the sorted intermediate data and for each unique intermediate key encountered, it passes the key and the corresponding set of intermediate values to the user's Reduce function.

   The output of the Reduce function is appended to a final output file for this reduce partition.

7. When all map tasks and reduce tasks are completed, the master wakes up the user program.

   At this point, the user program receives a final computation result (MapReduce call).

   When all map tasks and reduce tasks have been completed, the master wakes up the user program. At this point, the MapReduce call in the user program returns back to the user code.

Master Data Structures

Understanding the key data structures maintained by the Master node in the MapReduce framework and its core responsibilities in task coordination.

The Master node is actually the "brain" of the entire MapReduce execution process, maintaining the following important data:

1. Task State Records: For each map and reduce task, the master node records its current state:

   • idle: pending tasks
   • in-progress: tasks assigned to workers but not yet completed
   • completed: finished tasks

2. Worker Machine Identity: For non-idle state tasks, the master node records the identity of the worker machine executing that task, used for tracking task execution and handling failures

3. Intermediate File Metadata: For completed map tasks, the master stores location and size information of intermediate result files produced by that task

Master as Information Transfer Channel

The Master node plays the role of an information transfer channel for intermediate result location information. When a map task completes, it informs the master node of which intermediate files were produced, along with their location and size information.

Additionally, Google's MapReduce implementation has job-level encapsulation, where each job contains a series of tasks, namely Map Tasks and Reduce Tasks.

To maintain metadata for a running job, it's necessary to save the state of all executing tasks and their machine IDs.

This information is crucial for reduce tasks, as reduce tasks need to know where to obtain the data they need to process.

Moreover, the Master also serves as an information channel from Map Task output to Reduce Task. The master node incrementally pushes this information to worker nodes executing reduce tasks.

When each Map Task ends, it notifies the Master of the location information of its output intermediate results, the Master then forwards this to the corresponding Reduce Task, and the Reduce Task fetches the corresponding size of data from the corresponding location.

Note: Since Map Task completion times are not uniform, this notification → forwarding → fetching process is incremental. It's reasonable to infer that the sorting of intermediate data on the reduce side should be a continuous merge process, unlikely to be global sorting after all data is in place. — from Muniao

The master keeps several data structures. For each map task and reduce task, it stores the state (idle, in-progress, or completed), and the identity of the worker machine (for non-idle tasks).

The master is the conduit through which the location of intermediate file regions is propagated from map tasks to reduce tasks. Therefore, for each completed map task, the master stores the locations and sizes of the R intermediate file regions produced by the map task. Updates to this location and size information are received as map tasks are completed. The information is pushed incrementally to workers that have in-progress reduce tasks.

Fault Tolerance

Distributed systems must gracefully handle various errors on distributed machines when processing large amounts of data. The paper roughly divides failures into three types:

1. Worker Failure
2. Master Failure
3. Semantics in the Presence of Failures

Worker Failure

The master periodically pings each worker node, and if there's no response, the master marks it as a failed node.

At this time, both completed and uncompleted map tasks are marked back to the initial idle state, then wait to be scheduled to other normal workers.

These map tasks are then re-executed and re-stored to local disk, and the master continues to report this information to Reduce workers.

If a Reducer worker has already processed one of the map tasks, it doesn't need to fetch processing data again from the master's provided information; if not processed, it continues to fetch.

On the Reducer worker side, when it discovers that the map worker processing a certain segment of map tasks has failed, for example, this Reduce program is R5, processing map tasks (41-51) on Worker-37, when R5 processes M47 and discovers a failure, its countermeasures are as follows:

• Transmission Interruption Handling: If R5's connection is interrupted while pulling data from Worker-37, it triggers exception handling

// Simplified pseudo-code
try {
    fetchMapOutput(worker37, mapTaskId47);
} catch (FetchFailureException e) {
    // Wait for Master notification of new data location
    waitForNotification(mapTaskId47);
    // Retry with new data location
    fetchMapOutput(worker51, mapTaskId47);
}

• Data Consistency: R5 discards incomplete data partially pulled from Worker-37

if (partialData && dataSource != currentSourceForTask) {
    discardPartialData();
    fetchFromNewSource();
}

• Notification Mechanism: Master notifies Reduce tasks through the following ways

1. Heartbeat responses include updated Map output location information
2. RPC calls notify state changes
3. Reduce tasks periodically poll Master for latest mapping information

Master Failure

Google's paper handles Master failure relatively simply: saving state through checkpoint mechanisms, but terminating the entire job when Master actually fails.

This design is based on two considerations:

1. Single Point Characteristic: There's typically only one Master node in the system, with relatively low failure probability
2. Simplified Design: Simple failure handling mechanisms reduce system complexity

However, in critical production environments, this simple handling approach obviously cannot meet high availability requirements.

With the development of distributed systems, more robust Master failure handling mechanisms should be seriously considered.

Semantics in the Presence of Failures

MapReduce provides a key promise: In deterministic operations, the results of distributed parallel execution are completely consistent with sequential execution.

This feature greatly simplifies the complexity of distributed programs.

How is this "computational result consistency" achieved?

Key Mechanism: Atomic Commits

MapReduce achieves result consistency through carefully designed atomic commit mechanisms:

1. Temporary file strategy
2. Task completion flow
3. Redundant execution handling

Core Role of Atomic Operations

Atomic operations are the foundation of the entire fault tolerance mechanism, mainly reflected in two levels:

1. Master data structure updates
2. File system atomic renaming - for example, atomic renaming operations when Reduce tasks complete; only one execution instance can successfully rename, meaning if there's a successfully named file, it indicates one instance has been successfully executed

Locality

A commonly used principle in computer science is called locality of reference (specifically spatial locality), which states that when a program executes sequentially, after accessing a block of data, it's highly likely to access data physically adjacent to that data next. This simple assertion is the foundation for all cache effectiveness, leading to the formation of a storage hierarchy system from slow to fast, cheap to expensive, large to small (hard disk → memory → cache → registers).

In distributed environments, this hierarchy system requires at least one more layer — network I/O. This is also the first sentence in the paper: "Network bandwidth is a relatively scarce resource in our computing environment."

In MapReduce systems, we also fully utilize input data locality. However, this time, instead of loading data over, we schedule the program to go there (Moving Computation is Cheaper than Moving Data). If input exists on GFS, it manifests as a series of logical blocks, each of which may have several (typically three) physical replicas. For each logical block of input, we can run Map Tasks on machines where one of its physical replicas resides (if it fails, try another replica), thereby minimizing network data transmission and reducing latency while saving bandwidth.

Network bandwidth is a relatively scarce resource in our computing environment. We conserve network bandwidth by taking advantage of the fact that the input data (managed by GFS) is stored on the local disks of the machines that make up our cluster. GFS divides each file into 64 MB blocks, and stores several copies of each block (typically 3 copies) on different machines. The MapReduce master takes the location information of the input files into account and attempts to schedule a map task on a machine that contains a replica of the corresponding input data. Failing that, it attempts to schedule a map task near a replica of that task's input data (e.g., on a worker machine that is on the same network switch as the machine containing the data). When running large MapReduce operations on a significant fraction of the workers in a cluster, most input data is read locally and consumes no network bandwidth.

Task Granularity

Deep analysis of the core design principles, influencing factors, and best practices of Task Granularity in the MapReduce framework.

The paper has an important viewpoint: "M and R should be much larger than the number of worker machines."

M and R Should Be Much Larger Than Worker Machine Count

1. Dynamic Load Balancing

   // Simplified load balancing scenario
   100 Map tasks, 10 machines
   - Machines 1-9: Each processes 10 tasks, uniform load
   - Machine 10: Slower hardware, only completes 5 tasks
   - Task allocator automatically redistributes remaining 5 tasks to machines that completed tasks
   

   When each machine processes multiple small tasks rather than single large tasks, fast machines can process more tasks while slow machines process fewer, naturally forming performance-based work distribution.

2. Accelerated Failure Recovery

   Assumption:
   - 2000 machines, each executing about 100 Map tasks
   - Single Worker-37 fails, has completed 92 Map tasks
   
   Impact:
   - Traditional design (1 large task per machine): Lose entire Worker-37 computation results
   - Fine-grained design: Only need to re-execute 92 small tasks, distributed to other 1999 machines
   - Recovery speed: About 1/20 of traditional method (average less than 1 additional task per machine)
   

   When a worker fails, its completed multiple small tasks can be quickly redistributed to other machines in the cluster for re-execution, significantly accelerating recovery speed.

Backup Tasks

This mechanism was designed by Google to solve the "straggler" problem.

One of the common causes that lengthens the total time taken for a MapReduce operation is a "straggler": a machine that takes an unusually long time to complete one of the last few map or reduce tasks in the computation.

An important optimization introduced to solve slow task problems.

Straggler Problem: Key Challenge in Distributed Systems

"Stragglers" refer to machines that complete tasks abnormally slowly, severely slowing down the completion time of entire MapReduce jobs. This problem is particularly prominent in large-scale distributed environments.

Main Causes

1. Hardware Issues:

   - Disk errors: Correctable errors reduce read speed from 30MB/s to 1MB/s
   - Network problems: Network card failures cause bandwidth reduction
   - CPU or memory failures: Significantly reduced processing capability
   

2. Resource Competition:

   - Multi-task scheduling conflicts: Other jobs occupy CPU, memory resources
   - I/O contention: Multiple processes compete for disk or network I/O
   - Memory pressure: Insufficient memory causes frequent page swapping
   

3. Software Issues:

   - Configuration errors: Such as Google's processor cache disabled bug (100x performance degradation)
   - GC pauses: Long pauses caused by garbage collection
   - System updates: Background services or updates consuming resources
   

Impact of Stragglers

In MapReduce jobs, job completion time is limited by the last completing task. When 99% of tasks complete quickly but a few tasks are abnormally slow, the entire job's completion time will be dominated by these slow tasks.

Google designed the backup task mechanism as a simple yet effective strategy, using moderate resource redundancy to reduce overall execution time.

Refinements

The paper provides several extensions beyond the basic Mapper and Reducer primitives that have become recognized as standard components: Partitioner, Combiner, and Reader/Writer.

Partitioning Function

The Partitioning Function in MapReduce is an important extension mechanism connecting the Map and Reduce phases.

Core Concept

The partitioning function is a key component in MapReduce that connects the Map and Reduce phases, determining which intermediate key-value pairs are sent to which Reduce task.

// Basic definition of partitioning function
type PartitionFunc func(key interface{}, numPartitions int) int

Core functions:

• Determines which Reduce task processes Map output intermediate key-value pairs, ultimately affecting output file organization
• Controls the number and content organization of final output files
• Affects data distribution and load balancing across the cluster

Default Hash Partitioning Mechanism

MapReduce provides a simple and efficient default partitioning strategy:

// Default hash partitioning implementation
public int getPartition(K key, V value, int numReduceTasks) {
    return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks;
}

Characteristics:

• Simple and efficient: Low computational overhead, suitable for most scenarios
• Relatively balanced: Uses hash function to evenly distribute keys across partitions
• Deterministic: Same keys always map to the same partition, ensuring correct aggregation

Data flow diagram:

    Map Output          Partition Function       Reduce Tasks
 [K1:V1, K2:V2]       hash(K) % R        ->    Reduce-0
 [K3:V3, K4:V4]                          ->    Reduce-1
 [K5:V5, K6:V6]                          ->    Reduce-2
       ...                                       ...

Custom Partitioning Function Use Cases

The paper points out that certain scenarios require specific partitioning logic, such as when processing URL data where you want all URLs from the same host to go into the same output file.

Combiner Function

Core Concept and Working Principle

Combiner is a key optimization component in the MapReduce framework that performs partial data aggregation on the Map side, reducing network transmission volume.

Workflow:
Map output → Combiner local aggregation → Network transmission → Reducer final aggregation

Applicable conditions:

• Map output intermediate keys have lots of duplicates
• Reduce function has commutativity and associativity (such as sum, max)

Performance Advantage Example

Using word count as an example:

Without Combiner:
Map1 output: <"hello",1>, <"world",1>, <"hello",1>, <"hello",1> // 4 records transmitted
Map2 output: <"hello",1>, <"hadoop",1>, <"hello",1> // 3 records transmitted
Total network transmission: 7 records

With Combiner:
Map1 output: <"hello",3>, <"world",1> // 2 records transmitted
Map2 output: <"hello",2>, <"hadoop",1> // 2 records transmitted
Total network transmission: 4 records (43% reduction)

For data following Zipf distribution (like word frequency), Combiner can significantly reduce network transmission and improve performance.

Difference from Reduce

• Reducer output writes to final result files
• Combiner output writes to intermediate files, then transmitted to Reducer

Combiner is an important optimization in the MapReduce framework for improving data processing efficiency through "pre-aggregation," significantly reducing data transmission and processing time, especially effective for aggregation operations.

Input and Output Types

Support for different input data formats:

1. TextInputFormat (default)
   - Each line as one record
   - Key: line offset (LongWritable)
   - Value: line content (Text)
   - Smart splitting: ensures splitting at line boundaries

2. KeyValueTextInputFormat
   - Split each line into key-value by delimiter (default Tab)
   - Suitable for: simple structured text data

3. SequenceFileInputFormat
   - Read binary sequence files (key-value pairs)
   - Support compression, efficient random access
   - Commonly used for passing data between MapReduce jobs

4. DBInputFormat
   - Read records from relational databases
   - Support SQL queries as data source

This demonstrates the flexibility and extensibility of the framework.

MapReduce's input/output interface design enables it to handle diverse data sources:

- HDFS files
- Local file systems
- S3, Azure Blob and other cloud storage
- HBase, MongoDB and other NoSQL databases
- Kafka streaming data

By implementing appropriate InputFormat/OutputFormat, developers can integrate MapReduce with almost any data source/target, demonstrating the framework's powerful extensibility and making it suitable for various big data processing scenarios.

Performance

The paper mentions two performance tests representing MapReduce framework's capability to handle two typical big data processing scenarios:

Test Scenario Analysis

1. Pattern Search Test: Search for specific patterns in about 1TB of data

   • Represents the "extract small amounts of valuable information from large datasets" computation pattern
   • Typical applications include log analysis, anomaly detection, specific record finding, etc.

2. Large Data Sorting Test: Sort about 1TB of data

   • Represents the "transform data from one representation to another" computation pattern
   • Typical applications include ETL processes, data preprocessing, data reorganization, etc.

Below, the entire MapReduce distributed framework will be analyzed from five angles:

1. Cluster Configuration
2. Grep
3. Sort
4. Effect of Backup Tasks
5. Machine Failures

Cluster Configuration Analysis

First, a cluster configuration is given:

// Cluster configuration overview
type ClusterConfig struct {
    Nodes           int     // About 1800 machines
    CpuPerNode      int     // 2 × 2GHz Intel Xeon per node (with hyperthreading)
    MemoryPerNode   string  // 4GB per node (actual usable 2.5-3GB)
    DisksPerNode    int     // 2 × 160GB IDE disks per node
    NetworkBandwidth string  // Gigabit Ethernet
    NetworkTopology string   // Two-level tree switching network
    RootBandwidth   string   // 100-200Gbps aggregate bandwidth
    Latency         string   // <1ms latency between nodes
}

Analysis:

• This is a balanced cluster design for both computation and I/O, particularly suitable for MapReduce's divide-and-conquer model
• Storage-wise, each node has over 300GB total storage space, providing sufficient local storage for TB-level data processing
• Network-wise, tree topology is simple but may form bottlenecks during shuffle phase
• Low inter-node latency (<1ms) is extremely beneficial for data transmission in the reduce phase
• Cluster scale (1800 nodes) enables effective parallelization when processing TB-level data

Grep

This is a typical "extract small amounts of information from massive data" scenario:

// Grep task configuration
type GrepJobConfig struct {
    InputSize       string  // About 1TB (10^10 100-byte records)
    Pattern         string  // Three-character pattern (matches 92,337 records)
    InputSplits     int     // M=15000 (about 64MB per block)
    ReduceTasks     int     // R=1 (single output file)
    PeakScanRate    string  // >30GB/s (with 1764 workers)
    TotalTime       int     // About 150 seconds (including 60 seconds startup overhead)
}

Performance Analysis:

1. Scalability Performance: From Figure 2, as worker count increases, scan rate linearly improves to 30GB/s, showing excellent horizontal scaling capability for map-intensive tasks
2. I/O Bound Characteristics: Grep is essentially I/O intensive work; the test achieving 30GB/s throughput approaches the theoretical upper limit of total disk I/O for 1764 nodes
3. Optimization Opportunities: About 60 seconds startup overhead (40% of total time) shows an optimization point - GFS metadata operations and task distribution can be further optimized
4. R=1 Design: Single reduce design suits this "filtering" scenario, but also means final result collection could become a bottleneck (not manifested in this example due to small data volume)

Sort

Comprehensive test of the entire MapReduce framework capability:

// Sort task characteristic analysis
type SortJobAnalysis struct {
    InputSize       string  // About 1TB (10^10 100-byte records)
    InputRate       string  // Peak 13GB/s (lower than Grep due to writing intermediate data)
    ShufflePattern  string  // Two-phase pattern, related to reduce task batching
    OutputRate      string  // 2-4GB/s (dual replica writes, actual physical writes 4-8GB/s)
    MapTasks        int     // M=15000 (about 64MB per block)
    ReduceTasks     int     // R=4000 (partitioning strategy leverages key distribution knowledge)
    TotalTime       int     // 891 seconds (close to TeraSort benchmark 1057 seconds)
}

Technical Analysis:

1. Data Pipeline: Test clearly shows MapReduce three-phase pipeline - map phase (0-200 seconds), shuffle phase (200-600 seconds), and reduce phase (600-850 seconds)
2. Resource Bottleneck Shifting:
   • 0-200 seconds: Bottleneck in disk I/O and CPU (parsing data)
   • 200-600 seconds: Bottleneck shifts to network bandwidth (shuffle)
   • 600-850 seconds: Bottleneck in sorting computation and output disk I/O
3. Locality Optimization Effect: Input rate (13GB/s) higher than shuffle rate mainly due to data locality optimization, with most reads going through local disk rather than network
4. Replication Overhead: Output rate (2-4GB/s) relatively low mainly due to GFS dual replica strategy, actual physical writes are twice this rate

Effect of Backup Tasks

Figure 3 also shows:

// Backup task impact analysis
type BackupTaskAnalysis struct {
    WithBackup      int     // Normal execution, total time 891 seconds
    WithoutBackup   int     // 1283 seconds, 44% increase
    StragglerDelay  int     // Last 5 reduce tasks took additional 300 seconds
    EfficiencyGain  string  // Backup task mechanism improves performance by 44%
}

Professional Interpretation:

1. Severity of Straggler Problem: Data clearly demonstrates the severity of the "straggler problem" in distributed systems - just 5 slow tasks increased total time by 44%
2. Root Cause Analysis: Stragglers typically stem from:
   • Hardware anomalies (like disk performance degradation, memory errors)
   • Resource competition (like interference from other processes)
   • Data skew (some reduce tasks process significantly more data than others)
3. Cloud-Native Environment Significance: In shared resource cloud environments, straggler problems are more prevalent; backup task mechanisms are key to ensuring performance predictability

Machine Failures

Figure 3 also shows:

// Fault tolerance capability analysis
type FaultToleranceAnalysis struct {
    NodesKilled     int     // 200 nodes (about 11.5% of nodes)
    RecoveryPattern string  // Brief negative input rate, then quick recovery
    TotalTime       int     // 933 seconds, only 5% increase
    KeyMechanism    string  // Automatically detect failures and re-execute tasks
}

Analysis:

1. Failure Impact Visualization: Negative input rate in the graph intuitively shows how node failures cause loss of completed work and re-execution requirements
2. Quick Recovery Principle:
   • Task state tracking: master node continuously tracks each task's state
   • Heartbeat detection: detect worker failures through periodic heartbeats
   • Task rescheduling: reassign failed node tasks to healthy nodes
   • Redundant execution: key is MapReduce design allowing any node to handle any task
3. Comparison with Traditional Systems: Traditional MPP databases typically fail completely or have 50%+ performance degradation with 11% node failure; MapReduce's 5% performance loss highlights its excellent fault tolerance capability

Experience

This content is excerpted from Jeff Dean and Sanjay Ghemawat's MapReduce paper, detailing MapReduce's early development history and application within Google.

Technical Development History

The first version of the MapReduce library was developed in February 2003, with major enhancements in August of the same year, including:

• Locality optimization
• Dynamic load balancing of task execution across worker nodes
• Other performance optimizations

Wide Application Scope

MapReduce gained widespread application within Google, covering multiple domains:

1. Large-scale machine learning problems
2. Clustering problems for Google News and Froogle (early Google Shopping) products
3. Popular query report data extraction (such as Google Zeitgeist)
4. Attribute extraction from large-scale web corpora (such as geographical locations for localized search)
5. Large-scale graph computations

Explosive Growth

From the graph, MapReduce usage within Google showed exponential growth:

• Early 2003: Near 0 instances
• End of September 2004: Nearly 900 instances

This rapid growth indicates MapReduce gained extremely high recognition and application value within Google.

Success Factor Analysis

Key factors for MapReduce success:

1. Simplified Distributed Computing: Enables developers to write simple programs that run efficiently on thousands of machines
2. Accelerated Development Cycles: Significantly shortened development and prototyping cycles
3. Lowered Technical Barriers: Allows programmers without distributed/parallel systems experience to easily leverage large-scale computing resources

Scale and Efficiency Analysis (August 2004 Data)

From table data, the following insights emerge:

1. Wide Usage: Executed 29,423 MapReduce jobs in a single month
2. High Processing Efficiency: Average job completion time was 634 seconds (about 10.5 minutes)
3. Large Computation Scale:
   • Used equivalent of 79,186 machine-days of computation time
   • Processed 3,288 TB of input data
   • Generated 758 TB of intermediate data
   • Output 193 TB of results
4. Task Distribution Characteristics:
   • Each job used average of 157 worker machines
   • Average 1.2 worker node failures per job (indicating good fault tolerance)
   • Average 3,351 map tasks and 55 reduce tasks per job
5. Code Reusability:
   • 395 unique map implementations
   • 269 unique reduce implementations
   • 426 unique map/reduce combinations

Conclusions

The paper's conclusions explain why this paper is so famous:

Three Key Success Factors of MapReduce

1. Simple and Easy-to-Use Programming Model

MapReduce's primary success factor is its concise programming interface:

// Users only need to define these two functions, without worrying about distributed system complexity
func Map(key, value string) []KeyValue { /* User-defined mapping logic */ }
func Reduce(key string, values []string) string { /* User-defined reduction logic */ }

This design is extremely developer-friendly because it:

• Hides the complex details of parallelization
• Automatically handles fault tolerance mechanisms
• Built-in locality optimization
• Provides transparent load balancing

This enables even programmers without distributed systems experience to easily write efficient distributed programs.

2. Powerful Expressiveness

The MapReduce model can easily express various types of computational problems, widely applied within Google for:

• Web search service data generation
• Large-scale sorting
• Data mining
• Machine learning
• Many other systems

This versatility makes MapReduce a fundamental computing framework within Google.

3. Excellent Scalability

MapReduce implementation can scale to large clusters containing thousands of machines:

// Pseudo-code: MapReduce scheduling process
func Schedule(input []string, mappers int, reducers int) Result {
    // Automatically handles:
    // 1. Task allocation and parallelization
    // 2. Machine failure detection and recovery
    // 3. Data locality optimization
    // 4. Intermediate result management
}

This enables efficient handling of large-scale computational problems encountered by Google, laying the foundation for big data processing.

Three Key Insights from Research Team

1. Value of Restricted Programming Models

Research shows that by consciously restricting the programming model, enormous system advantages can be gained:

• Easy parallelization and distributed computing
• Natural implementation of fault tolerance mechanisms
• Reduced development and maintenance costs

This "less is more" philosophy contrasts sharply with other systems attempting to provide completely general parallel programming environments.

2. Network Bandwidth is a Scarce Resource

The research team found network bandwidth is a precious resource in distributed systems, so many optimizations target reducing network transmission:

• Locality optimization: Prioritize reading data from local disks, reducing cross-network data transmission
• Local intermediate data storage: Write intermediate results to local disks rather than distributed storage, saving network bandwidth

These designs are particularly important in large-scale clusters, where data transmission can become system bottlenecks.

3. Importance of Redundant Execution

Redundant execution is a key innovation in MapReduce, used for:

• Reducing impact of slow machines (stragglers)
• Gracefully handling machine failures
• Preventing data loss

// Pseudo-code: Redundant task scheduling in MapReduce
func scheduleBackupTasks(slowTasks []Task) {
    for _, task := range slowTasks {
        if time.Now() - task.StartTime > slowThreshold {
            // Launch backup copy of same task on another machine
            launchDuplicateTask(task)
        }
    }
}

This mechanism significantly improves reliability and performance consistency of large distributed systems.

Technical Legacy of MapReduce

The paper's conclusions reveal that MapReduce is not just a technical innovation, but a new paradigm for large-scale data processing:

1. Architectural Influence: MapReduce design philosophy influenced later big data processing frameworks like Hadoop, Spark
2. Programming Model Innovation: Proved that simplified programming models can solve complex distributed computing problems
3. Engineering Practice Revolution: Changed methods for building large-scale data processing systems, from expert systems to general frameworks
4. Commercial Value Creation: Laid foundation for later big data ecosystems, creating enormous commercial value

In summary, MapReduce successfully solved core challenges of large-scale distributed data processing through simple yet powerful abstractions, making massive data processing accessible. This is why it achieved tremendous success both within Google and across the industry.

Overall Architectural Design Principles

In high-concurrency API management scenarios, gateway architecture must adhere to the following core design principles:

1. Layered Architecture Design

   Traffic Ingress Layer → Processing Layer → Routing Layer → Backend Service Layer
   

   Adopt a strictly layered architecture with separation of concerns at each layer, making the system more scalable horizontally and optimizable vertically.

2. Stateless Design

   Design stateless gateway nodes to ensure any gateway instance can handle any request - this is fundamental to supporting high concurrency. Session state and user information should be stored in distributed caches or dedicated state storage systems.

High-Performance Technology Stack Selection

1. Core Technology Selection

   1. Data Plane: Based on high-performance proxies like Envoy, NGINX, or custom components built with Go/Rust
   2. Control Plane: Employ efficient configuration management and service discovery mechanisms

2. Asynchronous I/O Model

   1. Adopt non-blocking I/O models (such as Go's goroutine+channel, Rust's tokio, Node.js event loop)
   2. Avoid traditional thread pool models to reduce context switching overhead

Multi-Level Caching Architecture

1. Global Distributed Cache Layer

   Client → CDN → Edge Cache → API Gateway Local Cache → Service Cache
   

2. Multi-Dimensional Caching Strategy

   1. Routing Information Cache: Local high-speed cache with periodic updates
   2. Authentication Information Cache: Distributed token validation result caching
   3. Response Data Cache: Intelligent caching strategy based on content characteristics

Dynamic Scaling Design

1. Flexible Deployment Architecture

   Multi-Region → Multi-AZ → Multi-Cluster → Multi-Instance
   

2. Elastic Scaling Strategy

   1. Predictive Scaling: Predict scaling needs based on historical traffic patterns
   2. Reactive Scaling: Trigger scaling based on real-time metrics (CPU, memory, request queue depth)
   3. Graceful Scale-Down: Ensure graceful connection closure and request completion processing

Efficient Traffic Control Mechanisms

1. Multi-Level Rate Limiting Design

   flowchart LR
       Client --> GlobalLimit["Global Rate Limiting Layer"]
       GlobalLimit --> ServiceLimit["Service-Level Rate Limiting"]
       ServiceLimit --> APILimit["API-Level Rate Limiting"]
       APILimit --> UserLimit["User-Level Rate Limiting"]
       UserLimit --> Backend["Backend Services"]
   

2. Adaptive Flow Control Algorithms

   1. Token Bucket + Leaky Bucket Combination: Balance burst traffic and steady traffic
   2. Priority-Based Differential Processing: Prioritize core API protection
   3. Adaptive Rate Limiting: Dynamically adjust rate limiting thresholds based on backend service health

Gateway Cluster High Availability Design

1. Multi-Region Deployment Architecture

   1. Geographic-Level Redundancy: Cross-region deployment ensures regional-level fault isolation
   2. Proximity Access: Intelligent DNS or global load balancing for traffic proximity routing

2. Fault Isolation Strategy

   Client Grouping → Gateway Instance Grouping → Backend Service Grouping
   

   1. Bulkhead Pattern: Isolate client requests to different gateway instance groups
   2. Circuit Breaker Mechanism: Intelligent circuit breaker design based on multi-dimensional metrics like error rates and latency
   3. Degradation Strategy: Define clear service degradation paths and fallback mechanisms

Request Processing Optimization

1. Request Processing Pipeline

   Request Reception → Authentication & Authorization → Request Transformation → 
   Routing Decision → Load Balancing → Backend Invocation → Response Processing
   

2. Performance Optimization Techniques

   1. Batch Processing: Merge fragmented requests to reduce network round trips
   2. Request Collapsing: Merge concurrent requests for the same resource
   3. Parallel Processing: Parallelize cross-service request processing
   4. Streaming Response Processing: Stream transmission for large responses
   5. Zero-Copy Technology: Reduce data copying stages

Efficient Communication Protocols

1. Protocol Support and Optimization

   1. HTTP/2 Multiplexing: Reduce connection establishment overhead
   2. gRPC Support: Efficient binary transmission and stream processing
   3. WebSocket Optimization: Long connection management and heartbeat mechanisms

2. Connection Pool Management

   1. Dynamically adjustable backend connection pools
   2. Long connection reuse and keep-alive strategies
   3. Connection warm-up mechanisms to avoid cold start latency

End-to-End Observability

1. Multi-Dimensional Monitoring System

   Infrastructure Metrics → Gateway Performance Metrics → API Call Metrics → Business Metrics
   

2. Real-Time Monitoring and Alerting

   1. Health Checks: Combination of active and passive health detection
   2. Performance Analysis: Key metrics like request latency distribution and queue depth
   3. Anomaly Detection: Machine learning-based abnormal behavior identification

Dynamic Configuration Update Mechanism

1. Dynamic Configuration Architecture

   1. Distributed configuration center + local cache
   2. Configuration change event notification mechanism
   3. Incremental configuration updates to reduce resource consumption

2. Canary Release Capabilities

   1. Canary deployment for configuration changes
   2. Smooth traffic migration switching
   3. Emergency rollback mechanisms

While browsing through technical articles, I came across this insightful piece on microservices architecture, which resonated with our company's actual business scenarios.

Challenges at Scale

The article discusses how client applications access microservices in a distributed architecture. The main challenges include:

1. API Granularity Mismatch: Microservices typically expose fine-grained APIs, while clients need aggregated data, forcing multiple service interactions
2. Client Requirement Diversity: Different clients (desktop browsers, mobile apps, third-party applications) require different data structures and payloads
3. Network Performance Variations: Mobile networks are typically slower with higher latency compared to internal networks, impacting user experience
4. Dynamic Service Instance Changes: Service instances and their locations (host + port) change dynamically
5. Protocol Heterogeneity: Backend services may use different protocols, some of which are not web-friendly

I'll skip the detailed explanations here - feel free to reach out for deeper technical discussions.

Solution Architecture

The author proposes implementing the API Gateway pattern as the solution:

1. Unified Entry Point: Design an API gateway as the single entry point for all clients
2. Request Processing Strategies:
   • Simple proxy/routing to corresponding services
   • Fan-out requests to multiple services with result aggregation
3. Client-Specific APIs: Provide customized APIs for different clients rather than one-size-fits-all solutions
4. Cross-Cutting Concerns: Implement authentication, authorization, SSL termination, caching, and other shared functionalities

The article also introduces a pattern variant: Backends for Frontends (BFF), which involves designing dedicated API gateways for each client type (web applications, mobile apps, third-party applications).

Personal Analysis

Advantages

Centralized Management: All API traffic flows through a single entry point, enabling unified monitoring, tracing, and governance

A financial institution reduced security incident response time from hours to minutes 
after implementation, as all suspicious traffic could be quickly identified at the gateway layer.

Unified Cross-Cutting Concerns: Authentication, authorization, rate limiting, and other functionalities need to be implemented only once at the gateway layer

Protocol Translation & Aggregation: Convert internal protocols to unified external interfaces, reducing client request counts. An e-commerce platform reduced mobile app loading time by 68% by aggregating 7 independent calls into a single API

Decoupling Implementation: Clients are decoupled from internal microservice structure, allowing services to evolve independently without affecting clients

Traffic Control: Provides intelligent routing, load balancing, and circuit breaker capabilities, enhancing system resilience

These benefits are well-established in the industry.

Disadvantages

Potential Single Point of Failure: All traffic depends on a single component, which could become a system bottleneck if not properly designed - though this depends on the distributed architecture team's capabilities

Increased Latency: Additional network hops may increase response times (typically 5-10ms)

Complexity Escalation: As scale increases, the gateway may become bloated and difficult to maintain

A large enterprise's API gateway accumulated over 200 custom processing rules, eventually reaching a point where no one dared to modify the codebase.

Team Collaboration Challenges: Gateway changes require cross-team coordination, potentially creating development bottlenecks

Applicable Scenarios

1. Small to Medium Microservices Architecture: Moderate service count (10-50 services) with limited client types
2. High-Security Systems: Financial, healthcare, and other domains requiring unified security policies
3. Mixed Protocol Environments: Internal services using different protocols (REST, gRPC, AMQP, etc.)
4. Centralized API Governance: Enterprise applications requiring centralized API governance strategies

Core Functionality & Implementation Effects

The author outlines three core functions provided by API gateways:

1. Reverse Proxy/Gateway Routing: Using Layer 7 routing to redirect HTTP requests
2. Request Aggregation: Aggregating multiple internal microservice requests into single client requests
3. Cross-Cutting Concerns & Gateway Offloading: Centralized implementation of common functionalities

Benefits

1. Client-Microservice Decoupling: Isolates clients from microservice architecture details
2. Optimal API Provision: Provides customized APIs for various client types
3. Reduced Request Count: Single requests retrieve multi-service data, reducing network overhead
4. Simplified Client Logic: Complex call logic migrates from client to API gateway
5. Protocol Translation: Converts public web-friendly API protocols to internal protocols

Drawbacks

1. Increased Complexity: Introduces a new component requiring development, deployment, and management
2. Increased Response Time: Additional network hops may increase latency (minimal impact for most applications)
3. Single Point of Failure Risk: A single API gateway may become a bottleneck or single point of failure

These align with my initial assessment.

Production Deployment Strategies

┌───────────┐  ┌───────────┐  ┌─────────────┐
│  Web BFF  │  │ Mobile BFF│  │ Third-party │
│           │  │           │  │   BFF       │
└─────┬─────┘  └─────┬─────┘  └─────┬───────┘
      │              │              │
      ▼              ▼              ▼
┌─────────────────────────────────────┐
│        Core API Gateway             │
│ (Auth, Monitoring, Rate Limiting,   │
│      Basic Routing)                 │
└─────────────────────────────────────┘
      │              │              │
      ▼              ▼              ▼
┌─────────┐     ┌───────────┐     ┌───────────┐
│Service  │     │Service    │     │Service    │
│Cluster 1│     │Cluster 2  │     │Cluster 3  │
└─────────┘     └───────────┘     └───────────┘

Scale Considerations

1. Small Applications (5-15 microservices):
   • Recommendation: Single API gateway
   • Rationale: Simple and efficient, avoiding over-engineering
2. Medium Applications (15-50 microservices):
   • Recommendation: Basic API gateway + key BFFs
   • Rationale: Balances complexity and client optimization needs
3. Large Applications (50+ microservices):
   • Recommendation: Complete three-tier architecture (Edge Gateway + BFF Layer + Internal Gateway)
   • Rationale: Supports organizational scaling and complex business domains

Decision Factors

When selecting API gateway strategies, consider:

1. Organizational Structure: Conway's Law suggests system design often reflects organizational structure
2. Scaling Expectations: Consider business growth and diversification over 3-5 years
3. Operational Capabilities: Assess team capacity for managing multiple gateways
4. Consistency Requirements: Business requirements for API behavior consistency
5. Performance Budget: Evaluate latency impact of additional network hops

Regardless of the chosen pattern, maintain lightweight gateway layers, avoid excessive business logic drift leading to "smart pipes" anti-patterns, and establish robust monitoring systems to ensure gateway layer performance and stability.

2025-03-07 Update: Recently, Xiaohongshu's gateway practices follow many of the principles below. —— 《Xiaohongshu Launches Self-Developed Rust High-Performance Layer 7 Gateway ROFF》

This article is divided into two parts:

1. Performance Bottleneck Analysis Methods
2. Hands-on Analysis of APISIX, Kong, and Nginx

Performance Bottleneck Analysis Methods

Performance Bottleneck Analysis Methods (The Real Deal)

This section covers the actual methods.

Monitoring Metrics Analysis

• Latency Metrics: Monitor request latency and integration latency
• Error Rate Metrics: Monitor 4XX and 5XX error rates to identify system failure points
• Throughput Metrics: Request count and requests per second (RPS)
• Cache Hit Metrics: Monitor CacheHitCount and CacheMissCount ratios

System Resource Monitoring

• CPU utilization analysis
• Memory consumption analysis
• Network I/O monitoring
• Disk I/O monitoring (especially for logs and cache)

Request Flow Analysis

• Request processing chain tracing
• Hot path identification
• Serial processing bottleneck discovery

Infrastructure-Level Optimization

Hardware Resource Optimization

• Increase CPU cores and memory capacity
• Use SSD storage to improve I/O speed
• Deploy high-performance network cards to reduce network latency
• Properly evaluate and adjust container resource limits (in Kubernetes environments)

Network Optimization

• Optimize network topology structure
• Reduce network hops
• Use CDN to accelerate static content
• Implement DNS optimization
• Deploy gateways and backend services in the same network zone

Operating System Tuning

• Adjust TCP/IP stack parameters (such as increasing connection queues)
• Optimize file descriptor limits
• Adjust kernel parameters (such as somaxconn, tcpfintimeout)
• Optimize client thread count in Linux environments

Gateway Configuration-Level Optimization

Connection Pool Optimization

• Configure appropriate database connection pool sizes (recommended to match expected client count) —— Oracle Database Administrator Essentials
• Adjust backend service connection pool parameters
• Set connection timeout and retry strategies
• Implement connection keep-alive mechanisms

HTTP Optimization

• Enable HTTP keep-alive to reuse TCP connections —— Oracle Performance Practices
• Configure appropriate chunked encoding strategies
• Adjust HTTP header size limits
• Set reasonable request/response timeout values

Caching Strategy Optimization

• Implement response caching to reduce backend calls —— Amazon API Optimization Practices
• Configure cache key strategies and TTL (Time To Live)
• Implement multi-level caching strategies
• Optimize cache validation and invalidation strategies

Request Processing Optimization

Concurrent Processing Optimization

• Increase worker thread count
• Implement asynchronous processing patterns
• Optimize thread pool configuration
• Use event-driven architecture for high-concurrency request handling

Load Balancing Strategies

• Implement intelligent load balancing algorithms (weighted round-robin, least connections, etc.)
• Configure dynamic load balancing strategies
• Implement service health checks and automatic failover
• Adjust weights based on backend service capacity

Routing Optimization

• Optimize routing lookup algorithms
• Implement routing caching
• Configure routing warm-up strategies
• Implement intelligent routing based on traffic characteristics

Data Processing Optimization

Message Processing Optimization

• Configure "spill to disk" strategies for large messages (e.g., set >4MB messages to write to disk) 3
• Implement request/response compression 1
• Optimize serialization/deserialization processes
• Reduce unnecessary data transformations

Protocol Optimization

• Use HTTP/2 to reduce latency and improve parallel processing capability
• Use WebSocket and other long-connection protocols in appropriate scenarios
• Consider using gRPC to improve microservice communication efficiency
• Implement protocol upgrade strategies

Monitoring and Logging Optimization

Log Optimization

• Reduce unnecessary trace information (set production environment to ERROR or FATAL level) 3
• Disable or reduce access log recording 3
• Disable transaction logs to reduce disk I/O burden 3
• Implement asynchronous log writing strategies

Monitoring Optimization

• Disable real-time monitoring to reduce overhead 3
• Disable or reduce traffic monitoring 3
• Configure reasonable monitoring sampling rates
• Implement intelligent alerting thresholds to avoid excessive monitoring system load

Security Performance Optimization

Authentication Optimization

• Cache authentication results
• Use lightweight token validation
• Implement tiered authentication strategies
• Optimize JWT processing workflows

SSL/TLS Optimization

• Use session reuse to reduce handshake overhead
• Configure OCSP stapling
• Implement TLS connection pools
• Select efficient cipher suites

Advanced Optimization Strategies

Circuit Breaking and Rate Limiting

• Implement request rate limiting to protect backend systems
• Configure circuit breakers to prevent system overload
• Implement backoff algorithms for retry handling
• Traffic shaping to optimize request distribution

Service Mesh Integration

• Integrate with service meshes like Istio/Envoy
• Delegate some gateway functions to sidecar proxies
• Implement collaborative strategies between gateways and service meshes
• Leverage traffic management capabilities provided by service meshes

Architectural Optimization

• Consider multi-tier gateway architecture (edge gateway + internal gateway)
• Implement edge computing to reduce latency
• Domain-driven API gateway design
• Gateway sharding based on traffic characteristics

By systematically analyzing performance bottlenecks in the above aspects and applying corresponding optimization techniques, gateway performance, throughput, and reliability can be significantly improved. The key is to select the most suitable combination of optimization strategies based on actual system characteristics and business requirements, and maintain high system performance through continuous monitoring and tuning.

Hands-on Analysis

Nginx Gateway

Main Performance Bottlenecks

1. Connection Handling Capacity Bottlenecks

   • Default worker process configuration may not match server CPU cores
   • Connection pool size limits cause connection rejection under high concurrency
   • File descriptor limits cause "too many open files" errors

2. Configuration Complexity Bottlenecks

   • Static configuration files require manual modification and reloading
   • Large-scale routing rules make configuration maintenance difficult
   • Configuration changes require reloading, potentially causing request interruptions

3. SSL Processing Bottlenecks

   • High SSL handshake overhead, CPU usage spikes under high concurrency
   • Inefficient key exchange algorithms
   • Improper session cache configuration causes repeated handshakes

Optimization Methods

1. Worker Process and Connection Optimization

   • Set worker_processes to match CPU core count
   • Increase worker_connections value (typically set to 4096 or higher)
   • Use worker_cpu_affinity to bind worker processes to specific CPU cores
   • Adjust system file descriptor limits (ulimit -n)

2. Event Processing Optimization

   • Enable multi_accept and accept_mutex
   • Use epoll event handling model (on Linux systems)
   • Adjust worker_aio_requests to improve async I/O performance

3. HTTP Optimization

   • Configure keepalive_timeout and keepalive_requests parameters
   • Enable sendfile, tcp_nopush, and tcp_nodelay options
   • Implement gzip compression to reduce transmitted data
   • Set client_body_buffer_size and client_max_body_size to limit request size

4. SSL Performance Optimization

   • Enable ssl_session_cache shared to improve session reuse rate
   • Configure OCSP stapling to reduce handshake latency
   • Use ECC certificates to reduce computational overhead
   • Prioritize high-performance cipher suites (like AES-GCM)

Kong Gateway

A reliable and well-established gateway.

Main Performance Bottlenecks

1. Database Dependency Bottlenecks

   • PostgreSQL/Cassandra database queries become performance bottlenecks
   • Database pressure increases during configuration changes
   • Database consistency challenges in distributed deployments

2. Lua Script Processing Bottlenecks

   • Overly long plugin execution chains increase request latency 1
   • Improper Lua VM memory usage causes performance degradation
   • JIT compilation limitations affect dynamic script performance

3. JWT Validation Bottlenecks

   • High JWT validation processing overhead, noticeable in high-percentile latency 2
   • Kong's 99.99 percentile latency can reach 3 times that of NGINX 2

Optimization Methods

1. Database Optimization

   • Use DB-less mode to reduce database dependencies
   • Increase database connection pool size (pg_pool parameter in kong.conf)
   • Implement database read-write separation (master-slave architecture)
   • Consider using declarative configuration instead of database storage

2. Plugin Chain Optimization

   • Enable only necessary plugins to reduce processing chain length
   • Adjust plugin execution order (lightweight, high-frequency plugins first)
   • Configure independent caches for plugins (like Redis cache for rate-limiting plugin)
   • Monitor and optimize long-running plugins

3. Cache Optimization

   • Configure lua_shared_dict cache size
   • Adjust plugin-level cache TTL
   • Use external Redis cache to improve hit rates
   • Configure entity cache to reduce database queries

4. Connection Pool Optimization

   • Adjust upstream_keepalive parameter (typically 100-200)
   • Increase nginxupstreamkeepalive_timeout value
   • Set reasonable nginxupstreamkeepalive_requests value
   • Increase nginxhttpclientbodybuffer_size for large request bodies

APISIX Gateway

Main Performance Bottlenecks

1. etcd Dependency Bottlenecks

   • etcd cluster stability affects gateway configuration propagation
   • Frequent configuration changes increase etcd pressure
   • etcd read-write latency affects dynamic routing updates

2. Route Matching Bottlenecks

   • Large numbers of fine-grained routes increase matching latency
   • Complex regex routes reduce matching efficiency
   • Untimely route cache updates cause routing errors

Optimization Methods

1. etcd Optimization

   • Build highly available etcd clusters
   • Optimize etcd configuration (like setting reasonable heartbeat intervals)
   • Implement etcd sharding to reduce single-node pressure
   • Increase config_center.timeout parameter value (default 30 seconds)

2. Route Optimization

   • Use prefix matching instead of full regex
   • Increase route cache TTL
   • Reduce route rule complexity, split overly complex rules
   • Use domain or hostname pre-filtering

Envoy Gateway

Main Performance Bottlenecks

1. xDS Configuration Update Bottlenecks

   • Dynamic configuration updates cause resource reallocation overhead
   • Control Plane communication latency affects configuration delivery
   • Large numbers of listener and cluster configurations cause high memory usage

2. Filter Chain Processing Bottlenecks

   • Long HTTP filter chains increase processing latency
   • Complex filter logic causes high CPU usage
   • Lua filters are less efficient than native filters

Optimization Methods

1. xDS Configuration Optimization

   • Implement incremental xDS to reduce configuration update overhead
   • Optimize Control Plane communication (use gRPC streams instead of polling)
   • Set reasonable configuration cache TTL
   • Use Aggregated Discovery Service (ADS) to ensure configuration consistency

2. Filter Optimization

   • Reduce filter chain length, keep only necessary filters
   • Prioritize native C++ filters over Lua or WASM
   • Adjust filter execution order (high-frequency filters first)
   • Enable statistical monitoring for critical filters

Performance Comparison and Selection Recommendations for Different Gateways

Performance Comparison

• In standard API call tests, NGINX API Management Module performance can be 2x better than Kong Data Support
• In terms of latency, NGINX adds 20-30% lower latency than Kong Data Support
• In CPU efficiency, NGINX is about 40% more efficient than Kong Data Support
• In JWT validation scenarios, NGINX can handle 2x more API calls than Kong Data Support

Selection Recommendations

1. NGINX Suitable Scenarios:

   • Stable API gateway needs with static routing configuration
   • Scenarios prioritizing performance and low latency
   • Lightweight gateway needs in resource-constrained environments
   • Primarily providing reverse proxy and load balancing functions

2. Kong Suitable Scenarios:

   • Need rich API management features (authentication, rate limiting, transformation, etc.)
   • Teams pursuing development convenience (RESTful API configuration)
   • Microservice architectures with dynamic routing needs
   • Can accept some performance loss for feature richness

3. APISIX Suitable Scenarios:

   • Teams pursuing balance between dynamic routing and high performance
   • Cloud-native architectures with service discovery integration needs
   • Scenarios requiring fine-grained traffic control

4. Envoy Suitable Scenarios:

   • Kubernetes/Istio service mesh infrastructure
   • Modern cloud architectures requiring advanced observability
   • DevOps teams pursuing programmability and extensibility

By understanding the performance bottleneck characteristics and optimization methods of different gateways, you can choose the most suitable gateway type based on your business characteristics and technology stack, and implement targeted performance optimization to achieve the best balance of gateway performance and functionality.

Summary

Gateway performance optimization is systematic work that requires analysis and optimization from multiple levels. Through the analysis methods, optimization techniques, and code examples provided above, you can implement targeted optimization for different performance bottlenecks. The key points are:

1. Establish comprehensive performance monitoring systems to detect performance bottlenecks promptly 1
2. Adopt multi-level caching strategies to reduce redundant calculations and network requests
3. Optimize connection pool management to improve connection reuse efficiency
4. Implement efficient load balancing algorithms for intelligent request distribution
5. Use regular benchmark testing to continuously evaluate and optimize performance

For gateways in cloud-native environments, you can also consider leveraging Kubernetes' auto-scaling capabilities to dynamically adjust gateway instance counts in response to traffic changes.

Overall Architecture Overview

Thanks to Zhang Yanfei for the diagram.

After you basically understand what network card drivers, hardware interrupts, software interrupts, and ksoftirqd threads are, you can provide a kernel packet reception path diagram as shown above.

The general process is as follows:

1. When the network card receives data, it writes the received frames to memory using DMA, then sends an interrupt to the CPU to notify that data has arrived.
2. When the CPU receives the interrupt request, it calls the interrupt handler registered by the network device driver.
3. The network card's interrupt handler doesn't do much work - it issues a software interrupt request and quickly releases CPU resources.
4. The ksoftirqd kernel thread detects the software interrupt request, calls poll to start polling for packet reception, and forwards received packets to various protocol stack layers for processing. For TCP packets, they are placed in the user socket's receive queue.

Foundation Work Before Everything Else

Before Linux drivers, kernel protocol stacks, and other modules can receive network card data packets, extensive preparation work must be done:

1. Pre-create ksoftirqd kernel threads;
2. Register processing functions for various protocols;
3. Pre-initialize the network device subsystem;
4. Start up the network card.

Initialization Work

Creating ksoftirqd Kernel Threads

All Linux software interrupts are handled in dedicated kernel threads (ksoftirqd), so it's essential to understand how these threads are initialized.

First, there isn't just one thread, but N threads, where N equals the number of cores on your machine.

During system initialization, the smpboot_register_percpu_thread function is called in kernel/smpboot.c, which further executes spawn_ksoftirqd (located in kernel/softirq.c) to create softirqd threads, as shown below:

Related code:

static struct smp_hotplug_thread softirq_threads = {
    .store            = &ksoftirqd,
    .thread_should_run    = ksoftirqd_should_run,
    .thread_fn        = run_ksoftirqd,
    .thread_comm        = "ksoftirqd/%u",
};

static __init int spawn_ksoftirqd(void)
{
    cpuhp_setup_state_nocalls(CPUHP_SOFTIRQ_DEAD, "softirq:dead", NULL,
                  takeover_tasklets);
    BUG_ON(smpboot_register_percpu_thread(&softirq_threads));

    return 0;
}
early_initcall(spawn_ksoftirqd);

After ksoftirqd is created, it enters its thread loop functions ksoftirqdshouldrun and run_ksoftirqd, then checks if there are any software interrupts to process.

Software interrupts include not only network software interrupts but other types as well. The Linux kernel defines all software interrupt types in interrupt.h:

// file: include/linux/interrupt.h
enum
{
    HI_SOFTIRQ=0,
    TIMER_SOFTIRQ,
    NET_TX_SOFTIRQ,
    NET_RX_SOFTIRQ,
    BLOCK_SOFTIRQ,
    IRQ_POLL_SOFTIRQ,
    TASKLET_SOFTIRQ,
    SCHED_SOFTIRQ,
    HRTIMER_SOFTIRQ,
    RCU_SOFTIRQ,    /* Preferable RCU should always be the last softirq */

    NR_SOFTIRQS
};

Network Subsystem Initialization

During network subsystem initialization, softnetdata is initialized for each CPU, and processing functions are registered for RXSOFTIRQ and TX_SOFTIRQ, as shown in Figure 2.4.

The Linux kernel initializes various subsystems by calling subsys_initcall.

Key Point!!! The network subsystem initialization mentioned here executes the netdevinit function.

This is the netdevinit function in subsysinitcall(netdev_init). Code as follows:

static int __init net_dev_init(void)
{
    ......

    /*
     *    Initialise the packet receive queues.
     */

  /*
   * Allocate a softnet_data structure for each CPU. The poll_list in this structure
   * is used to wait for driver programs to register their poll functions, which can
   * be seen later when network card drivers initialize.
   */
        for_each_possible_cpu(i) {
        struct softnet_data *sd = &per_cpu(softnet_data, i);

        memset(sd, 0, sizeof(*sd));
        skb_queue_head_init(&sd->input_pkt_queue);
        skb_queue_head_init(&sd->process_queue);
        sd->completion_queue = NULL;
        INIT_LIST_HEAD(&sd->poll_list);
    ......
  }
  ......
    /*
     * open_softirq registers a processing function for each type of software interrupt.
     * NET_TX_SOFTIRQ processing function is net_tx_action;
     * NET_RX_SOFTIRQ processing function is net_rx_action;
     */
  open_softirq(NET_TX_SOFTIRQ, net_tx_action);
    open_softirq(NET_RX_SOFTIRQ, net_rx_action);
}

subsys_initcall(net_dev_init);

Following opensoftirq further reveals that this registration method is recorded in the softirqvec variable. Later, when the softirqd thread receives software interrupts, it will use this variable to find the corresponding processing function for each type of software interrupt.

void open_softirq(int nr, void (*action)(struct softirq_action *))
{
    softirq_vec[nr].action = action;
}

Protocol Stack Registration

Network Card Driver Initialization

Each driver uses module_init to register an initialization function with the kernel. When the driver is loaded, the kernel calls this function.

After completion, the Linux kernel knows the driver's relevant information, such as the igb network card driver's igbdrivername and igb_probe function address.

When a network card device is identified, the kernel calls its driver's probe method. (Continuing with the igb network card driver example), the igbdriver's probe method is igbprobe.

The purpose of the igb_probe method is to get the device into a ready state as quickly as possible.

Additionally, there's a critical step: registering the poll function required by the NAPI mechanism, which for the igb network card driver is igb_poll.

After Initialization Completion

Starting the Network Card

After all the above initialization is complete, the network card can be started. The general startup sequence is similar, as shown below:

igb_open code:

static int __igb_open(struct net_device *netdev, bool resuming)
{
  // Allocate transmit descriptor arrays
  err = igb_setup_all_tx_resources(adapter);
  // Allocate receive descriptor arrays
    err = igb_setup_all_rx_resources(adapter);

  // Register interrupt handler
  err = igb_request_irq(adapter);
    if (err)
        goto err_req_irq;

  // Enable NAPI
      for (i = 0; i < adapter->num_q_vectors; i++)
        napi_enable(&(adapter->q_vector[i]->napi));
    ......
}

The igbopen function calls igbsetupalltxresources and igbsetupallrxresources. In the igbsetupallrx_resources step, RingBuffer is allocated and the mapping relationship between memory and Rx queues is established.

static int igb_setup_all_rx_resources(struct igb_adapter *adapter)
{
    ......

    for (i = 0; i < adapter->num_rx_queues; i++) {
        err = igb_setup_rx_resources(adapter->rx_ring[i]);
        ...
    }

    return err;
}

Using a for loop with the igbsetuprxresources function, several queues are created. The igbsetuprxresources function:

int igb_setup_rx_resources(struct igb_ring *rx_ring)
{
    struct device *dev = rx_ring->dev;
    int size;

    // 1. Allocate igb_rx_buffer array memory
    size = sizeof(struct igb_rx_buffer) * rx_ring->count;

    rx_ring->rx_buffer_info = vzalloc(size);
    if (!rx_ring->rx_buffer_info)
        goto err;

    /* Round up to nearest 4K */
    // 2. Allocate e1000_adv_rx_desc DMA array memory
    rx_ring->size = rx_ring->count * sizeof(union e1000_adv_rx_desc);
    rx_ring->size = ALIGN(rx_ring->size, 4096);

    rx_ring->desc = dma_alloc_coherent(dev, rx_ring->size,
                       &rx_ring->dma, GFP_KERNEL);
    if (!rx_ring->desc)
        goto err;

    // 3. Initialize queue members
    rx_ring->next_to_alloc = 0;
    rx_ring->next_to_clean = 0;
    rx_ring->next_to_use = 0;

    return 0;

err:
    vfree(rx_ring->rx_buffer_info);
    rx_ring->rx_buffer_info = NULL;
    dev_err(dev, "Unable to allocate memory for the Rx descriptor ring\n");
    return -ENOMEM;
}

From the above source code, you can see that internally, a RingBuffer doesn't have just one circular queue array, but two:

1. igbrxbuffer array: This array is used by the kernel, allocated via vzalloc;
2. e1000advrxdesc array: This array is used by network card hardware, allocated via dmaalloc_coherent.

Then there's the final step of interrupt function registration, which can be seen in igbrequestirq.

OK, that's all the preparation work! Next comes receiving data packets.

Starting to Receive Data Packets

This section includes hardware interrupt processing.

Hardware Interrupt Processing

First, when data frames arrive at the network card from the network cable, the first stop is the network card's receive queue. The network card searches for available memory locations in its allocated RingBuffer, and when found, the DMA engine will DMA the data to the memory previously associated with the network card. At this point, the CPU is unaware.

When the DMA operation completes, the network card sends a hardware interrupt to the CPU, notifying it that data has arrived. The hardware interrupt processing is as follows:

In the "Starting the Network Card" section mentioned earlier, the network card's hardware interrupt registered processing function is igbmsixring.

// file: drivers/net/ethernet/intel/igb/igb_main.c
static irqreturn_t igb_msix_ring(int irq, void *data)
{
    struct igb_q_vector *q_vector = data;

    /* Write the ITR value calculated from the previous interrupt. */
    igb_write_itr(q_vector);

    napi_schedule(&q_vector->napi);

    return IRQ_HANDLED;
}

The igbwriteitr only records hardware interrupt frequency. Following the napischedule call, you'll discover that Linux only completes simple necessary work in hardware interrupts, leaving most processing to software interrupts. From the above code, you can see that the hardware interrupt processing is really very short - it just records a register, modifies the CPU's polllist, and then issues a software interrupt. That's it, the hardware interrupt work is complete.

ksoftirqd Kernel Thread Processing Software Interrupts

Network packet reception processing mainly occurs in the ksoftirqd kernel thread, where all software interrupts are processed, as shown below:

Network Protocol Stack Processing

The netifreceiveskb function processes packets according to their protocol. For UDP packets, packets are sent sequentially to protocol processing functions like iprcv, udprcv, etc., as shown below:

IP Layer Processing

Linux IP layer operations are in the code file net/ipv4/ip_input.c.

Summary

The network module is the most complex module in the Linux kernel. The entire process involves interactions between many kernel components, such as network card drivers, protocol stacks, kernel ksoftirqd threads, etc. It looks complex, but the overall picture is actually quite clear. Simple summary as follows.

After a user executes a recvfrom call, the user process enters kernel mode through the system call. If the receive queue has no data, the process enters sleep state and is suspended by the operating system. This part is relatively simple. Next comes the work between various Linux kernel components.

First, before starting packet reception, Linux must do extensive preparation work:

• Create ksoftirqd kernel threads, set up their thread functions, and rely on them to handle software interrupts later;
• Protocol stack registration: Linux implements many protocols like ARP, ICMP, IP, UDP, and TCP. Each protocol registers its processing function, making it convenient to quickly find the corresponding processing function when packets arrive;
• Network card driver initialization: Each driver has an initialization function that the kernel will initialize. During this initialization process, prepare DMA and tell the kernel the NAPI poll function address;
• Start the network card: Allocate RX and TX queues, register interrupt corresponding processing functions.

After preparation work is complete, data arrives. The first to greet it is the network card:

• The network card DMAs data frames to memory's RingBuffer, then sends an interrupt notification to the CPU;
• CPU responds to the interrupt request, calls the interrupt processing function registered when the network card started;
• The interrupt processing function only issues a software interrupt request and does nothing else;
• Kernel thread ksoftirqd discovers a software interrupt request has arrived, first disables hardware interrupts;
• ksoftirqd thread starts calling the driver's poll function to receive packets;
• The poll function sends received packets to the protocol stack's registered ip_rcv function;
• The iprcv function sends packets to the udprcv function (for TCP packets, sent to tcprcvv4).

Some Conclusions

Question 1: What exactly is RingBuffer, and why does RingBuffer drop packets?

RingBuffer is a special area in memory, a circular queue array. In fact, this data structure includes igbrxbuffer circular queue arrays, e1000advrx_desc circular queue arrays, and numerous skbs.

If RingBuffer represents pointer arrays, they are pre-allocated. If they are skbs, they are dynamically allocated during the packet reception process.

Question 2: What are software interrupts and hardware interrupts respectively?

Key flow of data packet reception in Linux network stack:

1. Hardware Stage: Network card places received data packets into RingBuffer
2. Hardware Interrupt Trigger: Network card generates hardware interrupt to notify CPU
3. Hardware Interrupt Processing: Add network card device to poll_list doubly linked list in softnet_data structure
4. Software Interrupt Trigger: Trigger NET_RX_SOFTIRQ software interrupt
5. Software Interrupt Processing: Traverse poll_list, execute network card driver's poll function to collect network packets
6. Protocol Stack Processing: Forward data packets to protocol processing functions like ip_rcv, udp_rcv, tcp_rcv_v4

This describes the Linux NAPI (New API) mechanism, an efficient method for handling network data packets.

Actual Application of RingBuffer in Network Stack

Network Card RX/TX Ring Buffers

/* Simplified network card RX Ring structure */
struct e1000_rx_desc {
    __le64 buffer_addr;    /* Data buffer address */
    __le16 length;         /* Packet length */
    __le16 checksum;       /* Checksum */
    __u8  status;          /* Descriptor status */
    __u8  errors;          /* Error code */
    __le16 special;
};

In practice, an Intel network card's RX Ring might contain 256 such descriptors, forming a circular structure.

Actual Example of Hardware and Software Interrupt Cooperation

In Intel 82599 network card driver:

/* Hardware interrupt handler */
static irqreturn_t ixgbe_msix_lsc(int irq, void *data)
{
    struct net_device *netdev = data;

    /* Disable network card interrupt */
    ixgbe_disable_interrupt();

    /* Add device to poll_list */
    napi_schedule(&adapter->q_vector[vector]->napi);

    return IRQ_HANDLED;
}

/* NAPI poll function */
static int ixgbe_poll(struct napi_struct *napi, int budget)
{
    struct ixgbe_q_vector *q_vector = container_of(napi, struct ixgbe_q_vector, napi);
    struct ixgbe_adapter *adapter = q_vector->adapter;
    int work_done = 0;

    /* Batch receive packets from RingBuffer, process at most budget packets */
    work_done = ixgbe_clean_rx_irq(q_vector, budget);

    /* If work is incomplete, stay in poll_list */
    if (work_done < budget) {
        napi_complete(napi);
        ixgbe_enable_interrupt();
    }

    return work_done;
}

Check Out My Results

My requirement: I want to read Linux source code, manage my Linux source code reading with git, and save my source code reading with my self-built GitLab. Then I discovered that simply using git push for a 2GB Linux source code file would cause problems.

So I wrote a script to solve this problem.

The script content is as follows:

#!/bin/bash

# Configuration parameters (modify as needed)
REMOTE_NAME="origin"              # Remote repository name
BRANCH_NAME="main"                # Target branch name
REMOTE_URL="git@gitlab.cheverjohn.me:CheverJohn/linux.git"  # Remote repository URL
BATCH_SIZE=500                    # Number of files per batch
COMMIT_MESSAGE="Batch file upload"      # Commit message

# Color configuration
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No color

# Check if we're inside a git repository
if ! git rev-parse --is-inside-work-tree > /dev/null 2>&1; then
    echo -e "${YELLOW}Current directory is not a git repository, initializing...${NC}"
    git init
    echo -e "${GREEN}Git repository initialized${NC}"
else
    echo -e "${GREEN}Git repository already exists${NC}"
fi

# Check if remote repository is configured
if ! git remote | grep -q "$REMOTE_NAME"; then
    echo -e "${YELLOW}Adding remote repository $REMOTE_NAME: $REMOTE_URL${NC}"
    git remote add $REMOTE_NAME $REMOTE_URL
    echo -e "${GREEN}Remote repository added${NC}"
else
    CURRENT_URL=$(git remote get-url $REMOTE_NAME 2>/dev/null || echo "")
    if [ "$CURRENT_URL" != "$REMOTE_URL" ]; then
        echo -e "${YELLOW}Updating remote repository URL: $REMOTE_URL${NC}"
        git remote set-url $REMOTE_NAME $REMOTE_URL
        echo -e "${GREEN}Remote repository URL updated${NC}"
    else
        echo -e "${GREEN}Remote repository correctly configured${NC}"
    fi
fi

# Ensure main branch exists
if ! git show-ref --quiet refs/heads/$BRANCH_NAME; then
    echo -e "${YELLOW}Creating branch $BRANCH_NAME...${NC}"
    git checkout -b $BRANCH_NAME
    echo -e "${GREEN}Branch $BRANCH_NAME created${NC}"
else
    echo -e "${GREEN}Branch $BRANCH_NAME already exists${NC}"
    git checkout $BRANCH_NAME
fi

# Get all untracked and modified files
echo -e "${BLUE}Getting list of files to upload...${NC}"
FILES=($(git ls-files --others --exclude-standard) $(git diff --name-only))
TOTAL_FILES=${#FILES[@]}

if [ $TOTAL_FILES -eq 0 ]; then
    echo -e "${YELLOW}No files found to add, trying to add all files...${NC}"
    git add -A
    FILES=($(git diff --name-only --cached))
    TOTAL_FILES=${#FILES[@]}

    if [ $TOTAL_FILES -eq 0 ]; then
        echo -e "${RED}Error: No files found to push${NC}"
        exit 1
    fi
else
    # Add all files to staging area
    echo -e "${BLUE}Adding all files to staging area...${NC}"
    git add -A
fi

echo -e "${GREEN}Found $TOTAL_FILES files to upload${NC}"

# Calculate number of batches
BATCH_COUNT=$(( ($TOTAL_FILES + $BATCH_SIZE - 1) / $BATCH_SIZE ))
echo -e "${GREEN}Will upload in $BATCH_COUNT batches${NC}"

# First commit all files
echo -e "${BLUE}Committing all files...${NC}"
git commit -m "$COMMIT_MESSAGE"

# Push in batches
echo -e "${BLUE}Starting batch push...${NC}"
git push -u $REMOTE_NAME $BRANCH_NAME

if [ $? -eq 0 ]; then
    echo -e "${GREEN}All files successfully pushed to remote repository${NC}"
else
    echo -e "${YELLOW}Regular push failed, trying batch push method...${NC}"
    # Using git batch push method (referenced from search results examples)
    # Based on git_batch_push.sh concept, but simplified implementation

    # Use git rev-list to get all commits
    COMMITS=($(git rev-list --reverse HEAD))
    TOTAL_COMMITS=${#COMMITS[@]}

    if [ $TOTAL_COMMITS -eq 0 ]; then
        echo -e "${RED}Error: No commits found${NC}"
        exit 1
    fi

    echo -e "${GREEN}Found $TOTAL_COMMITS commits, will push in batches${NC}"

    # Calculate number of batches (500 commits per batch)
    COMMIT_BATCH_SIZE=500
    COMMIT_BATCH_COUNT=$(( ($TOTAL_COMMITS + $COMMIT_BATCH_SIZE - 1) / $COMMIT_BATCH_SIZE ))

    echo -e "${GREEN}Will push commits in $COMMIT_BATCH_COUNT batches${NC}"

    # Push commits in batches
    for ((i=0; i<$COMMIT_BATCH_COUNT; i++)); do
        START=$(($i * $COMMIT_BATCH_SIZE))
        END=$((($i + 1) * $COMMIT_BATCH_SIZE))

        if [ $END -gt $TOTAL_COMMITS ]; then
            END=$TOTAL_COMMITS
        fi

        BATCH_END_INDEX=$((END - 1))
        TARGET_COMMIT=${COMMITS[$BATCH_END_INDEX]}

        echo -e "${BLUE}Pushing batch $((i+1))/${COMMIT_BATCH_COUNT} (commits $((START+1))-$END)...${NC}"

        if git push $REMOTE_NAME $TARGET_COMMIT:refs/heads/$BRANCH_NAME; then
            echo -e "${GREEN}Batch $((i+1)) successfully pushed${NC}"
        else
            echo -e "${RED}Batch $((i+1)) push failed${NC}"
            echo -e "${YELLOW}Trying alternative push method...${NC}"

            # If above method fails, try another batch push method
            if [ $i -eq 0 ]; then
                # First batch, create new branch
                git push $REMOTE_NAME $BRANCH_NAME
            else
                # Get the end commit of previous batch
                PREV_END=$(($START - 1))
                PREV_COMMIT=${COMMITS[$PREV_END]}
                CURR_COMMIT=${COMMITS[$BATCH_END_INDEX]}

                echo -e "${BLUE}Using range push $PREV_COMMIT..$CURR_COMMIT${NC}"
                git push $REMOTE_NAME $PREV_COMMIT:refs/heads/$BRANCH_NAME $CURR_COMMIT:refs/heads/$BRANCH_NAME
            fi
        fi

        echo
    done
fi

echo -e "${GREEN}Batch upload process completed!${NC}"

The usage method is simple.

chmod +x git_batch_push.sh

Then run the script in the current folder.

Final Results

This article focuses on practical implementation with a brief introduction to the underlying principles.

Throughout this article, I'll first cover the basic principles, then walk through three main sections: how to create GPG keys, how to manage your keys (key management commands), and hands-on practice: using self-created GPG keys to decrypt transmitted encrypted files.

Principle Overview

The principle behind using GPG for encrypting and decrypting documents is straightforward, as described in the documentation:

The procedure for encrypting and decrypting documents is straightforward with this mental model. If you want to encrypt a message to Alice, you encrypt it using Alice's public key, and she decrypts it with her private key. If Alice wants to send you a message, she encrypts it using your public key, and you decrypt it with your key.

The entire process is: I encrypt the file I want to send you with my private key. Then you use my public key to decrypt the file I want to send you, which has been encrypted by me.

This is the basic process.

This article provides a quick overview of how GPG encrypts document content.

This article is divided into two parts, starting with creating a usable GPG key.

How to Create GPG Keys

[Se] gpg --list-keys
gpg: checking the trustdb
gpg: no ultimately trusted keys found

First, let's list whether my machine currently has any GPG keys. As we can see, there are none. This mainly refers to whether there are GPG public keys. Let's also check for private keys.

[Se] gpg --list-secret-keys
[Se]

Nothing there either.

One Command to Rule Them All

Let's start creating.

The detailed steps begin with gpg --full-generate-key, as follows:

[Se] gpg --full-generate-key
gpg (GnuPG) 2.4.0; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
   (9) ECC (sign and encrypt) *default*
  (10) ECC (sign only)
  (14) Existing key from card
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (3072) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y

GnuPG needs to construct a user ID to identify your key.

Real name: Chenwei Jiang
Email address: cheverjonathan@gmail.com
Comment: Used for learning
You selected this USER-ID:
    "Chenwei Jiang (Used for learning) <cheverjonathan@gmail.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

gpg: revocation certificate stored as '/home/cheverjohn/.gnupg/openpgp-revocs.d/D1DED33E8FE9CA4B315A488968CC99424BF9EF81.rev'
public and secret key created and signed.

pub   rsa4096 2023-09-19 [SC]
      D1DED33E8FE9CA4B315A488968CC99424BF9EF81
uid                      Chenwei Jiang (Used for learning) <cheverjonathan@gmail.com>
sub   rsa4096 2023-09-19 [E]

Detailed Stage-by-Stage Command Explanation

Let me break down what happens at each stage.

Stage 1: Select Encryption and Signing Algorithm

In the first stage, you need to select the encryption algorithm. I chose option 1, which means both encryption and signing use the RSA algorithm.

[Se] gpg --full-generate-key
gpg (GnuPG) 2.4.0; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
   (9) ECC (sign and encrypt) *default*
  (10) ECC (sign only)
  (14) Existing key from card
Your selection? 1

Oh, and there's also a copyright notice.

Stage 2: Select Key Length

Longer keys are more secure. I chose 4096 here.

RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (3072) 4096
Requested keysize is 4096 bits

Stage 3: Set Key Validity Period

Set the validity period.

Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y

For this demonstration, I configured Key does not expire at all.

Stage 4: Personal Information

This section will ultimately be used to generate your user ID.

GnuPG needs to construct a user ID to identify your key.

Real name: Chenwei Jiang
Email address: cheverjonathan@gmail.com
Comment: Used for learning
You selected this USER-ID:
    "Chenwei Jiang (Used for learning) <cheverjonathan@gmail.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O

Here I set my Real name to "Chenwei Jiang", my Email address to "cheverjonathan@gmail.com", and added a comment. These are some basic information I commonly use, primarily for demonstration purposes.

The result is a USER-ID: "Chenwei Jiang (Used for learning) cheverjonathan@gmail.com".

How to Manage Your Keys (Key Management Commands)

This section covers how to manage multiple keys on a host machine.

Listing Keys

There are two types of key listings: public keys and private keys.

[Se] gpg --list-keys
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
/home/cheverjohn/.gnupg/pubring.kbx
-----------------------------------
pub   rsa4096 2023-09-19 [SC]
      D1DED33E8FE9CA4B315A488968CC99424BF9EF81
uid           [ultimate] Chenwei Jiang (Used for learning) <cheverjonathan@gmail.com>
sub   rsa4096 2023-09-19 [E]

The above command shows the details.

The key section:

/home/cheverjohn/.gnupg/pubring.kbx
-----------------------------------
pub   rsa4096 2023-09-19 [SC]
      D1DED33E8FE9CA4B315A488968CC99424BF9EF81
uid           [ultimate] Chenwei Jiang (Used for learning) <cheverjonathan@gmail.com>
sub   rsa4096 2023-09-19 [E]

The first line shows the public key filename.

Exporting Keys

The public key file is located at ~/.gnupg/pubring.kbx and stored in binary format. Using the armor parameter converts it to ASCII display.

Using commands to export as public-key.txt and private-key.txt:

[gpg] pwd
/home/cheverjohn/Se/gpg
[gpg] ls
[gpg] ls -la
total 0
drwxr-xr-x. 1 cheverjohn cheverjohn  0 Sep 19 23:36 .
drwxr-xr-x. 1 cheverjohn cheverjohn 34 Sep 19 23:36 ..
[gpg] gpg --armor --output public-key.txt --export 'Chenwei Jiang (used for learning) <cheverjonathan@gmail.com>'
[gpg] ls
public-key.txt
[gpg] cat public-key.txt
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBGUJt+QBEADAzoLPAdK8GfJ/5Ouxh2rOrMsClMmoOznMm2GOBcqSaQsdmP4G
................................................................
oM3YFFujtMxK/cQ/KkbmwAtlMkWx5x8RT/dJ
=NMtR
-----END PGP PUBLIC KEY BLOCK-----
[gpg]

As shown above, this demonstrates the steps to export public-key.txt. The detailed command is:

gpg --armor --output public-key.txt --export 'Chenwei Jiang (used for learning) <cheverjonathan@gmail.com>'

Below are the steps to export private-key.txt:

[gpg] ls
public-key.txt
[gpg] gpg --armor --output private-key.txt --export-secret-keys
[gpg] ls
private-key.txt  public-key.txt
[gpg]

If you previously set a password, this operation requires the password. The command requiring password is:

gpg --armor --output private-key.txt --export-secret-keys

Uploading Public Keys

Public key servers are network servers specifically for storing user public keys. The --send-keys subcommand can achieve this.

Public key servers have no verification mechanism - anyone can upload public keys in your name, so there's no guarantee of reliability for public keys on servers.

Generally, we publish a public key fingerprint on our own website for others to verify that the downloaded public key is authentic. The --fingerprint subcommand can generate a public key fingerprint.

Hands-On Practice / Practice Makes Perfect

In this section, I'll demonstrate exporting public keys, encrypting with private keys, and decrypting with public keys on another device.

I'll show how to encrypt a file containing "hello world" and then decrypt it remotely.

The steps are divided into encryption and decryption.

Detailed steps:

1. Create file with the following commands:

   [gpg] touch demo.txt
   echo "hello world" > demo.txt
   [gpg] ls
   demo.txt  private-key.txt  public-key.txt
   [gpg] cat demo.txt
   hello world
   [gpg]
   

   The file creation commands are:

   touch demo.txt
   echo "hello world" > demo.txt
   

   Now we have a demo.txt file containing "hello world".

2. Encrypt the file:

   [gpg] ls
   demo.txt  private-key.txt  public-key.txt
   [gpg] gpg --recipient 'cheverjonathan@gmail.com' --output demo.gpg --encrypt demo.txt
   [gpg] ls
   demo.gpg  demo.txt  private-key.txt  public-key.txt
   [gpg] cat demo.gpg
   �
   ..............
   �&5p�.�)I�槹iQ%
   [gpg]
   

   Encryption command:

   gpg --recipient 'cheverjonathan@gmail.com' --output demo.gpg --encrypt demo.txt
   

   You can see that demo.gpg is the encrypted file.

3. Decrypt the file:

   [gpg] gpg --output demo --decrypt demo.gpg
   gpg: encrypted with rsa4096 key, ID 13C117D5FEC4F051, created 2023-09-19
         "Chenwei Jiang (Used for learning) <cheverjonathan@gmail.com>"
   [gpg] ls
   demo  demo.gpg  demo.txt  private-key.txt  public-key.txt
   [gpg] cat demo
   hello world
   [gpg]
   

   Decryption command:

   gpg --output demo --decrypt demo.gpg
   

   After entering this command, you need to input the password set earlier, then you can get the file.

   The above demonstrates the process of encrypting files using local keys and decrypting them locally.

   Next, let's begin...

   Obtaining Files on Other Computers, Encrypting Them, Then Having the Original Host Decrypt Using Private Keys

   First, we transfer public-key.txt to another host. You can use GitHub's functionality to upload/download files. For convenience, I'll upload the following file tree from the above process:

   cheverjohn@Dell-G33579 git:(doc/test-gpg*)% tree ~/workspace/Opensource/github.com/Chever-John/cheverjohn.me/docs/wait-for-publish/gpg
   .
   ├── demo
   ├── demo.gpg
   ├── demo.txt
   ├── private-key.txt
   └── public-key.txt
   

   After removing private-key.txt, I'll upload the files to GitHub.

   You can see I've downloaded the public-key.txt file on another host, as shown in the file tree:

   cheverjohn:wait-for-publish/ git:(doc/test-gpg*)$ tree gpg                                                                   [0:54:06]
   gpg
   ├── demo
   ├── demo.gpg
   ├── demo.txt
   └── public-key.txt
   
   1 directory, 4 files
   

   Next, I need to use this public-key.txt to encrypt a file, then transfer this file back to the host with the private key for decryption.

   First, we need to import this public-key.txt locally:

   gpg --import public-key.txt 
   

   Command execution result:

   cheverjohn:gpg/ git:(doc/test-gpg*)$ gpg --import public-key.txt                                                             [1:20:11]
   gpg: key 3BE465D20064251C: public key "Chenwei Jiang (Learning second) <cheverjonathan@gmail.com>" imported
   gpg: Total number processed: 1
   gpg:               imported: 1
   

   Now we need to use this public key to encrypt text.

   First, let's create a file hello-no-encrypt.txt:

   cheverjohn:gpg/ git:(doc/test-gpg*)$ touch hello-no-encrypt.txt                                                              [1:16:07]
   cheverjohn:gpg/ git:(doc/test-gpg*)$ nvim hello-no-encrypt.txt                                                               [1:16:15]
   cheverjohn:gpg/ git:(doc/test-gpg*)$ cat hello-no-encrypt.txt                                                                [1:16:24]
   Hello Sasa!
   cheverjohn:gpg/ git:(doc/test-gpg*)$ tree                                                                                    [1:16:28]
   .
   ├── demo
   ├── demo.gpg
   ├── demo.txt
   ├── hello-no-encrypt.txt
   └── public-key.txt
   
   1 directory, 5 files
   

   We need to encrypt this file and then send it to the host with the private key for decryption.

   The entire encryption process:

   cheverjohn:gpg/ git:(doc/test-gpg*)$ gpg --output hello-encrypted.gpg --encrypt --recipient 'Chenwei Jiang (Learning second) <cheverjonathan@gmail.com>' hello-no-encrypt.txt
   gpg: 617C5542800731C1: There is no assurance this key belongs to the named user
   
   sub  rsa4096/617C5542800731C1 2023-09-21 Chenwei Jiang (Learning second) <cheverjonathan@gmail.com>
    Primary key fingerprint: 5588 D37D AF51 50FD 9186  47E7 3BE4 65D2 0064 251C
         Subkey fingerprint: A038 0DA1 D481 62C7 517C  D6C6 617C 5542 8007 31C1
   
   It is NOT certain that the key belongs to the person named
   in the user ID.  If you *really* know what you are doing,
   you may answer the next question with yes.
   
   Use this key anyway? (y/N) y
   

   Encryption command:

   gpg --output hello-encrypted.gpg --encrypt --recipient 'Chenwei Jiang (Learning second) <cheverjonathan@gmail.com>' hello-no-encrypt.txt
   

   Then we need to upload the encrypted file hello-encrypted.gpg to the machine with the private key.

   Again, I'll use GitHub as the file transfer tool.

   You can see I've obtained the file on the host machine (the machine with the keys) and started decryption:

   [gpg] tree                                                                                                         git:(doc/test-gpg)
   .
   ├── demo
   ├── demo.gpg
   ├── demo.txt
   ├── hello-encrypted.gpg
   ├── hello-no-encrypt.txt
   └── public-key.txt
   
   1 directory, 6 files
   

   Next, we need to decrypt hello-encrypted.gpg:

   [gpg] gpg --output hello-decrypted.txt --decrypt hello-encrypted.gpg                                               git:(doc/test-gpg)
   gpg: encrypted with rsa4096 key, ID 617C5542800731C1, created 2023-09-21
         "Chenwei Jiang (Learning second) <cheverjonathan@gmail.com>"
   [gpg] ls                                                                                                        git:(doc/test-gpg*)  ✱
   demo  demo.gpg  demo.txt  hello-decrypted.txt  hello-encrypted.gpg  hello-no-encrypt.txt  public-key.txt
   [gpg] cat hello-decrypted.txt                                                                                   git:(doc/test-gpg*)  ✱
   Hello Sasa!
   

   Decryption command:

   gpg --output hello-decrypted.txt --decrypt hello-encrypted.gpg
   

Related Articles

• gnupg.org Official Manual

Hello!

I am Chever John, a software engineer.

Just as title says.

Background Description

While wandering around the community, I found a user confused about "how to use multilingual plugins in APISIX Ingress environment". I happen to be a user of go-plugin-runner and have a little knowledge of the APISIX Ingress project, so this document was born.

Proposal Description

Based on version 0.3 of the go-plugin-runner plugin and version 1.4.0 of APISIX Ingress, this article goes through building the cluster, building the image, customizing the helm chart package, and finally, deploying the resources. It is guaranteed that the final result can be derived in full based on this documentation.

go-plugin-runner: 0.3
APISIX Ingress: 1.4.0

kind: kind v0.12.0 go1.17.8 linux/amd64
kubectl version: Client Version: v1.23.5/Server Version: v1.23.4
golang: go1.18 linux/amd64

Begin

Build a cluster environment

Select kind to build a local cluster environment. The command is as follows:

cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
EOF

Build the go-plugin-runner executable

If you have finished writing the plugin, you can start compiling the executable to run with APISIX.

This article recommends two packaging build options.

1. put the packaging process into the Dockerfile and finish the compilation process when you build the docker image later.
2. You can also follow the scheme used in this document, building the executable first and then copying the packaged executable to the image.

How you choose the option should depend on your local hardware considerations. The reason for selecting the second option here is that I want to rely on my powerful local hardware to increase the building speed and speed up the process.

Go to the go-plugin-runner directory

Choose a folder address /home/chever/api7/cloud_native/tasks/plugin-runner and place our apisix-go-plugin-runner project in this folder.

After successful placement, the file tree is shown below:

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner$ tree -L 1
.
└── apisix-go-plugin-runner

1 directory, 0 files

Then you need to go to the apisix-go-plugin-runner/cmd/go-runner/plugins directory and write the plugins you need in that directory. This article will use the default plugin say for demonstration purposes.

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner/apisix-go-plugin-runner$ tree cmd
cmd
└── go-runner
    ├── main.go
    ├── main_test.go
    ├── plugins
    │   ├── fault_injection.go
    │   ├── fault_injection_test.go
    │   ├── limit_req.go
    │   ├── limit_req_test.go
    │   ├── say.go
    │   └── say_test.go
    └── version.go
    
2 directories, 10 files

After writing the plugins, start compiling the executable formally, and note here that you should build static executables, not dynamic ones.

The package compile command is as follows.

CGO_ENABLED=0 go build -a -ldflags '-extldflags "-static"' .

This successfully packages a statically compiled go-runner executable.

In the apisix-go-plugin-runner/cmd/go-runner/ directory, you can see that the current file tree looks like this:

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner/apisix-go-plugin-runner/cmd/go-runner$ tree -L 1
.
├── go-runner
├── main.go
├── main_test.go
├── plugins
└── version.go

1 directory, 4 files

Please remember the path apisix-go-plugin-runner/cmd/go-runner/go-runner, we will use it later.

Build Docker Image

The image is built here in preparation for installing APISIX later using helm.

Write Dockerfile

Return to the path /home/chever/api7/cloud_native/tasks/plugin-runner and create a Dockerfile in that directory, a demonstration of which is given here.

ARG ENABLE_PROXY=false

# Build Apache APISIX
FROM api7/apisix-base:1.19.9.1.5

ADD ./apisix-go-plugin-runner /usr/local/apisix-go-plugin-runner

ARG APISIX_VERSION=2.13.1
LABEL apisix_version="${APISIX_VERSION}"

ARG ENABLE_PROXY
RUN set -x \
    && (test "${ENABLE_PROXY}" != "true" || /bin/sed -i 's,http://dl-cdn.alpinelinux.org,https://mirrors.aliyun.com,g' /etc/apk/repositories) \
    && apk add --no-cache --virtual .builddeps \
        build-base \
        automake \
        autoconf \
        make \
        libtool \
        pkgconfig \
        cmake \
        unzip \
        curl \
        openssl \
        git \
        openldap-dev \
    && luarocks install https://github.com/apache/apisix/raw/master/rockspec/apisix-${APISIX_VERSION}-0.rockspec --tree=/usr/local/apisix/deps PCRE_DIR=/usr/local/openresty/pcre \
    && cp -v /usr/local/apisix/deps/lib/luarocks/rocks-5.1/apisix/${APISIX_VERSION}-0/bin/apisix /usr/bin/ \
    && (function ver_lt { [ "$1" = "$2" ] && return 1 || [ "$1" = "`echo -e "$1\n$2" | sort -V | head -n1`" ]; };  if [ "$APISIX_VERSION" = "master" ] || ver_lt 2.2.0 $APISIX_VERSION; then echo 'use shell ';else bin='#! /usr/local/openresty/luajit/bin/luajit\npackage.path = "/usr/local/apisix/?.lua;" .. package.path'; sed -i "1s@.*@$bin@" /usr/bin/apisix ; fi;) \
    && mv /usr/local/apisix/deps/share/lua/5.1/apisix /usr/local/apisix \
    && apk del .builddeps \
    && apk add --no-cache \
        bash \
        curl \
        libstdc++ \
        openldap \
        tzdata \
    # forward request and error logs to docker log collector
    && ln -sf /dev/stdout /usr/local/apisix/logs/access.log \
    && ln -sf /dev/stderr /usr/local/apisix/logs/error.log

WORKDIR /usr/local/apisix

ENV PATH=$PATH:/usr/local/openresty/luajit/bin:/usr/local/openresty/nginx/sbin:/usr/local/openresty/bin

EXPOSE 9080 9443

CMD ["sh", "-c", "/usr/bin/apisix init && /usr/bin/apisix init_etcd && /usr/local/openresty/bin/openresty -p /usr/local/apisix -g 'daemon off;'"]

STOPSIGNAL SIGQUIT

这份 Dockerfile 配置文档，来源于这个链接。我做的唯一修改如下：

ARG ENABLE_PROXY=false

# Build Apache APISIX
FROM api7/apisix-base:1.19.9.1.5

ADD ./apisix-go-plugin-runner /usr/local/apisix-go-plugin-runner

ARG APISIX_VERSION=2.13.1
LABEL apisix_version="${APISIX_VERSION}"

ARG ENABLE_PROXY

Package all the /apisix-go-plugin-runner files in the /home/chever/api7/cloud_native/tasks/plugin-runner directory into a Docker image. Note down the location of the executable apisix-go-plugin-runner/cmd/go-runner/go-runner and the location of the /usr/local/apisix-go-plugin-runner directory in the Dockerfile above to get the final location of the executable in the Docker image is located as follows.

/usr/local/apisix-go-plugin-runner/cmd/go-runner/go-runner

Please make a note of this address. We will use it in the rest of the configuration.

Begin to build Docker Image

Start building a Docker image based on the Dockerfile. The command is executed in the /home/chever/api7/cloud_native/tasks/plugin-runner directory. The command is as follows:

docker build -t apisix/forrunner:0.1 .

Command Explanation: Build an image with the name apisix/forrunner and mark it as version 0.1.

Load the image to the cluster environment

kind  load docker-image apisix/forrunner:0.1 

Load the image into the kind cluster environment to pull the custom local image for installation during the helm installation.

Install APISIX Ingress

Customize the helm chart

This section focuses on modifying the values.yaml file in the official helm package so that it can install locally packaged images and run the go-plugin-runner executable properly.

Fetch Official helm chart

First, fetch the latest apisix helm chart package with the following command:

helm fetch apisix/apisix

The file tree is as follows:

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner$ tree -L 1
.
├── apisix-0.9.1.tgz
└── apisix-go-plugin-runner

1 directory, 1 file

Unzip

Unzip the apisix-0.9.1.tgz file and prepare to rewrite the configuration. The unzip command is as follows.

tar zxvf apisix-0.9.1.tgz

The file tree is as follows:

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner$ tree -L 1
.
├── apisix
├── apisix-0.9.1.tgz
└── apisix-go-plugin-runner

2 directories, 1 file

Change values.yaml

Go to the apisix folder and modify the values.yaml file. The two changes are as follows:

image:
  repository: apisix/forrunner
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: 0.1

The first change sets the image for the helm installation to be a locally packaged image of your own.

extPlugin:
  enabled: true
  cmd: ["/usr/local/apisix-go-plugin-runner/cmd/go-runner/go-runner", "run"]

The second change sets the position of go-runner in the container after running the container.

Compress the modified helm chart

Once configured, compress the apisix file. The compression command is as follows:

tar zcvf apisix.tgz apisix/

The compressed file is obtained, at which point the file tree is as follows:

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner$ tree -L 1
.
├── apisix
├── apisix-0.9.1.tgz
├── apisix-go-plugin-runner
└── apisix.tgz

2 directories, 2 files

Execute the helm install command

Create namespace

Before installation, create namespaces with the following command:

kubectl create ns ingress-apisix

Then install APISIX using helm with the following command:

helm install apisix ./apisix.tgz --set gateway.type=NodePort --set ingress-controller.enabled=true --namespace ingress-apisix --set ingress-controller.config.apisix.serviceNamespace=ingress-apisix

Create httpbin service and ApisixRoute resources

Create an httpbin backend resource to run with the deployed ApisixRoute resource to test that the functionality is working correctly.

Create httpbin service

Create an httpbin service with the following command:

kubectl run httpbin --image kennethreitz/httpbin --port 80

Expose the port with the following command:

kubectl expose pod httpbin --port 80

Create ApisixRoute Resource

Create the go-plugin-runner-route.yaml file to enable the ApisixRoute resource, with the following configuration file:

apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
  name: plugin-runner-demo
spec:
  http:
  - name: rule1
    match:
      hosts:
      - local.httpbin.org
      paths:
      - /get
    backends:
    - serviceName: httpbin
      servicePort: 80
    plugins:
    - name: ext-plugin-pre-req
      enable: true
      config:
        conf:
        - name: "say"
          value: "{\"body\": \"hello\"}"

The create resource command is as follows:

kubectl apply -f go-plugin-runner-route.yaml

Test

The command is as follows to test if the plugin written in Golang is working correctly:

kubectl exec -it -n ${namespace of Apache APISIX} ${Pod name of Apache APISIX} -- curl http://127.0.0.1:9080/get -H 'Host: local.httpbin.org'

Here I derived from the kubectl get pods --all-namespaces command that the ${namespace of Apache APISIX} and ${Pod name of Apache APISIX} parameters here are ingress-apisix and apisix- 55d476c64-s5lzw, execute the command as follows:

kubectl exec -it -n ingress-apisix apisix-55d476c64-s5lzw -- curl http://127.0.0.1:9080/get -H 'Host: local.httpbin.org'

The expected response obtained is:

chever@cloud-native-01:~/api7/cloud_native/tasks/plugin-runner$ kubectl exec -it -n ingress-apisix apisix-55d476c64-s5lzw -- curl http://127.0.0.1:9080/get -H 'Host: local.httpbin.org'
Defaulted container "apisix" out of: apisix, wait-etcd (init)
hello