Daily Debugging: Minisforum N100 Random Disk Drop / System Freeze (DNS, 1+ Day Uptime)

This is a troubleshooting log template. Facts first, evidence second, conclusions last.

TL;DR

Symptoms: The DNS service becomes unresponsive; the whole machine stutters/freezes; sometimes it looks like storage I/O issues or a “disk drop”.
Pattern: Random, usually after 1+ day of continuous running.
Recovery: Power off and let it cool down for a while, then it works again.
Current conclusion (probabilistic): ____ (e.g., thermals / power / storage-controller reset).

Impact

Direct impact: ____ (e.g., LAN-wide DNS timeouts).
Indirect impact: ____ (services that depend on name resolution).
Time window: ____ (start time / duration / auto-recovery?).

Environment (reproducible)

Hardware

CPU: Intel® N100 (4C/4T, 6MB cache, up to 3.4GHz)
GPU: Intel® UHD Graphics
RAM: 8GB LPDDR5 (single channel, onboard, 4800MHz)
Storage: UFS 2.1 256G
Wireless: Intel AX200/201 (Wi‑Fi 6 / Bluetooth 5.2)
Ethernet: 2.5G RJ45 ×1 (PoE IEEE 802.3at)
Ports: USB 3.2 Gen2 Type‑A ×2; USB 3.2 Gen2 Type‑C ×1 (Alt DP/PD); HDMI ×1
Power: 65W USB‑C Power Delivery adapter

Software

Actual OS: Linux (current kernel: 6.8.12-5-pve; factory OS: Windows 11 Home)
Kernel (if Linux): 6.8.12-5-pve
Bootloader (GRUB): grub2 2.06-13+pmx2
DNS stack: ____ (dnsmasq / unbound / AdGuard Home / Pi‑hole / other)
Deployment: ____ (systemd / Docker / other)
Other services: ____
Logging policy: ____ (write frequency / rotation / same disk?).

Problem statement (facts only)

First seen: ____
Frequency: about every ____ hours/days (random).
Observable symptoms during failure:
- DNS: ____ (timeouts / refusal / high latency)
- System: ____ (SSH/RDP reachable? CPU/memory spike?)
- Storage: ____ (mount disappears / read-only remount / I/O error / device reset)
- Network: ____ (packet loss? NIC reset?)
Recovery: ____ (power off for ____ minutes; does reboot also work: ____)
Load correlation: ____ (QPS / logging / other tasks).

Immediate mitigation (timeline)

T+0: ____ (e.g., switch upstream DNS)
T+5m: ____ (stop DNS or high-IO workloads)
T+10m: ____ (collect logs and metrics snapshots)
Result: ____

Evidence model (align everything by timestamp)

One timeline: failure start time, temperature peak, I/O errors, service timeouts.
One core question: is DNS dead, or is the storage/system dead and DNS is just the first visible symptom?

Hypotheses (prioritized)

H1: Thermals / overheating → throttling or protection → instability
- Evidence: ____
H2: Power instability (PD/PoE/adapter) → controller reset
- Evidence: ____
H3: Storage media/controller issue → I/O stall or device drop
- Evidence: ____
H4: Software resource exhaustion (FD/memory/log amplification/IO saturation)
- Evidence: ____
H5: Kernel/driver bug (power management, device PM policy)
- Evidence: ____

Data to capture (during failure)

Temps/fans: ____ (sensors, /sys/class/thermal/, Windows tools)
System logs: ____ (journalctl, Windows Event Viewer)
Storage logs: ____ (I/O error, timeout, reset, read-only remount)
Resource metrics: ____ (CPU/load/iowait/memory/disk throughput)
DNS metrics: ____ (QPS, cache hit ratio, upstream latency, error rate)
Client checks: ____ (dig/nslookup outputs)

Key evidence (3–5 snippets)

text
[time] ____

text
[time] ____

text
[time] ____

Investigation (Attempt → Observation → Conclusion)

Attempt A: ____

Change: ____
Observation: ____
Conclusion: ____

Attempt B: ____

Change: ____
Observation: ____
Conclusion: ____

Attempt C: ____

Change: ____
Observation: ____
Conclusion: ____

Final conclusion (probability, not absolute claims)

Most likely root cause: ____
Secondary factors: ____
Ruled out: ____ (with evidence)

Fix plan (short-term vs long-term)

Short-term mitigation

____ (e.g., improve cooling)
____ (e.g., cap power / disable aggressive power saving)
____ (e.g., reduce disk writes / configure log rotation)

Long-term fix

____ (e.g., replace power adapter / replace storage / BIOS update)
____ (e.g., airflow redesign / thermal pads)
____ (e.g., kernel/OS change)

Validation

Uptime: ____ days without recurrence
Temperature: CPU ≤ ____°C; storage ≤ ____°C
Logs: no I/O errors / resets / read-only remount

Retrospective

Biggest learning: ____
What I will do earlier next time: ____ (monitoring, alerting, auto log capture)
Open questions: ____

Attempt plan: Monitor 192.168.22.18 (Minisforum) via a monitoring LXC (192.168.22.19) on PVE 192.168.22.12

Goal: turn “it feels like overheating / disk drop / DNS died” into an evidence-aligned timeline: temps + kernel/storage errors + real DNS query success/latency + host reachability.
Note: never put passwords into configs or the repo. Any credentials should live in root-only local files on the monitoring host.

Extra pitfall: port `22` is reachable, but `ssh` still hangs / drops (more common across subnets)

This one is misleading: nc says 22/tcp open, so people assume “network is fine”. But ssh can still stall during handshake and eventually time out (or die with Broken pipe).

Topology (what happened in this incident)

Client subnet: 192.168.11.0/24
Target host: 192.168.22.12 (PVE)
Same-subnet control point: 192.168.22.18 (SSH from here to .12 is stable)
Correlated signal: DNS server (e.g. 192.168.22.53:53) experienced timeouts — reverse lookups can amplify SSH handshake latency.

Symptoms (typical)

nc -vz -G 2 192.168.22.12 22 reports succeeded (TCP handshake is OK)
ssh root@192.168.22.12 hangs and later shows:
- ssh_dispatch_run_fatal: Connection to 192.168.22.12 port 22: Operation timed out
- or Read from remote host 192.168.22.12: Operation timed out / client_loop: send disconnect: Broken pipe

Quick triage (split “port is open” into smaller truths)

Check whether the SSH banner is returned immediately (protocol only, no auth):

bash
nc -v 192.168.22.12 22

You should quickly see SSH-2.0-OpenSSH_.... If the TCP connect succeeds but the banner is slow/missing, the issue is likely in sshd/system path (DNS reverse lookup stalls, high load, policy/ACL causing unstable data path), not “port blocked”.

Capture where SSH stalls (useful evidence):

bash
ssh -vvv -o ConnectTimeout=5 -o ServerAliveInterval=5 -o ServerAliveCountMax=1 root@192.168.22.12

Minimal fix (eliminate DNS/GSSAPI special cases first)

If you can SSH from a same-subnet host (e.g. .22.18) into .22.12, prefer the server-side fix:

On .22.12, edit /etc/ssh/sshd_config and make sure you have:

UseDNS no
GSSAPIAuthentication no

Validate and restart:

bash
sshd -t && systemctl restart ssh
systemctl status ssh --no-pager

Practical rationale: when DNS/reverse lookup is flaky, UseDNS yes can block parts of the handshake/audit path. Disabling it removes an entire class of “it depends on DNS today” failures.

If it is still flaky: continue with the “cross-subnet triad”

Return path routing (the target must know how to route back to 192.168.11.0/24):

bash
ip route get 192.168.11.12

Firewall/ACL/state tracking: allow not only 22/tcp inbound but also the correct return direction and conntrack behavior (especially with gateways/NAT/policy routing).
MTU/fragmentation: tunnels/PPPoE/VLAN paths can yield “SYN works but data is unstable”; validate with DF-sized pings on Linux and temporarily lower MSS/MTU to confirm.

Architecture (minimal, but sufficient)

192.168.22.19 (monitoring LXC on PVE 192.168.22.12): Prometheus + Grafana + blackbox_exporter (optional Pushgateway)
192.168.22.18 (target host): nodeexporter (host metrics) + (optional) smartctlexporter + custom textfile metrics (kernel/I/O error counter)

Data model:

Time series (Prometheus): scrape every 15s, keep at least 48h around failures.
Active probing (blackbox_exporter): probe .18/.54 from .19 to separate “host down” from “service down”.
Alerting: pin the incident timestamp immediately.

Attempt D: Bring up the monitoring stack (Prometheus + Grafana + blackbox_exporter)

D0. Pre-checks (network & ports)

From 192.168.22.19, verify reachability to 192.168.22.18:

bash
ping 192.168.22.18
Test-NetConnection 192.168.22.18 -Port 22

Ports used (defaults):

.18:9100 node_exporter
.18:9633 smartctl_exporter (optional)
.19:9090 Prometheus
.19:3000 Grafana
.19:9115 blackbox_exporter
.19:9091 Pushgateway (optional)

D1. Deploy node_exporter on 192.168.22.18 (systemd, stable)

Run on .18:

bash
export VER="1.7.0"
curl -fsSL -o /tmp/node_exporter.tar.gz "https://github.com/prometheus/node_exporter/releases/download/v${VER}/node_exporter-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/node_exporter.tar.gz
install -m 0755 /tmp/node_exporter-${VER}.linux-amd64/node_exporter /usr/local/bin/node_exporter

useradd --system --no-create-home --shell /usr/sbin/nologin nodeexp || true
mkdir -p /var/lib/node_exporter/textfile_collector
chown -R nodeexp:nodeexp /var/lib/node_exporter

cat >/etc/systemd/system/node_exporter.service <<'EOF'
[Unit]
Description=Prometheus Node Exporter
After=network-online.target
Wants=network-online.target

[Service]
User=nodeexp
Group=nodeexp
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=:9100 \
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now node_exporter
systemctl status node_exporter --no-pager

Verify (from .12 or any LAN host):

bash
curl -fsS http://192.168.22.18:9100/metrics | head

D2. (Recommended) Add a kernel/storage error counter as a custom metric

Use node_exporter textfile collector and a timer:

bash
cat >/usr/local/bin/collect_kernel_error_metrics.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

OUT="/var/lib/node_exporter/textfile_collector/kernel_errors.prom"
TMP="$(mktemp)"

COUNT="$(journalctl -k --since "10 min ago" --no-pager 2>/dev/null | \
  grep -Eic 'I/O error|timeout|reset|remount read-only|EXT4-fs error|BTRFS error|blk_update_request|nvme|ufs' || true)"

NOW="$(date +%s)"

{
  echo "# HELP kernel_error_events_10m Number of kernel error-like events in last 10 minutes"
  echo "# TYPE kernel_error_events_10m gauge"
  echo "kernel_error_events_10m ${COUNT}"
  echo "# HELP kernel_error_scrape_time_seconds Last collect time"
  echo "# TYPE kernel_error_scrape_time_seconds gauge"
  echo "kernel_error_scrape_time_seconds ${NOW}"
} >"${TMP}"

mv "${TMP}" "${OUT}"
EOF

chmod +x /usr/local/bin/collect_kernel_error_metrics.sh

cat >/etc/systemd/system/kernel-error-metrics.service <<'EOF'
[Unit]
Description=Collect kernel error metrics for node_exporter textfile collector

[Service]
Type=oneshot
ExecStart=/usr/local/bin/collect_kernel_error_metrics.sh
EOF

cat >/etc/systemd/system/kernel-error-metrics.timer <<'EOF'
[Unit]
Description=Run kernel error metrics collector every 60s

[Timer]
OnBootSec=30s
OnUnitActiveSec=60s
AccuracySec=1s

[Install]
WantedBy=timers.target
EOF

systemctl daemon-reload
systemctl enable --now kernel-error-metrics.timer

Verify:

bash
curl -fsS http://192.168.22.18:9100/metrics | grep -E '^kernel_error_events_10m|^kernel_error_scrape_time_seconds' || true

D3. (Optional) smartctl_exporter for disk health

If the storage device does not expose SMART/health, this may not work. Still worth trying.

bash
apt-get update
apt-get install -y smartmontools

export VER="0.12.0"
curl -fsSL -o /tmp/smartctl_exporter.tar.gz "https://github.com/prometheus-community/smartctl_exporter/releases/download/v${VER}/smartctl_exporter-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/smartctl_exporter.tar.gz
install -m 0755 /tmp/smartctl_exporter-${VER}.linux-amd64/smartctl_exporter /usr/local/bin/smartctl_exporter

useradd --system --no-create-home --shell /usr/sbin/nologin smartctl-exp || true

cat >/etc/systemd/system/smartctl_exporter.service <<'EOF'
[Unit]
Description=Prometheus Smartctl Exporter
After=network-online.target
Wants=network-online.target

[Service]
User=smartctl-exp
Group=smartctl-exp
ExecStart=/usr/local/bin/smartctl_exporter --web.listen-address=:9633
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now smartctl_exporter

D4. Deploy Prometheus + Grafana + blackbox_exporter on 192.168.22.12 (PVE) via LXC (recommended)

Since 192.168.22.12 is also a PVE host, avoid piling Docker onto the host OS. The simplest operationally is: one Debian LXC container running the monitoring stack with systemd.

D4.1 Create a Debian LXC container on PVE (.12)

You can do it in the PVE Web UI, or via CLI like this (replace <CTID>, <GW>; LXC IP is pinned to 192.168.22.19):

bash
# Run on the PVE host 192.168.22.12
pct create <CTID> local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname monitoring \
  --cores 2 --memory 2048 --swap 512 \
  --rootfs local-lvm:8 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.22.19/24,gw=<GW> \
  --unprivileged 1

pct set <CTID> -features keyctl=1,nesting=1
pct start <CTID>

D4.2 Install Prometheus in the container (systemd)

Enter the container:

bash
pct exec <CTID> -- bash

Inside the container:

bash
apt-get update
apt-get install -y ca-certificates curl tar

useradd --system --no-create-home --shell /usr/sbin/nologin prometheus || true
mkdir -p /etc/prometheus /var/lib/prometheus
chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

export VER="2.54.1"
curl -fsSL -o /tmp/prometheus.tar.gz "https://github.com/prometheus/prometheus/releases/download/v${VER}/prometheus-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/prometheus.tar.gz
install -m 0755 /tmp/prometheus-${VER}.linux-amd64/prometheus /usr/local/bin/prometheus
install -m 0755 /tmp/prometheus-${VER}.linux-amd64/promtool /usr/local/bin/promtool
cp -r /tmp/prometheus-${VER}.linux-amd64/consoles /etc/prometheus/
cp -r /tmp/prometheus-${VER}.linux-amd64/console_libraries /etc/prometheus/
chown -R prometheus:prometheus /etc/prometheus

Create /etc/prometheus/prometheus.yml (scrape .18 + active probes; DNS is the AdGuardDNS LXC at .54):

yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "minisforum-node"
    static_configs:
      - targets: ["192.168.22.18:9100"]

  - job_name: "minisforum-smartctl"
    static_configs:
      - targets: ["192.168.22.18:9633"]

  - job_name: "blackbox-icmp"
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
          - 192.168.22.18
          - 192.168.22.54
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

  - job_name: "blackbox-tcp"
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
          - 192.168.22.18:22
          - 192.168.22.18:9100
          - 192.168.22.54:53
          - 192.168.22.54:3000
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

Create the systemd service:

bash
cat >/etc/systemd/system/prometheus.service <<'EOF'
[Unit]
Description=Prometheus
After=network-online.target
Wants=network-online.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now prometheus

D4.3 Install blackbox_exporter in the container (systemd)

bash
useradd --system --no-create-home --shell /usr/sbin/nologin blackbox || true
mkdir -p /etc/blackbox_exporter
chown -R blackbox:blackbox /etc/blackbox_exporter

export VER="0.25.0"
curl -fsSL -o /tmp/blackbox.tar.gz "https://github.com/prometheus/blackbox_exporter/releases/download/v${VER}/blackbox_exporter-${VER}.linux-amd64.tar.gz"
tar -C /tmp -xzf /tmp/blackbox.tar.gz
install -m 0755 /tmp/blackbox_exporter-${VER}.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter

cat >/etc/blackbox_exporter/blackbox.yml <<'EOF'
modules:
  icmp:
    prober: icmp
    timeout: 5s

  tcp_connect:
    prober: tcp
    timeout: 5s
EOF

chown -R blackbox:blackbox /etc/blackbox_exporter

cat >/etc/systemd/system/blackbox_exporter.service <<'EOF'
[Unit]
Description=Prometheus Blackbox Exporter
After=network-online.target
Wants=network-online.target

[Service]
User=blackbox
Group=blackbox
ExecStart=/usr/local/bin/blackbox_exporter \
  --config.file=/etc/blackbox_exporter/blackbox.yml \
  --web.listen-address=:9115
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now blackbox_exporter

D4.4 Install Grafana in the container (systemd)

Do not commit any Grafana admin password. Install it, then set the password manually.

bash
apt-get install -y apt-transport-https software-properties-common wget gpg
mkdir -p /etc/apt/keyrings
wget -qO- https://apt.grafana.com/gpg.key | gpg --dearmor >/etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" >/etc/apt/sources.list.d/grafana.list
apt-get update
apt-get install -y grafana
systemctl enable --now grafana-server

Reset admin password (run in the container):

bash
grafana-cli admin reset-admin-password

Attempt E: Deploy AdGuardDNS on 192.168.22.18 via LXC (fixed IP: 192.168.22.54) + real DNS query probing

Goal: isolate DNS service from the host and pin the DNS IP to 192.168.22.54 so monitoring targets never drift.

E0. Create the AdGuardDNS LXC container on PVE (.18) with IP 192.168.22.54

Example (replace <CTID>, <GW>):

bash
# Run on the PVE host 192.168.22.18
pct create <CTID> local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname adguarddns \
  --cores 1 --memory 1024 --swap 256 \
  --rootfs local-lvm:4 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.22.54/24,gw=<GW> \
  --unprivileged 1

pct set <CTID> -features keyctl=1,nesting=1
pct start <CTID>

E1. Install AdGuard Home inside the container (as AdGuardDNS)