Temporary Egress with SSH Reverse SOCKS and KubeVirt

I needed to lend a Singapore server a temporary network egress for one weekend. The egress had to be my Ubuntu VM inside a local K3s cluster, not the Singapore server itself. It also had to survive terminal disconnects and transient network failures, stop after exactly four days, and leave no reusable credential behind.

The final design was small:

• OpenSSH remote dynamic forwarding exposed a SOCKS5 listener only on the Singapore server's loopback interface.
• A systemd service on the Ubuntu VM kept the SSH connection alive and restarted it after failures.
• An absolute systemd timer stopped and disabled the service at the deadline.
• Live socket inspection showed destination IPs and ports without pretending that HTTPS payloads were readable.
• Cleanup revoked the public key first, then removed the local private key and every tunnel artifact.

This post documents the complete path, including the mistakes. All public addresses, usernames, and key material below are sanitized examples.

TL;DR

The command that made the Ubuntu VM the egress was:

bashCopy
ssh -N -T \
  -i ~/.ssh/sg-egress-ed25519 \
  -o BatchMode=yes \
  -o IdentitiesOnly=yes \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=~/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  -o ServerAliveInterval=30 \
  -o ServerAliveCountMax=3 \
  -o ExitOnForwardFailure=yes \
  -R 127.0.0.1:1080 \
  sg-egress@203.0.113.10

The important detail is that -R 127.0.0.1:1080 has no fixed destination. With modern OpenSSH, that creates a remote dynamic SOCKS listener. Applications on the Singapore server connect to 127.0.0.1:1080; the Ubuntu-side SSH client then opens the destination connections, so the Internet sees the Ubuntu VM's upstream public address.

Verification from the Singapore server:

bashCopy
curl --proxy socks5h://127.0.0.1:1080 https://ifconfig.me/ip

If the result matches the Ubuntu VM's direct egress IP and differs from the Singapore server's direct IP, the data path is correct.

Requirement and safety boundaries

This was intentionally a narrow, temporary capability:

1. Only processes on the Singapore server could use the proxy.
2. The SOCKS port was never exposed on 0.0.0.0.
3. The SSH key belonged to an unprivileged account that could only open the required remote listener.
4. The tunnel recovered automatically, but the deadline could not slide after a reboot.
5. The key expired server-side at the same deadline and was revoked before its private half was deleted.

Binding the proxy to 127.0.0.1 matters. Using this instead:

textCopy
-R 0.0.0.0:1080

would request a public listener. The client address alone is not a sufficient guard: GatewayPorts yes on the SSH server forces remote forwards onto wildcard addresses even when the client requests 127.0.0.1. I therefore enforced GatewayPorts no and PermitListen 127.0.0.1:1080 for the tunnel account, then verified the live listener.

Topology and data path

The Ubuntu VM ran in KubeVirt on K3s. Its SSH service was published through a NodePort on the worker node.

The direction is easy to misread. The SSH control connection starts on the Ubuntu VM and goes to Singapore, but proxied application traffic starts on Singapore and exits from Ubuntu.

Confirm that I was entering the guest, not virt-launcher

The KubeVirt pod contained the QEMU process. Running kubectl exec into virt-launcher would enter the compute container, not the Ubuntu guest OS.

I first confirmed the VMI and its SSH Service:

bashCopy
kubectl -n kubevirt-vms get vmi ubuntu-vm -o wide
kubectl -n kubevirt-vms get service ubuntu-vm-ssh -o wide

The Service mapped NodePort 30022 to guest port 22, so guest access was:

bashCopy
ssh -p 30022 vmuser@192.168.22.31

For one-off administration I also used a local port-forward:

bashCopy
kubectl -n kubevirt-vms port-forward service/ubuntu-vm-ssh 2222:22
ssh -p 2222 vmuser@127.0.0.1

These ports have different meanings:

• 30022 is the persistent K3s NodePort.
• 2222 was a temporary port on my Mac created by kubectl port-forward.
• 1080 was the SOCKS listener on the Singapore server.

Also note the CLI detail: ssh uses lowercase -p for a port; scp uses uppercase -P.

Prepare a dedicated SSH credential and deadline

The private key lived only on the Ubuntu VM and was mode 0600:

bashCopy
install -d -m 700 ~/.ssh
install -m 600 /secure/input/sg-egress-ed25519 ~/.ssh/sg-egress-ed25519
ssh-keygen -lf ~/.ssh/sg-egress-ed25519

I calculated the deadline once and derived the OpenSSH key-expiry format from the same value:

bashCopy
deadline=$(date -u -d '+96 hours' '+%Y-%m-%d %H:%M:%S UTC')
key_expiry=$(date -u -d "$deadline" '+%Y%m%d%H%M%SZ')
public_key=$(ssh-keygen -y -f ~/.ssh/sg-egress-ed25519)
printf 'deadline=%s\nkey_expiry=%s\n' "$deadline" "$key_expiry"
printf 'expiry-time="%s" %s sg-egress-four-day\n' "$key_expiry" "$public_key"

I transferred only the final public-key line through an authenticated administrative channel. On the Singapore server, a separate administrator created a dedicated account and installed that line as /home/sg-egress/.ssh/authorized_keys:

bashCopy
sudo useradd --create-home --user-group --shell /usr/sbin/nologin sg-egress
sudo install -d -o sg-egress -g sg-egress -m 700 /home/sg-egress/.ssh
sudo install -o sg-egress -g sg-egress -m 600 \
  /secure/input/sg-egress-authorized-key \
  /home/sg-egress/.ssh/authorized_keys

I also installed /etc/ssh/sshd_config.d/sg-egress.conf:

textCopy
Match User sg-egress
    AuthenticationMethods publickey
    PasswordAuthentication no
    KbdInteractiveAuthentication no
    AllowAgentForwarding no
    AllowStreamLocalForwarding no
    AllowTcpForwarding remote
    GatewayPorts no
    PermitListen 127.0.0.1:1080
    PermitTTY no
    PermitTunnel no
    PermitUserRC no
    X11Forwarding no
    MaxSessions 0

Match all

MaxSessions 0 denies shell, command, and subsystem sessions while still permitting forwarding. AllowTcpForwarding remote rejects local forwards, and PermitListen limits the only allowed remote listener. I validated and reloaded the server configuration before using the key:

bashCopy
sudo sshd -t
sudo systemctl reload ssh
sudo sshd -T -C user=sg-egress,host=localhost,addr=127.0.0.1 | \
  grep -E '^(allowagentforwarding|allowstreamlocalforwarding|allowtcpforwarding|gatewayports|maxsessions|permitlisten|permittty|permittunnel|permituserrc|x11forwarding) '

Strict host-key checking also needs an explicit trust bootstrap. From an already authenticated administrative console on Singapore, I recorded the ED25519 host-key fingerprint:

bashCopy
sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

On the Ubuntu VM, running as vmuser, I scanned the public host key into a dedicated file but refused to install it until its fingerprint exactly matched the value obtained through that trusted channel. The tunnel never reads or modifies the shared known_hosts file:

bashCopy
pinned_hosts=$(mktemp ~/.ssh/sg-egress-known_hosts.XXXXXX)
trap 'test ! -e "$pinned_hosts" || unlink "$pinned_hosts"' EXIT

ssh-keyscan -t ed25519 203.0.113.10 > "$pinned_hosts"
scanned_fingerprint=$(ssh-keygen -lf "$pinned_hosts" | awk 'NR == 1 { print $2 }')
read -r -p 'Trusted ED25519 SHA256 fingerprint: ' trusted_fingerprint

if [ -z "$scanned_fingerprint" ] || \
   [ -z "$trusted_fingerprint" ] || \
   [ "$scanned_fingerprint" != "$trusted_fingerprint" ]; then
  printf 'host-key fingerprint mismatch\n' >&2
  exit 1
fi

chmod 600 "$pinned_hosts"
mv "$pinned_hosts" ~/.ssh/sg-egress-known_hosts
trap - EXIT

ssh-keygen -F 203.0.113.10 -f ~/.ssh/sg-egress-known_hosts
ssh-keygen -lf ~/.ssh/sg-egress-known_hosts

ssh-keyscan only collected the presented key; it was not the trust source. The independently obtained fingerprint was the trust source. Every tunnel command explicitly uses this file and ignores global host-key files, so exactly the verified key is trusted without changing unrelated SSH state.

Do not paste a production private key into a ticket, chat, shell history, or repository. Use a dedicated short-lived key and an authenticated transfer mechanism. If a private key is ever disclosed, revocation is mandatory; deleting one local copy is not enough.

Why the first -R idea was wrong

The first mental model was a reverse SSH login port:

bashCopy
ssh -N -R 127.0.0.1:10022:127.0.0.1:22 sg-egress@203.0.113.10

That makes Singapore's 127.0.0.1:10022 forward to the Ubuntu VM's SSH port. It lets the Singapore server log back into the VM, but it does not make Ubuntu a general Internet egress.

The corrected command removed the fixed destination:

bashCopy
ssh -N -T \
  -i ~/.ssh/sg-egress-ed25519 \
  -o BatchMode=yes \
  -o IdentitiesOnly=yes \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=~/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  -o ServerAliveInterval=30 \
  -o ServerAliveCountMax=3 \
  -o TCPKeepAlive=yes \
  -o ConnectTimeout=15 \
  -o ExitOnForwardFailure=yes \
  -R 127.0.0.1:1080 \
  sg-egress@203.0.113.10

Now 1080 is a SOCKS5 endpoint, and destinations are selected per client request.

Make it survive a weekend with systemd

A foreground terminal is not a reliability mechanism. I installed this unit as /etc/systemd/system/sg-egress-socks.service:

iniCopy
[Unit]
Description=Reverse SOCKS5 tunnel using Ubuntu VM as SG egress
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=0

[Service]
Type=simple
User=vmuser
Environment=LANG=C
ExecStart=/usr/bin/ssh -N -T \
  -i /home/vmuser/.ssh/sg-egress-ed25519 \
  -o BatchMode=yes \
  -o IdentitiesOnly=yes \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=/home/vmuser/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  -o ServerAliveInterval=30 \
  -o ServerAliveCountMax=3 \
  -o TCPKeepAlive=yes \
  -o ConnectTimeout=15 \
  -o ExitOnForwardFailure=yes \
  -R 127.0.0.1:1080 \
  sg-egress@203.0.113.10
Restart=always
RestartSec=5s
TimeoutStopSec=15s
NoNewPrivileges=yes
PrivateTmp=yes

[Install]
WantedBy=multi-user.target

ExitOnForwardFailure=yes is essential. Without it, SSH could stay connected even when it failed to bind the remote port, leaving systemd with a green process and a broken service.

Before enabling the unit, I validated it:

bashCopy
systemd-analyze verify /etc/systemd/system/sg-egress-socks.service

Then:

bashCopy
sudo systemctl daemon-reload
sudo systemctl enable --now sg-egress-socks.service
systemctl is-enabled sg-egress-socks.service
systemctl is-active sg-egress-socks.service

Stop after exactly 96 hours

RuntimeMaxSec=4d looks attractive, but combining a runtime limit with an automatic restart policy creates ambiguous deadline behavior. I wanted a deadline that survived reboots and never moved. The deadline and key_expiry values were already calculated from the same instant when the key was prepared; the timer had to consume that exact deadline rather than a copied example timestamp.

The stop action was /etc/systemd/system/sg-egress-socks-stop.service:

iniCopy
[Unit]
Description=Stop and disable the SG egress SOCKS tunnel

[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl disable sg-egress-socks.service
ExecStart=/usr/bin/systemctl stop sg-egress-socks.service

I rejected an empty or expired value, checked how systemd parsed it, and generated the timer directly from the shell variable:

bashCopy
test -n "${deadline:-}" || {
  printf 'deadline is not set\n' >&2
  exit 1
}
deadline_epoch=$(date -u -d "$deadline" '+%s')
test "$deadline_epoch" -gt "$(date -u '+%s')" || {
  printf 'deadline is not in the future: %s\n' "$deadline" >&2
  exit 1
}
systemd-analyze calendar "$deadline"

sudo tee /etc/systemd/system/sg-egress-socks-stop.timer >/dev/null <<EOF
[Unit]
Description=Stop the SG egress SOCKS tunnel after the four-day window

[Timer]
OnCalendar=$deadline
AccuracySec=1s
Persistent=yes
Unit=sg-egress-socks-stop.service

[Install]
WantedBy=timers.target
EOF

Persistent=yes ensures that a missed deadline fires after the VM returns. The stop action also disables the tunnel unit, so a later reboot cannot reopen it. The matching expiry-time on the Singapore server prevents a new SSH authentication after the same deadline even if the client-side timer fails.

bashCopy
sudo systemctl daemon-reload
sudo systemctl enable --now sg-egress-socks-stop.timer
systemctl list-timers --all sg-egress-socks-stop.timer

Verify the actual egress

I compared three observations:

bashCopy
# On the Ubuntu VM
curl -4fsS https://ifconfig.me/ip

# Directly on the Singapore server
curl -4fsS https://ifconfig.me/ip

# On Singapore, through the reverse SOCKS tunnel
curl -4fsS \
  --proxy socks5h://127.0.0.1:1080 \
  https://ifconfig.me/ip

The expected relationship is:

textCopy
SG via SOCKS IP == Ubuntu direct IP
SG direct IP     != Ubuntu direct IP

I also checked the listener on Singapore:

bashCopy
ss -lntp 'sport = :1080'

It had to show 127.0.0.1:1080, not 0.0.0.0:1080.

Finally, I killed the SSH main process once and watched systemd recover. The PID changed, NRestarts increased, and the SOCKS test worked again seven seconds later. This tested the failure mode I actually cared about instead of merely checking that the initial start succeeded.

Observe live traffic without inventing visibility

The SSH client process opens destination sockets on the Ubuntu VM. Its live connections can be inspected with:

bashCopy
tunnel_pid=$(systemctl show sg-egress-socks.service -p MainPID --value)
sudo lsof -nP -a -p "$tunnel_pid" -i

For a persistent terminal view:

bashCopy
tmux new-session -A -s sg-egress-monitor
watch -n 1 'pid=$(systemctl show sg-egress-socks.service -p MainPID --value); printf "tunnel_pid=%s\n" "$pid"; if [ "$pid" != 0 ]; then sudo -n lsof -nP -a -p "$pid" -i; else printf "service is not running\n"; fi'

Detach with Ctrl-b, then d.

This view has strict limits:

• It shows active destination IPs and ports.
• It may miss very short connections between one-second samples.
• It does not reconstruct historical flows.
• HTTPS payloads, credentials, and full URLs remain encrypted.
• DNS names may be visible only when ordinary DNS or additional auditing captures them.

If historical accounting is a real requirement, enable consented flow logging before the window starts. A live lsof screen is not an audit log.

Failures encountered

1. One IP-check endpoint failed

api.ipify.org:443 returned Connection refused, and a Cloudflare endpoint timed out from the Ubuntu egress. Other independent endpoints returned the same valid public address.

The lesson was not to change the tunnel because one measurement target failed. Test DNS, TCP reachability, and at least two unrelated endpoints before blaming the data path.

2. Remote port 1080 was occupied

The service later accumulated 1,428 failures over roughly two hours:

textCopy
Error: remote port forwarding failed for listen port 1080

ExitOnForwardFailure=yes made each failed bind visible to systemd, which retried every five seconds. The evidence proved a port conflict on Singapore; it did not prove whether the owner was a stale SSH session or another manually started tunnel.

The correct investigation was:

bashCopy
ss -lntp 'sport = :1080'
ps -fp <listener-pid>

Do not kill a listener based only on a port number. Identify the owning process and connection first. Once the conflicting listener disappeared, the managed service acquired 1080 and stayed active.

3. A multiline paste broke the tmux monitor

The shell received actual newlines inside the service name and options:

textCopy
sg-egress-
socks.service
-p
MainPID

That produced socks.service: not found, -p: not found, and an empty PID that made lsof -a invalid. Terminal visual wrapping is harmless; embedded newline characters are not. Replacing the stored tmux command with one literal shell line fixed the monitor.

4. A green control connection does not imply proxy traffic

The monitor always showed one established socket from Ubuntu to Singapore port 22. That was the SSH control connection. Additional destination sockets appeared only while a client actually used the SOCKS proxy.

During a ten-second sample, Singapore had no established client to 127.0.0.1:1080 and no speed-test process. The correct conclusion was simply “the proxy is idle now,” not “the monitoring is broken.”

Complete revocation and cleanup

The order matters. Stop the active tunnel first, revoke the public key through a separate administrative account, verify that the old key cannot create a new tunnel, and only then delete the private key.

The remote scripts below use stdin for their bodies, so they cannot also use stdin for a sudo password. Before changing any state, I verified that the separate administrator had pre-existing non-interactive sudo access:

bashCopy
ssh \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=~/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  admin@203.0.113.10 \
  sudo -n true

If that check fails, stop here and run the privileged command bodies directly from an already authenticated Singapore management console. Do not send a sudo password through the script stream.

1. Stop the active tunnel but keep the private key

bashCopy
sudo systemctl disable --now sg-egress-socks-stop.timer
sudo systemctl disable --now sg-egress-socks.service
sudo systemctl stop sg-egress-socks-stop.service

Stopping first closes the already-authenticated SSH connection. Removing an authorized key does not terminate sessions that are already established.

2. Remove only the matching public key on Singapore

From the Ubuntu VM, derive the public blob without printing private material. Use a separate administrative account on Singapore to pass it as one controlled argument, require exactly one match, and replace authorized_keys atomically. The restricted tunnel key cannot and must not execute this cleanup script:

bashCopy
key_blob=$(ssh-keygen -y -f ~/.ssh/sg-egress-ed25519 | awk '{print $2}')

ssh \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=~/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  admin@203.0.113.10 \
  sudo -n bash -s -- "$key_blob" <<'REMOTE'
set -eu
key_blob=$1
auth_file=/home/sg-egress/.ssh/authorized_keys
matches=$(awk -v key_blob="$key_blob" \
  'index($0, key_blob) { count++ } END { print count+0 }' \
  "$auth_file")

if [ "$matches" -ne 1 ]; then
  printf 'refusing to edit: expected 1 key, found %s\n' "$matches" >&2
  exit 1
fi

temp_auth=$(mktemp /home/sg-egress/.ssh/authorized_keys.cleanup.XXXXXX)
trap 'test ! -e "$temp_auth" || unlink "$temp_auth"' EXIT

awk -v key_blob="$key_blob" \
  'index($0, key_blob) == 0' \
  "$auth_file" > "$temp_auth"

chmod --reference="$auth_file" "$temp_auth"
chown --reference="$auth_file" "$temp_auth"
mv "$temp_auth" "$auth_file"
REMOTE

An OpenSSH public-key blob contains a restricted base64 alphabet, but the match-count guard remains important: cleanup must not silently remove zero keys or multiple keys.

Then verify that the old key cannot authenticate a new forwarding connection. Because the managed service is already stopped, port 1080 is free; require an explicit Permission denied (publickey) result rather than treating a timeout or network error as successful revocation:

bashCopy
timeout 10s ssh -N -T \
  -i ~/.ssh/sg-egress-ed25519 \
  -o BatchMode=yes \
  -o IdentitiesOnly=yes \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=~/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  -o ConnectTimeout=5 \
  -o ExitOnForwardFailure=yes \
  -R 127.0.0.1:1080 \
  sg-egress@203.0.113.10

The expected result is Permission denied (publickey).

The administrator can now remove the dedicated account and its scoped SSH configuration:

bashCopy
ssh \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=~/.ssh/sg-egress-known_hosts \
  -o GlobalKnownHostsFile=/dev/null \
  admin@203.0.113.10 \
  sudo -n bash -s <<'REMOTE'
set -eu
userdel -r sg-egress
unlink /etc/ssh/sshd_config.d/sg-egress.conf
sshd -t
systemctl reload ssh
REMOTE

3. Remove the Ubuntu-side runtime artifacts

bashCopy
tmux kill-session -t sg-egress-monitor

sudo unlink /etc/systemd/system/sg-egress-socks.service
sudo unlink /etc/systemd/system/sg-egress-socks-stop.service
sudo unlink /etc/systemd/system/sg-egress-socks-stop.timer
sudo systemctl daemon-reload

unlink ~/.ssh/sg-egress-ed25519
unlink ~/.ssh/sg-egress-known_hosts

Remove any explicitly known temporary copies as well. Avoid broad globs when credentials are involved.

System journals were retained as an audit trail. Selectively erasing one unit's records is not a normal journal operation, while vacuuming the whole journal would destroy unrelated operational evidence.

4. Final checks

bashCopy
systemctl show sg-egress-socks.service -p LoadState --value
systemctl list-timers --all 'sg-egress-socks*'
pgrep -af '[s]sh.*-R.*1080'
tmux has-session -t sg-egress-monitor

The final state was:

• all three units: not-found
• no matching timer
• no ssh -R process
• no monitor tmux session
• no private key, dedicated host-key file, or temporary unit copies on Ubuntu
• no dedicated sg-egress account or scoped SSH configuration on Singapore
• the old key rejected by Singapore
• no listener on Singapore 127.0.0.1:1080

What I would keep for the next temporary egress

1. Treat reverse login forwarding and reverse dynamic forwarding as different tools.
2. Bind temporary proxies to loopback unless broader exposure is explicitly required.
3. Combine SSH keepalives with systemd restart and ExitOnForwardFailure=yes.
4. Use an absolute persistent deadline, then disable the service at expiry.
5. Verify the egress by comparing direct and proxied public addresses.
6. Decide before launch whether live observation or historical auditing is required.
7. Revoke the server-side public key before deleting the local private key.
8. Preserve logs, but remove credentials and runtime artifacts.

The tunnel itself was one line. Making it reliable, bounded, observable, and revocable was the actual engineering work.