Scribbles for my memory

My tweet-queue service listens only on its Tailscale IP. Great for safety, annoying when I'm off the tailnet and want to post. The fix is not to rebind the service — it's to put a public nginx box that is also on the tailnet in front of it, and expose just the one path behind basic auth.

On the edge host (already a tailnet member, with a wildcard *.dev DNS record pointing at it):

server {
    listen 80;
    server_name x.dev.example.com;
    client_max_body_size 32m;          # match the backend's own body limit

    location = /add {
        auth_basic           "x-auto-poster";
        auth_basic_user_file /etc/nginx/.htpasswd;   # reuse the existing user file
        proxy_pass http://100.91.4.56:8090;          # backend's Tailscale IP
        proxy_set_header Authorization "";           # don't leak Basic creds upstream
    }

    location / { return 404; }
}

Then sudo certbot --nginx -d x.dev.example.com --redirect adds TLS and the HTTP→HTTPS redirect. With wildcard DNS there's no DNS step at all.

Three details that matter:

• location = /add plus a catch-all 404 means only the endpoint you meant to share is reachable — not every route the backend happens to have.
• proxy_set_header Authorization "" stops nginx's Basic header from reaching the backend. If the backend ever grows its own auth (or uses Authorization for something else, like an API key selector), a forwarded Basic header produces confusing 401/403s.
• Reusing one .htpasswd across sites is convenient, but every user in it can now hit every site that points at it. Fine for a single-user box; split the file once that stops being true.

Test it from outside: no creds should give 401, other paths 404.

curl -s -o /dev/null -w "%{http_code}\n" https://x.dev.example.com/add   # 401
curl -u user:pass -X POST https://x.dev.example.com/add \
  -H 'Content-Type: application/json' -d '{"text":"hello from anywhere"}'

For backing up small state files (a SQLite control.db, config, keys), Cloudflare R2's S3-compatible API drags in a client, signing and a separate pair of S3 credentials. The plain Cloudflare REST API can PUT and GET objects directly — one curl, one API token, no SDK:

curl -sS -X PUT "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/r2/buckets/$BUCKET/objects/$KEY" \
  -H "Authorization: Bearer $TOKEN" \
  --data-binary @snapshot.tar.age -H "Content-Type: application/octet-stream"

GET is the same with -o snapshot.tar.age, DELETE with -X DELETE. No egress fees on R2, and the free tier (10 GB storage, 1M Class A writes/month) dwarfs an hourly-snapshot workload.

Minting the token needs no dashboard click either — the Global API Key can do it through the API. The shape that finally worked (after a few 400s worth of resources-expression spelunking):

curl -sS -X POST "https://api.cloudflare.com/client/v4/user/tokens" \
  -H "X-Auth-Key: $CF_API_KEY" -H "X-Auth-Email: $CF_API_EMAIL" \
  -H "Content-Type: application/json" --data '{
  "name": "snapshots-backup",
  "policies": [{"effect": "allow",
    "resources": {"com.cloudflare.api.account.<ACCOUNT_ID>": "*"},
    "permission_groups": [
      {"id": "b4992e1108244f5d8bfbd5744320c2e1"},
      {"id": "bf7481a1826f439697cb59a20b22293e"}
    ]}],
  "expires_on": "2027-10-17T00:00:00Z"
}'

Those two group ids are Workers R2 Storage Read and Write. The response carries the token value exactly once — pipe it straight to its destination file (umask 077, owner root, 0600) and never let it touch a terminal or a shell variable you might echo.

Two traps when verifying from the host that holds the credential:

• ssh -n redirects stdin from /dev/null and silently wins over < file — the credential file lands 0 bytes. Drop -n when piping stdin.
• If the credential file is root-only, curl -H "Bearer $(cat ...)" in your ssh shell reads nothing and the API answers 9106: Missing Authorization header. Run the whole probe under sudo sh -c so the expansion happens where the file is readable.

Encrypt before upload so the host leak doesn't matter: age with the public key on the host and the private key in your password manager. Restore = GET on the operator machine, decrypt, PRAGMA integrity_check.

My Windows 11 dev box running AI agents through MSYS2/Git Bash lost about 2 GB of RAM a day to kernel paged pool. A reboot fixed it, and then it started climbing again. PoolMon-style tag stats put almost all of it under Toke (security token objects): ~3 million live tokens.

The tell-tale sign: millions of token objects, but only ~1,400 token handles system-wide. No process is holding them, so you won't find the culprit in Task Manager or with handle.exe. Something in the kernel is keeping a reference.

The culprit turns out to be a Windows 11 25H2/26H2 bug in win32kfull.sys (seen on 10.0.26100.94xx). Every new process that attaches to a console leaks one reference to its primary token whenever ForegroundLockTimeout is non-zero. Shell-heavy workloads hit it hard: each bash → child → grandchild spawn leaves a token behind. My WPR pool trace showed ~50 process starts per second from agent polling loops, leaking ~11 tokens per second.

The workaround is to set the timeout to 0. The registry value alone does nothing, though: at every logon win32k overwrites the live value with 2147483647, whatever HKCU\Control Panel\Desktop says. You have to set the live value with SystemParametersInfo, from inside the interactive session:

Add-Type -Namespace W -Name U -MemberDefinition @'
[DllImport("user32.dll", SetLastError=true)] public static extern bool SystemParametersInfo(uint a, uint b, ref uint c, uint d);
[DllImport("user32.dll", SetLastError=true)] public static extern bool SystemParametersInfo(uint a, uint b, IntPtr c, uint d);
'@
$v = [uint32]0
[void][W.U]::SystemParametersInfo(0x2000, 0, [ref]$v, 0)      # SPI_GETFOREGROUNDLOCKTIMEOUT
"before=$v"
[void][W.U]::SystemParametersInfo(0x2001, 0, [IntPtr]0, 2)    # SPI_SETFOREGROUNDLOCKTIMEOUT, SPIF_SENDCHANGE
[void][W.U]::SystemParametersInfo(0x2000, 0, [ref]$v, 0)
"after=$v"

The value resets at every logon, so run that script from a scheduled task with a logon trigger in interactive mode. This even works from an elevated SSH session:

schtasks /create /tn token-leak-flt-fix /sc onlogon /delay 0001:00 /it /f ^
  /tr "powershell.exe -NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -File C:\ProgramData\flt-fix.ps1"
schtasks /run /tn token-leak-flt-fix

Running the script over plain SSH would change session 0's value, which doesn't help. /it makes the task run in the logged-on user's session, where the console processes are. Mine went from before=2147483647 to after=0, and the leak rate dropped from 11.5 tokens/s to 0.26 tokens/s within minutes.

There are two catches. First, at 0 any app can steal foreground focus. Second, tokens that already leaked are freed only by a real Restart: with Fast Startup on, "Shut down" saves and restores the kernel, leak included.

I spent an hour on pool tracing before searching for "windows toke leaking", which turned up the answer right away (microsoft/terminal#20781, git-for-windows/git#6471, and the logon-fix write-up at gunrgbcr/windows-token-leak-logon-fix). When a symptom is this specific, search for it first.

My always-on home server, an old Dell Latitude with an Intel 82579LM NIC, went silent for 22 minutes. The machine itself was fine. A reboot brought the network back, but a reboot doesn't tell you why it went down.

The kernel log from the previous boot does:

sudo journalctl -k -b -1 | grep -c "Detected Hardware Unit Hang"
sudo journalctl -k -b -1 | grep "Hardware Unit Hang" | sed -n '1p;$p'

There were 678 hits, two seconds apart, from the moment it dropped until I rebooted. The TX descriptor ring was stuck (TDH never moved), and the driver never managed to reset the card. The NIC simply stopped transmitting.

This is a long-known e1000e erratum on 82579/82574-era Intel NICs. The usual trigger is TCP segmentation offload. The standard mitigation is to stop the card doing TSO/GSO and let the CPU do it, which costs almost nothing at home-network speeds:

sudo apt install ethtool
sudo ethtool -K eno1 tso off gso off
sudo ethtool -k eno1 | grep -E '^(tcp-segmentation|generic-segmentation)-offload'

ethtool -K doesn't survive a reboot. If NetworkManager owns the interface, you don't need a udev rule or a systemd unit: NM can apply ethtool features itself on every activation:

sudo nmcli con modify "Wired connection 1" ethtool.feature-tso off ethtool.feature-gso off
nmcli -g ethtool.feature-tso,ethtool.feature-gso con show "Wired connection 1"

Applying it live only blips the link for a second, so it's safe even when you're working over that same connection, or from a VM running on the box.

If the hang ever comes back with offload off, the next suspects are Energy Efficient Ethernet and PCIe ASPM (pcie_aspm=off on the kernel command line).

ssh vm 'mytool --version' returns command not found while ssh vm followed by the same command works. The tool is installed, the interactive shell is fine — the difference is which startup files the command actually gets.

ssh vm 'command -v mytool'   # nothing, or only /usr/bin tools
ssh -t vm                     # interactive: mytool is right there

sshd runs a remote command as $SHELL -c 'cmd'. That shell is neither login nor interactive, so on Debian it reads neither ~/.profile (login only) nor ~/.bashrc (which returns at its case $- in *i*) ;; *) return;; esac guard). The session keeps sshd's stock PATH=/usr/local/bin:/usr/bin:/bin:/usr/games. Anything you installed to ~/.local/bin — cargo, pipx, uv tool, most Rust and Python tool installers — is invisible. Chasing it in ~/.bashrc won't work, because ~/.bashrc is not read at all.

The fix that survives every session type is PAM, not shell config. /etc/pam.d/sshd already has a pam_env.so user_readenv=1 line, and pam_env re-reads ~/.pam_environment on every session — no sshd edit, no restart:

cat > ~/.pam_environment <<'EOF'
PATH=/home/you/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
EOF
chmod 600 ~/.pam_environment
ssh vm 'command -v mytool'   # resolves now

Two details bite here. pam_env replaces PATH rather than appending, so the file must spell out the full list, not just the directory you care about. And it must use the bare PATH=… form: PATH DEFAULT=… only applies when the variable is unset, and sshd always exports PATH, so DEFAULT would silently do nothing.

Pair it with a block at the very top of ~/.bashrc, above the non-interactive guard, for nested shells and tmux children that do source .bashrc:

for _pb in "$HOME/.local/bin" "$HOME/bin"; do
    [ -d "$_pb" ] && case ":$PATH:" in *":$_pb:"*) ;; *) PATH="$_pb:$PATH" ;; esac
done
unset _pb
export PATH

The case ":$PATH:" check keeps it idempotent, and it matters more than it looks: once pam_env supplies ~/.local/bin, Debian's own unguarded PATH="$HOME/.local/bin:$PATH" in ~/.profile appends it a second time in every login shell. Harmless, but it misleads the next person reading echo $PATH.

The one case PAM cannot cover is a fully stripped environment — env -i mytool or a systemd unit with a hardcoded PATH= — where even login, interactive and ~/.bashrc are all bypassed. There the only options are to make bash read the file (BASH_ENV=/etc/profile) or put the entry in a system directory, e.g. sudo ln -s ~/.local/bin/mytool /usr/local/bin/mytool, which is already on every default PATH.

Check what a session really sees, rather than guessing:

ssh vm 'echo $PATH; command -v mytool'
ssh vm 'bash -lc "echo $PATH; command -v mytool"'   # login shell