Scribbles for my memory

My Windows 11 dev box running AI agents through MSYS2/Git Bash lost about 2 GB of RAM a day to kernel paged pool. A reboot fixed it, and then it started climbing again. PoolMon-style tag stats put almost all of it under Toke (security token objects): ~3 million live tokens.

The tell-tale sign: millions of token objects, but only ~1,400 token handles system-wide. No process is holding them, so you won't find the culprit in Task Manager or with handle.exe. Something in the kernel is keeping a reference.

The culprit turns out to be a Windows 11 25H2/26H2 bug in win32kfull.sys (seen on 10.0.26100.94xx). Every new process that attaches to a console leaks one reference to its primary token whenever ForegroundLockTimeout is non-zero. Shell-heavy workloads hit it hard: each bash → child → grandchild spawn leaves a token behind. My WPR pool trace showed ~50 process starts per second from agent polling loops, leaking ~11 tokens per second.

The workaround is to set the timeout to 0. The registry value alone does nothing, though: at every logon win32k overwrites the live value with 2147483647, whatever HKCU\Control Panel\Desktop says. You have to set the live value with SystemParametersInfo, from inside the interactive session:

Add-Type -Namespace W -Name U -MemberDefinition @'
[DllImport("user32.dll", SetLastError=true)] public static extern bool SystemParametersInfo(uint a, uint b, ref uint c, uint d);
[DllImport("user32.dll", SetLastError=true)] public static extern bool SystemParametersInfo(uint a, uint b, IntPtr c, uint d);
'@
$v = [uint32]0
[void][W.U]::SystemParametersInfo(0x2000, 0, [ref]$v, 0)      # SPI_GETFOREGROUNDLOCKTIMEOUT
"before=$v"
[void][W.U]::SystemParametersInfo(0x2001, 0, [IntPtr]0, 2)    # SPI_SETFOREGROUNDLOCKTIMEOUT, SPIF_SENDCHANGE
[void][W.U]::SystemParametersInfo(0x2000, 0, [ref]$v, 0)
"after=$v"

The value resets at every logon, so run that script from a scheduled task with a logon trigger in interactive mode. This even works from an elevated SSH session:

schtasks /create /tn token-leak-flt-fix /sc onlogon /delay 0001:00 /it /f ^
  /tr "powershell.exe -NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -File C:\ProgramData\flt-fix.ps1"
schtasks /run /tn token-leak-flt-fix

Running the script over plain SSH would change session 0's value, which doesn't help. /it makes the task run in the logged-on user's session, where the console processes are. Mine went from before=2147483647 to after=0, and the leak rate dropped from 11.5 tokens/s to 0.26 tokens/s within minutes.

There are two catches. First, at 0 any app can steal foreground focus. Second, tokens that already leaked are freed only by a real Restart: with Fast Startup on, "Shut down" saves and restores the kernel, leak included.

I spent an hour on pool tracing before searching for "windows toke leaking", which turned up the answer right away (microsoft/terminal#20781, git-for-windows/git#6471, and the logon-fix write-up at gunrgbcr/windows-token-leak-logon-fix). When a symptom is this specific, search for it first.

My always-on home server, an old Dell Latitude with an Intel 82579LM NIC, went silent for 22 minutes. The machine itself was fine. A reboot brought the network back, but a reboot doesn't tell you why it went down.

The kernel log from the previous boot does:

sudo journalctl -k -b -1 | grep -c "Detected Hardware Unit Hang"
sudo journalctl -k -b -1 | grep "Hardware Unit Hang" | sed -n '1p;$p'

There were 678 hits, two seconds apart, from the moment it dropped until I rebooted. The TX descriptor ring was stuck (TDH never moved), and the driver never managed to reset the card. The NIC simply stopped transmitting.

This is a long-known e1000e erratum on 82579/82574-era Intel NICs. The usual trigger is TCP segmentation offload. The standard mitigation is to stop the card doing TSO/GSO and let the CPU do it, which costs almost nothing at home-network speeds:

sudo apt install ethtool
sudo ethtool -K eno1 tso off gso off
sudo ethtool -k eno1 | grep -E '^(tcp-segmentation|generic-segmentation)-offload'

ethtool -K doesn't survive a reboot. If NetworkManager owns the interface, you don't need a udev rule or a systemd unit: NM can apply ethtool features itself on every activation:

sudo nmcli con modify "Wired connection 1" ethtool.feature-tso off ethtool.feature-gso off
nmcli -g ethtool.feature-tso,ethtool.feature-gso con show "Wired connection 1"

Applying it live only blips the link for a second, so it's safe even when you're working over that same connection, or from a VM running on the box.

If the hang ever comes back with offload off, the next suspects are Energy Efficient Ethernet and PCIe ASPM (pcie_aspm=off on the kernel command line).

ssh vm 'mytool --version' returns command not found while ssh vm followed by the same command works. The tool is installed, the interactive shell is fine — the difference is which startup files the command actually gets.

ssh vm 'command -v mytool'   # nothing, or only /usr/bin tools
ssh -t vm                     # interactive: mytool is right there

sshd runs a remote command as $SHELL -c 'cmd'. That shell is neither login nor interactive, so on Debian it reads neither ~/.profile (login only) nor ~/.bashrc (which returns at its case $- in *i*) ;; *) return;; esac guard). The session keeps sshd's stock PATH=/usr/local/bin:/usr/bin:/bin:/usr/games. Anything you installed to ~/.local/bin — cargo, pipx, uv tool, most Rust and Python tool installers — is invisible. Chasing it in ~/.bashrc won't work, because ~/.bashrc is not read at all.

The fix that survives every session type is PAM, not shell config. /etc/pam.d/sshd already has a pam_env.so user_readenv=1 line, and pam_env re-reads ~/.pam_environment on every session — no sshd edit, no restart:

cat > ~/.pam_environment <<'EOF'
PATH=/home/you/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
EOF
chmod 600 ~/.pam_environment
ssh vm 'command -v mytool'   # resolves now

Two details bite here. pam_env replaces PATH rather than appending, so the file must spell out the full list, not just the directory you care about. And it must use the bare PATH=… form: PATH DEFAULT=… only applies when the variable is unset, and sshd always exports PATH, so DEFAULT would silently do nothing.

Pair it with a block at the very top of ~/.bashrc, above the non-interactive guard, for nested shells and tmux children that do source .bashrc:

for _pb in "$HOME/.local/bin" "$HOME/bin"; do
    [ -d "$_pb" ] && case ":$PATH:" in *":$_pb:"*) ;; *) PATH="$_pb:$PATH" ;; esac
done
unset _pb
export PATH

The case ":$PATH:" check keeps it idempotent, and it matters more than it looks: once pam_env supplies ~/.local/bin, Debian's own unguarded PATH="$HOME/.local/bin:$PATH" in ~/.profile appends it a second time in every login shell. Harmless, but it misleads the next person reading echo $PATH.

The one case PAM cannot cover is a fully stripped environment — env -i mytool or a systemd unit with a hardcoded PATH= — where even login, interactive and ~/.bashrc are all bypassed. There the only options are to make bash read the file (BASH_ENV=/etc/profile) or put the entry in a system directory, e.g. sudo ln -s ~/.local/bin/mytool /usr/local/bin/mytool, which is already on every default PATH.

Check what a session really sees, rather than guessing:

ssh vm 'echo $PATH; command -v mytool'
ssh vm 'bash -lc "echo $PATH; command -v mytool"'   # login shell

A Host block only tells SSH where to connect and which key to offer. A new VM can be reachable and still return Permission denied (publickey) until the guest account trusts that public key.

First confirm the client selected the intended host, user, and identity:

ssh -G vm-alias | grep -E '^(hostname|user|identityfile) '

Then, from the VM console or hypervisor, find the account's real home instead of assuming it is /home/$USER; some Lima guests use a suffixed path:

home=$(getent passwd "$(id -un)" | cut -d: -f6)
install -d -m 700 "$home/.ssh"
cat /path/to/client_key.pub >> "$home/.ssh/authorized_keys"
chmod 600 "$home/.ssh/authorized_keys"

Only the client's .pub key belongs on the guest; keep the private key on the client. Test the complete path with:

ssh -o BatchMode=yes vm-alias 'hostname; id -un'

ssh -G validates client configuration, not server-side authorization.

A GUI is not required to configure Wi-Fi on Ubuntu, Debian, or Linux Mint. If NetworkManager is running, nmcli can scan for networks, create a persistent connection profile, and reconnect automatically after reboot.

Scan for nearby networks:

nmcli radio wifi on
nmcli device wifi rescan
nmcli device wifi list

Connect without putting the password in shell history:

read -rsp 'Wi-Fi password: ' WIFI_PASSWORD
echo
sudo nmcli device wifi connect 'SSID_NAME' \
  password "$WIFI_PASSWORD" \
  ifname wlan0
unset WIFI_PASSWORD

Replace SSID_NAME and wlan0 with the network name and wireless interface shown by nmcli device status. NetworkManager saves the new profile and normally enables autoconnect automatically.

Verify the address, route, and connectivity:

nmcli device status
nmcli connection show
ip addr show wlan0
ip route
ping -c 3 1.1.1.1

For a hidden network, add hidden yes; for an open network, omit the password argument. Do not switch networks over SSH unless you have another way back in, because the current connection may drop immediately.