Scribbles for my memory

My always-on home server, an old Dell Latitude with an Intel 82579LM NIC, went silent for 22 minutes. The machine itself was fine. A reboot brought the network back, but a reboot doesn't tell you why it went down.

The kernel log from the previous boot does:

sudo journalctl -k -b -1 | grep -c "Detected Hardware Unit Hang"
sudo journalctl -k -b -1 | grep "Hardware Unit Hang" | sed -n '1p;$p'

There were 678 hits, two seconds apart, from the moment it dropped until I rebooted. The TX descriptor ring was stuck (TDH never moved), and the driver never managed to reset the card. The NIC simply stopped transmitting.

This is a long-known e1000e erratum on 82579/82574-era Intel NICs. The usual trigger is TCP segmentation offload. The standard mitigation is to stop the card doing TSO/GSO and let the CPU do it, which costs almost nothing at home-network speeds:

sudo apt install ethtool
sudo ethtool -K eno1 tso off gso off
sudo ethtool -k eno1 | grep -E '^(tcp-segmentation|generic-segmentation)-offload'

ethtool -K doesn't survive a reboot. If NetworkManager owns the interface, you don't need a udev rule or a systemd unit: NM can apply ethtool features itself on every activation:

sudo nmcli con modify "Wired connection 1" ethtool.feature-tso off ethtool.feature-gso off
nmcli -g ethtool.feature-tso,ethtool.feature-gso con show "Wired connection 1"

Applying it live only blips the link for a second, so it's safe even when you're working over that same connection, or from a VM running on the box.

If the hang ever comes back with offload off, the next suspects are Energy Efficient Ethernet and PCIe ASPM (pcie_aspm=off on the kernel command line).

ssh vm 'mytool --version' returns command not found while ssh vm followed by the same command works. The tool is installed, the interactive shell is fine — the difference is which startup files the command actually gets.

ssh vm 'command -v mytool'   # nothing, or only /usr/bin tools
ssh -t vm                     # interactive: mytool is right there

sshd runs a remote command as $SHELL -c 'cmd'. That shell is neither login nor interactive, so on Debian it reads neither ~/.profile (login only) nor ~/.bashrc (which returns at its case $- in *i*) ;; *) return;; esac guard). The session keeps sshd's stock PATH=/usr/local/bin:/usr/bin:/bin:/usr/games. Anything you installed to ~/.local/bin — cargo, pipx, uv tool, most Rust and Python tool installers — is invisible. Chasing it in ~/.bashrc won't work, because ~/.bashrc is not read at all.

The fix that survives every session type is PAM, not shell config. /etc/pam.d/sshd already has a pam_env.so user_readenv=1 line, and pam_env re-reads ~/.pam_environment on every session — no sshd edit, no restart:

cat > ~/.pam_environment <<'EOF'
PATH=/home/you/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
EOF
chmod 600 ~/.pam_environment
ssh vm 'command -v mytool'   # resolves now

Two details bite here. pam_env replaces PATH rather than appending, so the file must spell out the full list, not just the directory you care about. And it must use the bare PATH=… form: PATH DEFAULT=… only applies when the variable is unset, and sshd always exports PATH, so DEFAULT would silently do nothing.

Pair it with a block at the very top of ~/.bashrc, above the non-interactive guard, for nested shells and tmux children that do source .bashrc:

for _pb in "$HOME/.local/bin" "$HOME/bin"; do
    [ -d "$_pb" ] && case ":$PATH:" in *":$_pb:"*) ;; *) PATH="$_pb:$PATH" ;; esac
done
unset _pb
export PATH

The case ":$PATH:" check keeps it idempotent, and it matters more than it looks: once pam_env supplies ~/.local/bin, Debian's own unguarded PATH="$HOME/.local/bin:$PATH" in ~/.profile appends it a second time in every login shell. Harmless, but it misleads the next person reading echo $PATH.

The one case PAM cannot cover is a fully stripped environment — env -i mytool or a systemd unit with a hardcoded PATH= — where even login, interactive and ~/.bashrc are all bypassed. There the only options are to make bash read the file (BASH_ENV=/etc/profile) or put the entry in a system directory, e.g. sudo ln -s ~/.local/bin/mytool /usr/local/bin/mytool, which is already on every default PATH.

Check what a session really sees, rather than guessing:

ssh vm 'echo $PATH; command -v mytool'
ssh vm 'bash -lc "echo $PATH; command -v mytool"'   # login shell

A GUI is not required to configure Wi-Fi on Ubuntu, Debian, or Linux Mint. If NetworkManager is running, nmcli can scan for networks, create a persistent connection profile, and reconnect automatically after reboot.

Scan for nearby networks:

nmcli radio wifi on
nmcli device wifi rescan
nmcli device wifi list

Connect without putting the password in shell history:

read -rsp 'Wi-Fi password: ' WIFI_PASSWORD
echo
sudo nmcli device wifi connect 'SSID_NAME' \
  password "$WIFI_PASSWORD" \
  ifname wlan0
unset WIFI_PASSWORD

Replace SSID_NAME and wlan0 with the network name and wireless interface shown by nmcli device status. NetworkManager saves the new profile and normally enables autoconnect automatically.

Verify the address, route, and connectivity:

nmcli device status
nmcli connection show
ip addr show wlan0
ip route
ping -c 3 1.1.1.1

For a hidden network, add hidden yes; for an open network, omit the password argument. Do not switch networks over SSH unless you have another way back in, because the current connection may drop immediately.

Your server has no GUI — just the bare console TTY — and you want to read and type Chinese on it. Installing fonts does nothing: the kernel console draws with a ~256-glyph VGA font baked into the driver, so CJK renders as diamonds no matter what you install. The fix has two halves: fbterm draws text onto the framebuffer itself (display), and fcitx5 with its fbterm frontend supplies the input method (typing). Here's the whole setup, including the three gotchas that cost me an evening.

Everything below was done on Ubuntu 24.04 (Mint 22.3) over SSH; the target machine is an old laptop used headless. Swap davidwei for your user.

Display: fbterm

sudo apt install fbterm fonts-wqy-microhei
sudo usermod -aG video $USER   # needed for /dev/fb0; re-login to apply

Log in on the physical console and run fbterm — it opens a full terminal on the framebuffer using fontconfig fonts. It cannot start over SSH (no framebuffer there, and you don't need it: your SSH client renders CJK itself).

Font gotcha #1: fbterm renders TrueType fonts well but tends to clip outline fonts at the right/bottom of the character cell, and --font-width/--font-height only change the cell metrics — they don't scale the glyphs. WenQuanYi Zen Hei Mono with an 8px cell worked for me where Noto CJK and default settings clipped. Put it in ~/.fbtermrc:

font-names=WenQuanYi Zen Hei Mono
font-size=16
font-width=8

I run a lid-closed laptop with two external monitors as a remote-access host. Remote desktop into it works perfectly, but the physical monitors sitting on my desk were pitch black. Cables fine, monitors powered, nothing in the logs.

The screens weren't broken — they were just DPMS-blanked, and nothing I did remotely could wake them:

$ sudo XAUTHORITY=/var/run/lightdm/root/:0 DISPLAY=:0 xset q
DPMS (Display Power Management Signaling):
  Standby: 600    Suspend: 0    Off: 900
  DPMS is Enabled
  Monitor is Off

One command brings them back:

sudo XAUTHORITY=/var/run/lightdm/root/:0 DISPLAY=:0 xset dpms force on

The reason it happened at all is the part worth remembering: xrdp gives you a second X server. The console session is Xorg :0; your RDP session is Xorg :10. They're separate processes with separate idle timers. Every keystroke and mouse move you make remotely resets the idle timer on :10 — and :0 never hears about any of it. So :0 sits there accumulating idle time forever and blanks on schedule, no matter how busy you look on the other display.

If the host is supposed to stay lit, disable blanking in the console user's session, not yours. On XFCE:

xfconf-query -c xfce4-power-manager -p /xfce4-power-manager/dpms-enabled -s false
for k in blank-on-ac blank-on-battery dpms-on-ac-sleep dpms-on-ac-off \
         dpms-on-battery-sleep dpms-on-battery-off; do
  xfconf-query -c xfce4-power-manager -p /xfce4-power-manager/$k -s 0
done

Those only take effect when xfce4-power-manager restarts, so log the console user out and back in. Note the settings persist to ~/.config/xfce4/xfconf/xfce-perchannel-xml/, not ~/.config/xfconf/ — I looked in the wrong place first.

One trap after all that: I logged back in, confirmed DPMS is Disabled, and declared victory too early. xset q still read timeout: 600. That's X's own built-in screensaver, a completely separate mechanism that the power manager's blank-on-ac=0 never touched. DPMS controls whether the monitor gets powered down; the X screensaver just paints over the screen while it stays on. From a chair three feet away the two are indistinguishable, and remote input can't dismiss either one. Kill both, from the console user's autostart:

xset s off s noblank -dpms

Check for timeout: 0 and DPMS is Disabled before you believe it.