Posts in category “Linux”

kitty showed huge gaps between characters — `monospace` was resolving to a CJK font

Every character in kitty had a massive gap after it, like the cell width was double what it should be. The obvious first move — swap font_family in kitty.conf — did nothing. Neither did the version a second pair of hands tried. Two failed attempts, and the config looked completely normal.

The config wasn't the problem. monospace is just a fontconfig alias, and on this machine fontconfig was handing it to a CJK font:

$ fc-match monospace
NotoSansCJK-Regular.ttc: "Noto Sans Mono CJK SC" "Regular"

kitty bases its cell width on the primary font, and a CJK font's cell is full-width — so narrow ASCII glyphs end up stranded in the middle of wide blank cells. The same kitty.conf was perfectly fine on another machine, because over there fc-match monospace returns DejaVu Sans Mono.

What was poisoning the alias was a file Ubuntu's language-selector drops in:

$ grep -A6 '<family>monospace</family>' \
    ~/.config/fontconfig/conf.d/64-language-selector-prefer.conf
		<family>monospace</family>
		<prefer>
			<family>Noto Sans Mono CJK SC</family>
			<family>Noto Sans Mono CJK JP</family>
			...

It prepends the Noto CJK mono fonts to the monospace alias, so they outrank every Latin mono. Comment out that one <alias> block and monospace goes back to DejaVu:

$ fc-match monospace
DejaVuSansMono.ttf: "DejaVu Sans Mono" "Book"

The reload trap that made this take an hour

Here's the part that hurts. After fixing fontconfig, pressing ctrl+shift+F5 to reload kitty... still showed the gaps. It looked exactly like the fix hadn't worked — the same dead end that had already wasted two attempts.

kitty caches its fontconfig resolution in-process. ctrl+shift+F5 re-reads kitty.conf, but it never re-queried fontconfig, so a change to how monospace resolves never reached the running kitty. Only quitting and reopening kitty — a fresh process — picked it up.

So the rule, painfully earned: when a kitty font change "doesn't work," before you conclude the fix is wrong, fully restart kitty and re-check. The fix may have been right all along; you were just looking at a cached font. And when the symptom is uniformly wide gaps across every character, run fc-match monospace before touching kitty.conf at all.

Migrating a vagrant-libvirt VM to a newer host: skip `vagrant package`, hand it to virsh

Moving a vagrant-libvirt VM between Linux hosts looks like a two-step (vagrant package then vagrant up on the other side), and that's exactly what I tried. Both steps died in the same place: fog-libvirt's stream upload/download of a large qcow2 from/to libvirt's storage pool reset mid-flight (Cannot recv data: Connection reset by peer, hung at 0%).

The streaming bug is in fog-libvirt's vol upload/download. The fix is to bypass vagrant-libvirt's vol-upload/vol-download entirely: flatten the overlay qcow2 against its backing file, drop the result in a libvirt storage pool by hand, then virsh define + virsh start. Treat vagrant-libvirt as the boot-time scaffolding only; the running VM is plain libvirt after that.

1. Make the box self-contained

vagrant package exists to bake the VM's disk + metadata into a .box file. With a libvirt provider the disk is usually a qcow2 with a backing file (qemu-img info ... | grep "backing file"), and vol-download only streams the overlay — you'd ship an incomplete box. Skip it and flatten manually:

…more

WSL SSL Certificate Errors on Corporate Networks

If curl throws SSL certificate problem: unable to get local issuer certificate every time in WSL, it's usually a stale CA bundle — especially common on corporate networks running SSL inspection (Zscaler, etc.).

First, refresh the bundle:

sudo apt-get update && sudo apt-get install -y ca-certificates
sudo update-ca-certificates

That fixes most cases. If not, try a full reinstall:

sudo apt-get install --reinstall ca-certificates
sudo update-ca-certificates --fresh

On corporate networks, you likely need your company's root CA certs. You probably already have them somewhere in your Windows filesystem (.crt or .cer files). Copy them all into the trusted store:

sudo cp /c/Certificates/*.crt /usr/local/share/ca-certificates/
sudo cp /c/Certificates/*.cer /usr/local/share/ca-certificates/
sudo update-ca-certificates

Skip .pfx files — those contain private keys and are a different format.

If you don't have the certs handy, export them from Windows:

# PowerShell — list all root certs, look for your company name
Get-ChildItem Cert:\LocalMachine\Root | Select-Object Subject, Issuer | Sort-Object Subject

Or use certmgr.msc (Win+R → certmgr.msc) → Trusted Root Certification Authorities → find your company's cert → right-click → Export → Base-64 encoded X.509.

Test with curl https://google.com after each step.

whoseport: Find What's Listening on a Port (With Working Directory)

ss -tlnp and lsof -i :PORT tell you the PID and command name, but for Node.js or Python processes, "node" or "python" alone doesn't tell you which project is running. The working directory is what you actually need — and it's sitting right there in /proc/$pid/cwd.

Put this in ~/.bashrc:

whoseport() {
  if [ -z "$1" ]; then
    sudo ss -tlnp | tail -n +2 | while read -r line; do
      port=$(echo "$line" | grep -oP ':\K[0-9]+(?=\s)')
      pid=$(echo "$line" | grep -oP 'pid=\K[0-9]+' | head -1 | tr -dc '0-9')
      [ -n "$pid" ] && echo "PORT: $port | PID: $pid | CMD: $(ps -p $pid -o comm=) | CWD: $(readlink /proc/$pid/cwd)"
    done
  else
    whoseport | grep 'PORT: '$1
  fi
}

Usage:

$ whoseport 6173
PORT: 6173 | PID: 1326886 | CMD: node | CWD: /home/davidw/Projects/ccode_viewer/server

$ whoseport          # list all listening ports
PORT: 61217 | PID: 1356 | CMD: tailscaled | CWD: /
PORT: 22     | PID: 1385  | CMD: sshd      | CWD: /

A few things that went wrong before arriving at this version:

Don't use an alias for this — $1 in a single-quoted alias gets swallowed by inner sh -c calls, and local variables don't survive xargs boundaries. A function avoids both problems. The tr -dc '0-9' on the PID is not cosmetic — ss output can carry trailing whitespace or newlines that break ps -p with a "process ID list syntax error".

Linux suddenly preferring IPv6 and breaking connectivity? Fix gai.conf

Your Linux box resolves a host to its AAAA (IPv6) record, connects, but the remote isn't actually listening on IPv6. You see telnet hang on an IPv6 address. This happens when your system starts preferring IPv6 over IPv4.

One-off fix — force IPv4 for a single command:

telnet -4 api.z.ai 80

Permanent fix — edit /etc/gai.conf and uncomment this line:

precedence ::ffff:0:0/96  100

This tells getaddrinfo() to return IPv4-mapped addresses with higher precedence than IPv6. Changes take effect immediately — no restart needed. gai.conf is re-read on every getaddrinfo() call, so the next DNS lookup picks up the new rule. Existing connections are unaffected.

If /etc/gai.conf doesn't exist, just create it with that single line.