Scribbles for my memory

An XRDP/Xfce connection is tied to a desktop and user-manager lifecycle, not only to the RDP window. If that session tears down, the default tmux server, kitty processes, and user services can disappear together without anyone running tmux kill-server.

On XPS, xrdp-sesman logged the window manager exiting with signal 15, then cleaned up the X server. XFCE reported broken PipeWire, D-Bus, and ICE connections; systemd subsequently stopped the user manager and kitty scopes. A simultaneous NVIDIA “fallen off the bus” failure was a strong suspect, but the old logs could not identify the original signal sender.

Keep long-lived agents on an explicit tmux socket supervised outside the desktop session:

sudo systemctl enable --now [email protected]
tmux -L mat list-sessions
tmux -L mat attach -t <session>

The mat-keeper session is only the service sentinel. MAT agent sessions share the mat server and should survive an XRDP switch. The default tmux a command still refers to the desktop-bound default socket, so use -L mat when recovering the fleet.

To make the next teardown attributable, enable auditd with a persistent signal rule and inspect the caller:

sudo ausearch -k process-signal -ts recent -i
sudo ausearch -k service-control -ts recent -i

Look for comm, exe, pid, ppid, and the target in OBJ_PID; these distinguish xrdp-sesman, systemd, and other senders. Auditd cannot reconstruct an earlier incident, and a desktop process can also exit because of D-Bus or GPU failure without a signal, so correlate its records with /var/log/xrdp-sesman.log, .xsession-errors, and journalctl.

A bash pipeline that is quiet locally but fails only on one self-hosted runner may be a signal-disposition problem, not a data or code problem. Our required test failed in unrelated files with printf: write error: Broken pipe, and every failure landed on the same Linux runner.

The useful check is the effective job-shell behavior, not only the runner listener:

systemctl show actions.runner.SHUKE-LABS.<runner>.service -p IgnoreSIGPIPE

PATH=/usr/local/libexec/actions-runner:$PATH bash -c '
  set -o pipefail
  printf "%s\n" {1..10000} | head -n1 >/dev/null
  printf "pipeline rc=%s\n" "$?"
'

The normal result is a SIGPIPE exit (usually 141). With SIGPIPE inherited as SIG_IGN, bash's builtin printf receives EPIPE instead, prints the broken-pipe diagnostic, and pipefail turns it into a failure.

The fix has two parts. Add a systemd drop-in:

[Service]
IgnoreSIGPIPE=no

Then put a root-owned launcher first in the runner's .path file:

#!/usr/bin/perl
$SIG{PIPE} = "DEFAULT";
exec "/usr/bin/bash", @ARGV or die "exec bash: $!";

The launcher matters because the Actions Runner's Node/Listener/Worker chain can set SIGPIPE to ignored again after systemd starts the service. Verify with a fresh job shell and the pipeline test; the listener process's SigIgn alone is not a reliable success criterion.

Check the whole Linux fleet, not only the runner named in the first failure. The same defect was present on all three sydney2 runners and on the mbp15-vm Lima guest. After fixing the signal boundary, two independent failures surfaced: a jq 1.6-incompatible test mock and newer-main tmux/test timing regressions. Keeping those layers separate prevented a correct runner diagnosis from being mistaken for a complete CI diagnosis.

The final CI-shaped local run passed 166/166, and the GitHub required jobs passed after the fleet fix.

A systemctl --user service kept starting, then stopping ~12s later, every 5 minutes — flooding a notification channel each time. But crontab -l was empty and systemctl --user list-timers had no matching timer. So who was cycling it?

Usually it's not a scheduled job. A user-level service's lifetime is tied to the per-user systemd manager (user@<uid>.service), and without linger that manager only runs while the user has at least one login session:

• Someone logs in (even a 2-second SSH/rsync) → the manager starts → it reaches default.target → your WantedBy=default.target service gets pulled up.
• ~10s after the last session exits → the manager tears down → your service stops with it.
• Next login repeats the whole dance.

So the service's "restart cadence" is really the login cadence. The system journal makes it obvious:

journalctl --since "-15min" | grep -iE "Accepted publickey|New session|Removed session|Reached target exit.target"

In my case the culprit was another box running */5 * * * * rsync … host:/backup/… — a short SSH connection every 5 minutes that lit up the entire user manager and dropped it again. The moment the host had a persistent session (a lingering tmux), the symptom vanished — which is the strongest tell: it only happens when nobody is logged in.

The fix is to detach the manager from login so it stays resident:

sudo loginctl enable-linger <user>
# verify
loginctl show-user <user> --property=Linger   # Linger=yes

With linger on, the manager starts at boot and no longer stops when sessions end, so the service becomes a real background daemon. For a user-level service that must run while no one is logged in, enable --now is not enough — enable-linger is the missing prerequisite.

一个 systemctl --user 服务莫名每 5 分钟启动、十几秒后又停,反复刷屏。crontab -l 空的,systemctl --user list-timers 也没有对应 timer——那到底是谁在动它?

答案往往不是定时任务,而是 user 级服务的生命周期绑在 per-user systemd manager(user@<uid>.service)上。没开 linger 时,这个 manager 只在该用户"至少有一个登录会话"时存活:

• 有人(哪怕一条 2 秒的 SSH/rsync)登录 → manager 启动 → 到达 default.target → 你那个 WantedBy=default.target 的服务被拉起。
• 会话退出约 10 秒后 → manager 拆除 → 服务跟着停。
• 下次登录再来一遍。

所以服务的"重启节奏"其实是登录节奏。用系统 journal 一看就露馅:

journalctl --since "-15min" | grep -iE "Accepted publickey|New session|Removed session|Reached target exit.target"

我这次的真凶是另一台机器上的 */5 * * * * rsync … host:/backup/…——每 5 分钟一条短 SSH 连接,把整个 user manager 点起来又放下。一旦本机有了常驻会话(比如一个挂着的 tmux),现象立刻消失——这就是最强的判定线索:只在"没人登录"时发生。

治本是让 manager 脱离登录、常驻:

sudo loginctl enable-linger <user>
# 确认
loginctl show-user <user> --property=Linger   # Linger=yes

开 linger 后 manager 开机即起、不随会话退出而停,服务真正变成后台常驻。要让一个 user 级服务在无人登录时也长跑,enable --now 是不够的,enable-linger 才是前提。

Every character in kitty had a massive gap after it, like the cell width was double what it should be. The obvious first move — swap font_family in kitty.conf — did nothing. Neither did the version a second pair of hands tried. Two failed attempts, and the config looked completely normal.

The config wasn't the problem. monospace is just a fontconfig alias, and on this machine fontconfig was handing it to a CJK font:

$ fc-match monospace
NotoSansCJK-Regular.ttc: "Noto Sans Mono CJK SC" "Regular"

kitty bases its cell width on the primary font, and a CJK font's cell is full-width — so narrow ASCII glyphs end up stranded in the middle of wide blank cells. The same kitty.conf was perfectly fine on another machine, because over there fc-match monospace returns DejaVu Sans Mono.

What was poisoning the alias was a file Ubuntu's language-selector drops in:

$ grep -A6 '<family>monospace</family>' \
    ~/.config/fontconfig/conf.d/64-language-selector-prefer.conf
		<family>monospace</family>
		<prefer>
			<family>Noto Sans Mono CJK SC</family>
			<family>Noto Sans Mono CJK JP</family>
			...

It prepends the Noto CJK mono fonts to the monospace alias, so they outrank every Latin mono. Comment out that one <alias> block and monospace goes back to DejaVu:

$ fc-match monospace
DejaVuSansMono.ttf: "DejaVu Sans Mono" "Book"

The reload trap that made this take an hour

Here's the part that hurts. After fixing fontconfig, pressing ctrl+shift+F5 to reload kitty... still showed the gaps. It looked exactly like the fix hadn't worked — the same dead end that had already wasted two attempts.

kitty caches its fontconfig resolution in-process. ctrl+shift+F5 re-reads kitty.conf, but it never re-queried fontconfig, so a change to how monospace resolves never reached the running kitty. Only quitting and reopening kitty — a fresh process — picked it up.

So the rule, painfully earned: when a kitty font change "doesn't work," before you conclude the fix is wrong, fully restart kitty and re-check. The fix may have been right all along; you were just looking at a cached font. And when the symptom is uniformly wide gaps across every character, run fc-match monospace before touching kitty.conf at all.