Scribbles for my memory

If you're still using terminal tabs, tmux windows do the same job with extras you didn't know you needed.

The hierarchy: Session > Window > Pane. A session holds multiple windows, each window can be split into panes. The bottom status bar shows all windows at a glance:

[main] 0:bash* 1:vim- 2:server

* marks the active window, - marks the previous one. You always know where you are.

Key bindings to internalize:

What tmux gives you over plain terminal tabs:

• Survives disconnects — SSH drops? Reconnect with tmux attach, everything is still there. Terminal tabs are gone the moment the connection dies.
• Panes — Split any window horizontally or vertically without opening another tab.
• Scriptable — Create and arrange windows/panes from a script or config.

The one real risk: if the machine itself reboots, the session is gone. tmux-resurrect and tmux-continuum can save/restore layouts, but in practice most people use tmux to survive network hiccups, not server reboots. A reboot means re-opening a session, which takes seconds.

Both prevent a process from dying when you close the terminal, but they work at different stages and with different guarantees.

nohup intervenes before launch. It sets the process itself to ignore SIGHUP via sigaction, so no matter who sends the signal, the process won't react. It also redirects stdout/stderr to nohup.out (or a file you specify). POSIX-standard, portable to any shell.

disown intervenes after launch. It removes the process from the shell's job table so the shell won't notify it on exit — but the process has no built-in signal immunity. Some SSH/terminal implementations broadcast SIGHUP to the entire process group on disconnect, bypassing the shell entirely. In that case disown alone won't save you.

# nohup: know you need it upfront
nohup python train.py > train.log 2>&1 &

# disown: the "oh crap I need to leave" rescue
./long_task.sh        # already running...
# Ctrl+Z to suspend, then:
bg
disown %1

# Belt and suspenders — use both
nohup cmd > out.log 2>&1 &
disown

The combination is the safest one-liner: nohup gives the process its own signal shield, disown cleans up the job table so closing the terminal is truly a no-op.

For anything long-lived (a server, a daemon), neither is the right tool — use systemd, supervisord, or tmux. nohup and disown are emergency measures, not service managers.

One gotcha with nohup: if you don't redirect output yourself, it writes to ./nohup.out, which can quietly grow to fill a disk. Always pair it with an explicit > log 2>&1.

When a Linux box slowly bleeds memory over hours, the kernel OOM killer kicks in too late — by then the machine is already frozen. Two complementary fixes: earlyoom kills a hog before the system locks up, and a periodic memory snapshot gives you post-mortem data next time it happens.

earlyoom

Install and go — on Debian it's one command and the service auto-enables:

sudo apt-get install -y earlyoom
systemctl status earlyoom.service

Defaults (Debian package): kills when available RAM drops below 10% and available swap drops below 10%. Sends SIGTERM first, then SIGKILL. Configurable via /etc/default/earlyoom or override flags in the systemd unit.

Memory snapshot every 5 minutes

When the machine eventually does die, you want to know what was growing. A systemd timer + shell script + logrotate gives you that with near-zero overhead.

The capture script (/usr/local/bin/memsnap-capture):

You have a screen session running in one terminal and want to interact with it from a second terminal on the same host. Two modes:

Share it — both terminals see and type in the same session:

screen -x <session>

Great for pair debugging. Both clients are attached simultaneously.

Take it over — detach the other client and bring the session here:

screen -d -r <session>

Use this when you're switching from SSH to local, or reclaiming a session you accidentally left open elsewhere.

Find your session name first:

screen -ls

If the session belongs to a different user, you'll need that user's permissions (or sudo).

When dispatching commands between tmux sessions, tmux send-keys -t <target> '<command>' Enter exits 0 and the command text appears in the target's input buffer — but the Enter never commits. The agent sits idle. The dispatcher sees success. Nothing happened.

This hits hardest with long commands that trigger tmux's bracketed paste mode: the trailing Enter gets absorbed into the paste block instead of executing it. Two other failure modes: the target app is in a modal state (dialog, secondary prompt) that swallows Enter, or the input handler is loaded and drops the keystroke.

The fix is a verify-and-retry loop after every dispatch:

dispatch() {
  local target="$1" cmd="$2"

  # 1. Send
  tmux send-keys -t "$target" "$cmd" Enter

  # 2. Wait for the target to start processing
  sleep 3

  # 3. Verify it's actually working, not just staged
  if tmux capture-pane -t "$target" -p -S -10 | tail -10 \
      | grep -qE "Working|Exploring|Reading|Analyzing"; then
    return 0
  fi

  # 4. Retry — empty Enter kicks the paste buffer
  tmux send-keys -t "$target" "" Enter
  sleep 2

  # One more check, then give up
  if tmux capture-pane -t "$target" -p -S -10 | tail -10 \
      | grep -qE "Working|Exploring|Reading|Analyzing"; then
    return 0
  fi

  # 5. Fallback: write to file, let target poll for it
  echo "$cmd" > "/tmp/dispatch-${target##*.}.cmd"
  return 1
}

Cap retries at 2. If both fail, fall back to writing the command to a file the target reads on its own polling cycle — this sidesteps the pty entirely.

The broader lesson: any async channel where the sender gets a local exit code instead of a structured delivery ACK has this failure class. Pty pipes, shell sessions, anything interactive. If you're building multi-agent coordination over interactive channels, bake verification into the dispatch protocol and don't trust exit code 0.

Apply the same pattern in reverse for callbacks: workers should verify their "done" message was received by the dispatcher, not just fire-and-forget.