Scribbles for my memory

When a Linux box slowly bleeds memory over hours, the kernel OOM killer kicks in too late — by then the machine is already frozen. Two complementary fixes: earlyoom kills a hog before the system locks up, and a periodic memory snapshot gives you post-mortem data next time it happens.

earlyoom

Install and go — on Debian it's one command and the service auto-enables:

sudo apt-get install -y earlyoom
systemctl status earlyoom.service

Defaults (Debian package): kills when available RAM drops below 10% and available swap drops below 10%. Sends SIGTERM first, then SIGKILL. Configurable via /etc/default/earlyoom or override flags in the systemd unit.

Memory snapshot every 5 minutes

When the machine eventually does die, you want to know what was growing. A systemd timer + shell script + logrotate gives you that with near-zero overhead.

The capture script (/usr/local/bin/memsnap-capture):

You have a screen session running in one terminal and want to interact with it from a second terminal on the same host. Two modes:

Share it — both terminals see and type in the same session:

screen -x <session>

Great for pair debugging. Both clients are attached simultaneously.

Take it over — detach the other client and bring the session here:

screen -d -r <session>

Use this when you're switching from SSH to local, or reclaiming a session you accidentally left open elsewhere.

Find your session name first:

screen -ls

If the session belongs to a different user, you'll need that user's permissions (or sudo).

When dispatching commands between tmux sessions, tmux send-keys -t <target> '<command>' Enter exits 0 and the command text appears in the target's input buffer — but the Enter never commits. The agent sits idle. The dispatcher sees success. Nothing happened.

This hits hardest with long commands that trigger tmux's bracketed paste mode: the trailing Enter gets absorbed into the paste block instead of executing it. Two other failure modes: the target app is in a modal state (dialog, secondary prompt) that swallows Enter, or the input handler is loaded and drops the keystroke.

The fix is a verify-and-retry loop after every dispatch:

dispatch() {
  local target="$1" cmd="$2"

  # 1. Send
  tmux send-keys -t "$target" "$cmd" Enter

  # 2. Wait for the target to start processing
  sleep 3

  # 3. Verify it's actually working, not just staged
  if tmux capture-pane -t "$target" -p -S -10 | tail -10 \
      | grep -qE "Working|Exploring|Reading|Analyzing"; then
    return 0
  fi

  # 4. Retry — empty Enter kicks the paste buffer
  tmux send-keys -t "$target" "" Enter
  sleep 2

  # One more check, then give up
  if tmux capture-pane -t "$target" -p -S -10 | tail -10 \
      | grep -qE "Working|Exploring|Reading|Analyzing"; then
    return 0
  fi

  # 5. Fallback: write to file, let target poll for it
  echo "$cmd" > "/tmp/dispatch-${target##*.}.cmd"
  return 1
}

Cap retries at 2. If both fail, fall back to writing the command to a file the target reads on its own polling cycle — this sidesteps the pty entirely.

The broader lesson: any async channel where the sender gets a local exit code instead of a structured delivery ACK has this failure class. Pty pipes, shell sessions, anything interactive. If you're building multi-agent coordination over interactive channels, bake verification into the dispatch protocol and don't trust exit code 0.

Apply the same pattern in reverse for callbacks: workers should verify their "done" message was received by the dispatcher, not just fire-and-forget.

When you need a clean slate — testing, debugging, or isolating a project — each major AI coding CLI lets you override its config directory via an environment variable:

# Claude Code
CLAUDE_CONFIG_DIR=/tmp/claude-clean claude

# GitHub Copilot CLI
COPILOT_HOME=/tmp/copilot-clean copilot

# OpenAI Codex CLI
CODEX_HOME=/tmp/codex-clean codex

These replace the entire config directory (~/.claude/, ~/.copilot/, ~/.codex/), so existing sessions, permissions, and plugins won't carry over. Copy the default directory contents if you need to preserve anything.

Copilot also has a separate COPILOT_CACHE_HOME for marketplace cache and update packages — setting COPILOT_HOME alone won't relocate those.

In the HTTP era, virtual hosting was trivial: one IP serves many domains, the Host header tells them apart. HTTPS breaks this because TLS handshake happens before any HTTP header — the server must pick a certificate before knowing which domain the client wants.

SNI — the standard fix

Server Name Indication is a TLS extension where the client sends the target hostname during handshake. All modern clients support it. Nginx uses it automatically — just define separate server blocks:

server {
    listen 443 ssl;
    server_name a.com;
    ssl_certificate /etc/ssl/a.com.crt;
    ssl_certificate_key /etc/ssl/a.com.key;
}

server {
    listen 443 ssl;
    server_name b.com;
    ssl_certificate /etc/ssl/b.com.crt;
    ssl_certificate_key /etc/ssl/b.com.key;
}

Each domain gets its own certificate. Nginx routes based on SNI. Zero extra config needed.

When you'd rather use one certificate

For a handful of related domains, a SAN certificate (Subject Alternative Name) covers multiple names in one cert:

server {
    listen 443 ssl;
    server_name a.com b.com c.com;
    ssl_certificate /etc/ssl/multi.crt;
    ssl_certificate_key /etc/ssl/multi.key;
}

Let's Encrypt makes this painless:

certbot --nginx -d a.com -d b.com -d c.com

For lots of subdomains (*.example.com), a wildcard certificate is the way to go.

SNI + Let's Encrypt covers 99% of real-world setups — free and automatic.