Scribbles for my memory

My Windows 11 dev box running AI agents through MSYS2/Git Bash lost about 2 GB of RAM a day to kernel paged pool. A reboot fixed it, and then it started climbing again. PoolMon-style tag stats put almost all of it under Toke (security token objects): ~3 million live tokens.

The tell-tale sign: millions of token objects, but only ~1,400 token handles system-wide. No process is holding them, so you won't find the culprit in Task Manager or with handle.exe. Something in the kernel is keeping a reference.

The culprit turns out to be a Windows 11 25H2/26H2 bug in win32kfull.sys (seen on 10.0.26100.94xx). Every new process that attaches to a console leaks one reference to its primary token whenever ForegroundLockTimeout is non-zero. Shell-heavy workloads hit it hard: each bash → child → grandchild spawn leaves a token behind. My WPR pool trace showed ~50 process starts per second from agent polling loops, leaking ~11 tokens per second.

The workaround is to set the timeout to 0. The registry value alone does nothing, though: at every logon win32k overwrites the live value with 2147483647, whatever HKCU\Control Panel\Desktop says. You have to set the live value with SystemParametersInfo, from inside the interactive session:

Add-Type -Namespace W -Name U -MemberDefinition @'
[DllImport("user32.dll", SetLastError=true)] public static extern bool SystemParametersInfo(uint a, uint b, ref uint c, uint d);
[DllImport("user32.dll", SetLastError=true)] public static extern bool SystemParametersInfo(uint a, uint b, IntPtr c, uint d);
'@
$v = [uint32]0
[void][W.U]::SystemParametersInfo(0x2000, 0, [ref]$v, 0)      # SPI_GETFOREGROUNDLOCKTIMEOUT
"before=$v"
[void][W.U]::SystemParametersInfo(0x2001, 0, [IntPtr]0, 2)    # SPI_SETFOREGROUNDLOCKTIMEOUT, SPIF_SENDCHANGE
[void][W.U]::SystemParametersInfo(0x2000, 0, [ref]$v, 0)
"after=$v"

The value resets at every logon, so run that script from a scheduled task with a logon trigger in interactive mode. This even works from an elevated SSH session:

schtasks /create /tn token-leak-flt-fix /sc onlogon /delay 0001:00 /it /f ^
  /tr "powershell.exe -NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -File C:\ProgramData\flt-fix.ps1"
schtasks /run /tn token-leak-flt-fix

Running the script over plain SSH would change session 0's value, which doesn't help. /it makes the task run in the logged-on user's session, where the console processes are. Mine went from before=2147483647 to after=0, and the leak rate dropped from 11.5 tokens/s to 0.26 tokens/s within minutes.

There are two catches. First, at 0 any app can steal foreground focus. Second, tokens that already leaked are freed only by a real Restart: with Fast Startup on, "Shut down" saves and restores the kernel, leak included.

I spent an hour on pool tracing before searching for "windows toke leaking", which turned up the answer right away (microsoft/terminal#20781, git-for-windows/git#6471, and the logon-fix write-up at gunrgbcr/windows-token-leak-logon-fix). When a symptom is this specific, search for it first.

Diffwalk's own agent skill is a one-shot author: the agent reads the whole change, writes an ordered explanation for every block, and hands you a finished walkthrough. That's great for documenting your own PR, but when you're reviewing a colleague's PR it anchors you to the agent's verdict before you've formed your own.

pair-review turns the same CLI into a dialogue. The agent slices the diff into review units (a signature change travels with its call sites), opens the diff in the browser, then takes one unit per round: says what it does, raises its own suspicions, and stops for your take. The agreed text — not the debate — goes into the walk's explanations.yaml, so an interrupted review resumes at the next pending unit instead of starting over. A GitHub comment is only posted after you confirm its exact draft.

The newest bit: when the agent talks about a unit, the browser scrolls there and highlights the exact lines. It works through the Chrome DevTools Protocol. split-view launches a dedicated Chrome/Edge (separate profile, CDP port 9333) in the top half of the screen with the terminal below. Then each round the agent runs something like:

node scripts/focus.mjs change-003 --at src/Orders/OrderService.cs:42-50 --at src/Orders/OrderService.cs:40:old

focus.mjs attaches to that browser with Playwright's connectOverCDP, expands the file, and marks rows in the diff's shadow DOM. This works in split or unified layout, on the new side or (:old) the deleted side. The first range is scrolled to the centre of the screen. Your terminal keeps keyboard focus throughout, unlike the old approach of typing an anchor URL into the omnibox.

It needs a patched Diffwalk build: shukebeta/diffwalk branch local-both adds a Wrap toggle for long lines and shows each block's change-0NN id on the page, so "change-012" is something you can actually see. Build it with the bun version CI pins. A different bun silently tree-shakes the report's web components, and every diff renders unstyled:

git clone -b local-both https://github.com/shukebeta/diffwalk.git
cd diffwalk
npx --yes [email protected] install
npx --yes [email protected] run build
alias diffwalk="node $PWD/dist/diffwalk.js"   # add to your shell profile

Then install the skill and its one dependency (playwright-core, no browser download — it drives the Chrome you already have):

npx skills add shuke-personal/pair-review-skill --skill pair-review -g
cd ~/.claude/skills/pair-review/scripts && npm i

Start a new agent session and say "pair review this PR". Split view plus highlighting is scripted for Windows and Linux/X11. On macOS, start the CDP browser by hand (the command is in SKILL.md §3) and focus.mjs works unchanged.

When you run a fleet of coding agents, most of their "I'm stuck, human please" pages are not really for you. A runner is wedged, a lock is stale, a relay died, or the question is one you already answered in a runbook or on an old PR. We built an oncall agent that reads every escalation first, acts as you, and only pages you with what truly needs you. The whole thing is small, and it runs on stock my-ai-team (mat) features. It took two days to go from idea to running system.

The flow:

agent ── notify-user --action ──► Telegram (you, unchanged)
   └── ssh oncall-host oncall-inbox        (same message on stdin)
             │
        oncall agent: triage (env / decision / noise), investigate over ssh
             ├── ssh <agent-host> send-relay %<pane> reply.md   → answer the stuck agent
             ├── incident record, committed to a repo
             ├── notify-user                                    → FYI to you
             └── notify-user --action + a GitHub issue          → only when you must decide

1. Forward action messages. On every agent host, point mat at the oncall host (any ssh alias that logs in without a prompt):

git config --global mat.devopsHost oncall-host

You're logged into gh and can browse every private repo with it, but plain git pull over HTTPS still fails with fatal: could not read Username for 'https://github.com'. git doesn't know how to read gh's token store — nobody introduced them. (A GH_TOKEN env var makes every gh command work and can hide exactly this breakage, because plain git ignores it.)

You don't need Microsoft's Git Credential Manager (it drags in a .NET runtime). gh ships its own credential helper; one command wires it in:

gh auth setup-git

That adds a single stanza to your ~/.gitconfig:

[credential "https://github.com"]
    helper = !/usr/local/bin/gh auth git-credential

From then on, any HTTPS clone/pull/push of private repos authenticates through gh's stored token — global config, so it works for every repo on the machine. No SSH keys, no per-remote remote-url surgery, no token pasting.

Verify without letting an env token mask the result:

env -u GH_TOKEN git ls-remote https://github.com/<you>/<private-repo> HEAD

If that prints a ref, you're done.

We run codex CLI agents behind a local reverse proxy (pool selector, per-seat failover). The agent config looked airtight:

model_provider = "codex"

[model_providers."codex"]
base_url = "http://gateway.internal:9949"
wire_api = "responses"
requires_openai_auth = false

Yet when a seat hit its weekly cap, the agent stayed stuck on the dead seat: disabling it server-side changed nothing, only a fresh session worked. Every proxy log said the proxy was innocent. After a day of reading proxy code we put a sniffer on the client box instead — and the codex process was dialing chatgpt.com:443 directly. The config was never being used.

The trigger: the launcher exported OPENAI_API_KEY into the agent's environment. When codex sees that variable, it prefers its built-in OpenAI/ChatGPT backend and ignores the custom model_providers.*.base_url entirely — even with requires_openai_auth = false. The fix in our launcher is one line:

# codex launch env plan
unset OPENAI_API_KEY OPENAI_BASE_URL CODEX_ACCESS_TOKEN

One more trap: agents launched before the fix keep the stale env until the process actually restarts. A process's environ is frozen at exec time — check it directly with:

tr '\0' '\n' </proc/<pid>/environ | grep OPENAI_API_KEY

The general lesson: when a client "should" be going through your proxy, don't trust configs or client-side logs. Ask the kernel. Per-process sockets (ss -tnp), a tcpdump SYN filter on port 443, and DNS queries at the moment of a failing request settle in one minute what a day of log archaeology can't.