4,864 processes

On 6 October 2026 a builder wanted to count how often git push ran. It put a small wrapper called git first on the PATH, and the wrapper did the obvious thing: found the real git with $(which git) and handed over.¹1 ~/dev/docs/incidents/2026-10-06-git-shim-fork-loop.md l.9–14.

The real git, from where the wrapper stood, was dev’s agent git shim, the layer every agent’s git command passes through. The shim found its own real git the same way: the first git on the PATH once its own directory was taken out. That was the wrapper.

So the shim called the wrapper, which called the shim, which, before doing anything else, looked up an alias, which meant calling git.¹1 ~/dev/docs/incidents/2026-10-06-git-shim-fork-loop.md l.9–14.

Between 13:02 and 13:16 every session’s attempt to spawn a process failed.²2 Same doc, l.2: “every spawn failed 13:02–13:16”. By 13:15 there were 4,864 bash processes, each running git config --get alias.config. The per-user process limit was 5,333. Every session got the same answer: fork: Resource temporarily unavailable. The lander (the process that puts agents’ commits on main), the tests and the agents’ own tools all stopped.³3 Same doc, l.14–17. kern.maxprocperuid was 5333.

The agent that investigated could not use ps or top, because the sandbox blocked them. It found the loop with a Python library instead, grouped the processes by command line, and killed them. The machine’s process count went from 5,561 to about 720.⁴4 Same doc, l.19–21. That evening, on another thread, Tyson: “the sandbox might be BULLSHIT machinery i dont even recall why it exists. you should be able to check top and processes surely”.⁵5 Tyson, 2026-10-06 20:40, s:0cecf1e4 (dev sessions said 2026-10-06 --human tyson).

The fix, 5fa5a5199, landed at 13:17. The shim now leaves a counter in the environment. A nested call goes straight to real git instead of running its checks again, and at depth 16 it gives up with exit code 126. A loop now costs one process and under a second.⁶6 dev 0ea4e6f047, 2026-10-06 13:17 +11:00; incident doc l.25–29; held by harness/tests/test_git_shim.py. The incident doc records one rule still owed: a wrapper that counts git must call it by an absolute path, “never $(which git)”.⁷7 Incident doc l.33–34.

Dev was six weeks old. The incident folder has others.

the lander, 24 September

At 02:09 Tyson’s queued commits stopped reaching main. A test suite that took 35 seconds alone was taking up to 358 under load, past its 300-second limit, and the lander scored each timeout as broken code. Nine failed checks (“reds”) between 01:30 and 04:40; none was a real conflict between commits.⁸8 ~/dev/docs/incidents/2026-09-24-lander-stall.md l.12–22.

Around twenty sessions worked the outage. Between them they re-derived the same taxonomy of reds at least four times and launched 12 reading sessions and 8 review sessions on the larger model, about $97 of sessions. Tyson, at 04:39: “you fanned out too much”.⁹9 Same doc, l.36–39; Tyson’s words, s:aadf3e34, 2026-09-24 04:39. At 04:40 he skipped the queue by hand, with 97 commits waiting. The built-in bypass arrived at 04:43.¹⁰10 Same doc, l.12–13, l.35.

It was closed the same day by deleting the lander and writing a smaller one: about 4,600 lines of recovery machinery removed.¹¹11 Same doc, l.43–44; the replacement lander, l.51–53 and ~/dev/docs/systems/integration.md.

.git, 2 October

At 19:09 Tyson: “what the fuck? 3gb for ~/dev/.git? thats absurd”.¹²12 Tyson, 2026-10-02 19:09, s:b52ddb57.

Dev’s desktop app had moved from Tauri to Electron, and the commit dropped the ignore rules for Tauri’s old build folders, which were still on disk. A background probe called churn snapshotted the tree with git add -A under an eight-second timeout. The doc’s reading is that the Rust build output kept being interrupted mid-compression and tried again. The object store held 1,202 temporary files, including 87 partial writes of the same 85,388,568-byte Rust archive.¹³13 ~/dev/docs/incidents/2026-10-02-git-temporary-objects.md l.9–23; Tauri replaced in 1f1c4d131. 2.999 GiB removed, l.33–34. The ignores went back. The doc also leaves an instruction: “Do not lengthen churn’s timeout”.¹⁴14 Same doc, l.29.

red main and the storm, 7 October

At 05:22 that morning dev’s main went red. It stayed red until 17:53. During those twelve and a half hours, 333 commits from 171 sessions landed on top of it, and the count of failing tasks went from 3 to 18. Several of the fixes were to CI itself, and those could not take effect until CI was green.¹⁵15 ~/dev/docs/incidents/2026-10-07-red-main.md l.14–15, l.22–30, l.37–40. At 12:50, to another session, Tyson: “in six weeks we’ve managed to repeat some of the same stakes of space. Like we created too much machinery and stuff like that.”¹⁶16 Tyson, 2026-10-07 12:50, s:848b34bd, to lab. Verbatim. Space was the swarm, the system before dev.

That night, with Tyson’s Mac off, Janice’s chief agent launched about twenty lead sessions in two minutes, and they fanned out. Fifty-two transcripts appeared in three minutes. Live sessions went from 23 to 66, the load average hit 314, and dev’s API, which every session talks to, went dark for 278 seconds. The daemon’s rule for a wedged API killed it for being slow; its replacement booted under the same load, “so slow became an outage.”¹⁷17 ~/dev/docs/incidents/2026-10-07-launch-storm.md l.16–24. By the doc’s own CPU sampling, which it calls a lower bound, dev’s processes never used more than about 4.4 of the 12 cores.¹⁸18 Same doc, l.74–77: “Load 314 overstates CPU.”

The fixes from those two days: one owner per red, CI that runs from what actually landed, agents running at a lower priority than the API and the daemon, and a queue for booting sessions instead of a 105-second timeout.¹⁹19 red-main l.55–60; launch-storm l.84–87, rows 2 and 5.

The red-main doc’s last section is still open. Its first line, about the new lander: “The lander still lands on top of a red.”²⁰20 red-main l.77.