# Handoff — a watch for a running program (`PORTING.md` Tier 1, item 5) Written because the session was wound down for budget. **The work is finished and `dune test` is green**, run twice. This file exists so the reasoning is not re-derived, and so the two things I would have done next are named. --- ## The finding that reframes the task **Most of item 5 was already built before this session.** The brief reads as though a watch had to be built from nothing; it did not. What existed at `344e571`: - `runtime/flan_dev.c` — the watch table: 64 slots, a name and rendered text per slot, a per-slot seqlock, an `on`/`off` flag so a watch call costs a load and a not-taken branch when nobody is looking, and four scalar entry points (`flan_dev_watch_i64`/`_u64`/`_f64`/`_str`). - `vendor/agent/flan_agent.c` — `watch`, `watch on`, `watch off` on the agent socket. - `lib/dev.ml` — `watch_read`, `watch_enable`, and the `:op "watch"` / `:op "watch-enable"` arms. - `emacs/flan-watch.el` — the watch buffer *and* the inline ghost text, both fed from one reply. - `test/test_dev.ml` — a block driving a real daemon and a real running program. That is `watch.clj`'s `spy`, end to end. **What was missing was `spy-num`** — the numeric accumulator for hot loops, which is precisely the part `PORTING.md` calls "least obvious and most valuable". That is what this session built. --- ## The accumulator decision This is the expensive part to re-derive, so it is stated in full. The long-form version is in `BUILT.md`, "A hot loop keeps five numbers, and the window is the editor's"; `runtime/flan_dev.c` carries it at the code. ### What a slot keeps **Count, min, max, last, mean.** Five numbers per label. The scalar watch keeps one value per name. Sampled from a hot inner loop that is nearly useless — you see whichever of the 91,200 cells happened to run last. `watch.clj`'s own docstring says so and is why `spy-num` exists there. Each of the five answers a question you can ask *without building a query*: - **`n`** — how many times the expression actually ran. This is the first thing that is wrong when a loop is wrong. A count that tracks the frame counter rather than the cell count is a loop that is not running. - **`min` / `max`** — the range. This is the thing a single sample can never show you, and it is what you are looking for when you suspect an index or a velocity is leaving the region it should stay in. - **`last`** — the one sample. Kept because it is what the scalar watch would have given you, and losing it on the way to something richer would be a regression. - **`mean`** — carried as a running `sum` and divided at read time. A mean accumulated as a mean drifts; a sum does not. ### What it deliberately does not keep **A ring, or a history.** A small ring of the last N samples was the serious alternative. It loses on the only ground that matters here: N samples out of 91,200 is a sample of the *tail* of the loop, not of the loop. It answers "what did the last few cells do" when the question is "what did the cells do". Past five numbers, every richer answer is a UI for building a query — and a query builder is the one thing this whole design exists not to be. The ghost text section of `BUILT.md` already settled the same question the same way for the scalar case ("a watch inside a loop shows the last value written... every better answer is a UI for building a query. Settled, not open."). ### The write path does no formatting This is the feature, not an optimisation. An `snprintf` per sample at thousands of samples a frame is a HUD that costs more than the game. A sample is: one relaxed load of the epoch, five compares and stores, and the slot's seqlock. The **reader** — the agent's listener thread, once per editor tick — turns the five numbers into text. `watch.clj` reaches the same place with a `double-array` per label and a `render` function Emacs calls. This is also why the slot grew a `num` flag rather than getting a second table: the read path already copies under a seqlock, so it copies five doubles instead of 192 bytes and renders them out of locals *after* the counter check. ### The window is since the last reset — a deliberate divergence from `watch.clj` `watch.clj` is cumulative until `reset-spies!` is called by hand. **This is not**, and the disagreement is on purpose. Cumulative is the wrong default for a frame loop. A `min` and a `max` over a whole session reach the session's extremes within a few seconds of play and then never move again — so the two most useful of the five numbers go dead *exactly* when you start interacting with the thing you are debugging. This tool exists to show you a number while you drag the mouse. So `flan-watch--tick` sends `:reset t` beside its read, and what you see is "since you last looked": a fifth of a second, about a dozen frames, which keeps the range tracking the present. A caller who wants cumulative numbers gets them by not resetting. The setting lives in the editor, not the runtime. ### Reset is its own message, never a side effect of reading A destructive read was the tempting shape and is wrong: it makes *looking* change what is there. Anything that polls — a test's `await`, a second editor, a person reading twice — would silently shorten the window and come back with a count that is noise. So `watch reset` is its own agent command, `:reset t` is a field on the read op, and `dev.ml` sends the reset *after* the read so a tick reports the window it just closed. This was settled before the test was written rather than after, which is the only reason the test can assert anything about `n` at all. ### The reader never writes the table Reset bumps one global epoch counter and touches no slot. A slot clears itself on its **next sample**, when it notices the epoch moved, and does so *inside* its own odd-generation window so a reader can never catch a half-cleared slot. The game thread stays the only writer of the table — the invariant the whole watch section of `flan_dev.c` rests on. The cost: a new window begins when the program next runs, not at the instant of the reset. For a frame loop that is the only moment it could sensibly begin. **This bit me in the test** (see "What did not work" below) and the test now waits for it and says why. ### `i64` accumulates as a double A magnitude past 2^53 loses precision in the sum and in the ends of the range. Recorded rather than designed around: a count, a coordinate and a tile index are what this is pointed at, and a second integer accumulator for a case nobody has would be two code paths for one tool. `watch-num-i64` and `watch-num-f64` are two entry points only so a program need not cast at the call site. ### A whole number prints as one `watch.clj`'s `fmt-num` does this and the reason survives the port: a watch on an array index that reads `66.0000` sends you looking for a rounding bug that is not there. --- ## What was built, file by file All of it is **working** — built, run against a real daemon and a real running program, and covered by the test. Nothing is stubbed and nothing is half-written. | File | Change | State | |---|---|---| | `runtime/flan_dev.c` | `watch_slot` grew `num`, `epoch` and five doubles; `watch_begin` clears `num`; new section "A number sampled thousands of times a frame" with `watch_record`, `flan_dev_watch_reset`, `flan_dev_watch_num_i64`, `flan_dev_watch_num_f64`, `watch_num_str`, `watch_render_num`; `flan_dev_watch_read` renders a num slot instead of copying its (unused) text | working | | `vendor/agent/flan_agent.c` | `flan_dev_watch_reset` declaration; a `watch reset` command beside `watch on`/`watch off` | working | | `lib/dev.ml` | `watch_read` takes `~reset` and sends `watch reset` *after* the read; the `:op "watch"` arm reads a `:reset` field the same way `watch-enable` reads `:on` | working | | `emacs/flan-watch.el` | `flan-watch-ghost-call-regexp` `?` → `*`; docstring updated; `flan-watch--tick` sends `(:op "watch" :reset t)` | working | | `test/programs/dev-watch.flan` | `declare-c watch-num-i64`; a `loop-cells` hot loop sampling `"cell"` eight times a step at 0,3,…,21 | working | | `test/test_dev.ml` | assertions inside the existing watch block: the row exists, `n > 8` (per-sample not per-step), `min < max` (a range, not just `last`), a plain read does not reset, `:reset t` reopens the window | working | | `BUILT.md` | new section "A hot loop keeps five numbers, and the window is the editor's", after the ghost text one | working | | `PORTING.md` | item 5 struck in house style with the accumulator decision | working | | `NEXT.md` | "What is left on `PORTING.md`'s list" corrected: item 5 done, item 6 is next | working | ### Verified end to end, not just compiled Against a real `flan dev` daemon and the real running program, the `"cell"` row read back: ``` n=184 min=0 max=21 last=21 mean=10.5000 (first read) n=368 min=0 max=21 last=21 mean=10.5000 (second read — a plain read does not reset) n=16 min=0 max=21 last=21 mean=10.5000 (after :reset t — the window reopened) ``` The loop is 0,3,…,21, so `min`, `max`, `mean` and the per-step count of 8 are each pinned by a different part of it. ### Test result `dune test` is **green**, run twice in full. `test_dev.ml`'s first block — the separately-known-flaky socket bind race — failed on one earlier run with "the daemon never listened" and passed on both final runs. That is exactly the documented flake and another agent is on it; it is not related to anything here. --- ## The ghost text path — the brief's assumption, checked The brief said to reuse the ghost text path rather than invent a second one. **It fits, and it needed one character.** The important property is in `BUILT.md`: ghost text anchors on the **string literal in the source**, not on a source location carried in the table. `flan-watch--ghost-sites` scans buffers shown in a window for `( ""` and matches the name against the rows. So a new *kind* of watch needs nothing new in the daemon, nothing new in the wire, and no new display path — the accumulator's rendered value is a string like `n=368 min=0 max=21 last=21 mean=10.5` and flows through the existing `:watch` rows untouched. **Is it pull-based?** No, and the brief's phrasing is worth correcting for whoever reads this next. The watch is **push-based** and that is load-bearing: `watch.clj` is pull (Emacs polls a render function), and `flan_dev.c`'s comment explains at length why that does not port — an evaluation here compiles a module and `dlopen`s it, so a 5Hz poll would be hundreds of `.so` files a minute. So the program pushes into a table and the editor reads *memory*. Ghost text is not a poller at all: it is painted from `flan-watch--absorb`, off the same single reply that paints the buffer, which is what keeps the two from disagreeing and preserves the one-request-in-flight invariant. The accumulator sits inside that unchanged. **The one thing that did not fit.** `flan-watch-ghost-call-regexp` was `watch\(?:-[[:alnum:]]+\)?` — *one* optional hyphenated segment. `[[:alnum:]]` does not match a hyphen, so on `(watch-num-i64 "cell" ...` the regexp matched `watch-num`, then required whitespace, found `-i64`, and backtracked to failure. A numeric watch therefore appeared normally in the watch buffer and got **no inline value at all** — a silent half-failure. `*` for `?` is the whole fix, verified in `emacs --batch` against all four head shapes. That is worth recording as the predictable cost of the decision the ghost text section defends. Anchoring on the name rather than on the head is what makes the head a `defcustom`, and a `defcustom` with an enumerated default is a default that needs widening whenever the set of heads grows. Cheaper than the alternative, but not free. --- ## What remains, in the order a fresh session should do them Item 5 is complete. These are the two gaps I would close next, both small. 1. **`emacs/test-flan-watch.el` — add a `watch-num-i64` fixture site.** The regexp fix is covered by *nothing* automated; I verified it by hand in `emacs --batch`. Add a fixture whose buffer text contains `(watch-num-i64 "cell" x)` and assert `flan-watch--ghost-sites` returns `("cell" . …)`, alongside the existing `watch-i64` fixtures. No daemon needed — the file is already in the suite via `test-flan-cider.el` and, per `BUILT.md`, ghost text is a pure function from rows plus buffer text to overlays. **This is the highest-value remaining item**: it is the one change in this commit with no regression test behind it. 2. **`test/test_dev.ml` — assert the `n=0` render.** `watch_render_num` in `runtime/flan_dev.c` has a branch for a window with no samples that emits `n=0 last=` (the range and the mean have nothing behind them, but `last` is still the last value the name ever had). No test reaches it: the program in `dev-watch.flan` samples every step, so a window is never empty in practice. Reaching it needs a reset followed by a read inside one step, or a program with a watch call that runs only sometimes. Neither blocks anything. Item 6 (frame rollback as a worked example) is the next real piece of work and `NEXT.md` now points at it. --- ## What I tried that did not work - **Two Flan type errors in `dev-watch.flan`.** `(* i 3)` over a `let`-bound `0` is `i32`, and `watch-num-i64` wants `i64` — fixed with an explicit `(i64 …)`. Then `loop-cells` was declared `i64` and returned the `i32` counter; declared `i32`. Both are ordinary, but they cost two daemon round trips to find because a build error goes to the daemon's stdout, not to the client — worth knowing if you are driving `flan dev` by hand. - **My first probe script spoke the wrong protocol.** `Wire.send` is **length-prefixed** (`"%d\n%s"`), not newline-delimited. A newline-delimited client gets the connection closed with no error text. If you write a throwaway client, read `lib/wire.ml:32` first. - **Two attempts at the reset assertion failed before the third stuck**, and both failures were informative rather than noise: 1. *"the window did not reopen: 8 after 8"* — one step writes eight samples, so two reads back to back leave no room *under* the count for a reset to show in. The test now runs the accumulator up past 32 first (which is itself the other half of the claim: every one of those polls is a plain read, and a plain read must not reset, or the count could never climb). 2. *"the window did not reopen: 40 after 40"* — reading immediately after `:reset t` still sees the old count, because the reset bumps an epoch and the **slot clears on its next sample**. That is the design and not a bug; the test now `await`s the drop and the comment explains why the laziness is what keeps the reader out of the table. - **Not attempted, deliberately:** the `(watch "hp" hp)` arm in `check.ml`. Both `PORTING.md` and `flan_dev.c` say it is only wanted for *composites* (a struct, a slice, a union), which need `Render.render` pointed at the `flan_dev_watch_emit_*` emitters. A scalar and an accumulator both reach the runtime by plain `declare-c` and need no compiler change, which is the property that kept this change out of `check.ml` entirely. - **One worktree note.** This worktree was checked out at `2c232dd`, an ancient commit, rather than at `dev-loop`'s tip — none of `PORTING.md`, `NEXT.md`, `lib/dev.ml` or `runtime/` existed in it. `git reset --hard dev-loop` on a clean tree fixed it. Also, `dune` finds the parent repo's root from inside a worktree, so every build here is `dune build --root .` / `dune test --root .`.