From 37d4e7b32b0b724563492e23fd8d45ca676390b2 Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Mon, 21 Sep 2026 19:23:42 +0700 Subject: [PATCH] The standing rules go in the repository, and three dated reports leave it --- CLAUDE.md | 116 ++++++++++++++ docs/DIAGNOSTICS-AUDIT.md | 225 ---------------------------- docs/README.md | 46 ++++-- docs/REVIEW-production-readiness.md | 220 --------------------------- docs/overview.md | 54 ------- 5 files changed, 149 insertions(+), 512 deletions(-) create mode 100644 CLAUDE.md delete mode 100644 docs/DIAGNOSTICS-AUDIT.md delete mode 100644 docs/REVIEW-production-readiness.md delete mode 100644 docs/overview.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..7a0c9195 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,116 @@ +# Working in this repository + +For any agent working here, including a lane in its own worktree. These are +standing rules, not preferences. Where a rule has a reason, the reason is given +once — the rules without one are the ones that have already cost something. + +## Never + +- **Never touch anything under `/home/joe/Development/fnm/`.** That is the + author's own working copy of the falling-sand game, open in an editor with a + live session attached. `fnm/flan/sand.flan` is not the same file as this + repository's `sand.flan` and is never to be read-and-written-back, moved or + edited. The copy in this repository is ordinary tracked source and may be + edited like any other file. +- **Never execute `examples/*.flan`, `sand.flan`, or anything that links + raylib.** They open a window on the author's desktop, which strobes it. + Compile them, emit them, diff them — never run them. +- **Never run `dune clean`.** It walks up out of a worktree and deletes the main + checkout's `_build`, which takes the author's `flan` binary with it. This has + happened. Run `dune` with `--root .` from inside your own worktree. +- **Never kill a `flan dev` daemon you did not start.** The author keeps a live + one attached to their editor all day. Orphans from finished lanes are fair + game; anything whose cwd is under `fnm/` is not. +- **Never put a `Co-Authored-By`, a `Claude-Session` trailer, or any other + watermark in a commit message.** + +## Commits and branches + +`master` is the trunk and tracks `origin/master`. Lanes work in their own git +worktree off it and do not merge or push — merges are resolved by the session +that dispatched the lane. + +A commit message is a single declarative sentence in the repository's voice, +saying what is now true rather than what was done: "A thunk named for the order +it was minted in is a different signature after a reorder", not "fix thunk +naming bug". A body is for the reasoning, when there is any. + +## Tests + +`dune test --root .` is the suite and takes seconds. Run it constantly; it must +be green before a lane reports. + +`dune build @checks` is `@page`, `@x86` and `@cells` — the reference page's +examples still compile and print what the page says, the hand-written x86 +backend still agrees with LLVM, and a `--dev` build still calls through its +indirection cells. `@sanitize` and `@valgrind` run the corpus under ASan/UBSan +and memcheck and take tens of minutes. + +Those three stay out of a lane's own run. They are swept in a batch after +several lanes have merged, and the fixes are batched with them. Keeping the +default run fast is deliberate: a suite that takes tens of minutes is a suite +nobody runs. + +Note that ASan does not see a stack-lifetime bug — uninitialised stack reads +need `@valgrind`. + +## Evidence + +Read the source, do not recall it. `docs/REFERENCES.md` lists the reference +clones on this machine — Odin, SBCL, Zig, Carp, clojure-mode, CIDER, raylib and +the rest — and what each one answers. A claim in these notes that came from one +of them was read out of the clone; the ones that were recalled instead have been +wrong before. + +The same applies to this repository. Cite a file and a line because you opened +it. When a brief hands you a list of facts, verify them before building on them +— a brief written from a stale tree has sent a lane a full day in the wrong +direction. + +## Diagnostics + +Elm's shape: the source line, a caret, what the compiler understood, and the fix +named. Beyond that: + +- A message is written for someone who has never used Flan and does not know its + history. Never "X is now Y", never "X was renamed", never our rationale. If + `defvar` no longer exists, the message says `defvar` does not exist — not that + it became something else. +- Every suggestion a message prints must compile. +- An assertion about the compiler's own invariants is prefixed `internal:` and + says it is a compiler bug. A user never caused one. + +## Writing + +User-facing prose — the README, `web/index.html`, the Emacs manual — is plain. +Odin's website is the reference: declarative rather than second-person, the +concept defined before the mechanics, the code example after the prose that +frames it, and rationale given its own subsection rather than mixed in. + +No aphorisms, no closing lines built for effect, no rhetorical inversions. If a +sentence's only content is its own style, cut it. This applies to reports and +commit messages as much as to documentation. + +## The records + +- `FIX.org` — decisions, dated, in the author's own words where they were + spoken. A lane records its decision here. On a merge conflict in this file, + keep both sides: two lanes recording two decisions is not a conflict. +- `NEXT.md` — what is in flight and what is known to be missing. +- `docs/BUILT.md` — why the parts that exist are shaped the way they are. The + largest and most load-bearing document here. +- `docs/DISCUSS.md` and `DISCUSS.org` — open questions, deliberately unanswered. +- `docs/handoffs/` — one report per finished work session. +- `docs/README.md` indexes all of it. + +## The shape of the work + +The author dogfoods the language in a separate project, hits friction, and +reports it. Work is dispatched as parallel lanes in isolated worktrees; every +lane is reviewed by an independent agent before it merges; the dispatching +session resolves the merges. + +A review is adversarial. Verify by building and running rather than by reading, +measure both sides of a claimed fix, and say what you actually measured. A lane +that reports a number you did not reproduce is a lane with a number you should +reproduce. diff --git a/docs/DIAGNOSTICS-AUDIT.md b/docs/DIAGNOSTICS-AUDIT.md deleted file mode 100644 index 2fe99669..00000000 --- a/docs/DIAGNOSTICS-AUDIT.md +++ /dev/null @@ -1,225 +0,0 @@ -# Diagnostics audit — Flan against Elm - -Read-only audit, 2026-09-20. Feeds the fix pass that starts once the two live -lanes merge. Untracked on purpose: this is a worklist, not a spec. - -## The contract being graded against - -Elm's, plus this repo's own two additions: - -- **(S) Show / locate** — the exact code, a caret, at the *most specific* span. -- **(U) Understood vs conflicted** — say what the compiler took the code to - mean, and what that collides with. -- **(F) Fix by name** — name the thing to write instead, not the category. -- **(R) Register** — plain language. FIX.org's literary voice is banned here. - -Grades are per-dimension letters. A = Elm-class, B = good with one gap, -C = states the fact and stops, D = misleading or actively unhelpful. - -## The machinery that already exists (lib/loc.ml) - -Worth stating up front, because most of the worklist below is *underuse*, not -absence. - -- Spans, not points (`Loc.t` carries `eline`/`ecol`), and `squiggle` draws a - multi-column underline from them. -- **Secondary notes with their own locations and severities** (`Loc.note`, - rendered by `report` as their own `file:line:col: info:` entry, marked `-` - rather than `^`, sorted into source order). This is Elm's "this here… but - that there" and it is fully built. -- `kind` — a stable id per diagnostic (`"check/unknown-field"`), deliberately - never printed. -- `expansion` — names the macro a form came out of, automatically, off the - location. -- GNU `file:line:col:` first line, so `M-x compile` and `next-error` work. - -What it does **not** have: any notion of "expected because of *that* -signature" beyond a free-form note; anything a runtime trap can use (the -runtime has only preformatted loc strings, no access to source text). - ---- - -## The worst 20, ranked by (badness × how often a user hits it) - -| # | Message | Site | Trigger (one-liner) | S | U | F | R | Fix direction | -|---|---------|------|---------------------|---|---|---|---|---------------| -| 1 | `unknown name p.x` | check.ml:2732, 4046 | `(defstruct P [x i32])` then `p.x` anywhere | B | D | D | C | The dot-infix habit from C/Go/Odin. The checker can see the head `p` is bound and is a struct with field `x` — split on `.`, and say "field access is written `(.x p)`". The repo's own comment in examples/core-2d-camera.flan:20 writes `camera.rotation` in prose while the code is `(.rotation camera)`, so the author trips it too. | -| 2 | `dyn +: int and text, and it takes two numbers — (+ 3 "hi")` | flan_dyn.c:724 (`trap2`), :734 (`trap1`), :746 (`trap_range`), :814 (`trap_oom`) | `(defn add [x y] dyn (+ x y))` + `(add 3 "hi")`, then `flan run` | D | B | D | B | **No location at all** — no file, no line, nothing. In a dynamic-first language these *are* the type errors. flan_rt.c's bounds and arith traps already take an emitter-threaded `(loc, loclen)` pair, so the ABI precedent exists; the dyn entry points simply were never given one. Highest-leverage runtime fix. | -| 3 | `expected i32, found string` | check.ml:1866 (`expect`), 6292 (call args) | `(defn add [a i32 b i32] i32 …)` + `(add 1 "two")` | B | C | D | A | The most-hit message in the compiler. Caret is right; it never says *which* argument of *which* function, and never points at the parameter that wanted it. Elm's whole hallmark. `Loc.note` on the `Ast.field`'s `floc` is a five-line change: "the 2nd argument of `add`" + "`b` is declared `i32` here". | -| 4 | `unknown function prinltn` / `unknown name n` | check.ml:6259, 2732 | `(prinltn "hi")` | B | C | D | B | No did-you-mean, although `near_miss` (check.ml:670) is written, tested and wired — to **types only**. Point it at `env.fns`, `env.globals` and the local scope. Cheapest structural win on the list. | -| 5 | `expected bool, found i32` | check.ml:1866 via `check_truthy` (3596) | `(let [x 1] (if x …))` | A | C | D | A | Caret is exactly right (the `check_truthy` loc work paid off). But the message never states Flan's truthiness rule — bool or dyn, nothing else — and never names the fix (`(not= x 0)`). Special-case the condition position. | -| 6 | `binding 5 has no value — let takes name/value pairs` | parse.ml:670 | `(let [x i32 5] …)` | C | D | D | B | A type annotation in `let` is the single most natural thing for someone arriving from a typed language, and `let` has none. The message reads as if the user miscounted. Detect "middle form names a type" and say so: "`let` bindings take no type annotation — write `[x 5]`". | -| 7 | `get takes 2 arguments, given 1` against the **user's own** `(defn get [p P] …)` | check.ml:5296 (builtin dispatch) + 6233 | `(defn get [p P] i32 …)` + `(get p)` | D | D | D | B | A user defn whose name collides with a builtin is silently shadowed, and then the arity refusal is measured against the *builtin*, pointing at the user's call. Either refuse the shadowing definition at its `dloc` with a note, or report the arity against the definition the user can see. **Settled 2026-09-20, neither way: the author chose "allow shadowing but warn" — the defn wins at every call site in its own file and the compiler warns once at the definition. See FIX.org, "Shadowing a builtin".** | -| 8 | `unhandled Boom` | flan_rt.c:646 | `(defstruct Boom [why i32])` + `(error (Boom {.why 7}))`, `flan run` | D | C | D | B | Three words. No location (not even the `error` site, which the emitter knows), no field values, no list of the handlers that were in scope. The condition system is a headline feature and this is its failure mode. | -| 9 | `the collection nosuch: is a directory named nosuch somewhere above /…/. , and there is none` | load.ml:120 | `(import zz "nosuch:thing")` | B | C | C | D | Reads as an assertion immediately contradicted. Also emits a bare `/.` on the path. Rewrite as a plain statement of the search ("no directory named `nosuch` between here and the root") and list what collections *were* found. | -| 10 | `and`'s last operand gets the previous operand's caret | parse.ml `shortcircuit`, via check.ml:3632 | `(println (and true true (vec-new i32)))` | D | B | C | A | Already diagnosed in FIX.org:1036 with three rejected fixes; the accepted one — `check_if` preferring the arm that is not a compiler temp when choosing which to blame — is a check.ml change nobody owned. This pass owns check.ml. | -| 11 | `unterminated string` | reader.ml:92 | `(println "oops` | C | C | C | A | Caret is one column on the opening quote, and there is **no** "the input ends here" note — unlike `reader/unclosed` (241) and `reader/mismatched-closer` (250), which both have one. Copy their shape. | -| 12 | `unknown type i — did you mean i8? A parameter with no type is dyn, so this would otherwise be read as a second parameter called i` | check.ml:850 | `(defn idx [v i] dyn …)` | A | B | C | C | Locates and explains well, but the did-you-mean fires on a *lowercase* name the user plainly meant as a parameter, so the suggestion is a false accusation. Suppress `near_miss` when the name is lowercase and in a parameter vector; lead with the dyn-parameter reading instead. | -| 13 | `expected a type, found 1. This is the return type, which every defn states -- a function that returns nothing writes ()` | parse.ml:1156 | `(defn f [x i32] (+ x 1))` | C | B | B | C | Says the fix, which is good. Two warts: the caret lands on the `1` deep inside the body rather than on the position where the return type belongs; and the literal `--` where the house uses `—` everywhere else. | -| 14 | `f64 is not a struct, so it has no fields` | check.ml:4028 | `(match s (Circle c) (.r c))` — single-field case binds the payload directly | B | C | D | A | The user wrote what looks like a destructuring pattern and got a type fact. Say what the pattern bound (`c` is the payload, an `f64`) and that the field is already in hand. | -| 15 | `% is a constant` | check.ml:4043 | `(defconst k 3)` + `(set k 4)` | A | C | D | A | Four words. Needs a `Loc.note` at the `defconst` and the named fix (`defonce`). `no_container_defconst` (7447) already shows how the house writes this well — imitate it. | -| 16 | `% is a parameter, and parameters are not assignable places (spec-memory.md) — bind a local with let` | check.ml:4037 | `(defn f [x i32] i32 (set x 1) x)` | A | B | B | C | Names the fix. Register wart: a diagnostic should not cite a spec filename at the user; put the rule in words and drop `(spec-memory.md)`. Same for `(plan.org, Types)` at 4609/4614 and `(see plan.org)` at 423. | -| 17 | `% is not implemented yet — milestone %d (see plan.org)` | check.ml:423 (`unimplemented`), used widely | `(Result i32 string)` in a type position | B | B | D | D | Sends the user to a planning document. Say what is missing in one clause and what to write in the meantime, or nothing. | -| 18 | `% takes numbers, found string` | check.ml:4219 (`binary`) | `(+ "a" "b")` | C | B | C | A | Caret covers the whole form rather than the offending operand — the exact "whole form vs operand" regression class this repo already fixed once in `check_truthy`. The operand's `Tast.eloc` is right there. Also: no mention of `str-cat`/the concatenation route, which is what the user wanted. | -| 19 | `expanding this declaration produced %d of them` / `expanding this expression produced %d forms, and an expression is one` | parse.ml:1541, 1569 | a `defmacro` returning two forms | B | C | D | C | "of them" has no antecedent. The `expansion` field on the diagnostic can name the macro automatically; use it, and suggest `do`. | -| 20 | `this fn has %d parameters and %s was wanted here` / `nothing here says what this fn's parameters are` | check.ml:2801, 2809 | passing an `fn` literal where no signature is in view | B | B | C | A | Correct and readable; the arity one should note the parameter list it was measured against, which is the one thing the reader cannot see from the caret. | - -### Also seen, not ranked - -- `an index is an i32, and %s is wider — write (i32 …), because …` (4113) is - **fine** — S/U/F/R ≈ A/A/A/B, listed only so the fix pass does not touch it. -- The `--warn-memory` / `check/no-gc` sites (8566, 8723) are warnings rather - than refusals and were not exercised; they build `Loc.diag` directly and - carry no notes. - ---- - -## The house's best — imitate these, not just Elm - -These are internal precedent and they already satisfy the contract. The fix -pass should copy **their shape**, so the corpus converges on one voice. - -**1. Unknown field, with the declaration shown.** check.ml:3706 / 2433 / -3793, via `declared_note` (169). This is the model. - -``` -v6.flan:3:23: P has no field z - 3 | (let [p (P {.x 1 .z 9})] (println (.x p)))) - | ^ -v6.flan:1:1: info: P is declared here, with x - 1 | (defstruct P [x i32]) - | --------------------- -``` - -Primary span on the exact offending key; secondary span on the declaration; -the available names enumerated. All four dimensions, in five lines. - -**2. Non-exhaustive match.** check.ml:3983. - -``` -w4.flan:3:3: this match is not exhaustive — Shape.Tri has no arm. Add it, or a _ arm for the rest - 3 | (match s - | ^^^^^^^^ -w4.flan:1:1: info: Shape is declared here, with Circle, Square, Tri -``` - -Names the missing case *and* both fixes, with the declaration alongside. -Elm-class. - -**3. The defconst pair**, landed 2026-09-20. `const_defconst_init` (7527) -and `no_container_defconst` (7447). No secondary span, but the prose does the -whole contract: what was understood ("the constant `n` is computed"), what it -conflicts with ("a defconst is what the linker writes into the image and has -nowhere to run"), and two named ways out (`defonce`, or a folded literal). The -best *prose-only* message in the tree. - -**4. The dyn view-lifetime refusal.** `view_not_permanent` (1570). Explains -the rule, gives the one shape that does work (`defonce g …`), and enumerates -what is refused. Borderline long — it is the closest thing in the tree to the -banned essay register, and a fix pass should cut it by a third rather than -lengthen anything toward it. - -**5. Reader's unclosed / mismatched-closer.** reader.ml:241, 250. The only -two-span diagnostics outside check.ml, and both are right: - -``` -u4.flan:2:25: expected ')' to close '(', found ']' -u4.flan:2:12: info: '(' is opened here -``` - -**6. `flan_arith_fail`'s overflow sentence** (flan_rt.c:911). A runtime trap -that explains *why* one pair of operands overflows a division. Good register, -good detail — it just lacks nothing except company. - ---- - -## Structural gaps — things Elm does that no Flan message does - -**1. Two-span "this here… but that there" as a habit, not an exception.** -*Not absent — underused.* `Loc.note` is fully built and rendered, and roughly -eight of ~250 sites use it (`check/duplicate-field`, `check/unknown-field`, -`check/defined-twice`, `check/duplicate-parameter`, -`parse/enum-autoincrement-collision`, `reader/unclosed`, -`reader/mismatched-closer`, `check/non-exhaustive-match`). **Feasible today, -no machinery needed** — the gap is that `expect` (1866) and the call-argument -path (6292) don't have the wanting-side location threaded to them. `Ast.field` -already carries `floc`; the defn record is reachable from the call site. This -is the single highest-value structural change and it is plumbing, not design. - -**2. Did-you-mean on anything but types.** `near_miss` (670) is a real -one-edit matcher with a real candidate set, wired at exactly one call site -(771). **Feasible today**: the candidate sets for functions, globals and -locals are all in `ctx.env` / the scope stack at the raise points -(check.ml:6259, 2732, 4046). Field names are already gathered by -`declared_note`, so field-typo suggestions are nearly free too. - -**3. Error titles / anchors / doc links.** Elm prints `-- TYPE MISMATCH ---` -and links a hint page. Flan has the ingredient — every diagnostic carries a -stable `kind` — and loc.ml:109 says explicitly that it is "never printed as -the reason". **Feasible today** at zero cost: print it as a trailing -`[check/unknown-field]` or route it to a docs anchor. Needs a decision, not -work. (Caveat: the first line is load-bearing for compilation-mode, so the id -belongs at the end of the line or on an indented continuation.) - -**4. Any caret at all at runtime.** Runtime traps carry a preformatted -`file:line:col` string and nothing else — flan_rt.c has no access to the -source text, and flan_dyn.c has not even the string. **Partly feasible**: the -`flan_trap_hook` (flan_rt.c:598) already hands control to the dev session, -which *is* in-process with the compiler and can read the file. Rendering a -squiggle there is the natural home — but dev.ml belongs to another lane, so -this is a hand-off, not a row in this pass. The cheap half — threading a loc -into flan_dyn.c's four trap functions — is entirely within this pass. - -**5. "What I understood" as structure rather than prose.** No message in the -tree separates the two halves the way Elm's body does ("This function expects -… / But you gave it …"). The good ones (defconst, view-lifetime) achieve it -with a paragraph. **Feasible as convention**, not machinery: the `diag` -record has `dmsg` plus notes and nothing between, so the shape would have to -be a writing rule — "first clause is what was read, second clause is the -collision, last clause names the fix" — enforced by review rather than types. - -**6. Multiple errors without cascade.** `Loc.sink` exists and finishes at -phase boundaries only, which loc.ml:178 admits is "deliberately crude". Not -graded here (no program was written to exercise cascade), but it is the -remaining Elm behaviour with machinery that only half exists. - ---- - -## Register notes for the fix pass - -- **Never cite a repo file at the user.** `(spec-memory.md)`, `(plan.org, - Types)`, `(see plan.org)` appear in at least six messages. Say the rule. -- **One dash convention.** parse.ml:1156 uses `--`; every other message uses - `—`. -- **"%s of them" / "this one"** — pronouns with no antecedent once the - message is read cold. parse.ml:1541 is the clearest case. -- **Length.** The tree's long messages are mostly *earning* their length - (defconst, runaway-instantiation, predicate-not-carried). The one to watch - is `view_not_permanent`, which is a paragraph with a subordinate clause - nested three deep. The rule the repo wants — plain language, no essay - register — bites there and nowhere else so far. - ---- - -## Coverage — honest statement - -- **Harvested**: every `Loc.fail` / `Loc.failk` / `Loc.diag` / `Loc.note` - call site in lib/check.ml, lib/parse.ml, lib/reader.ml and lib/load.ml — - roughly 250 sites — plus every `fprintf(stderr, …)` trap in - runtime/flan_rt.c and runtime/flan_dyn.c. -- **Rendered by hand**: about 30 messages, by writing the triggering program - and running `_build/default/bin/main.exe check` or `run` on it, and reading - the full output including carets and secondary lines. Scratch programs are - in the session scratchpad, not the repo. -- **Graded from source only**: the remaining ~220. Their S grade is inferred - from which location value they are raised against, which is reliable for - "is it the operand or the whole form" only where the code makes it obvious. - Treat those as provisional. -- **Out of scope**: lib/emit.ml, lib/x86.ml, lib/cimport.ml, lib/session.ml - and lib/macro.ml internals; the wasm32 and JS backends; anything printed by - `flan dev`'s break loop. -- **Not touched**: lib/dev.ml, which belongs to another lane. Gap 4's dev-side - half is written up as a hand-off for that reason. -- **Not exercised**: the `--warn-memory` warning path, macro-expansion - diagnostics with a non-trivial `expansion` chain, and multi-error cascade - behaviour through `Loc.sink`. diff --git a/docs/README.md b/docs/README.md index 4cb8e069..9638e5b8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -34,10 +34,26 @@ standing rule that a claim in these notes was read out of a clone rather than re ## The reports +Each of these was written once, from a reading or a measurement, and is not +maintained afterwards. They are kept because the reasoning in them is the +reason the code looks the way it does, and deriving it again would cost more +than reading it. + | File | What it is | |---|---| | `SPIKE-GENERICS.md` | Milestone 5's parametric polymorphism, run early and out of order as a spike. The spike succeeded and generics landed, so this is the report of finished work rather than a live plan — but its findings about where the cost falls are still the reason the implementation looks the way it does. **Historical, findings still stand.** | -| `overview.md` | The first brainstorm, from before the language had S-expressions. It says in its own first line that it is superseded and no longer accurate. Kept for history only; its table of what changed is the only part worth reading. **Superseded.** | +| `SPIKE-INFERENCE.md` | Whether the checker should infer more than it does. A reading rather than a branch: nothing was built. | +| `SPIKE-DYNAMIC.md` | The dynamic-value runtime — NaN-boxed values over a mark-sweep heap. The design the `dyn` type and `runtime/flan_dyn.c` were built from. | +| `SPIKE-DUPLICITY.md` | Where the dyn side and the native side meet, and where one of them repeats the other. The source of the rule that a capability is written once per side and never twice on the same side. | +| `SBCL-REDEFINITION-NOTES.md` | What SBCL does when a struct is redefined under a running image, read out of the clone. Written for the open question of whether Flan's hard refusal should become a warning and a kept old layout. The question is still open. | +| `BUGS-2026-09-18.md` | A five-agent sweep of the runtime, checker, backends, dev loop and Emacs client. The bugs are fixed; the file is kept because `spec-memory.md` cites one of its findings as evidence. | + +Three files that used to be here are gone, and `git log` is where they live +now: `overview.md`, the first brainstorm from before the language had +S-expressions, which had said in its own first line that it was superseded; +`DIAGNOSTICS-AUDIT.md`, the worklist the diagnostics rewrite was done from, +which the rewrite closed; and `REVIEW-production-readiness.md`, a review of a +tree a thousand commits behind this one. ## `handoffs/` @@ -48,18 +64,22 @@ describes a session that has ended — but they are cited by name from source co because the reasoning behind a guard or a calling convention is often only written down once and this is where it was written. -Six of them are the hand-written x86-64 backend, read in the order the work happened: -`handoffs/HANDOFF-x86-rt.md` set the remaining-items list that the four after it close, `handoffs/HANDOFF-x86-redef.md` -built the redefinition emitter, `handoffs/HANDOFF-x86-aggregates.md` took that across the struct boundary, -`handoffs/HANDOFF-x86-guards.md` settled the two guards nothing reaches, `handoffs/HANDOFF-x86-debug.md` added debug -information, and `handoffs/HANDOFF-x86-cost.md` measured what the backend costs and set the survey running on its -own so a refusal cannot sit unnoticed again. +Nine of them are the hand-written x86-64 backend, read in the order the work happened: +`HANDOFF-x86-rt.md` set the remaining-items list the others close, `HANDOFF-x86-redef.md` +built the redefinition emitter, `HANDOFF-x86-aggregates.md` took that across the struct boundary, +`HANDOFF-x86-guards.md` settled the two guards nothing reaches, `HANDOFF-x86-debug.md` added debug +information, and `HANDOFF-x86-cost.md` measured what the backend costs and set the survey running on its +own so a refusal cannot sit unnoticed again. Then `HANDOFF-x86-devloop.md` for making it the dev loop's +default, `HANDOFF-x86-abi-marker.md` and `HANDOFF-x86-annotate.md`, and +`HANDOFF-x86-macro-visibility.md` for why a macro module's symbols are hidden. -The other six sessions are unrelated to each other. `handoffs/HANDOFF-arith.md` is why a divide by zero is a condition -rather than a `SIGFPE`. `handoffs/HANDOFF-raylib-ports.md` is the running record of the last two raylib example -ports, whose lasting findings were folded into `PORTING.md`, and `handoffs/HANDOFF-cimport-ptr.md` is the arm +The rest are unrelated to each other. `HANDOFF-arith.md` is why a divide by zero is a condition +rather than a `SIGFPE`. `HANDOFF-raylib-ports.md` is the running record of the last two raylib example +ports, whose lasting findings were folded into `PORTING.md`, and `HANDOFF-cimport-ptr.md` is the arm `cimport.ml`'s header check had been promising in a comment and not implementing — which is what those two ports -found, worked around and wrote up. `handoffs/HANDOFF-devtest-noise.md` is the linker -error `dune test` used to print on every run. `handoffs/HANDOFF-emacs-flake.md` is the `test_emacs` flake that +found, worked around and wrote up. `HANDOFF-devtest-noise.md` is the linker +error `dune test` used to print on every run. `HANDOFF-emacs-flake.md` is the `test_emacs` flake that turned out to be `SIGPIPE` killing the daemon mid-reply, and it is worth reading for the two mechanisms it -rules out as much as for the one it found. `handoffs/HANDOFF-tidy.md` is this reorganisation. +rules out as much as for the one it found. `HANDOFF-dyn-m1.md` is the first milestone of the dynamic +runtime, `HANDOFF-lowering-buffer.md` the buffer that shows what a form lowered to, `HANDOFF-rot.md` a +sweep for prose that had stopped being true, and `HANDOFF-tidy.md` is this reorganisation. diff --git a/docs/REVIEW-production-readiness.md b/docs/REVIEW-production-readiness.md deleted file mode 100644 index 9c46fad4..00000000 --- a/docs/REVIEW-production-readiness.md +++ /dev/null @@ -1,220 +0,0 @@ -# Production-readiness review — 2026-09-17 - -Three review lanes (C runtime, compiler robustness, tooling/UX) plus a suite run on the -merged tree (`e9d0b99`, 232 checks, 0 failures, `@x86` 104 MATCH / 0 DIFFER). This file is -written to be implemented from: each item says what is wrong, where, and what the fix is. -Items marked **(known)** are already recorded in NEXT.md/FIX.org and are listed so the -ranking is complete, not because they are news. - -**The verdict in one paragraph.** The development loop is the most finished part of the -project and the shipping loop is the least. A solo developer in Emacs on this machine can -build a small raylib game today and the experience is genuinely good — the module system, -the Emacs client's failure handling, the `import-c` header diffing, the diagnostics (a real -span-and-notes system, ~365 located refusal sites, zero warning suppressions), and full -LLVM/x86 backend parity are production-grade. Nearly every gap sits at the boundary where a -second person, a second machine, or a shipped binary appears — plus three memory-safety -holes in the runtime that nothing currently mentions. - ---- - -## Tier 1 — correctness blockers - -These produce silent wrong values or memory corruption in programs that look fine. - -### 1.1 Every number→string conversion aliases one static buffer **(known, unenforced)** -`runtime/flan_rt.c:217-218` — `flan_i64_to_bytes` / `flan_f64_to_bytes` / `flan_u64_to_bytes` -all return `{scratch, len}` into one shared `static char scratch[64]`, and `emit.ml:2266-2270` -never copies the bytes out. Holding two results — `(vec-push! v (str i))` in a loop, a -`(str n)` stored in a struct — silently reads clobbered bytes. No crash, no diagnostic, so -sanitizers never see it. NEXT.md's "Sharp edges" records it; the prelude's `append-i64!` -family works around it; nothing *enforces* it. -**Fix to decide, then do:** either make the runtime conversions allocate from -`context/temp` (semantic change, kills the hazard everywhere), or have the checker type -these results as a distinct short-lived slice that may not be stored or outlive the next -conversion. The first is simpler and matches the Odin idiom already adopted for arenas. - -### 1.2 `cap * size` overflow at container growth -`runtime/flan_rt.c:1336` (Vec), `:1574` (Pool, `cap*size + cap*sslot`). `flan_vec_reserve` -(`:1374`) forwards arbitrary `n` and the `1<<40` clamp at `:1333` is bypassed when `want` -exceeds it (`cap = want; break;`). A wrap to a small positive allocates a tiny block while -`v->cap` stores the unwrapped value; the next push memcpys far past the block (`:1386`). -Same family, lower reach: `flan_over_budget` (`:785`), `flan_map_block_size` (`:2080`). -**Fix:** `__builtin_mul_overflow` (or `size > INT64_MAX/cap`) at every grow/reserve site; -overflow reports as `StorageExhausted` like any other allocation failure. Small, mechanical. - -### 1.3 `Map` has no removal -`runtime/flan_rt.c:1788`, `:2439` — deferred, no tombstones. A `Map` you cannot delete a -key from is a daily-use gap, not an edge case (entity tables, caches). -**Fix:** Robin Hood backward-shift deletion (no tombstones needed with the existing -cache-line-run layout), a `map-remove!` builtin through check/emit/x86, tests on both -backends. Medium-sized, self-contained. - -### 1.4 The dev allocation registry is read cross-thread with no synchronisation -Writer: game thread inside every alloc/free (`runtime/flan_dev.c:1130`, `:1180`). Reader: -the agent's listener thread (`vendor/agent/flan_agent.c:1081`, `:1146`) on a *running* -program — the `reg` verb has no stopped-gate, unlike the frame chain. No seqlock, no -atomics, and `flan_reg_compact` (`flan_dev.c:1099`) memsets and reinserts the whole table -mid-scan. A torn `(type, typelen)` pair is an out-of-bounds read in the listener. -Ranked Tier 1 because the dev loop is the project's priority. -**Fix:** either gate the `reg` verb on stopped (matching the frame chain — smallest change), -or give the registry the same per-slot seqlock the watch table already has -(`flan_dev.c:380-870` is the template in the same file). - -### 1.5 `(addr (.field x))` on an `Option` — confirm, then fix -FIX.org records `Tast.Addr (Tast.Pfield ...)` failing on both backends (working route: -`Prim (AddrOf, [Field ...])`). `check.ml:4715` builds `Tast.Addr` from user-writable -`(addr )` and `check.ml:2968` builds `Pfield` from `(.field x)`, so the combination -is expressible today; neither `emit.ml:1371-1375` nor `x86.ml:2070` has an Option arm. -**Fix:** first write the failing program to confirm reachability from source; then either -lower `Addr(Pfield)` through the AddrOf route in `check.ml`, or add the Option arm to both -backends. If unreachable from source, add the refusal-by-name that the house rule requires. - ---- - -## Tier 2 — the install and shipping story - -One coherent problem: the compiler runs only from its checkout, and its output runs only on -this machine. This is the single largest thing between "the author's language" and -"a language someone else can try." - -### 2.1 There is no install path -`dune-project` is two stanzas — no `(package ...)`, so `dune install` cannot work; the -documented way to run the compiler is `dune exec`. Worse, the README's suggested workaround -(copy the binary onto PATH) silently breaks the flagship feature: the merged `flan dev` -needs `flan.cmxa` + `flan.a` *beside the binary* (`lib/dev.ml:3198-3203`) and `ocamlfind` -on PATH at runtime (`lib/dev.ml:2898`). The failure message names the escape hatches -(`FLAN_LIBDIR`, `--two-process`) but neither README nor `emacs/MANUAL.md` mentions them. -**Fix:** a `(package)` stanza with install rules that place `flan.cmxa`/`flan.a` where the -binary's own lookup finds them; document `FLAN_LIBDIR`; make the README's install section -truthful about what `flan dev` needs. - -### 2.2 A built game runs only on machines configured like this one -No `-static` anywhere in `lib/build.ml`; `vendor/raylib/link` names `-l:libraylib.so.550` -by exact soname (a Fedora-ism — no unversioned symlink). The binary is otherwise genuinely -standalone (the C runtime is embedded in the compiler, `lib/dune:25-41` — right design). -**Fix:** a `flan build --static` (or bundled-raylib) option, and per-target link lines that -do not hardcode one distro's soname. - -### 2.3 The env-var surface is real and entirely undocumented -`FLAN_CLANG`, `FLAN_EMCC`, `FLAN_LD`, `FLAN_LLC`, `FLAN_WASM_SYSROOT`, `FLAN_WASM_BUILTINS`, -`FLAN_CACHE_DIR`, `FLAN_OCAMLFIND`, `FLAN_LIBDIR`, `FLAN_RAYLIB_WEB` — none in the README. -Also undocumented: the live loop needs `llc`/`ld` at an LLVM version matching clang -(`lib/build.ml:886-962`), and a mismatch breaks `C-c C-c` while `flan build` keeps working. -**Fix:** an env-var table in the README, and a sentence about the llc/clang version coupling. - ---- - -## Tier 3 — the standard library - -The containers, strings/UTF-8, sequences, random, and printing layers are decent. The gaps: - -- **No clock of any kind** — no `now`, no monotonic time, no `sleep`, not in prelude or - runtime. A raylib game gets time from raylib; a non-graphical tool cannot time anything. - Blocker for the "or tool" half of day-to-day use. A `time`/`sleep` builtin pair backed by - `clock_gettime` is small. -- **Math is five f32 functions** (`lib/prelude.ml:624-677`: sqrt, sin, cos, atan2, pow). - No tan/asin/acos/log/exp/fmod/abs/hypot, no f64 variants, no PI constant. -- **File IO is whole-file only** (`slurp`/`barf`). No streaming, stdin, directory listing, - metadata, delete/rename/mkdir. -- **No env vars, no process spawn.** `argv` and `exit` are the whole OS surface. -- **The prelude is an OCaml string literal** (`lib/prelude.ml:32`) — cannot be read as Flan, - extended, or replaced without rebuilding the compiler. Its own docstring promises a - `core:` package "at milestone 3" that does not exist. The `import-c` header-diffing is a - strong mitigation (users can bind libc and be told when they get it wrong), but the - promised `core:` migration is the structural fix. - ---- - -## Tier 4 — robustness and polish (small, high-value) - -- **`Sys_error` uncaught in the CLI** — `flan check nosuch.flan` → - `Fatal error: exception Sys_error(...)`. The daemon already has the arm - (`lib/dev.ml:2613-2621`); copy it into `bin/main.ml:8-32`. One line. Add a `Not_found` - backstop arm at the same time — nothing reaches it today, but the failure would be a - message-less `Fatal error: exception Not_found`. -- **Three tests leave `Fatal error: exception Flan.Loc.Error(_)` on stderr** - (test_acceptance, test_session, test_web; suite still passes). Some spawned compiler - process dies without going through the error printer — locate the spawn (grep the dune - log attribution), and either wrap it or assert on the formatted message instead. -- **`abort()` in the dev runtime** — `flan_dev.c:47-50`, `:103`: reload-name-table - exhaustion, intern OOM, and "global changed size" kill the game instead of signalling. - Against the grain of everything else in the runtime; route through `flan_error`. -- ~~**Six trap paths bypass `flan_exit_hook`** and end a merged `flan dev` session~~ — - fixed 2026-09-18. Two corrections to the entry as written. The hook bounds and - arithmetic park through is `flan_break_hook`, not `flan_exit_hook` — that second one is - normal termination, the one `main` reaches when it ends. And the six could not be routed - through it as it stands: `flan_break_hook`'s contract is that the loop may answer by - aiming a transfer channel, and these six are called by emitted code that falls off the - end with no channel in the call at all, so a restart chosen against one would be accepted - and silently dropped. So there is a second hook, `flan_trap_hook`, and a break that - refuses the resume with a reason rather than a process that exits before anyone can ask - a question. All six park, for two reasons rather than one. Four are guards that fire - *before* the operation they guard (`if (!a)` and the capability test both precede - `a->proc(...)`), so nothing is half done. The other two — `flan_transfer_fail` and - `flan_restart_unarmed` — fire mid-transfer, with this frame's defers possibly half run, - and park only to be *looked at*: stopping on a torn unwind is strictly more than exiting - before anyone can ask what tore it. Standalone is unchanged: the - hook is null in a program that did not import the agent, and the trap still exits 134 - with the same sentence. -- **Unchecked `malloc` in `flan_argv`** (`flan_rt.c:162`) — the one in the file. -- ~~**`v->gen` stale-slice word is maintained and never consulted**~~ — deleted - 2026-09-18 with the second round of the repeal; the header is five words now. -- **`flan_slurp_into` conflates elements and bytes and skips `flan_vec_check`** - (`flan_rt.c:2746`) — safe only because check.ml pins slurp to `(Vec u8)`; a latent trap. -- **`flan run` swallows build flags as program arguments** (`bin/main.ml:634-651`) — - `flan run game.flan --debug` hands `--debug` to the game. Filter or refuse by name like - every other subcommand. Also: no `-O` control anywhere (`Build.default` pins `-O2`; - `--debug` is the only route to `-O0`). -- **`main` signature errors print `:0:0`** (`check.ml:6033`, `:6038`) — - `env.locs` already holds the decl location; use the existing `find_opt` idiom. -- **`flan_shim_cstr` accepts embedded NULs** that `flan_path_cstr` refuses - (`lib/shim.ml:310` vs `flan_rt.c:2639`) — pick one policy. -- **No `-Wall -Wextra` on the runtime's C compile** (`lib/build.ml:827`, `:1069`). -- **Package visibility** — everything in a package is public except `main` - (`lib/load.ml:25-31`); `rl/get-color-raw` is the recorded symptom. -- **Emacs client**: 30s hard deadline with no retry on long builds (`flan.el:155-170`); - `accept-process-output` loops can freeze Emacs up to 60s on a hung daemon (`:521`, - `:576-600`); no package headers, so not installable off MELPA or by path alone. -- **No CI** — README states it openly and records two silent-failure incidents. `@checks` - exists; a workflow that runs `dune build @checks` on push is the whole job. Note `@x86` - parity is only under `@checks`, so routine `dune test` does not protect parity. - -## Documentation corrections (cheap, decision-relevant) - -- **`docs/DISCUSS.md:762` is stale and argues the opposite of the truth**: it lists the - condition/restart family under "no plan" for the x86 backend; `x86.ml:1587-1615` lowers - all of it and the survey shows 104/104 parity. Anyone reading it for a production - decision concludes wrongly. -- README documents 4 of 11 subcommands — `import-c`/`generate-c`, the most valuable - undocumented feature, are missing (`README.md:105-119` vs `bin/main.ml`). -- `lib/prelude.ml:8-10` promises the nonexistent `core:` package. -- Root clutter: working artifacts (`MY-NOTES.org`, `plan.org`, committed binaries, - `sand.js`/`sand.wasm`, `old-ocaml/`) a newcomer must ignore. - ---- - -## Deliberately out of scope — do not pick these up from this report - -- **`drop` / recursive teardown** — parked on `worktree-agent-a18e9e62485eaedb5` with - `docs/handoffs/HANDOFF-drop.md`; the arena route replaced it (FIX.org item 4, merged). -- **JavaScript backend** — held (FIX.org item 6). -- **wasm32/browser** — explicitly deprioritised; the ILP32 `(size_t)` truncations and the - emscripten gaps are recorded here but not queued. -- **macOS/Windows portability** (`aligned_alloc`, `MSG_NOSIGNAL`, `__atomic_*` vs - `stdatomic.h`, `long ftell` 2 GiB cap) — real, recorded, not current-machine problems. -- **Known and accepted**: seqlock memcpy formal-UB (correctly fenced, retry-bounded); - one-`.so`-leak-per-reload (cells hold module text addresses by design); the arena route's - compile-time→runtime-trap trade (stated in FIX.org); string-literal write-through - (waiting on provenance, plan.org decision #3). - -## Suggested order - -1. **Quick wins, one sitting**: Sys_error + Not_found arms; `flan_argv` malloc check; - `main`-signature locations; `flan run` flag filtering; DISCUSS.md:762 correction; - README subcommand + env-var tables. -2. **Runtime correctness**: mul-overflow guards (1.2), registry sync (1.4), - scratch-buffer decision + fix (1.1), `map-remove!` (1.3), dev-runtime aborts → signals. -3. **Confirm and fix** `Addr(Pfield)` on Option (1.5); locate the stderr `Loc.Error` fatals. -4. **Install story** (2.1, 2.3), then **shipping** (2.2). -5. **Stdlib**: clock first, then math, then IO/env — each is independent. -6. **CI**: `dune build @checks` on push. diff --git a/docs/overview.md b/docs/overview.md deleted file mode 100644 index 9ab67baa..00000000 --- a/docs/overview.md +++ /dev/null @@ -1,54 +0,0 @@ -# Overview — superseded - -This was the first brainstorm. It is kept for history and is **no longer -accurate**. Read instead: - -- `../plan.org` — the design, the build sequence, and the open decisions -- `../spec-memory.md` — ownership, containers, places, generics, function values -- `../spec-conditions.md` — conditions and restarts, operational semantics -- `../sand.flan` — the first acceptance program -- `../syntax-sketch.flan` — the syntax, annotated with the decisions above - -## What changed since - -| This document said | Now | -|---|---| -| "C-like with Roc syntax" | S-expressions, Clojure's brackets; C's value model | -| "Start with interpreter, output C later" | Frontend → typed IR → LLVM IR as text → `clang`. An interpreter is acceptable for milestone 2 only | -| "lists, slice, fixed-length array, matrices" | Four container types, distinct ownership: `[n T]`, `[T]`, `(Vec T)`, `(Map K V)` — see spec-memory.md | -| `arr[4]`, `arr[1..]` index syntax | `(at a i)`, `(slice a lo hi)` — no infix, no bracket indexing | -| `const by default?` | Locals are assignable places; parameters are not; `const` qualifies slices and pointers | -| Rust-style iterator chains ending in `.collect()` | `->>` threading over slices; every collecting operation allocates from an explicit allocator, usually the frame arena | -| `Ptr a` | Kept, as `(Ptr a)`. Cross-references use `(Handle a)` instead | -| option/result in the stdlib | Kept, and layered with conditions — see plan.org "Error handling, layered" | - -## Original text - -``` -Concepts: - -- Primitives - - u8, i32, f32, bool, char - - lists, slice, fixed-length array builtin, matrices -- Control flow - - for, while, break -- Structs & tuples -- Let bindings -- `Ptr a` -- Pattern matching -- ADTs -- Mutability - - const by default? -- Functions -- Array syntax - - ranges `arr[1..]`, `arr[..3]` - - index `arr[4]` -- Stdlib - - string - - vec/dynarray - - hashtable - - option/result - -C-like with Roc syntax. -Start with interpreter. Output C later down the track -```