2719 lines
212 KiB
Markdown
2719 lines
212 KiB
Markdown
# Overnight, 2026-09-13 into 09-14 — one decision waiting for you
|
||
|
||
**`ArithError` landed, but without the restarts you asked for.** The design agreed at the keyboard was
|
||
a condition plus `use-value` everywhere and `saturate` where clamping is meaningful. What landed is the
|
||
condition and the guards; no restart is established. The reason given is mechanical rather than a matter
|
||
of taste, and it is worth your reading of it: `runtime/flan_rt.c:70-86` allocates a restart frame **in the
|
||
`restart-case` that offers it, on that frame's own stack**, so the runtime cannot push one on a program's
|
||
behalf. Offering `use-value` at a division would mean an alloca plus a push and pop emitted at every
|
||
division in every checked build — which is the identical cost `lib/prelude.ml:70-76` already refuses, in
|
||
prose and on the record, for indexing. `docs/handoffs/HANDOFF-arith.md` is the full write-up.
|
||
|
||
So the question back to you: **is signal-and-die-with-a-location enough for arithmetic, the way it is for
|
||
`BoundsError`, or do you want `saturate` on the cast arm** — which is the one case where clamping is
|
||
canonical rather than arbitrary, and where casts are rare enough that the cost does not matter? That
|
||
follow-up is named in the handoff and not built. Nothing else about the feature is in question: both
|
||
backends agree byte for byte on all ten cases, and float division is deliberately unguarded because
|
||
IEEE already answers it.
|
||
|
||
# Where this is — end of 2026-09-13, second handoff
|
||
|
||
**This section is the record of that evening and is no longer where the tree is.** It is kept because the
|
||
reasoning in it is still the reasoning, but read the section above this one first, and take the state below
|
||
as a snapshot: it pins `dev-loop` at `e725a5a`, and since then the arithmetic-condition lane, the
|
||
redefinition emitter, the aggregate case across the reload boundary, the two unreached guards, DWARF for
|
||
`--x86`, the cost measurements, the last two raylib ports and the `dune test` noise have all landed. The
|
||
x86 survey it quotes at 97 now reports **103 MATCH, 0 DIFFER, 0 refused by name, 38 skipped**, and of the
|
||
seven remaining items it lists from `docs/handoffs/HANDOFF-x86-rt.md`, items 1, 3, 4, 5, 6 and 7 are done —
|
||
each has a `docs/handoffs/HANDOFF-x86-*.md` of its own.
|
||
|
||
**Branch `dev-loop` at `e725a5a`, working tree clean, `dune test` green, every lane merged.**
|
||
Nothing is running and nothing is half-built.
|
||
|
||
## The x86 backend is correct, and is not yet the dev backend
|
||
|
||
`spike/x86/survey.sh` builds every program in `test/programs` and every probe in `spike/x86` both
|
||
ways and compares **stdout, stderr and exit status**: **97 MATCH, 0 DIFFER, 0 refused by name.** That
|
||
is up from 41/1/41 at the start of the day. Conditions, bounds checks, `Fnval`'s indirection cell and
|
||
`--x86 --dev` all landed. `docs/DISCUSS.md` items 16, 17 and 18 are the three reports.
|
||
|
||
**`docs/handoffs/HANDOFF-x86-rt.md` at the root has the remaining seven items in order.** Item 1 is the substantial
|
||
one and is the reason the backend exists:
|
||
|
||
> **A redefinition emitter in `lib/x86.ml`** — the counterpart to `Emit.redefinition`, producing a
|
||
> `.so`. Until it exists the backend builds whole programs and cannot serve a single `C-c C-c`, which
|
||
> is where `docs/DISCUSS.md` item 13's ~21ms of a 35ms redefinition actually lives.
|
||
|
||
**And the constraint the last lane found, which is the thing not to rediscover:** publishing a cell an
|
||
LLVM-built module can store into puts an edge on `lib/x86.ml`'s own licence that *"a dev build is
|
||
compiled entirely by this backend and a release build entirely by LLVM, and the two never meet in one
|
||
process."* The two conventions agree on scalars and **disagree on every aggregate**, so an
|
||
`Emit.redefinition` module dlopened into an `--x86` host is correct right up until the first redefined
|
||
function takes or returns a struct. The answer is a redefinition emitter in `x86.ml`, not an aggregate
|
||
classifier. It is written into both files' headers.
|
||
|
||
Items 2, 4 and 5 on that list are small. Item 3 (`f64` → `i64` out of range, `INT64_MIN / -1`) is a
|
||
**language decision and not backend work** — the two backends differ there today, deliberately.
|
||
Items 6 and 7 are debug information and a size/speed measurement, neither started.
|
||
|
||
## Two things that were true this morning and are not now
|
||
|
||
Worth stating because the notes elsewhere may still read the old way:
|
||
|
||
- **`Rt` with an aggregate return was never a gap.** Every `rt` builder in `check.ml` answers `Unit`,
|
||
an `Int`, a `Ptr`, an `Alloc` or a `Handle`, with aggregates leaving through an out-pointer, and
|
||
`crossable` refuses an aggregate return from a `declare` outright. The refusal in `call_native` is
|
||
unreachable and building sret behind it would have been *wrong* — that path is the C boundary, where
|
||
SysV returns a 16-byte slice in `rax:rdx` rather than by hidden pointer. `Vec`, `Map` and `Pool` all
|
||
match; item 16's "unexercised" line was stale.
|
||
- **The header check is not opt-in.** `vendor/raylib/raylib-5.5.h` is committed and
|
||
`vendor/raylib/headers` names it directly. There is no `FLAN_RAYLIB_H`.
|
||
|
||
## The dev loop, as it now stands
|
||
|
||
Everything below works today on the LLVM path, which is the default and unaffected by any of the
|
||
backend work.
|
||
|
||
`C-c C-c` a top-level form, `C-x C-e` an expression, `C-u` before either to mark a pause.
|
||
`C-c C-m` expands a macro one step and `C-u C-c C-m` to the fixpoint, into `*flan-macroexpansion*`
|
||
where `m` expands again in place. `C-c C-b` lists restarts at a break. `M-x flan-inspect-address`,
|
||
`M-x flan-allocations`, `M-x flan-leaks`. A watch keeps count, min, max, last and mean per label,
|
||
windowed to the editor's last tick.
|
||
|
||
Macros come from imported packages, qualified like a `defn`, and a session expands the buffer's own.
|
||
`vendor/raylib/modes.flan` has five `with-*` macros over raylib's begin/end pairs, used at 35 call
|
||
sites. A failed evaluation no longer kills the session — `lib/dev.ml`'s `serve` guards the whole
|
||
request and `Session.eval` commits below the checker.
|
||
|
||
## Queued, nothing blocked on a decision
|
||
|
||
1. `docs/handoffs/HANDOFF-x86-rt.md`'s seven items, item 1 first.
|
||
2. `text_codepoints_loading` and `textures_image_processing` — ordinary porting now that
|
||
`slice-from-ptr` exists.
|
||
3. Generic structs and `$n` array lengths — decided, unblocked, sections below. **Stopped once
|
||
already because it was started against the author's freeze; do not start it without asking.**
|
||
4. `any` and `drop` — **stashed by the author deliberately.** Both designed, neither has a customer.
|
||
Do not schedule either.
|
||
|
||
# Decided by the author, end of 2026-09-13: the backend, then a feature freeze
|
||
|
||
**Priority is finishing the x86 backend and tightening the dev workflow.** After the backend is done,
|
||
**feature freeze for the rest of the day.** Work in flight may finish; nothing new starts.
|
||
|
||
**Stashed, explicitly not wanted now:** `any` and `drop`. Both are designed, neither has a customer,
|
||
and the author's words are that they feel like extra language features. `any` is plan.org's opt-in
|
||
tagged union (two words, a pointer and a typeid, no GC). `drop` is `spec-memory.md`'s hook for owning
|
||
something that is not memory. Their old entries stand; do not schedule either.
|
||
|
||
## ~~Queued: `slice-from-ptr`'s run-time refusal names a range the caller never wrote~~ — **done**
|
||
|
||
**Built.** `flan_slice_promise_error` is its own function in `runtime/flan_rt.c`, shaped like the two
|
||
beside it — it signals `BoundsError`, walks the handlers, offers the break loop, and only then falls
|
||
through to a message and a status. The sentence names the promise and the number that broke it, and a
|
||
second line says what is *not* checked. See [`docs/BUILT.md`](docs/BUILT.md), "`slice-from-ptr` refuses in its
|
||
own words". The original entry follows.
|
||
|
||
Small, and known rough at the time it shipped. `(slice-from-ptr p n)` with a **computed** negative
|
||
length is caught at run time — the check is `icmp sge i64 n, 0`, signed, because `check_slice`'s own
|
||
comparisons are unsigned and a negative `i32` sign-extended to `i64` becomes a huge `u64` that passes
|
||
both clauses. The catch is right. The message is not:
|
||
|
||
```
|
||
slice [0 -2) is out of bounds for length 0
|
||
```
|
||
|
||
It reuses `@flan_slice_error`, so it reports a range and a length the caller never wrote. The
|
||
condition actually violated is `0 <= n`, which is the same *shape* as a reversed range, and that is
|
||
why the existing function accepted it — but `slice-from-ptr` is the one form in the language where the
|
||
compiler cannot check the thing that matters, so its refusal is the place the promise should be
|
||
spelled out and it is the one place it is not.
|
||
|
||
The fix is a function in `runtime/` — that directory was held by another lane when this landed, which
|
||
is the whole reason it was left. Say what the caller promised and what was passed.
|
||
|
||
## The step after the backend's list, and it is the one the backend exists for
|
||
|
||
**Named here because it is not on the list below and would otherwise be discovered late.**
|
||
|
||
Finishing items 1-6 below gives a second **whole-program** compiler. `--x86` is refused with `--dev`
|
||
today, deliberately, and `docs/DISCUSS.md` item 16 says it has no cells. So at the end of that list the
|
||
backend can build `sand.flan` and cannot serve a single `C-c C-c`.
|
||
|
||
What the backend was for is `docs/DISCUSS.md` item 13's measurement: `llc` and `ld` are **about 21ms of a
|
||
35ms redefinition**, and removing them is the only thing that touches that number. Collecting it means
|
||
emitting a **redefinition module** — one function, compiled and installed into a running process
|
||
through an indirection cell — which is what `Build.shared` does on the LLVM path.
|
||
|
||
Concretely, and none of it is scoped yet:
|
||
|
||
- **`Fnval`'s indirection cell.** Item 4 below. `FnAddr (Fnval n)` emits the symbol, which is right
|
||
for a whole-program build and wrong the instant anything is redefined into it. This is the
|
||
prerequisite and it is already on the list; what follows is not.
|
||
- **A module rather than a program.** `Build.shared` compiles one function to a `.so` that
|
||
`flan_dev.c` loads. The x86 path writes an assembly file for a whole program; emitting a relocatable
|
||
object for one function, with the right symbols exported and the cells left external, is different
|
||
work.
|
||
- **The dev registry.** A run-time-new name needs `flan_dev_cell`. `Emit.redefinition` already
|
||
declares 256 siblings for an imported package and that cost was measured at +3.6ms per eval; the
|
||
x86 path has no equivalent.
|
||
- **DWARF.** `flan dev --debug` exists and `--x86` refuses `--debug`. A dev backend that cannot be
|
||
debugged is a narrower tool than the one it replaces.
|
||
- **Then measure, because the whole case rests on it.** If a hand-written backend does not actually
|
||
beat 35ms end to end, the argument for it was wrong and that is worth knowing. `docs/DISCUSS.md` item 13
|
||
says the backend is decided at step 4 on a measurement taken at step 3, and this is that
|
||
measurement finally being takeable.
|
||
|
||
## The backend, in the order that finishes it
|
||
|
||
`docs/DISCUSS.md` item 16 is the report and its verdict is the sequencing: *"the wiring is done and it was
|
||
the easy half. What is left is conditions, and the measurement moved them from 'first obstacle' to
|
||
'the only obstacle'."* 40 of 111 programs lower today; the other 40 are refused by name.
|
||
|
||
1. **Conditions, entire.** The transfer-channel guard after every call, the landing pad, the transfer
|
||
exit, `fdefers` on it, `emit_restart_case` and `emit_with_alloc`. Several hundred lines of
|
||
`emit.ml` **reimplemented from `spec-conditions.md` rather than ported**, because `emit.ml` writes
|
||
LLVM control flow and this writes bytes. The semantics are settled and the LLVM path is the oracle:
|
||
compare program output, never disassembly.
|
||
|
||
The mechanism is three pieces. The transfer channel is a pointer passed as the last argument; after
|
||
every call, check whether it is set; a set channel jumps to a per-function landing pad that runs
|
||
the defers and either handles or re-propagates.
|
||
|
||
`check_no_transfer` is what makes today's omission sound rather than hopeful — it walks the linked
|
||
program and refuses by name. It is also the measure of progress: the 40 refusals are 27
|
||
`restart-case`, 7 `signal`, 4 `handler-bind`, 1 `with-allocator`, 1 `fdefers`.
|
||
|
||
2. **Bounds checks**, which are the same work — `check_at` and `check_slice` cannot exist without the
|
||
guard. **Until they do, `--x86` is silently a `--no-bounds-checks` build**, and `bounds.flan` is
|
||
the one program that DIFFERs (exit 139 against LLVM's 134). Item 16 says this out loud and calls
|
||
deciding what a bounds violation means in a build with no handler one of two cheap things worth
|
||
doing first. Do that before item 1, not after.
|
||
|
||
3. **`Rt` with an aggregate return**, and with it most of the container runtime. `Vec`, `Map` and
|
||
`Pool` have not been exercised through this backend at all.
|
||
|
||
4. **`Fnval`'s indirection cell.** `FnAddr (Fnval n)` emits the symbol, which is right for a
|
||
whole-program build and wrong the instant anything is redefined into it. There are no cells and no
|
||
`--dev` here, deliberately — **but this is the one that decides whether the backend ever serves the
|
||
dev loop**, which is the reason it exists. Item 15's question 5, still waiting.
|
||
|
||
5. **Size and speed, measured.** Nothing has a number on it. Every value is in memory, every
|
||
intermediate is a frame temporary, and a block copy is `rep movsb`. That is the trade the brief
|
||
asked for and nobody has checked what it cost.
|
||
|
||
6. **Debug information.** None; `--x86` and `--debug` are refused together.
|
||
|
||
Two known divergences that are deliberate and should stay written down: `(uninit)` is stable garbage
|
||
rather than `poison`, and `unreachable` is `ud2` rather than UB. Item 16 says to take item 15's
|
||
question 4 seriously now that there are two backends that can disagree.
|
||
|
||
## Tightening the dev workflow — what is already known to want doing
|
||
|
||
- ~~**Finish the macro branch**~~ — done. Macros are importable from a package, qualified like a
|
||
`defn`, and the suite is green. Its customer, `with-drawing`/`with-mode-2d`, is now unblocked and
|
||
being written.
|
||
- ~~**`slice-from-ptr`**~~ — **built**. `examples/text-rectangle-bounds.flan` is the port it unblocked,
|
||
and the first example to call the hand-written `Font` surface. The other two it named —
|
||
`text_codepoints_loading` and `textures_image_processing` — are now ordinary porting work.
|
||
- ~~**`merged_serve`'s 10s warning path**~~ — tested. `test/programs/dev-noagent.flan` plus the last
|
||
block of `test_dev.ml`: the session still answers `describe` after the wait runs out, which is the
|
||
policy that distinguishes it from `two_process`. `HANDOFF-f1.md` is gone with it.
|
||
- ~~**The memcheck half of the registry**~~ — done. `flan_arena_proc`'s `free-all` issues memcheck's
|
||
`MAKE_MEM_UNDEFINED` over the arena, vendored rather than included; it is a control in
|
||
`test_valgrind.ml` now, and `docs/BUILT.md` carries the cost. The registry answer and the memcheck
|
||
answer stay two claims.
|
||
|
||
## Not in the freeze, because it is already decided and unblocked
|
||
|
||
**Generic structs and `$n` array lengths** — queued below with the Odin citations. It is a language
|
||
feature and the freeze says do not start it. Left here so it is not lost, not so it is picked up.
|
||
|
||
# Where this is — end of 2026-09-13
|
||
|
||
**Branch `dev-loop` at `861f591`, working tree clean, `dune test` green.** Read this section first;
|
||
everything below it is the standing queue and the decision record.
|
||
|
||
## Landed today, merged
|
||
|
||
- **Generics.** Monomorphisation, checked abstractly, `$t` at the binding site and bare `t` at a use.
|
||
`{:where (ordered? $t)}` as a Clojure-style map at the head of a body — five predicates: `ordered?`,
|
||
`equal?`, `hashable?`, `numeric?`, `copyable?`. **A type variable is move-only by default**; `copyable?`
|
||
is the opt-out. Prelude went 80 → 69 defns. A generic may also key a map now, which was a hole closed
|
||
the same day. `Tast` and every backend are untouched — instantiation lives entirely in `check.ml`.
|
||
- **The x86 backend runs.** `flan build --x86` writes `.s` and hands it to the same clang. **40 of 111
|
||
test programs build through it and print exactly what the LLVM build prints.** Off by default.
|
||
- **`pause` marking, both halves.** `C-u` before an eval marks the innermost form, `C-u C-u` the
|
||
top-level one.
|
||
- **`spy-num`.** A watch slot keeps count, min, max, last, mean, windowed to the editor's last tick.
|
||
- **Frame rollback**, `docs/PORTING.md` Tier 1 item 6. `restore` goes in the `continue` clause, not a defer.
|
||
- **The dev allocation registry is finished.** `M-x flan-inspect-address`, `M-x flan-allocations`,
|
||
`M-x flan-leaks`.
|
||
- **15 raylib examples**, an idiomatic layer over the bindings, and `vendor/raylib/vector.flan`.
|
||
- **The raylib header is tracked** at `vendor/raylib/raylib-5.5.h`. `FLAN_RAYLIB_H` is gone and the
|
||
check runs on every build.
|
||
- **`{K V}` is dropped.** `(Map K V)` is the only map type spelling.
|
||
- **Two test-infrastructure fixes worth not re-deriving:** the socket flake was an ordering bug, not a
|
||
race (the test checked for a socket before completing any round-trip); and `the daemon never
|
||
listened` was never a race either — it was an llc-and-link taking 6.6s against a 5s await, because
|
||
`Build.cachedir` sat under dune's per-run `TMPDIR` and every build was cold. Both fixed.
|
||
|
||
## Unmerged, and deliberately so
|
||
|
||
~~**`worktree-agent-a859480edc827ab73` — macros importable from a package.**~~ **Finished.** The suite
|
||
is green, the acceptance wiring asserts both halves of the rule, and the dev loop is covered. See
|
||
docs/BUILT.md, "A package may declare a macro, and its name is the package's".
|
||
|
||
The finding is worth keeping: the old refusal claimed collecting a package's macros needed a second
|
||
import resolver at the Form level. It did not. The file being compiled is parsed before `Load` runs
|
||
too, so no shape of the feature could have left import resolution where it was — `Load.program` takes
|
||
forms now and uses the one resolver that always existed.
|
||
|
||
~~One gap left, and it is older than this work: **`C-x C-e` expands no macros at all.**~~ **Done** —
|
||
see [`docs/BUILT.md`](docs/BUILT.md), "`C-x C-e` expands too, and a declaration is not an expression". It was
|
||
the wrap `Parse.decl` already had, and the decision it was waiting on came out as: an expression that
|
||
expands to a declaration is **refused by name**, in `Parse.expr`'s head dispatch, so a `defn` nested
|
||
anywhere in the expansion and a `defvar` typed by hand get the same sentence. The spin refusal fires
|
||
on this path; the ring never can, because a ring is refused while its own package is parsed. What
|
||
expands is the prelude's macros and the imported packages' — and, since the lane recorded just below,
|
||
the file's own as well.
|
||
|
||
Two things left behind it, neither this lane's. ~~**A session expands no macro the buffer itself
|
||
declares**~~ — **done**, see [`docs/BUILT.md`](docs/BUILT.md), "A session expands the buffer's own macros". The
|
||
session seeds its file's own `defmacro`s in `Session.create`, from the same read that produced
|
||
`decls`, and an evaluated `defmacro` joins the set so the next evaluation can call it — the shape
|
||
`defn` already has. Re-reading the file was the alternative and was refused: it would put
|
||
unsaved-versus-saved skew inside macro expansion. The commit point stays below the checker. And **`Dev.eval_expr` catches only `Loc.Error`** around `Session.eval_expr`: the two non-termination
|
||
refusals are that, so they answer as replies, but `C-x C-e` can now reach `Build.macro_module` for
|
||
the first time, and a macro that fails to *compile* raises `Failure` from the clang driver or
|
||
`Loc.Errors` from the checker. Neither is caught, and an unhandled exception there is a dead session.
|
||
Pre-existing for `C-c C-c`, which reaches the same builder; `lib/dev.ml` is another lane's file.
|
||
|
||
Two things found while finishing it. `Session` held the imported macro set but **replaced** it on
|
||
every evaluation, and the one form `C-c C-c` sends carries no import — so a package macro worked on
|
||
the build and was an unknown name on the first reload. It unions now, and `test_session` drives two
|
||
evaluations because one proves nothing. And cold build cost roughly doubles for a program importing a
|
||
package that declares macros, because a macro module is built per round and the package's rounds are
|
||
its own; warm is unchanged at ~70ms.
|
||
|
||
Its customer is `with-drawing`/`with-mode-2d` over raylib's begin/end pairs, which a lane tried to add
|
||
and could not. Unblocked, not written.
|
||
|
||
**`worktree-agent-a4dca263c97eb0b66`** is an older WIP, "the inspector's address root, half wired on
|
||
the Emacs side". **Superseded** — the registry lane built that properly. Delete it.
|
||
|
||
## What I would do next
|
||
|
||
1. **`with-drawing` and `with-mode-2d`**, now that a package may declare a macro. They belong in
|
||
`vendor/raylib`, and they remove a class of unbalanced-pair bug from every raylib program.
|
||
2. **The pointer-length question** — its own queued section below. It blocks three raylib examples and
|
||
leaves the hand-written `Font` surface with no example caller.
|
||
3. **Generic structs and `$n` array lengths** — queued below, decided, and now unblocked since the
|
||
generics lane has merged.
|
||
|
||
## For siam-farmer
|
||
|
||
`docs/PORTING.md`'s verdict is unconditional now: the game's state is **fixed arrays with counts**, so it
|
||
fits in `defvar` globals, nothing is move-only, and generics is off its critical path. Tier 0 and Tier 1
|
||
items 4, 5 and 6 are all done. Nothing blocks writing it.
|
||
|
||
## Landed: a pointer from C needs a length before it can be indexed
|
||
|
||
> **Built, 2026-09-13.** `(slice-from-ptr p n)` is in `lib/check.ml`, `Tast.SliceFromPtr` in `lib/tast.ml`, one
|
||
> `insertvalue` pair in `lib/emit.ml`. The section is kept as it was written, because the reasoning is the part
|
||
> worth having; what was built and what it refuses is in this box.
|
||
|
||
**Option 1, and only option 1.** The reasoning below still holds: option 2 cannot reach `font.recs`, where the count
|
||
is a sibling *field*, so option 1 was the floor and turned out to be the whole thing.
|
||
|
||
**No marker on the name.** A `!` in this language means *mutates* (`map-next!`) and a `?` means *asks*
|
||
(`font-valid?`); this does neither, and `zeroed` — the nearest neighbour, a value conjured rather than derived —
|
||
carries no marker either. `ptr` is the marker: a `(Ptr T)` only ever arrives from a `declare-c`.
|
||
|
||
**Nothing new in the representation.** A `Types.Slice` is `{ptr, i64}` in `emit.ml` and in `x86.ml` already, which
|
||
is exactly ptr+len, so the form is two `insertvalue`s and no layout change. `x86.ml`'s `_ -> unsupported` arm takes
|
||
the new constructor without an edit.
|
||
|
||
**What it refuses.** A first argument that is not a `(Ptr T)`; a negative literal length, at check time; a negative
|
||
computed length, at run time, through `signal_block` and `@flan_slice_error` — reusing that message rather than
|
||
growing the runtime a function, because the violated condition is `0 <= n`, which is a reversed range spelled the
|
||
other way. The run-time test is *signed* on purpose: `check_slice`'s comparisons are unsigned and a negative i32
|
||
sign-extended to i64 walks straight through them. It is behind `f.md.checks` like the other two, so it is on at -O0
|
||
and -O2 alike and off only when checks as a whole were asked off.
|
||
|
||
**It owns nothing, and needed no analysis to say so.** The result is a `Types.Slice`, which is not move-only and
|
||
carries no allocator, so `free` refuses it by the rule it already had — the same rule that refuses `(as-slice v)`.
|
||
|
||
**Where the promise is written.** `rl/font-recs` and `rl/font-glyphs` in `vendor/raylib/raylib.flan`, because that
|
||
is the one place raylib's invariant (both arrays hold `glyph-count` entries) is knowable. `examples/text-rectangle-bounds.flan`
|
||
is the port that motivated all of it and it runs; `test/programs/slice-from-ptr.flan` covers the form with no
|
||
raylib, no window and no C.
|
||
|
||
### The reasoning, as it was written
|
||
|
||
**Sequenced after the generics lane**, which holds `lib/check.ml`.
|
||
|
||
`indexed` in `lib/check.ml` accepts `Array` and `Slice` only, so a `(Ptr T)` that came back from C is readable at
|
||
element 0 through `deref` and nowhere else. Confirmed empirically rather than inferred:
|
||
`(at (.recs (rl/get-font-default)) 0)` answers `(Ptr rl/Rectangle) cannot be indexed`.
|
||
|
||
**It blocks real work now**, which is why it moved from a note to a queue entry. Three upstream examples are
|
||
unportable — `text_rectangle_bounds` (walks `font.recs[i]` and `font.glyphs[i]`), `text_codepoints_loading`
|
||
(`LoadCodepoints` returns `int*`) and `textures_image_processing` (`LoadImageColors` returns `Color*`) — and the
|
||
first of those is the only *procedural* font example, so the whole hand-written `Font` surface has no example
|
||
caller. Every other text example needs a TTF on disk.
|
||
|
||
The problem is that C hands back a pointer and no length. Two shapes:
|
||
|
||
1. **`(slice-from-ptr p n)`** — the caller states the length and owns being right about it. Honest, one form, and
|
||
the unsafety is written at the site where somebody had to know the answer anyway. It is the same trust
|
||
`declare-c` already extends.
|
||
2. **A per-binding declaration naming which argument carries the count.** raylib's convention is an out-parameter
|
||
(`LoadCodepoints(text, &count)`), so the information usually *is* there. This makes the binding carry it once
|
||
instead of every call site repeating it, and it fits the existing `bindings` file. It does not cover
|
||
`font.recs`, where the count is a sibling *field* (`font.glyphCount`) rather than an argument.
|
||
|
||
They are not exclusive, and 2 does not cover every case, so 1 is probably the floor whatever else is built.
|
||
|
||
## Queued, decided by the author: generic structs and generic array lengths
|
||
|
||
**Sequenced after the core generics lane, not beside it.** Both change `Types.t`, which that lane is inside.
|
||
|
||
The author's question settled it: Odin has both, and the normal way to write them is together.
|
||
`core/math/linalg/extended.odin:488` binds three things in one pattern —
|
||
|
||
```odin
|
||
is_nan_array :: proc(x: $A/[$N]$T) -> (out: [N]bool) where IS_FLOAT(T)
|
||
```
|
||
|
||
`A` is the array type, `N` its length, `T` its element, and `[N]bool` in the *return* type uses `N`, so the result
|
||
is an array of the same length as the argument. And `core/container/small_array/small_array.odin:23` is the generic
|
||
struct over both, with a predicate over a **value** rather than a type:
|
||
|
||
```odin
|
||
Small_Array :: struct($N: int, $T: typeid) where N >= 0 { ... }
|
||
```
|
||
|
||
### Why they are one project
|
||
|
||
`Types.Named` is a bare string with no parameters and `Types.Array` is `int64 * t`. Either feature means `Types.t`
|
||
gains a parameterised case, and `Types.t` is consumed by `emit.ml`'s layout calculator, `x86.ml`, `render.ml`, the
|
||
DWARF path and the map key-pair emitter. **Same price for one as for both**, which is the whole reason to do them
|
||
together.
|
||
|
||
What the spike measured for the length half specifically: `Ast.len` is `Lint | Lname`, so `$n` parses today with no
|
||
reader or parser change — the same free ride the type sigil got. Everything after that is the expensive part.
|
||
`bind_ty` gains a length-binding case (`Array (n, p)` against `Array (m, a)` binds `n := m`), `mangle_ty` gains a
|
||
number, and `array_len` has to answer "a variable" rather than failing — which means the checker's compile-time
|
||
constant folding has to accept that a length can stay symbolic until instantiation.
|
||
|
||
### The motivating case, and it is not `$n` on its own
|
||
|
||
`$n` alone buys little. `swap!` does not need it — it takes a slice and two indices, and the length is a runtime
|
||
field. Nor does a `pop` from a `Vec`. The case that wants both is **a fixed-capacity array with a count and no
|
||
allocation**, which is Odin's `Small_Array` and a good fit for a game that refuses to allocate in a frame:
|
||
|
||
```lisp
|
||
(defvar enemies (Small-Array 64 Enemy))
|
||
```
|
||
|
||
`docs/PORTING.md`'s recommendation for the game's state was fixed arrays with counts held in `defvar` globals, kept
|
||
deliberately away from `Vec` so nothing is move-only. `(Small-Array $n $t)` is that pattern with a type behind it
|
||
instead of two variables kept in step by hand. That is the argument for building this, and it is stronger than
|
||
"Odin has it".
|
||
|
||
### Also worth deciding when it is built
|
||
|
||
Odin's `where N >= 0` is a predicate over a **value**, not a type. The `where` mechanism just decided here takes
|
||
type predicates (`ordered?`, `copyable?`). Whether it also takes value predicates over a length parameter is a
|
||
separate question and should be answered deliberately rather than falling out of the implementation.
|
||
|
||
## ~~To discuss: two decisions parked during the ownership discussion~~ — **both answered and built**
|
||
|
||
1. **The header is tracked now.** `vendor/raylib/raylib-5.5.h` is committed beside the package and
|
||
`vendor/raylib/headers` names it directly. **`FLAN_RAYLIB_H` is gone** — there is no variable to export and no
|
||
way for the check to silently not run, which is what `vendor/raylib/web/` being gitignored had been doing to
|
||
every worktree. The old argument for opt-in was that requiring a header makes a build need raylib-devel; that
|
||
dissolved once the header shipped with the repository. Measured before committing to it: a warm build of
|
||
`raylib-font.flan` is 0.10-0.11s against a 0.10s baseline, because the header read is cached and keyed like the
|
||
object cache. Verified it fires by putting `f64` where raylib says `int`.
|
||
|
||
2. **A `bindings` gap stops a build, and says which file to edit.** The findings stayed split by kind but both are
|
||
now fatal. The objection had really been to the *wording* — a coverage gap arrived wrapped in "the package
|
||
disagrees with the header", which describes a layout bug and sends the reader to the wrong file. Gating it to
|
||
`generate-c` instead meant an unchecked enum member went unreported on every ordinary build, and "remember to
|
||
run generate-c" is the same shape as "remember to check the header by eye", which is what this replaced.
|
||
|
||
## Decided 2026-09-13: generics by monomorphisation, checked abstractly, with `where` predicates
|
||
|
||
The spike answered it (`docs/SPIKE-GENERICS.md`, on `worktree-agent-afcd2406f3660629b`): **it runs**, the whole feature
|
||
is `lib/check.ml` and nothing else in `lib/`, and instantiation is under the noise floor at 1.8ms whole-program
|
||
re-check. The bill is `llc`+`ld` at **+1.7ms per extra body**, so a generic used at three types adds ~7ms to a
|
||
redefinition. `plan.org` is updated; what follows is the decision and the reasoning that does not belong there.
|
||
|
||
### The fork that turned out to be a false one
|
||
|
||
`plan.org` posed a choice nobody needed to make. It said an unconstrained `=`, `<`, `+` or `hash` is *rejected*,
|
||
and that this was "Odin's model". **Odin does not do that** — it checks a polymorphic body only per instantiation,
|
||
so `a + b` over a `$T` compiles and fails only when someone instantiates at a type without `+`. The spike had to
|
||
write a second pass to get plan.org's rule.
|
||
|
||
Both options were bad in the way the other was good. Rejecting abstractly gives the error at the definition and
|
||
makes every call site carry a comparison: `(sort! xs)` becomes `(sort-by! xs (fn [a b] (< a b)))` everywhere.
|
||
Checking per instantiation keeps the call short and moves the error into code the caller did not write, which is
|
||
worse here than in most languages because a hot-reload session may have been running for an hour before the call
|
||
site is reached.
|
||
|
||
**What dissolves it is the thing plan.org ruled out while citing Odin: Odin has constraints.**
|
||
`core/slice/slice.odin:289` is `where intrinsics.type_is_ordered(T)`, and there are 41 such predicates. A `where`
|
||
clause tells the abstract pass what it may assume, so the body checks at the definition *and* the call stays
|
||
`(sort! xs)`.
|
||
|
||
### What is being built
|
||
|
||
- A type variable binds as `$t` in a signature and is used bare. Already decided; see the sigil entry.
|
||
- A generic body is checked **abstractly**, once, with nothing substituted.
|
||
- The clause is written as a **Clojure-style map at the head of the body**, `{:where (ordered? $t)}`, chosen by
|
||
the author over a bare keyword. It disambiguates because a bare `{}` in expression position is already refused
|
||
(`parse.ml:110`), so a `{}` there can be nothing else, and Clojure's `{:pre [...] :post [...]}` is the
|
||
precedent. It leaves room for further keys without new syntax. ~~**One catch to settle first:** `{K V}` is
|
||
currently a legal return type, so `(defn f [...] {string i32} {:where ...} body)` puts two braces in a row
|
||
meaning different things.~~ — resolved the way this predicted: `{K V}` went in favour of `(Map K V)`,
|
||
braces in type position are refused by name, and the two-braces-in-a-row case cannot arise.
|
||
- A `where` clause over **compile-time type predicates** admits the operators the body needs. ~~Four are
|
||
wanted — `ordered?`, `equal?`, `hashable?`, `numeric?`~~ — against Odin's forty-one. Five landed: `copyable?`
|
||
is the fifth, and it has no Odin counterpart because a `$T` there never has to answer whether it moves. The
|
||
prelude's nine non-collapsing functions need only the first two.
|
||
- Each instantiation checks the concrete type satisfies the predicates and refuses **that call site** if not.
|
||
|
||
**This is not a type class, and the distinction is the one to keep straight.** A type class carries
|
||
implementations, selected per instance, extensible by anyone, and needs dictionaries and coherence rules. A
|
||
predicate carries nothing; it gates a builtin the compiler already has. The ceiling is that nobody can supply a
|
||
user-defined `<` — and every operation the prelude and the containers need is a primitive, so the ceiling does not
|
||
bind. The day someone asks to sort by a comparison they wrote is the day to look at type classes again, and it
|
||
will be obvious.
|
||
|
||
### What collapsing the prelude actually buys — 27 → 15, not 27 → 6
|
||
|
||
Measured by the spike against the real sources, and worth knowing before the work starts:
|
||
|
||
- **13 collapse into 6 with nothing but a signature change** — `swap-*!`, `reverse-*!`, `map-*!`, `reduce-*`,
|
||
`filter-*`, `sort-*-by!`. They move elements or already take the operation as a function value. `filter` is the
|
||
strongest case: it allocates, `(vec-new t)` and `push` and returns `(Vec t)`, and the type-erased container
|
||
runtime needed **no changes at all**, because `SizeOf`/`AlignOf` are computed where the type is concrete. That
|
||
is the direct confirmation that `spec-memory.md`'s type-erased containers and monomorphised functions compose.
|
||
- **9 collapse into 4 and need a `where`** — `sort-*!`, `index-of-*`, `min-*`, `max-*`. Under the old no-constraint
|
||
rule these were the ones whose call sites all got longer. With predicates they do not.
|
||
- **5 do not collapse and should not** — `sum-i32`/`sum-f32` widen to `i64`/`f64` with an explicit cast, and "the
|
||
wider type `t` accumulates into" is a type-level function, which is a constraint system or an associated type.
|
||
A generic `sum` would have to take its accumulator and its `add`, at which point it *is* `reduce`.
|
||
`append-i64!`/`append-f64!` are two different primitives, `I64ToBytes` and `F64ToBytes`, and choosing between
|
||
them per instantiation is compile-time overloading, which is what multimethods are for.
|
||
|
||
The prelude keeps a per-type layer for the numeric ones. That is the honest number.
|
||
|
||
### Still open, and not blocking
|
||
|
||
0. ~~**Ownership.**~~ **Decided by the author.** A type variable is **move-only by default**, and a
|
||
`copyable?` predicate in the `where` clause is what tells the checker otherwise. Conservative in the safe
|
||
direction: move rules are the stricter ones, so assuming them can only refuse programs, never admit a bad one.
|
||
It costs nothing at run time — the predicate changes what the checker permits and emits no code.
|
||
|
||
**Odin is no help here and this is worth knowing before someone goes looking.** Odin has no move semantics and
|
||
no ownership tracking at all: `Raw_Dynamic_Array` (`base/runtime/core.odin:523`) is a plain struct of pointer,
|
||
length, capacity and allocator, freely copyable, and an alias is the programmer's problem to notice. Nothing in
|
||
its compiler tracks it, so `$T` never has to answer this. **The prior art is Rust**, where a parameter defaults
|
||
to move and `T: Copy` is the opt-in. One difference to keep: Rust's `Copy` is a trait a user may implement,
|
||
while `copyable?` is a predicate the compiler answers, because it already knows which types own heap storage.
|
||
Same ergonomics, none of the trait machinery, consistent with the `where` decision above.
|
||
|
||
What this means in the body: a generic may not use a parameter twice unless it declares `copyable?`.
|
||
`(defn twice [x $t] $t (+ x x))` is refused without it — correct at `i32`, wrong at `(Vec i32)`, and the
|
||
checker cannot tell which until it substitutes.
|
||
|
||
1. **The runaway refusal.** `(defn grow [x $t] () (grow [x x]))` asks for a copy at `[2 t]`, then `[2 [2 t]]`,
|
||
forever. Before the spike's cap it did not fail, it **hung** — and `Session.eval` runs the same code, so what
|
||
hangs is `C-c C-c` with the daemon wedged behind it and nothing to show. The cap is a depth counter refusing
|
||
past 32, and **the number is arbitrary**. A refusal that names the chain of instantiations rather than the
|
||
depth it gave up at is the designed version. **Odin has no cap to copy.**
|
||
2. **Ownership is not decidable abstractly, and plan.org does not mention it.** `Types.is_move_only (Var _)` is
|
||
false, but the same variable at `(Vec i32)` is move-only, so `spec-memory.md`'s dead-set analysis is sound only
|
||
per instantiation. Either ownership is checked per copy — a generic may then be accepted and its instantiation
|
||
refused — or type variables carry a move-only constraint, which is a constraint system of a different kind from
|
||
`where`, since it constrains what the *body* may do rather than what the type supports. Every other analysis in
|
||
the checker survives abstraction; this one does not.
|
||
3. **`C-c C-c` on a generic installs nothing, silently.** `Session.eval` reports `installs=false, fns=[]` because
|
||
a generic name never reaches `Tast.fns` by design — only its instantiations do. The fix is expanding to
|
||
instantiations transitively in `session.ml`; the cells already exist. This must land before generics is usable
|
||
in the dev loop, which is the whole point of the project.
|
||
4. **Deferred together: generic structs and `$n` length parameters.** `Types.Named` is a bare string and
|
||
`Types.Array` is `int64 * t`, so either one means `Types.t` gains a parameterised case — and `Types.t` is
|
||
consumed by `emit.ml`'s layout calculator, `x86.ml`, `render.ml`, the DWARF path and the map key-pair emitter.
|
||
Same price for both. If both are wanted they are one project; if only one is, drop `$n`.
|
||
5. **Generics across a real compilation-unit boundary.** `Load` flattens imports before checking so it works
|
||
today, but a package boundary that ever becomes a real unit boundary needs the generic's *body* to cross it —
|
||
which separate compilation cannot do, and is why C++ puts templates in headers.
|
||
6. ~~**`hashable?` gates the type and not the operations.**~~ **Closed the same day it landed.** It was real for
|
||
an afternoon: a generic could take and return a `(Map $t V)` and could not `get` or `put` into one, because the
|
||
hash and the equality pair are emitted as concrete symbols chosen from the key type and there is no symbol to
|
||
name while `$t` is a variable. The fix is that the five map operations that reach the pair — `put`, `get`,
|
||
`has-key?`, `reserve`, `clone` — are now **deferred to the instantiation**, joining `print` and `println` on
|
||
the one list of forms the abstract pass does not answer where they are written.
|
||
|
||
**What made that acceptable is the clause, and it is worth stating as a rule rather than as a special case.**
|
||
Every member of that list moves a refusal from the definition to a call site, which is the thing the abstract
|
||
pass exists to prevent. `print` and `println` pay nothing for it — every type prints, so the deferred check
|
||
always succeeds. The map operations *can* fail, and the reason they are still allowed on is that
|
||
`{:where (hashable? $t)}` is in the signature: an instantiation at a type with no usable equality is refused
|
||
against a requirement the author wrote down, naming the call site, the type it asked for and the predicate it
|
||
failed. That is categorically different from an unconstrained `(+ a b)` failing deep in a body with nothing to
|
||
blame. **A generic that does not declare the predicate gets no deferral** — `deferred_key` checks first, and
|
||
`map_type` has usually refused the signature before that. The membership rule for the list is therefore not a
|
||
headcount: either the operation cannot fail after substituting, or a declared predicate gives its failure
|
||
somewhere to land.
|
||
|
||
## Decided by the author, 2026-09-13: a type variable takes a `$` sigil
|
||
|
||
**This overrides plan.org**, which says under Types: *"Lowercase type names are variables, Capitalized are
|
||
concrete — no sigil."* That rule is withdrawn. Taken as provisional, to make progress rather than as a final
|
||
design, and expected to be revisited after some real use.
|
||
|
||
```lisp
|
||
(defn swap [xs [$t] i i32 j i32] () ...)
|
||
```
|
||
|
||
Three reasons, the third being the one that actually forced it.
|
||
|
||
1. **There is no binding site without a sigil.** Nothing distinguishes where a variable is introduced from where
|
||
it is used.
|
||
2. **Introducing one is invisible.** `check.ml:535-562` resolves builtins, aliases, structs, unions, enums, then
|
||
near-miss, and *then* treats any leftover lowercase name as a type variable. So a lowercase name nobody has
|
||
heard of silently becomes a type parameter, and a mistyped type makes the function **more** permissive instead
|
||
of failing. The `near_miss` guard at line 553 exists for exactly this hazard and only reaches names close to a
|
||
known one — the comment there says so. `[elem]` is not close to anything and would become generic silently.
|
||
3. **Brackets carry two opposite rules for the same lexical thing.** In `[n t]`, `check.ml:563` resolves `n`
|
||
against `env.consts` and *errors* if it is absent, while `t` becomes a *variable* if absent. Absence means
|
||
"mistake" on one side of the bracket and "new parameter" on the other.
|
||
|
||
The rule is also not the one plan.org states. It is not case-based: `i32`, `f32`, `bool` and `string` are all
|
||
lowercase and concrete. What the code actually does is **treat any unresolved lowercase name as a variable**,
|
||
which is an absence rule, and absence is a bad thing to give meaning to.
|
||
|
||
**Open, and deliberately not decided here:** whether a *use* after the binding is `$t` or bare `t`. Odin binds with
|
||
`$T` and uses bare `T`. Left to whoever builds it, to be reported with the reason.
|
||
|
||
**Also open: `$n` in length position**, Odin's `$N: int`, which is what would make a function generic over array
|
||
length rather than only element type. Not in the spike unless it falls out for free; the spike is to report what it
|
||
would cost.
|
||
|
||
## ~~To discuss: five gaps the raylib examples hit and could not close~~ — **three closed, two live elsewhere**
|
||
|
||
Found by the lane that ported `core-2d-camera`, `core-scissor-test`, `core-window-flags`,
|
||
`core-world-screen` and `core-window-should-close`. None blocked those five. All five blocked
|
||
something else, and each is a language or checker question rather than a missing binding, which is
|
||
why they are here and not in a binding list.
|
||
|
||
1. **A `(Ptr T)` returned from C cannot be indexed.** `indexed` in `lib/check.ml` accepts `Array`
|
||
and `Slice` only, so a C function answering `int*` is readable at element 0 through `deref` and
|
||
nowhere else. `LoadRandomSequence` is the case that hit it and `core_random_sequence` is
|
||
unportable until it moves. The question is what the answer should be: a length has to come from
|
||
somewhere before a pointer can become a slice, and C does not supply one. Possibly a
|
||
`(slice-from-ptr p n)` where the caller states the length and owns being right about it.
|
||
**This has its own queue entry at the top of this file now** — it blocks three more examples than this
|
||
section knew about, and both candidate shapes are written out there.
|
||
|
||
2. ~~**An enum-typed `defstruct` field is refused by the layout check.**~~ **Closed.** The layout
|
||
check now accepts an enum where the header says `int`, symmetrically, and still refuses
|
||
anything that is not four bytes. `Camera3D.projection` is a `CameraProjection` again and the
|
||
`rl/camera-projection` helper is gone; `.projection :perspective` resolves at the construction
|
||
site, so the keyword half of the problem went away with it. See docs/BUILT.md.
|
||
|
||
3. ~~**The header check does not reach `defconst` or `defenum`.**~~ **Closed.** It reaches both.
|
||
`bindings` gained `enum`, `const` and `constant` lines that say what a Flan constant is called in
|
||
C; every mapped name is compared by value, and a name the mapping cannot find, a rule that
|
||
reaches nothing, and a `defenum` with no line at all are each reported rather than skipped. All
|
||
eight raylib enums and all 16 `ConfigFlags` bits check out against 5.5. See docs/BUILT.md.
|
||
|
||
4. ~~**raymath is `static inline`, so there is no symbol to bind.**~~ **Closed for the vector half.**
|
||
`Clamp`, `Vector2Add`, `Remap` and the rest exist only in the header and `declare-c` has nothing
|
||
to name, so the arithmetic is written in Flan: `vendor/raylib/vector.flan`, a package file with
|
||
no `declare-c` in it at all — which is why it is a file of its own rather than more of
|
||
`raylib.flan`, the file the header check reads hand-written signatures out of. Vector2 and
|
||
Vector3 add/sub/mul/scale/negate/dot/length/distance/normalize/lerp, `v2-angle`, `v2-rotate`,
|
||
`v3-cross`, and `remap`, `inverse-lerp`, `wrap-f32` on scalars. raymath's semantics exactly,
|
||
including the zero-length guard in `normalize`. **`clamp` and `lerp` are deliberately absent**:
|
||
both are already in the prelude, and a second `lerp` would not even be the same function —
|
||
the prelude writes `(1-t)a + tb`, raymath writes `a + t*(b - a)`, and shipping both under names
|
||
one letter apart is a footgun. The C-shim alternative was rejected on the measurement
|
||
`examples/shapes-following-eyes.flan` already took: it would buy identical arithmetic for a
|
||
compilation unit in the build and a second place raylib's semantics are written down. rlgl's
|
||
matrix stack is still unbound for a different reason, so `core_2d_camera_mouse_zoom` is still
|
||
skipped.
|
||
|
||
5. **Four families are still refused by the importer for want of a `defstruct`.** `FilePathList`
|
||
(a `char**`, blocks `core_drop_files`), `Model`/`Mesh`/`Ray`/`BoundingBox` (the model and
|
||
3D-collision families), and **function-pointer parameters** (`SetTraceLogCallback`, which blocks
|
||
`core_custom_logging`, and the audio stream processors). The first three are ordinary widening —
|
||
write the `defstruct` and they import. The function-pointer one is not, and is the interesting
|
||
one: it is the callback direction of the FFI, which nothing has needed yet.
|
||
|
||
## Queued, 2026-09-13 (second session) — everything four lanes left behind
|
||
|
||
Four lanes ran in parallel worktrees and three landed; all are merged into `dev-loop` and `dune test` is green on
|
||
two consecutive full runs. Each lane wrote a handoff at the repository root, and those files are the detail — what
|
||
is below is the queue, not the reasoning. **Delete a handoff when its list is empty**, the same rule this file
|
||
follows.
|
||
|
||
**A note on the worktrees, because it nearly cost two lanes their work:** at least two were handed out at `2c232dd`,
|
||
roughly 485 commits stale, on a tree where `lib/dev.ml` does not exist. Both agents noticed and reset to the branch
|
||
tip themselves. A lane that did not notice would produce plausible work against the wrong tree. Check
|
||
`git log --oneline -1` before starting in a worktree.
|
||
|
||
### ~~From `HANDOFF-f2.md` — `pause` marking, the Emacs half~~
|
||
|
||
**Built.** All seven items landed: the `eval_expr` pause path, `flan-dev--eval` sending the position,
|
||
`flan-eval-defun` taking `C-u`, the overlay and its face, `test/programs/dev-pause.flan`, the `test_dev.ml` block and
|
||
the `emacs/test-flan-dev.el` one. `HANDOFF-f2.md` is deleted; the reasoning is in `docs/BUILT.md`, "A breakpoint is a
|
||
function call, and the editor only says where".
|
||
|
||
### ~~From `HANDOFF-f1.md`~~ — the socket flake is fixed, and everything that followed it is done
|
||
|
||
The fix is ordering, not timing: the test checked for `agent.sock` before completing any round-trip, and
|
||
`merged_serve` does not accept until its own 10s await on that path. Moving the existing `describe` request ahead of
|
||
the check took it from 2/8 failures to 0/10, and closed a second race that was burning the full 10s timeout.
|
||
`lib/dev.ml` was not touched.
|
||
|
||
**The `the daemon never listened` flake is closed at the root, and the root was a cold cache.** The wait is not for
|
||
a socket but for an llc-and-link of the whole program before `flan dev` binds, and that build was cold on every
|
||
invocation because `Build.cachedir` sat under `$TMPDIR` — which dune makes private per run. The cache now lives under
|
||
`$XDG_CACHE_HOME/flan/objcache` (`$FLAN_CACHE_DIR` overrides). Measured on an idle machine, and so not comparable
|
||
with the 6.5-6.8s this file used to quote, which was a cold build under dune's own parallelism — on `dev-repl.flan`,
|
||
launch to bound socket: **2.0s cold, 0.48s warm**; whole-program `flan build` **1.44s cold, 0.06s warm**; full `dune test` 25.7s/30.1s before
|
||
against 24.0s after, with user CPU down from ~50s to ~34s — the wall gain is small because the suite is bound by
|
||
sequential test steps, not by compiles.
|
||
|
||
The move was safe because the keys are total, which is the thing to check before moving a cache and not after:
|
||
`compile_c` digests the source *text*, the compiler binary's stamp and every flag; `wasm_resource_dir` digests the
|
||
builtins archive; `compiler_object` digests flan.cmxa and flan.a. Writes are already `.tmp`-then-`rename`, so a
|
||
shared durable cache is safe under dune's own parallelism. **One key was not total**: `Macro.key` was prelude text
|
||
plus the call's forms, with nothing about the compiler that emitted the `.so` it names — and that `.so` is dlopened
|
||
back into this binary, so a codegen or ABI change served a stale expander as a crash rather than an error. It now
|
||
carries the compiler's stamp, passed across `start_merged`'s exec in `FLAN_COMPILER_STAMP` because a merged dev
|
||
binary's own path is a per-session throwaway under /tmp (keying on it rebuilt a macro module every dev start —
|
||
~350ms, most of what this cache exists to save).
|
||
|
||
`lib/cimport.ml` spells its own copy of the old `cachedir` (line 828) and still sits under `$TMPDIR`. Another lane
|
||
holds that file; folding it onto `Build.cachedir` is a one-line change when that lane lands.
|
||
|
||
Worth not re-deriving: **the site was `test_dev.ml:77`**, not `test_emacs.ml`/`test_repl.ml` — the belief that only
|
||
those two carry that wording was false and sent one lane after the wrong file.
|
||
|
||
**The socket ordering fix is confirmed** at 30 sequential full `dune test` runs, 0 failures, no `never bound` in any
|
||
log — so the section this file used to carry about it is gone.
|
||
|
||
**It deserved a test, and has one; `HANDOFF-f1.md` is deleted.** The reason is not the log line, it is the policy
|
||
behind it: a program that never calls `agent/start` is the one condition on which the two shapes of `flan dev`
|
||
deliberately disagree — `two_process` kills its child and `failwith`s, `merged_serve` warns and serves anyway,
|
||
because the thing it would have to kill is itself. Nothing held that second answer in place, and copying the
|
||
daemon's answer back into the merged path is the obvious tidy-up: it would turn every agentless program into a
|
||
session that dies at startup, with no test to notice. `test/programs/dev-noagent.flan` and the last block of
|
||
`test_dev.ml` assert the session still answers `describe` after the wait runs out, with the warning text as the
|
||
secondary evidence that this is the branch that produced it.
|
||
|
||
The block costs the full ten seconds and there is no way to spend less — `accept_loop` is not reached until the
|
||
await expires, so no reply can arrive sooner. A timeout override in `lib/dev.ml` existing only for the test was the
|
||
alternative and is a worse trade.
|
||
|
||
**The `never listened` message named the wrong thing, and now it names which.** One timer covered two waits — a
|
||
build, then a bind — so the message was a wrong diagnosis, which costs more than no message. `listening` in
|
||
`test_dev.ml` (and the copy in `test_emacs.ml` and `test_repl.ml`) polls the daemon's process alongside the socket
|
||
and reports which of the two failed: "exited with status N before binding <sock>" when it died, "was still running
|
||
after 30s without binding <sock>, so it was the build that did not finish, not the socket" when it did not. A crash
|
||
now fails in milliseconds instead of costing the whole timeout, which is the half of "split the waits" that was
|
||
actually worth having. The timers are 30s, down from a minute, because the build they wait on is warm now.
|
||
|
||
1. **The two waits are still one wait.** Separating them needs a signal from `flan dev` that the build is done and
|
||
the bind has begun — a sentinel beside the socket in `start_merged` and `two_process`, or the existing
|
||
`flan dev: built %s in %.0fms` stderr line captured per daemon. The stderr route means redirecting stderr at
|
||
twelve spawn sites in `test_dev.ml` and polling a log for a substring; the sentinel is two lines in `lib/dev.ml`
|
||
and an unambiguous boundary, and is the one to do if this is picked up. Neither was done here because watching
|
||
the process already buys the fail-fast, and the message already says which wait it was.
|
||
|
||
### ~~From `HANDOFF-f3.md` — the watch is built; two tests are not~~
|
||
|
||
`spy-num` landed and `docs/PORTING.md` Tier 1 item 5 is closed. The `spy` half already existed — the finding that
|
||
reframed the task is that the table, the agent commands, the buffer and the ghost text were all there at `344e571`,
|
||
and only the numeric accumulator was missing. A slot keeps count, min, max, last and mean; the window is since the
|
||
editor's last tick rather than cumulative, because cumulative min/max reach the session's extremes within seconds of
|
||
play and then never move again. Reset is its own message and never a side effect of reading, so the reader never
|
||
writes the table.
|
||
|
||
The fixture is in: `emacs/test-flan-watch.el` pins both accumulator heads beside the one-segment ones, verified by
|
||
putting the old `?` back in `flan-watch-ghost-call-regexp` and watching exactly those two checks fail.
|
||
|
||
**Closed.** The dead `n=0` branch in `watch_render_num` is deleted and the reader stays epoch-blind on purpose: a
|
||
stopped program does not sample, so an epoch-aware reader would blank the watch for the whole of a break loop, which
|
||
is exactly when the numbers are being read. What was actually wrong was in the editor — `flan-watch--tick` sent
|
||
`:reset t` five times a second at a program that could not answer it — and it is now guarded on `flan-dev--stopped`,
|
||
with the read still going out every tick. `emacs/test-flan-watch.el` pins both halves. `HANDOFF-f3.md` is deleted;
|
||
the reasoning is in `docs/BUILT.md`, "A hot loop keeps five numbers, and the window is the editor's".
|
||
|
||
### Not started: five more raylib core examples
|
||
|
||
Ten are in `examples/`. Five more were queued and the lane was stopped before it wrote anything, so there is nothing
|
||
to pick up — only the brief. Pick for what they exercise of the bindings rather than for how they look, since the
|
||
point of the corpus is finding gaps in `vendor/raylib/raylib.flan`. Upstream is
|
||
`vendor/raylib/web/raylib-5.5/examples/core/`. `docs/PORTING.md` §1's rule binds: a raylib function on a real per-frame
|
||
path is hand-written and header-checked, not left to the opt-in `FLAN_RAYLIB_H` import, because the default build
|
||
has no `FLAN_RAYLIB_H` and still has to draw. An enum value has to be added by hand whatever happens — the importer
|
||
generates functions and only functions.
|
||
|
||
### Landed: frame rollback, `docs/PORTING.md` Tier 1 item 6
|
||
|
||
`test/programs/frame-rollback.flan`, with three acceptance rows beside `bounds-condition.flan`'s (plain, `-O0`,
|
||
dev). Nothing was added to the language, which was the premise: `snapshot` is `(set grid-backup grid)` and
|
||
`(set world-backup world)`, `restore` is the same two the other way, and that is the whole of what `engine.clj`
|
||
spends an `IntGrid` walk on and `engine.lisp` spends `sb-mop:class-slots` on.
|
||
|
||
**The ordering against defers was the decision.** Restore lives in the `continue` clause, so it is the last write
|
||
on the abandoned path and needs no agreement with any defer that ran on the way out. The rejected alternative —
|
||
restore in a `defer` inside the frame function — is the silent one: a defer runs on the ordinary return path too,
|
||
so it rolls back the frames that *succeeded*, and nothing reports that as an error. The test pins the ordering with
|
||
a counter inside the snapshot and a counter outside it rather than in prose, and carries a negative control (the
|
||
same bad frame, a `continue` that only counts) because "state equals snapshot" passes trivially on a program that
|
||
wrote nothing.
|
||
|
||
One edge left in the file's header comment rather than here: the snapshot covers plain values only. A defer that
|
||
frees a resource the snapshot holds a pointer to means restore resurrects a dangling one.
|
||
|
||
### Landed: sand.flan is the game the other ports are
|
||
|
||
765 lines to 201, at parity with `lisp/sand.lisp`, `clojure/src/fnm/sand.clj` and `src/fnm/sand.jank` in
|
||
`~/Development/fnm`. Audio, the brush textures and the embedded PNG, the render-texture scene, the HUD font, the
|
||
camera, the world cursor, the HUD and the input-state read-out are gone — none is in any reference version. The
|
||
single-use helpers are inlined. **The physics is untouched on purpose**: the three references disagree there, CL
|
||
using `landing-col`/`move-grain` and Clojure and jank using `step-cell` with velocity parking, so parity does not
|
||
name a target, and `settle` is what the pinned hash covers. It still prints `15595743031174623232` at `-O2`, `-O0`
|
||
and as a dev build, which is what says the inlining changed nothing.
|
||
|
||
Interactively the controls are now `r` and left-mouse only. `test_web.ml`'s assertion that brush.png's bytes reach
|
||
the wasm module went with the embed: `web-files.flan` is web-built and *run* under node and asserts the embedded
|
||
bytes print, which is the same property checked harder.
|
||
|
||
## ~~Queued: an idiomatic layer over the generated bindings~~ — **landed**
|
||
|
||
Thin Flan-shaped wrappers **over** the generated bindings, not instead of them. Built as three kinds, listed in the
|
||
header of `vendor/raylib/raylib.flan`:
|
||
|
||
- **A slice where C takes a pointer and a count.** The eleven vector-array drawing calls — `draw-line-strip`, the two
|
||
triangle batches, the five splines, the two `image-draw-triangle-*`, `draw-triangle-strip-3d` — under one section.
|
||
All eleven and not the three anybody calls: a subset puts the hole where the next caller looks. Each also guards the
|
||
empty slice, which is the part a hand-written call site gets wrong rather than merely writes out — raylib takes a
|
||
count of 0 happily, but `(addr (at pts 0))` is out of bounds before raylib is reached.
|
||
- **An `Option` where C signals with a sentinel.** `get-key-pressed` and `get-char-pressed`, raylib's two input
|
||
queues, both of which say "empty" with 0. What it buys is in `examples/text-input-box.flan`: the C shape reads the
|
||
queue in two places, once to prime the loop and once at the bottom of the body, and the Option shape reads it in
|
||
one.
|
||
- **An enum where the header says `int`.** `key-up?`, `key-pressed-repeat?`, `mouse-button-up?` — holes in families
|
||
whose other halves already took a `Key` or a `MouseButton`, so `(rl/key-down? :space)` compiled and
|
||
`(rl/key-up? :space)` did not. **These are not wrappers.** A C enum parameter has an int's ABI, so the hand-written
|
||
`declare-c` with the Flan type on it is the whole fix and a `defn` around it would be a rename.
|
||
|
||
The mechanism for the first two is the `name` directive in `vendor/raylib/bindings`: the generated declaration keeps
|
||
the symbol and gives up the name, so nothing about the C signature is hand-written and the generated half keeps its
|
||
agreement-by-construction with the header. The third is an `exclude` plus a hand-written line, exactly as `SetExitKey`
|
||
and `SetMouseCursor` already were.
|
||
|
||
**Not built: `with-drawing` and `with-mode-2d`.** An unbalanced begin/end is a real bug and a macro removes it. This
|
||
used to be blocked — a macro could not live in a package — and it no longer is: the expander collects `defmacro`s from
|
||
the prelude, from the file being compiled **and from imported packages**, qualified under the alias. They can be
|
||
written in `vendor/raylib` now, and have not been.
|
||
|
||
## ~~Queued: a restart is not a transaction, and the docs must say so~~ — **landed**
|
||
|
||
See [`docs/BUILT.md`](docs/BUILT.md), "A restart is not a transaction". **This entry was stale** — all three places already
|
||
carried the note when it was re-read: `conditions.org` under *Gotchas*, `spec-conditions.md` §5, and
|
||
`web/index.html`'s restart gotcha list, which was rewritten in place rather than gaining a second bullet beside the
|
||
existing one. Nothing was left to write.
|
||
|
||
What follows is the original entry, kept for the reasoning.
|
||
|
||
Raised by the author, and it is a real sharp edge rather than a gap. **If a frame mutates a global and then signals,
|
||
taking a `retry` re-runs the mutation.** Nothing rolls back. Common Lisp has exactly this property and offers no help
|
||
either — restarts are explicitly not transactional.
|
||
|
||
The discipline is that **the author chooses where the retry boundary is**: a `restart-case` at the top of a frame
|
||
re-runs everything including mutations already applied; one placed after the mutations re-runs only what follows. So
|
||
either put the restart before anything mutates, make the retried section idempotent, or snapshot what will be
|
||
re-applied.
|
||
|
||
This matters more here than in most Lisps because the intended use is a **game loop**, where the author's plan is to
|
||
skip a frame and carry on rather than die — exactly the case where a non-idempotent mutation bites.
|
||
|
||
## ~~Queued: a dev-build allocation registry~~ — **landed; one item left, and it is not a registry item**
|
||
|
||
Items 1 to 5 are built and the test is written. `docs/BUILT.md`'s *"An address answers with a type"* is the account of the
|
||
table, the note and the inspector's pointer arm; *"An address you have in your hand"* is the account of the reader —
|
||
the address root, the breakdown, the leak report, and what "at exit" turned out to mean.
|
||
|
||
**Left: nothing.** The memcheck half of item 6 is written. The registry knows an arena's `free-all` killed
|
||
everything in the region, so a later read through a pointer into it is *answerable*; memcheck is now told the same
|
||
fact on the next line, by a vendored `MAKE_MEM_UNDEFINED` over `[base, cap)`, so the same read is *reported*.
|
||
**The two are still not blurred** — the registry answer and the memcheck answer are different tools reaching
|
||
different people, and building one was not progress on the other; they were built separately and neither is evidence
|
||
for the other.
|
||
|
||
**What it does not cover, and does not need to:** stack locals and globals, which the shadow stack and the static type
|
||
table already answer by name. A stack address is deliberately not in the table, and a pointer to one still renders
|
||
`<ptr>` — the address root refuses such an address by name rather than rendering bytes at it.
|
||
|
||
**Note on classes:** `defclass` instances will carry shape metadata by design, so they get identification for free and
|
||
do not need the registry. This is for plain structs, `Vec`, `Map` and pool storage.
|
||
|
||
**On cost, as built.** One insert per allocation, always on in a dev build, no opt-out — the author's instruction,
|
||
followed literally. Nothing was built per-region, no range recording, no per-allocator opt-out. Revisit only if a real
|
||
program shows a problem, and `docs/BUILT.md` still names **two** places a release build is not quite free: the readers
|
||
added since are functions nothing in a release build calls, and the exit report is registered by `atexit` from inside
|
||
`flan_dev_reg_enable` rather than by a file-scope destructor, precisely so that it is not a third.
|
||
|
||
## Picked up first, 2026-09-13
|
||
|
||
Three things, in order. The first two are one line each and unblock a real game.
|
||
|
||
**1. `DrawTexturePro` is not bound — DONE.** It is `draw-texture-pro` in `vendor/raylib/raylib.flan` now, hand-written
|
||
beside `draw-texture-rec` and read off a raylib header rather than remembered. `image-from-image` and `window-ready?`
|
||
went in with it. The rule the gap exposed is written down in `docs/BUILT.md` and `docs/PORTING.md` §1: *a raylib function on a
|
||
game's per-frame path is hand-written and header-checked, not left to the opt-in import* — the import widens the
|
||
surface and must not be load-bearing, because the default build has no `FLAN_RAYLIB_H` and still has to draw.
|
||
|
||
**2. `Key` has no `left-shift` — DONE.** `left-shift 340`, and nothing else: `docs/PORTING.md` §5 checked every other enum
|
||
value the game touches and they were all already right.
|
||
**1. `DrawTexturePro` — done, and not by a hand-written line.** It was the one true blocker for `siam-farmer`
|
||
(see `docs/PORTING.md`: every tile in both implementations goes through it, and neither `DrawTextureRec` nor
|
||
`DrawTextureEx` substitutes). It was reachable only through the opt-in `FLAN_RAYLIB_H` import. The bindings are
|
||
committed now, so `rl/draw-texture-pro` is in `vendor/raylib/generated.flan` and a default build has it. **Nothing
|
||
should add it by hand** — a second `declare-c` for the same C symbol is refused for the whole program.
|
||
|
||
**2. `Key` has no `left-shift`.** Both implementations use shift+1..5 to pick the tilemap. One enum member, and
|
||
still a hand edit: the importer generates functions and only functions, so no `defenum` comes out of the header.
|
||
`vendor/raylib/raylib.flan` is where `Key` lives.
|
||
|
||
**3. An out-of-bounds index should signal a condition, not `exit(134)` — DONE.** A failed bounds check signals
|
||
`BoundsError` with `error`; `runtime/flan_rt.c`'s `flan_bounds_error`/`flan_slice_error` walk the handlers, then offer
|
||
the break loop, and tail into the old `flan_bounds_fail` message and status only if nothing answered. `Vec`'s two
|
||
checks are plumbed the same way, since `(at v i)` and `(at arr i)` are one form in the source.
|
||
|
||
**No restart is established at the failing index**, and `docs/BUILT.md` has the argument: `retry` exists for allocation and
|
||
for files because those attempts are *repeatable*, and nothing a handler can do makes index 51 valid for a length-50
|
||
array. `use-value` for the index would cost every indexing operation a restart frame and buy a silently different
|
||
element. What answers a bad index is the restart the program already had — a frame loop's `continue`, `sand.flan`'s
|
||
shape — which is on the restart stack and on the break loop's list without anything being pushed at the site.
|
||
|
||
**Defer** had to be answered rather than inherited, since the `noreturn`-then-`unreachable` shape is what the old note
|
||
followed from: an answered bounds failure leaves through the function's unwind block, which is `return`'s path, so it
|
||
runs the defers; an unanswered one still runs none. `test/programs/bounds-condition.flan` counts them.
|
||
|
||
Two tests, because there are two paths. `bounds-condition.flan` is the *answered* half — a `handler-bind` taking
|
||
`continue` over five routes to a bad index, at `-O2`, `-O0` and as a dev build. `dev-break-bounds.flan` in
|
||
`test_dev.ml` is the half this was built for: nothing handles it, the break loop reports `BoundsError`, the layout for
|
||
that name resolves to `low`/`high`/`length`, the only restart on offer is the program's own `continue`, and taking it
|
||
resumes with the session intact.
|
||
|
||
**Still dying, deliberately:** a `Map`'s bounds check and `flan_vec_stale_fail`. The stale-allocator case is a
|
||
different kind of failure — the region the container lived in was released — and there is no frame to go back to that
|
||
would not read freed memory. The map path was left alone rather than converted half-way.
|
||
|
||
### What is left on `docs/PORTING.md`'s list
|
||
|
||
Tier 0 is finished, and so is item 5. **The watch for a running program is done** — the `spy` half was already
|
||
built (a pushed table, the watch buffer, inline ghost text) and the `spy-num` half landed 2026-09-13: a hot-loop slot
|
||
keeps count/min/max/last/mean, the write path does no formatting, and the window is since the editor's last tick
|
||
rather than cumulative, which is a deliberate divergence from `watch.clj` argued in `docs/BUILT.md`, "A hot loop keeps
|
||
five numbers, and the window is the editor's". **Item 6 landed the same day**:
|
||
`test/programs/frame-rollback.flan` is the worked example — `snapshot` at the top of the frame, `restore` in the
|
||
`continue` clause — and `bounds-condition.flan`'s half-written abandoned frame is the thought it finishes. That was
|
||
the last Tier 1 item anyone was going to move. Items 7 (`drop`), 8 (generics) and 9 (`(read-edn T bytes)`) are still
|
||
on that list and still deferred with reasons written beside each; none is a blocker for this game.
|
||
|
||
**What `docs/PORTING.md` says NOT to build, with evidence:** escaping closures (one capture site, fixed by one parameter),
|
||
`Handle`/pools, `Result`/`try`, `handler-case`, `loop`/`recur` and tail calls, user allocators, structural typing —
|
||
**none has a customer in that code**. (`Handle` and the pool were built anyway, and on the other reason: they are the
|
||
gate on classes. The finding stands and is why they were built small — see [`docs/BUILT.md`](docs/BUILT.md). `loop`/`recur` was
|
||
built too, and the finding stands there as well: what it is not is **tail calls**, which are still not built and still
|
||
have no customer.) And **generics is not the blocker** there either: the element-changing maps are
|
||
five-line load-time loops. That last one hangs on a design decision the report states flatly — whether the game's
|
||
state holds fixed arrays or `Vec`s.
|
||
|
||
**raylib 6.0 is not urgent:** all seven struct layouts on the game's path and every enum value it touches are
|
||
byte-identical between 5.5 and the vendored 6.0.
|
||
|
||
### Left mid-flight when the session ended
|
||
|
||
Both lanes committed their main work and died on trailing polish; both are merged and the suite is green.
|
||
|
||
- ~~**`pause` marking from Emacs was not built.**~~ **Built**, both halves. See `docs/BUILT.md`, "A breakpoint is a
|
||
function call, and the editor only says where".
|
||
- ~~**Ghost text** for the watch window — values shown inline at the code they belong to — is noted and not designed.~~ **Built.** See `docs/BUILT.md`, "Ghost text finds its anchor in the buffer, not in the table".
|
||
- `tools/unit-return.py` is re-runnable; run it over any `.flan` file a lane wrote before the conversion landed.
|
||
|
||
# Where this is
|
||
|
||
## Start here — next session
|
||
|
||
**Branch `dev-loop`, 199 commits, working tree clean, `dune test` green.**
|
||
|
||
**[`docs/DISCUSS.md`](docs/DISCUSS.md) is what has been *asked* and not answered** — open questions with the repo context that
|
||
bears on each, so an investigation starts from what exists. Nothing in it is a decision or a task; when one becomes
|
||
either, it moves here.
|
||
|
||
**`NEXT.md` is what is left. [`docs/BUILT.md`](docs/BUILT.md) is why the existing parts are the shape they are** — the reload
|
||
primitive, cells, the agent, the session, the daemon, the Emacs client, conditions, the FFI shim, the layout, and the
|
||
order it all got built in. This file was half build log until it was split; do not let it become one again. When a
|
||
track here finishes, its explanation moves there and its entry here goes away.
|
||
|
||
The dev loop works end to end: `flan dev program.flan`, then `C-c C-c`, `C-x C-e` and `C-c C-r` in Emacs against the
|
||
running process.
|
||
|
||
**Conditions are three steps of four.** `(error c)` is the diverging variant — a handler that returns normally has not
|
||
answered it, so only a transfer gets past. The break loop is in, editor half included: an unhandled `error` stops the
|
||
program on the frame that erred, the daemon annotates every reply with `:stopped`/`:condition`, and `C-c C-b` lists the
|
||
restarts and resumes into the choice. A restart is chosen **by position**, off a snapshot taken when the break was
|
||
entered, because a name resolves to the innermost frame offering it and the stopped thread's stack does not hold still.
|
||
Restarts below the evaluation a break is inside are listed, marked, and refused with the reason.
|
||
|
||
**Restarts take parameters now** — §3's other half. `(use-value [v i32] ...)` binds them, `(invoke-restart 'use-value 21)`
|
||
supplies them, and what a clause takes against what was given is checked at run time and refused with both spellings,
|
||
because a restart is found by name on a dynamic stack and neither end of a transfer can see the other. The one path
|
||
that cannot yet supply a value is the break loop, which is item 2 below and is where the interesting half is.
|
||
|
||
Still open: **`handler-case`**, which §"What this does not settle" leaves open as possibly a macro over `handler-bind`
|
||
plus a transfer. **`find-restart` and `compute-restarts` are blocked on a type, not on effort** — §4 gives them
|
||
`(Option Restart)` and a list, and there is no `Restart` type and no list to return one in. The minibuffer prompt never
|
||
needed them; it reads the snapshot over the agent's socket. And a `restart-case` clause should still carry a **report
|
||
string**: `use-placeholder` is what `invoke-restart` needs, not what a person reading a list needs. §3 says to settle
|
||
that *before* parameters and it was not settled — the field is cheap and the accessor is cheap, but the only consumer
|
||
is the break loop's listing, which lives in the agent and the daemon, so it would have shipped as a field nothing
|
||
read. It belongs with item 2, where the listing is being changed anyway.
|
||
|
||
Read SBCL for what restarts should *mean* and ignore how it moves control: it transfers with `block`/`return-from`,
|
||
which §6 rules out.
|
||
|
||
### Landed — macros run, and `unless` is not a special form any more
|
||
|
||
The expander is written and the exit criterion plan.org set for milestone 5 is met: a conditional sugar moved out of
|
||
`parse.ml` and into `prelude.ml` as a `defmacro`, with the corpus that was written against the special form
|
||
unchanged. Running `test/programs/macro-unless.flan` means the compiler built a shared object, `dlopen`ed it into
|
||
itself and called a Flan function to find out what `(unless c a b)` means.
|
||
|
||
The full explanation is in [`docs/BUILT.md`](docs/BUILT.md), "Macros: the compiler dlopens the program". Four things worth
|
||
knowing before touching any of it, because each cost something to find:
|
||
|
||
- **A call inside a quasiquote is output, not a compile-order dependency.** A macro body that *calls* another macro
|
||
needs it compiled first; a macro body that *quasiquotes* a call to one needs nothing, because the call is part of
|
||
what it answers and the answer is expanded again. The first cycle test written for this got that wrong and was not
|
||
a cycle at all. The two non-termination failures are therefore different and are refused differently: a ring is
|
||
named, a macro that does not settle is bounded.
|
||
- **Quasiquote is desugared before the walk**, and that is load-bearing rather than tidy — with the quasiquote still
|
||
standing, the walk expands the call inside it against the wrong arguments.
|
||
- **`lib/dune` passes `-linkall`.** `lib/macro.ml` installs itself into `Parse.expander` and nothing references it, so
|
||
the linker would otherwise drop it from `bin/main.exe`. Installing by hand is not viable: `session.ml` parses for
|
||
`C-c C-c`, and `test_session.ml` drives the session library in-process.
|
||
- **Two parser bugs fell out of it**, both in the rule that tells a return type from the first form of a body. The
|
||
prelude's types were not in the set that rule consults, so `Form` in return position was read as a body form; and
|
||
adding them plainly made `(defn f [] (Rune {.code 65}) (bar))` a function returning a `Rune` with a one-form body,
|
||
silently, in every file in the language. Both are pinned in test_flan.ml's return-type section.
|
||
|
||
Costs: a build that names no macro is unchanged at 50ms; one that calls a macro is 310ms cold and 70ms warm, the
|
||
difference being a cached `.so`; and a hello-world carries eight bytes of it, because `Reach.link` drops the rest.
|
||
|
||
### Landed — a C header is read, so a binding is checked instead of trusted
|
||
|
||
`lib/cimport.ml`, `lib/cjson.ml`, a `headers` file beside `link`. Full reasoning in `docs/BUILT.md`, "The header is read
|
||
now"; docs/DISCUSS.md item 6 is rewritten down to the two decisions left, both the author's.
|
||
|
||
The gap closed is the one `docs/BUILT.md` recorded as *trusted*: `declare-c` generates the wrapper, the typedefs and the
|
||
prototype from one declaration, so they agree with each other by construction and only the library could disagree —
|
||
and nothing had a second opinion to disagree with. Now clang is asked for a JSON AST dump of the header (shelled out,
|
||
never libclang — the dependency plan.org rejected; Zig has since left it too, for Aro) and both halves are compared
|
||
against it.
|
||
|
||
**The evidence.** Against raylib 5.5, the version whose `.so` `vendor/raylib/link` names: **all 16 `defstruct`s and
|
||
all 172 hand-written `declare-c` agree exactly.** Against the 5.1-dev header also installed on this machine, ten real
|
||
differences — nine functions that version lacks and one that gained a parameter — so picking the wrong header is
|
||
loud. Both comparisons run at build time and stop the build; verified by permuting `Texture2D` and by putting `f64`
|
||
where raylib says `float`, which is the hazard `docs/BUILT.md` names and says only a test can catch.
|
||
|
||
**Costs, measured, because they decide the remaining question.** Release build +4ms warm — `Reach.link` already drops
|
||
a wrapper nothing reachable calls, confirmed on the wasm32 case it exists for with 256 extra declarations in play.
|
||
Redefinition 31.0ms → 46.5ms. Dev build +333ms cold, once per session, since `Build.shared` compiles no C. Reading the
|
||
header is cached (64ms → 17ms), keyed like the object cache; the cache was built against a measurement, not a guess.
|
||
|
||
**Re-measured, and the 15.5ms was misattributed** — see docs/BUILT.md, "Where that 15.5ms actually is". A `C-c C-c` reads
|
||
no header: `Session.eval` puts the forms through `Load`, and forms with no `(import …)` in them touch no package. The
|
||
15.5ms is `flan reload`'s, and `flan reload` is a fresh process — ~14.5ms of it is session startup and ~4ms of *that*
|
||
is the header. What a redefinition really pays for an imported package is **+3.6ms per eval** in `Check` and in
|
||
`Emit.redefinition` declaring 256 more siblings, and no cache touches that; it is the number to attack next. The
|
||
header is now cached in the session as well as on disk, so a repeat import (a `C-c C-k` of a buffer carrying its own
|
||
`import` line) costs nothing, and a header edited mid-session is not picked up until the session restarts — the same
|
||
rule a changed `.c` file follows.
|
||
|
||
**Opt-in on purpose.** `vendor/raylib/headers` is `?${FLAN_RAYLIB_H}`. "A build needs libraylib linkable and not
|
||
raylib-devel installed" is a property chosen deliberately, and requiring a header would take it from everyone to give
|
||
the check to whoever has one. Unset means off; set-and-wrong is an error naming the path.
|
||
|
||
Worth knowing before touching it:
|
||
|
||
- **The import is bounded by the package's own `defstruct`s**, not by a curated list. A function mentioning a struct
|
||
the package has not described is refused with that reason. Of raylib's 581 functions, 256 import, 153 are refused,
|
||
172 are already bound by hand and left alone. Widening the binding is a `defstruct`, not a list edit.
|
||
- **No `defstruct` is generated, and that is load-bearing.** Generate them and the header becomes the authority on
|
||
layout, and checking the package's layouts against it would be comparing the header with itself — which is exactly
|
||
why `docs/BUILT.md` rejected a `_Static_assert` as circular. Keeping them hand-written is what makes the check a second
|
||
source.
|
||
- **A refusal is a demotion, not a drop** — Zig's `failDecl`, which `Load.refuse_hidden` already implemented for
|
||
`main`. `rl/get-gamepad-name` is a name that exists, cannot be had, and says why at the use site.
|
||
- **`declare-c` and `declare` are untouched and still win.** A C symbol the package binds by hand is not imported, so
|
||
the escape hatch is the override.
|
||
- **`test/headers/sample.h`** is the importer's table — one function per decision, committed, no raylib needed. The
|
||
raylib acceptance case skips without `FLAN_RAYLIB_H`; that one does not.
|
||
|
||
Two things that are *not* done, and are 6a and 6b in docs/DISCUSS.md: whether the header stays a build-time read or becomes
|
||
a committed generator (`flan import-c` already prints the lines, so it costs nothing more to switch), and whether the
|
||
172 hand-written lines migrate. Neither is blocked on correctness. The argument for the first is weaker than it
|
||
looked — committing the generated lines would save ~4ms of session startup and none of the +3.6ms per redefinition,
|
||
since that cost is the 256 declarations existing at all and not where they came from; needing the header at every build — vendoring raylib.h or requiring raylib-devel — is the argument on the
|
||
second.
|
||
|
||
One smaller thing found and worth not re-deriving: an enum parameter imports as `i32`, because the header says
|
||
`KeyboardKey` and nothing tells the importer the package calls that `Key`. The ABI is identical, the face is worse,
|
||
and it is why `(rl/key-down? :space)` keeps its hand-written line. The idiomatic-layer lane closed the three holes
|
||
this left — `key-up?`, `key-pressed-repeat?` and `mouse-button-up?` were generated and therefore took an `i32`, so the
|
||
sibling of a call that worked did not — by excluding them and hand-writing the enum type, which is all it takes.
|
||
|
||
### Landed 2026-09-12 — six tracks, one session
|
||
|
||
Six agents in parallel worktrees. Kept short on purpose; the reasoning that outlives the change is in `docs/BUILT.md` or in
|
||
the commit that made it.
|
||
|
||
1. **`nth` removed**, an alias of `at` that was asymmetric — `check.ml` aliased them but `parse.ml` and
|
||
`place_of_expr` matched only `at`, so `(set (nth a i) x)` and `(addr (nth a i))` were refused while the `at` forms
|
||
worked.
|
||
2. **`println` and `print`**, compiler-provided and structural. `Session.render` was already the compile-time walk
|
||
plan.org asks for; it moved to `lib/render.ml` parameterised on an emitter and a slot allocator, so the REPL and
|
||
stdout share one copy. Found doing it: `field_addr` in `emit.ml` accepted only `Types.Named`, so a field of an
|
||
`Option` threw at emit time and **the walk's Option arm had never run** — the inspector would have failed on the
|
||
first `(Option T)` pointed at it. prelude.ml's claim that this had to wait for milestone 5 and generics was wrong,
|
||
and is gone: a printer selected per *concrete* type has nothing to dispatch on and no type variable in it.
|
||
3. **`restart-at`** — a restart is taken by position now. See "Start here".
|
||
4. **Names in DWARF.** `Tast.fn` carries `snames` beside `slots`, so a let-bound local is its own name under lldb
|
||
instead of `s0`; a slot the compiler invented keeps `s<index>`, because inventing a name puts a variable in the
|
||
debugger that is not in the file. Shadowing had to be decided rather than assumed: every `!DILocalVariable` is
|
||
scoped to the subprogram — the typed IR has no block structure to build a `!DILexicalBlock` from — so two slots
|
||
called `v` left lldb answering `p v` with the outer one while the body computed with the inner, and not listing the
|
||
inner at all. A repeat gets a `~2` suffix, unspellable in source. That is a way of not lying rather than a way of
|
||
being right; see "One line away". Also `flan dev --debug`, one flag for host and every redefinition module, off by
|
||
default because a debug build is an `-O0` build.
|
||
5. **Ten raylib core examples** in `examples/`, plus seven bindings and the colour palette. The gap list they produced
|
||
is under "Unblocked now, and ranked"; the top item, that no number could reach `draw-text`, is fixed — `(string b)`
|
||
reinterprets a `[u8]` as a `string`, which costs no instructions because they are already the same 16 bytes.
|
||
|
||
6. **The `print-*` family is gone.** `print` and `println` are the whole printing surface; ~500 call sites across 47
|
||
files rewrote, and `web/index.html` gained a `#printing` section, the first documentation either has had. Two pinned
|
||
outputs moved and both are corrections: `sand-headless`'s hash is `15595743031174623232` rather than
|
||
`-2851001042534928384` — the same 64 bits, printed unsigned now that `hash-grid`'s `u64` no longer goes through an
|
||
`(i64 …)` cast — and a trap column shifted because the call it names got shorter.
|
||
|
||
### Landed — `(Map K V)`, and `defer` in a `let`
|
||
|
||
Step 4 of the container build order, following Odin: open-addressed Robin Hood hashing at a 75% load factor,
|
||
cache-line cell packing, pointer-width integers through the probe loop. Two deliberate departures from Odin — no
|
||
tombstones, because `spec-memory.md` defers removal, which deletes the backward-shift loop entirely; and no capacity
|
||
tagged into the data pointer, because this header has room for it and tagging would make correctness depend on an
|
||
alignment that is only ever requested.
|
||
|
||
**One amendment to a frozen `spec-memory.md`, and it is the `defer` half**: the spec says under "When storage is
|
||
released" that `(defer (free v))` for a `let`-bound `v` is "not expressible today" and that no idiom may depend on it.
|
||
It is expressible now. A `let` at the top level of a function body has exactly the function's extent — a `let` is not a
|
||
frame here, and nothing is released at scope exit — so a `defer` in one always registers. A loop body and a branch stay
|
||
refused, by name, for the reason that does apply to them.
|
||
|
||
**One restriction the spec does not have**: a fixed array is a map key only when its elements compare bytewise, so an
|
||
array of structs or of strings is refused by name. A struct key holding the array works, because a struct key is walked
|
||
field by field.
|
||
|
||
The measured answer to the author's "is this another Python dict": **six times quicker cache-resident and slower at a
|
||
million entries**. Python's algorithm is fine — what makes it slow is a separately allocated refcounted object per key
|
||
and value, and hashing through calls that cannot be inlined. The second half of that result is the interesting one and
|
||
is written down rather than left out; see the unsettled list under the build order.
|
||
|
||
### Landed — the allocator, the arena, `(Vec T)`, `StorageExhausted`
|
||
|
||
The critical path, and the thing NEXT.md said was the only one standing between this and writing a game. Steps 1, 2 and
|
||
3 of the build order below are struck; `Map` is step 4 and is untouched. `Allocator` is a builtin opaque type and
|
||
needed nothing from milestone 5, which was the whole bet. Three amendments to a **frozen** `spec-memory.md`, made
|
||
deliberately and stated as amendments in [`docs/BUILT.md`](docs/BUILT.md): `free-all` is retain-capacity with `arena-destroy`
|
||
beside it; `context/allocator` is a dynamic variable rather than a literal calling-convention parameter; and the `Vec`
|
||
header is six words in every build rather than four in release. One addition the spec does not have: a budget on the
|
||
allocator, because `retry` needs a handler that can make the *same* request succeed.
|
||
|
||
### Landed — the runtime under a sanitizer
|
||
|
||
`--sanitize` is a build flag beside `--debug`; `dune build --root . @sanitize` builds twenty-eight programs twice, plain
|
||
and sanitized, and compares output and exit status. Its own alias and not `dune test`, because the sweep is about nine
|
||
minutes. The checked sweep is **clean**. How ASan and UBSan reach a language whose IR is written by hand, and why the
|
||
flag does not force `-O0` when `--debug` does, is in [`docs/BUILT.md`](docs/BUILT.md).
|
||
|
||
Two defects came out of it, both found by reading rather than by the tools, both fixed with a regression case:
|
||
`flan_bytes_to_i64`/`flan_bytes_to_f64` clamped a slice length with `(size_t)n` and so read 63 or 511 bytes off the end
|
||
of a negative-length slice; and the three `snprintf` shims published snprintf's return as a slice length, which is what
|
||
it *would* have written.
|
||
|
||
**What is left, and it is most of what the sweep was meant to settle:**
|
||
|
||
1. **UBSan sees no Flan code and no flag changes that.** Its checks are branches clang's C frontend emits inline, not a
|
||
pass, so shift UB (`(<< 1 32)`, see Sharp edges), alignment, and the f32→i32 cast on NaN or an infinity — the things
|
||
`floor-f32` guards by hand and nothing else does — are unreached. Either `Emit` grows those checks behind the flag,
|
||
which is a compiler feature of the same shape the bounds checks already have, or they belong to the checker. Not
|
||
decided. `test_sanitize` pins the current answer with a control that must *not* report, so a future clang changing
|
||
this is a test failure rather than a discovery.
|
||
2. **Three of the four named buffers now have evidence; one still does not.** Two lanes closed different pairs and
|
||
they combine. The 4K result cap and `condition_name[128]` are driven over the agent's socket from `test_agent.ml`
|
||
— a 5000-byte value comes back as 4096 ending in the ellipsis, a 198-character condition class comes back from
|
||
`status` as 127. The 4K cap and the **dev registry overflow guard** are also run directly by `test/dev_limits.c`,
|
||
a C main beside `reload_host.c`, one process per limit because the name table never shrinks and the overflow case
|
||
aborts. Only `SNAP_MAX`/`SNAP_NAMES` is still read rather than tested: sixty-five nested `restart-case`s are a lot
|
||
of program for a clamp. `escaped[ESCAPE_MAX]` was already covered, because `println.flan`
|
||
drives a 1100-character string through it on purpose — 1019 bytes out against a worst case of 1021 into 1024.
|
||
`scratch[SCRATCH]` never sees more than 20 characters of 64.
|
||
3. **Valgrind over the headless corpus, done.** `dune build --root . @valgrind` runs forty-nine programs under
|
||
memcheck, twelve of them again with `--no-bounds-checks`, in 91 seconds including the compiles — on a warm object
|
||
cache; 162s was measured on a cold one, which is the compiles and not the sweep. Clean. It needs no
|
||
instrumentation at all — memcheck works on the binary, so `Emit`'s hand-written IR arrives on the same footing as
|
||
clang's C, which is why it was reachable where MSan was not. The uninitialised read ASan is blind to is now a
|
||
control that must report: index 3 of a `Vec` with len 2 and cap 4, with `--track-origins` naming the
|
||
`aligned_alloc` in `flan_vec_push`. Two more controls pin a heap overrun and a padded `Map` key. `test/valgrind.supp`
|
||
holds **no suppressions** — nothing false came up to suppress. Details, and the measured fact that memcheck catches
|
||
0 of `bounds.flan`'s 6 cases where ASan catches 3, in [`docs/BUILT.md`](docs/BUILT.md).
|
||
|
||
**The arena half of the hole is closed; the interior half cannot be.** `free-all` is retain-capacity, so the pages
|
||
stay and `malloc` never learns the storage died — but memcheck can be told directly, and now is: `flan_arena_proc`
|
||
issues a vendored `MAKE_MEM_UNDEFINED` over `[base, cap)` beside its registry call, and round two of a reset arena
|
||
reading a byte it never wrote reports where it used to print round one's value in silence. Measured: 0 errors
|
||
before, 6 from 4 contexts after, with `--track-origins` naming the client request. It is the sweep's fourth
|
||
control. Cost outside valgrind is five no-op instructions plus the request block — 23 on the `free-all` path only,
|
||
about 1ns per reset — and the macro is vendored rather than `#include`d because the machine that runs the sweep
|
||
has `valgrind` and not `valgrind-devel`, so a guarded include would silently compile to nothing exactly here. See
|
||
[`docs/BUILT.md`](docs/BUILT.md).
|
||
|
||
**Interior overruns remain invisible**, for a structural reason no client request fixes — an arena is one `malloc`,
|
||
and the `Map`'s `keys | values | hashes | scratch` is one allocation too, so "probe overrun at high load" is not
|
||
clean, it is not observable. And all three positive controls had to be written by hand: no corpus program reaches
|
||
an observable uninitialised read, so the sweep is a regression net from here rather than an audit that found the
|
||
runtime sound. The corpus stayed clean across the arena change, which is its own small finding — nothing in it
|
||
reads stale arena storage.
|
||
|
||
Two things the sweep structurally cannot cover: raylib and libm are uninstrumented, so the windowed examples are noise;
|
||
and a redefinition module is built by `llc` and `ld` rather than clang, so the reload path carries no instrumentation
|
||
whatever the flag says.
|
||
|
||
### Managed classes are planned. Do not start them.
|
||
|
||
plan.org grew a `class` facility beside `struct`: identity, runtime shape metadata, an implementation-defined
|
||
representation, generic-function dispatch, and live schema change with an explicit migration at a frame boundary. Its
|
||
own last line is the rule — nothing until ordinary `struct`, `Handle` and reload semantics are working. **All three
|
||
now do**, `Handle` and the pool having landed, so this section is no longer "not yet" but "next, and deliberately not
|
||
started here". It is here so that a session reading plan.org cold does not take it as the next task. Three things
|
||
found while reviewing it, none of them in plan.org yet:
|
||
|
||
- **A generic function is a cell.** "A later module can add `(defmethod draw ((e Enemy)) ...)` without editing the
|
||
original" means every compiled call site of `draw` has to find the new method — which is the problem the indirection
|
||
cells already solve. A generic function is a cell whose body is a dispatch table and a reload extends the table. The
|
||
expensive half of classes is therefore already built and tested.
|
||
- **The pool is not one storage option among three.** `migrate-instances` has to *enumerate* live instances. A pool
|
||
behind generational `(Handle T)` gives that by construction; a world arena and an owned region do not obviously.
|
||
plan.org presents the three as a free choice and they are not. **The pool is built**, and `(len p)` with
|
||
`(pool-handle p i)` is that enumeration.
|
||
- **`Enemy@1` has to stay resolvable** for `migrate` to dispatch on it, so the session retains every layout version's
|
||
metadata for as long as any instance holds it. Same rule as "nothing is ever `dlclose`d", and worth stating as one.
|
||
|
||
### Open: can a condition be a class?
|
||
|
||
Unanswered, and it wants answering before `handler-case`, because it decides whether handler matching has one path or
|
||
two.
|
||
|
||
It would buy the thing conditions most lack: a **hierarchy**. §1 says flatly there is none, which is why nothing can say
|
||
"any condition" — no catch-all handler and nothing for a break loop to match on. Class inheritance gives it.
|
||
|
||
Three costs, one serious:
|
||
|
||
- **Signalling would allocate.** A struct condition is a stack value and `signal` takes its address; a class instance
|
||
needs a pool slot at the signal site. That is the failure path, sometimes the hot path, and sometimes the thing that
|
||
failed is allocation itself. plan.org also says no implicit allocation anywhere in the core.
|
||
- **§5's lifetime inverts.** Today the condition dies with the signalling frame and a handler that keeps it copies,
|
||
which is free for a value struct. A class instance survives the transfer — nicer, but now something owns and frees it.
|
||
- **Layout versions meet handler frames.** A struct condition cannot change layout; it is refused. A class can, and then
|
||
a frame pushed against `MyError@1` is on the stack while the signaller builds `MyError@2`.
|
||
|
||
The shape that probably wins is both: a struct condition stays exactly what it is — no allocation, matched by name hash,
|
||
dies with the frame — and a class condition is allocated, survives, and matches by walking its class chain. That is two
|
||
matching paths, which is the same bill the struct/class split already signs, so it is consistent rather than a new cost.
|
||
Either way it is an amendment to a **frozen** `spec-conditions.md`, not a gap in it.
|
||
|
||
|
||
**The dev loop is closed.** `C-c C-c` in Emacs recompiles the top-level form at point and installs it in a running
|
||
program, at that program's next frame boundary. Verified against sand: an unsaved buffer edit to `game-draw`, and 240
|
||
consecutive frames drew it.
|
||
|
||
Steps 1, 2 and 3 are done — see *The reload primitive* in `docs/BUILT.md`. A list of top-level forms can be recompiled and installed
|
||
into a running process; call sites compiled before they existed follow them, and a `defn` or `defvar` the process was
|
||
never built with can be added and then redefined again. That is the whole of `C-c C-c`, minus an editor: sand.flan takes
|
||
a redefinition over a socket and installs it between frames.
|
||
|
||
What is left is the *session* — something that holds the checker environment between evaluations, tracks which names the
|
||
running process was built with, and speaks a protocol an editor can talk to.
|
||
|
||
Milestone 4 is done: **sand.flan builds, links raylib and runs**, and its simulation has a headless acceptance case that
|
||
runs on the `dune test` path at `-O0` and `-O2`. Milestones 2 and 3 are behind it (`calc-me.flan` compiles and runs; the
|
||
interpreter was dropped — open decision #7, settled — see "Why there is no interpreter" in `docs/BUILT.md`).
|
||
|
||
```
|
||
reader ✅ → parse ✅ → load ✅ → check ✅ → emit ✅ → clang ✅
|
||
```
|
||
|
||
| File | What it does |
|
||
|---|---|
|
||
| `lib/loc.ml` | source locations + `Loc.Error`, the frontend's one exception |
|
||
| `lib/form.ml` | reader output: `Sym Kw Int Float Str Byte List Vec Map` |
|
||
| `lib/reader.ml` | hand-written S-expression reader, no menhir/ocamllex |
|
||
| `lib/ast.ml` | AST: `texpr`, `expr`, `place`, `pattern`, `decl` |
|
||
| `lib/parse.ml` | forms → AST; special forms, desugaring, declarations |
|
||
| `lib/load.ml` | **imports: a package directory → qualified declarations** |
|
||
| `lib/types.ml` | resolved types; structural equality, `Never` fits anywhere |
|
||
| `lib/tast.ml` | the typed IR the backend consumes |
|
||
| `lib/check.ml` | AST → typed IR; two passes, bidirectional |
|
||
| `lib/session.ml` | **a live program: what the process was built from, plus every change since** |
|
||
| `lib/wire.ml` | **the editor protocol: one s-expression per message, length framed** |
|
||
| `lib/agent.ml` | **the agent, called rather than connected to, when it is in this process** |
|
||
| `lib/dev.ml` | **`flan dev`: a session, an editor socket, and the program it is a thread inside** |
|
||
| `lib/prelude.ml` | printers + `rand-f32`, written in Flan |
|
||
| `lib/emit.ml` | typed IR → LLVM IR text |
|
||
| `lib/build.ml` | `.ll` + the shim + the packages' C → clang → executable |
|
||
| `runtime/flan_rt.c` | the host ABI: argv, stdout, exit, 4 conversions |
|
||
| `runtime/flan_dev.c` | **dev only: the by-name registry a run-time-new name needs** |
|
||
| `lib/shim.ml` | **`declare-c` -> the generated C that flattens a struct crossing** |
|
||
| `vendor/raylib/` | **the raylib package: `raylib.flan` and `link`, and no C at all** |
|
||
| `vendor/agent/` | **the dev agent: one verb table, a loader thread, install at a frame boundary** |
|
||
| `emacs/` | **`flan-mode.el`, `flan-dev.el`, `flan-repl.el`: the editor half of the dev loop** |
|
||
| `bin/main.ml` | `flan read \| parse \| check \| emit \| shim \| build \| run \| reload \| dev` |
|
||
| `test/test_flan.ml` | reader, parser and checker |
|
||
| `test/test_acceptance.ml` | expression/result pairs + whole programs + the traps |
|
||
| `test/test_reload.ml` | **the reload primitive: recompile one function, load it, call it** |
|
||
| `test/test_agent.ml` | **a running program taking a redefinition over a socket** |
|
||
| `test/test_session.ml` | **what a running process cannot be told, and recovering from a typo** |
|
||
| `test/test_dev.ml` | **the daemon, driven the way an editor drives it** |
|
||
| `test/test_repl.ml` | **`C-x C-e`: an expression evaluated inside a running program** |
|
||
| `test/programs/conditions.flan` | **`handler-bind` and `signal`, the accumulation case** |
|
||
| `conditions.org` | **a cheatsheet for driving conditions: what works, the exact refusals, the gotchas** |
|
||
| `conditions-play.flan` | **a program to poke at them with, built to be attached to by `flan dev`** |
|
||
| `test/programs/restarts.flan` | **`restart-case` and `invoke-restart`: the transfer, across two frames** |
|
||
| `test/test_emacs.ml` | **the client, driven against a real daemon and a real program** |
|
||
| `test/reload_host.c` | the C host that loads and installs two rebuilds, in one process |
|
||
| `test/wasm-run.mjs` | **a WASI host in twenty lines of `node:wasi`, so the table can run a wasm32 build** |
|
||
|
||
```
|
||
$ flan run calc-me.flan "1 + 2 * (3 - 0.5) / 2"
|
||
3.5
|
||
$ flan run test/programs/sand-headless.flan
|
||
15595743031174623232
|
||
$ flan run sand.flan # a window, 120 fps, hold space
|
||
```
|
||
|
||
## Decided 2026-09-12, by the author, and not yet built
|
||
|
||
Five questions were put and answered in one sitting. Each is a decision, not a preference — build against them, and
|
||
reopen one only with a reason rather than a taste.
|
||
|
||
~~**1. Assets are embedded at compile time, one file or one directory.**~~ **Built** — `(embed "p")`, `(embed "p" string)`, `(embed-dir "d")`. See docs/BUILT.md, "Assets are baked in". Odin's answer, and the reason it is the right
|
||
one here: it is a *compiler* feature, so it needs no build flags, no linker arguments and no per-target packaging, and
|
||
it works identically on desktop and web. That matters more here than it does for Odin, because `Load` gives link flags
|
||
only to directory packages — the single file doing `(rl/load-texture "brush.png")` is structurally the one file with
|
||
no link channel, which is what stopped the web lane from inventing a flag. Embedding has no such hole. Odin's
|
||
`#load` and `#load_directory` are the model (`src/parser.cpp:853`, `src/checker.cpp:3594`). emscripten's
|
||
`--preload-file` stays available later for assets that should load lazily rather than be baked in; the `@web` link
|
||
line already carries it if wanted.
|
||
|
||
~~**2. Reading a file works everywhere; writing is desktop-only and signals on web.**~~ **Built** — `barf` on the web signals `FileError` with reason `file-unsupported`, and `test/test_web.ml` runs it under node rather than asserting the artifact's shape. See docs/BUILT.md, "slurp, barf, and the two ways they fail". Odin stubs its whole file API on
|
||
js/wasm — every operation returns `.Unsupported`, and `core/os/file_js.odin`'s own comment says the stubs exist only
|
||
so importing `core:os` "panics cleanly". Take the restriction and not the mechanism. **Flan has no conditional
|
||
compilation** — nothing in `parse.ml` or `check.ml` reads the target — so "isolate this code to desktop" is not
|
||
expressible in source, and a build-time refusal would therefore be unusable. A **silent no-op is worse than either**:
|
||
it is how a save file disappears with nothing said. So `barf` on web signals a condition under a restart and the
|
||
program decides. This is the language having something Odin does not; use it. Per-package target isolation, if a
|
||
whole desktop-only package is ever wanted, is the `@native`/`@wasi`/`@web` link-line tagging the web lane built.
|
||
|
||
**3. Build the shadow stack.** ~~Not yet built.~~ **Built**, both halves — see docs/BUILT.md. Kept here as the decision it
|
||
was, with the measurement it asked for: +33% on call-heavy code over globals for the frames, +61% with the slot table,
|
||
and 0.06% of a 60fps frame.
|
||
|
||
plan.org:591 has specified it in the dev-build column since the beginning and nothing
|
||
has ever built it. It is the route to `(:op "backtrace")` *and* to locals, together, and it is dev-only so a shipped
|
||
game pays nothing. Chosen over the DWARF route deliberately: DWARF still owes a `!DILexicalBlock` per `Let` before
|
||
`p v` under shadowing is even honest, and that buys locals in lldb rather than in the break loop. The author's reason
|
||
is the one to keep in view — **the more a break loop can show, the less often a real debugger is needed** — which
|
||
makes this a dev-loop feature, not a debugger feature.
|
||
|
||
**4. Conditions get a parent link, not class inheritance.** A condition type may name a parent where it is declared;
|
||
matching walks that static chain. This buys the hierarchy §1 of `spec-conditions.md` says there is none of — a
|
||
catch-all handler, "any file error" — at compile-time cost only. **It is deliberately not the class answer** that the
|
||
"Open: can a condition be a class?" section below weighs: a class condition allocates at the signal site, which is
|
||
the failure path and sometimes the thing that failed; it inverts §5's lifetime, so something must own and free it;
|
||
and it lets a condition's layout change while a handler frame stands against the old one. A parent link has none of
|
||
those costs and leaves the frozen model otherwise intact. Real inheritance stays possible later if a case demands it;
|
||
this closes nothing off. That section stays open for the record but is no longer the blocking question for
|
||
`handler-case`.
|
||
|
||
~~**5. File I/O — `slurp` and `barf` — is the next stdlib work**~~ **Built.** It was the next stdlib work, after `Vec`, because `slurp` returns a string whose
|
||
length is not known until the file is read and therefore cannot exist before an allocator does.
|
||
|
||
## Decided later the same day, and queued
|
||
|
||
~~**6. A field label is written with a dot, not a colon, and the colon is reserved for keys.**~~ **Done.**
|
||
`{.x 1.0 .y 2.0}` is struct construction and `{inner .field}` is destructuring; the old spelling is refused, and the
|
||
refusal names the new one. `:keys` kept its colon — it names no field, so leaving it alone is what lets the dot mean
|
||
exactly one thing. 681 labels across 45 `.flan` files including `vendor/`, plus 94 more in the Flan embedded in
|
||
`lib/prelude.ml` and the tests. `Map` is now free to take `{:key value}` without colliding with struct literals. See
|
||
docs/BUILT.md, "The colon belongs to keys".
|
||
|
||
**What it left for the Emacs lane, both verified.** `render.ml` still *prints* a struct with colons, deliberately:
|
||
`emacs/flan-inspect.el:165` parses that output and hard-codes the colon when it reads a field out, so the printer
|
||
has to move in the same commit as its reader. That half is still open and belongs with whoever next opens the
|
||
inspector. ~~And `flan-mode.el:61` font-locks `:name` as a constant with nothing matching `.name`, so a field label
|
||
is now unfontified where it used to be coloured.~~ **The font-lock half is done:** a field is drawn as a constant in
|
||
both of the spellings that exist while the corpus moves, so `{.x 1}` and the accessor `(.x v)` read alike, and the
|
||
keyword rule stayed where it was because the colon still means an enum member and a map key.
|
||
|
||
**7. `Map` follows Odin's implementation.** Read `base/runtime/dynamic_map_internal.odin` before writing any of it;
|
||
the checkout is at `~/Repositories/Odin`. Three properties are the ones worth copying, and they are stated in its own
|
||
header comment:
|
||
|
||
- **Open-addressed Robin Hood hashing at a 75% load factor.** No buckets, no per-entry allocation, and probe
|
||
distances stay even because a later arrival steals a slot from an earlier one.
|
||
- **Cache-line-aligned `Map_Cell` packing**, so no single key or value ever straddles a cache line and a linear probe
|
||
walks memory in a cache-friendly order. This is the part a hand-rolled open-addressed map usually gets wrong.
|
||
- **`uintptr` throughout** for sizes, masks and offsets, to keep sign-extension and masking instructions out of the
|
||
probe loop.
|
||
|
||
Its static/dynamic split is the same type-erasure this project already committed to: `Map_Info` carries size,
|
||
alignment and offsets, and the compiler emits the hash and equality pair per key type. `spec-memory.md`'s
|
||
structural-key restriction holds this to the built-in key set, so there is no dispatch to design.
|
||
|
||
**Why this will not be Python's dict.** Worth recording because it is the question that prompted the decision.
|
||
Python's dict algorithm is fine; what makes it slow is that every key and value is a separately allocated, reference-
|
||
counted object, and hashing and comparison go through indirect calls that cannot be inlined. Flan stores raw bytes
|
||
and compiles the hash and comparison concretely at each use. That difference is most of the gap before any
|
||
algorithmic cleverness. **jank is not the model** — it is Clojure, so its maps are persistent with structural
|
||
sharing, which plan.org rules out by name because shared structure destroys the clear ownership that is the whole
|
||
reason there is no collector.
|
||
|
||
## Decided in discussion, queued
|
||
|
||
**Globals in the break buffer — built.** One section under the stack, holding the union of the globals every frame on
|
||
the current stack references, each entry annotated with the frames that touch it and ordered by the innermost one. It
|
||
is `(:op "globals")` in `dev.ml` and `flan-cnr--insert-globals` in the break buffer. See docs/BUILT.md, "Globals of a
|
||
stopped stack". ~~The one hole left open — the redefinition check is a fingerprint over a body's *slots*, so a new body
|
||
that names different globals while binding the same locals is not caught.~~ **Closed**: `Reach.ref_fingerprint` is a
|
||
second fingerprint over the set of globals a body names, carried beside the slot one in `%fninfo` and checked the same
|
||
way, and such a frame is now refused by name. Kept separate from the slot fingerprint deliberately, so `locals` still
|
||
reads a frame whose locals are fine and whose global attribution is not.
|
||
|
||
**The break buffer opens by itself when the program stops.** Today a condition stops the program and the buffer appears
|
||
only when `C-c C-b` is typed. `flan-dev--absorb` already inspects every reply for `:stopped` and a poll covers the case
|
||
where no reply is pending, so the client already knows the moment it happens and already moves the mode line from it —
|
||
this is a hook at a point that exists, not new plumbing.
|
||
|
||
Three things to settle while building it: whether it takes focus or only displays; whether `(pause)` should always take
|
||
the window, being a deliberate stop rather than a failure; and what it does when the program stops while point is
|
||
mid-edit in another buffer.
|
||
|
||
**~~`Handle` and the pool are the real gate on classes, and they are buildable now.~~ Built.** plan.org's rule is
|
||
that nothing starts on managed classes "until ordinary `struct`, `Handle`, and reload semantics are working". All
|
||
three now hold: structs work fully; reload works with one known hole (a changed signature is refused rather than
|
||
versioned); and `(Handle T)` and `(Pool T)` exist, with the enumeration primitive `migrate-instances` was blocked on.
|
||
See [`docs/BUILT.md`](docs/BUILT.md), "`(Handle T)` and the pool, which is what a stale reference answers with".
|
||
|
||
It was not an incidental precondition. `migrate-instances` has to *enumerate* live instances, and a pool behind a
|
||
generational `(Handle T)` gives that by construction while a world arena and an owned region do not. plan.org presents
|
||
the three storage strategies as a free choice and they are not: handles are the one that makes migration possible.
|
||
`(len p)` plus `(pool-handle p i)` is that enumeration, and it is two entry points rather than an iteration protocol.
|
||
|
||
Already banked, and it means classes are less work than plan.org implies: **a generic function is an indirection cell**
|
||
whose body is a dispatch table, which a reload extends. That is the expensive half of method dispatch, and it is built
|
||
and tested.
|
||
|
||
**Resource cleanup: `defer` stays the answer. `drop` is not built, and `with-cleanup` is not either.** Reached by
|
||
working the case through rather than by preference, so the reasoning is worth keeping.
|
||
|
||
`drop` was specified in `spec-memory.md` this morning. **This amends it: the hook is deferred, not built.** Three things
|
||
decided against it. It runs code somewhere the reader is not looking, which is the C++ behaviour the author explicitly
|
||
does not want. It would not even cover the motivating case — `Image` and `Texture2D` are *raylib's* types, and attaching
|
||
a hook to a foreign type is its own unsolved design question. And its one real advantage, cascading through a container,
|
||
is the case `Handle` makes rare: entities holding handles hold numbers, not resources.
|
||
|
||
`with-cleanup` / `unwind-protect` was also put and rejected: awkward with several resources, and it reads worse than
|
||
what already exists. The raylib begin/end pairs that seemed to motivate it are a macro problem, not a primitive one —
|
||
`with-drawing` and `with-mode-2d` are three-line macros once the expander lands.
|
||
|
||
**What to build instead is small: relax where `defer` may be written.** It is refused today inside a `let`, a loop or a
|
||
branch. The loop and branch refusals are right — `defer` is a *compile-time* construct, the cleanup copied into every
|
||
exit path, so "maybe registered" is not expressible and a loop body would fire once at function exit instead of once per
|
||
iteration. But **a `let` at the top level of a function body has exactly the function's extent** and always registers,
|
||
so it is as safe as function scope and is refused for a reason that does not apply to it. Relaxing it gives:
|
||
|
||
```
|
||
(defn load-brush []
|
||
(let [sheet (rl/load-image-from-memory ".png" brush-bytes)]
|
||
(defer (rl/unload-image sheet))
|
||
(set brush (rl/load-texture-from-image sheet))
|
||
(rl/image-flip-horizontal (addr sheet))
|
||
(set brush-mirrored (rl/load-texture-from-image sheet))))
|
||
```
|
||
|
||
Several resources are several defers, released in reverse, visible in acquisition order. A container of resources is an
|
||
ordinary loop inside the defer body — the manual cascade, three lines, at one level of nesting.
|
||
|
||
**Odin, for the record**, has no destructors, no drop and no finalizers: `delete` frees container memory and nothing
|
||
else, and resource release is `defer` at the acquisition site. That idiom does not transfer directly only because
|
||
Odin's `defer` is block-scoped; the relaxation above recovers most of it.
|
||
|
||
**The safety net, and the better use of effort: a debug tracking allocator.** ASan's leak detection covers memory
|
||
*instrumented* code allocated — the Flan allocator, and it is already wired up and clean. It does **not** cover a leaked
|
||
texture, because that memory belongs to uninstrumented raylib, which is the same reason the sanitizer sweep treats the
|
||
windowed examples as noise. But every raylib call goes through a *generated* wrapper, so a dev build can count
|
||
acquisitions against releases at that boundary and report what is still held at exit, by name. No hook, no type
|
||
annotation, nothing running at a distance — it does not change how code is written, it reports when something was
|
||
forgotten.
|
||
|
||
## ~~Before the batch below: read `docs/DISCUSS.md`'s "NEXT SESSION STARTS HERE"~~ — **answered, and being unwound**
|
||
|
||
The architectural question raised at the end of 2026-09-12 — **putting the compiler inside the running program's
|
||
process** — was researched (`docs/DISCUSS.md` §14), then built: `flan dev` is one binary and one process, and the editor
|
||
socket did not move. See "One process" in [`docs/BUILT.md`](docs/BUILT.md).
|
||
|
||
What it reopened is now being deleted one piece at a time, each with its own green run:
|
||
|
||
**One of the three was transport. The other two were concurrency, and were only mistaken for transport because the
|
||
socket was in front of them.**
|
||
|
||
1. ~~**The internal socket, the line protocol, and `Dev.deliver`/`result`/`ask`.**~~ **Done** — a delivery is a direct
|
||
call into the agent's verb table. Measured: the transport was ~50µs of a 23ms redefinition, so the end-to-end
|
||
number did not move. Code generation is 19 of the 22 milliseconds, which is the number any backend argument has to
|
||
start from.
|
||
2. ~~**The 4K `RESULT_MAX` cap** in `runtime/flan_dev.c`.~~ **Refused, with half of it deleted** — it is not a
|
||
transport buffer. The agent's second copy of the number and the drift check between the two files were, and those
|
||
are gone. The bound itself is the buffer the *game thread* writes into, so a growable one means the frame thread
|
||
calling `realloc`, and it would break the seqlock — which is a protocol about torn contents and assumes the address
|
||
it copies from does not move. Removing it is a redesign of the read, and belongs with moving the read to a frame
|
||
boundary.
|
||
3. ~~**`flan_agent.c`'s snapshot copying and generation stamping.**~~ **Refused, in full** — it was never about two
|
||
address spaces, it is about two threads, and there are still two. The break loop polls, a thunk it runs is
|
||
arbitrary Flan that pushes and pops the live restart list and the shadow stack, and the generation stamp is what
|
||
keeps a nested break from claiming a choice made against the outer one. A pointer is meaningful to the compiler
|
||
now; the frame it points into is no more alive for that.
|
||
4. **The render-thunk-per-inspection design for locals and globals.** A redesign rather than a deletion, and its own
|
||
lane: it is what unblocks "the inspector can retain a value".
|
||
|
||
Still reopened and still undecided: the watch design (push was chosen partly because polling costs a compile), and
|
||
whether an in-process JIT or a hand-written backend is needed at all.
|
||
|
||
## ~~Queued: a second tier of the standard library, after macros~~ — **landed**
|
||
|
||
See [`docs/BUILT.md`](docs/BUILT.md), "The prelude's second tier". The diagnosis here was right and the prelude had 44
|
||
allocation-free functions because there was nothing to allocate from; there are 24 more now, and the "Refused, by
|
||
name" block at the foot of `prelude.ml` is down from eight entries to four, each with a *different* reason rather
|
||
than the one shared sentence.
|
||
|
||
What landed: `append!`/`append-i64!`/`append-f64!` (the builder), `concat`, `join`, `split` returning a
|
||
`(Vec [u8])`, `repeat-bytes`, `replace-bytes`, `to-lower`, `to-upper`, `slices-new`; `format-f64` with a precision;
|
||
`atan2-f32` and `pow-f32`; `clamp` as a `defmacro`; and the slice family at two more element types —
|
||
`sort-f32!`, `reverse-f32!`, `swap-f32!`, `min-f32`, `max-f32`, `sum-f32`, `bytes<?`, `swap-bytes!`, `sort-bytes!`.
|
||
Tests: `programs/strings.flan`, `programs/format.flan`, `programs/algorithms.flan`, `programs/math2.flan`.
|
||
|
||
`string-from-bytes`, refused in that block, turned out to already exist: `string` is a builtin and
|
||
`(string (as-slice v))` is the round trip.
|
||
|
||
**What could not be built, and why each one could not.** All four want a compiler or runtime change, and none of
|
||
them wants a language decision.
|
||
|
||
- ~~**`Map` keys and values.**~~ **Iteration is built** — `flan_map_next` and the `map-next!` builtin, exactly the
|
||
shape this described. See [`docs/BUILT.md`](docs/BUILT.md), "`map-next!`, the one thing a Map could not do".
|
||
`map-keys`/`map-values` as *prelude functions* stay refused, and the reason is now generics rather than the
|
||
iterator: a `defn` has to name its types and `(defn map-keys [m {K V}] (Vec K))` has no `K`. The loop is three
|
||
lines at the call site, where `K` is known.
|
||
|
||
- ~~**`map`, `filter`, `reduce`, and a sort taking a comparator.**~~ **All four are in the prelude.** The diagnosis
|
||
was right and is now evidenced: **function values, not generics** — they arrived with no generics at all. See
|
||
[`docs/BUILT.md`](docs/BUILT.md), "Function values, with no capture". They are one copy per element type (i32 and f32), which
|
||
is the half generics would remove, and a `map` that *changes* the element type is the one shape that did not come
|
||
with them — one copy per ordered pair of types rather than per type.
|
||
|
||
**Capture is not built and escaping closures stay deferred.** An `fn` is lifted into a function of its own and
|
||
handed nothing but its parameters; a reference to an enclosing local is refused by name. That is what keeps a
|
||
function value a bare code address with no environment, and it is the next thing to want if a callback needs
|
||
state — `spec-memory.md`'s cases 1 and 2 are still the design to build from.
|
||
|
||
- **`(vec-new [u8])` is refused**, so a `(Vec [u8])` can only be made where the *context* names the type.
|
||
`check.ml`'s `vec_new_elem` accepts a single bare symbol naming a type and nothing else, and a `let` has no type
|
||
annotation to say it the other way round — so `split` needs a one-line `(defn slices-new [] (Vec [u8]) (vec-new))`
|
||
standing in as the place where the type is said. The fix is to let `vec_new_elem` take a type *expression* rather
|
||
than a name, which is the same parser that already reads `[u8]` in a parameter list.
|
||
|
||
- **An array literal cannot say it is `[f32]`.** A float literal defaults to `f64`, an array literal has no context,
|
||
and a `let` has no annotation, so `[3.5 -1.0]` is an `[f64]` and every element in `programs/algorithms.flan` is
|
||
written `(f32 3.5)`. Same shape of gap as the one above and probably the same fix.
|
||
|
||
Two smaller findings, both written down beside the code that ran into them:
|
||
|
||
- ~~**The prelude is never macro-expanded.**~~ **Fixed, and the diagnosis above was wrong in both halves** — see
|
||
[`docs/BUILT.md`](docs/BUILT.md), "A prelude function may call a prelude macro". The prelude *does* reach the expander; the
|
||
arity error came from the `Check.program` *inside* `Macro.compile`, where expansion is off. It is a cycle and not
|
||
an ordering — a macro module is compiled from the prelude — so moving the prepend would have changed nothing. What
|
||
fixed it is `Macro.reduce`, which makes the prelude smaller for that one build, plus dropping the prelude's own
|
||
macros from the forms fed back as `extra`. `format-f64` is `(clamp prec 0 9)` now. "A prelude macro may not call a
|
||
macro" stands and names itself when violated.
|
||
|
||
- **A returned `Vec` is a move, and the dead set spans the function**, so an early `(return v)` on one branch kills
|
||
the binding for the `v` at the foot of another. `replace-bytes` guards its empty-needle case with an `if` rather
|
||
than a `when`/`return` for that reason. Probably correct as it stands — the analysis is not path-sensitive and
|
||
making it so is a real piece of work — but it is a shape that reads as though it should compile.
|
||
|
||
Already present and easy to miss: an **EDN parser**, at `vendor/edn/edn.flan`.
|
||
|
||
## Decided: the Clojure patterns we are deliberately not copying
|
||
|
||
This language borrows Clojure's shape and is **not** trying to be Clojure-compatible, so its known wrinkles are ours
|
||
to avoid rather than inherit. Four, with what to do instead.
|
||
|
||
**1. One argument-order rule, held everywhere.** Clojure's sequence functions take the collection *last*
|
||
(`(map f coll)`) and its collection functions take it *first* (`(assoc m k v)`). The split is deliberate there, and it
|
||
is why Clojure needs **two** threading macros instead of one. **Our rule: the thing being operated on comes first.**
|
||
That is already what the language does — `(at a i)`, `(len xs)`, `(push v x)`, `(as-slice v)` — and `into` follows it
|
||
with the source first. Hold it; do not ship two of anything to paper over a split.
|
||
|
||
**2. A membership test says which thing it tests.** Clojure's `contains?` checks *keys*, so `(contains? [1 2 3] 1)` is
|
||
true because index 1 exists — the most-cited confusion in the language, and there is no built-in for "is this value in
|
||
this list". Half of this is already right here: `has-key?` on a `Map` is named for what it does. If a value-membership
|
||
test is added for sequences, name it for values and never overload one name across both meanings.
|
||
|
||
**3. A predicate returns a boolean; a search returns what it found.** Clojure's `some` returns the *value*, so
|
||
`(some even? [1 2])` is `true` but `(some identity [nil false])` is `nil` — one name doing two jobs. Keep them apart:
|
||
a `?` name answers yes or no, a finder answers the thing or nothing, and neither pretends to be the other.
|
||
|
||
**4. Composition reads in the same direction as threading.** Clojure's `comp` is right-to-left while `->` is
|
||
left-to-right, so the two compose mentally in opposite directions. If anything here ever composes operations, it reads
|
||
left to right, the way `into` does.
|
||
|
||
Sources are community consensus rather than a specification; the `contains?` complaint is documented in *Getting
|
||
Clojure*. Recorded because these are cheap to honour now and expensive to unpick once a standard library depends on
|
||
them.
|
||
|
||
## ~~Queued: `into`, fused transformation without transducers~~ — **landed**
|
||
|
||
Decided in conversation. **Not transducers, and not Rust's iterators — a macro that fuses the chain at compile time.**
|
||
|
||
```
|
||
(into xs (vec-new i32) (map double) (filter even?))
|
||
```
|
||
|
||
Argument order is **source, destination, then any number of transforms**, matching the `into->` macro the author
|
||
already uses in Clojure (`from to xform & xforms`). It reads as a sentence — take this, put it there, doing these —
|
||
and the variadic transforms have to trail anyway, which is the mechanical reason they cannot sit in the middle.
|
||
Clojure's own `into` composes them into one `xform` first, which is why that macro exists at all.
|
||
|
||
**Why a macro and not transducers.** Transducers compose at *runtime*: they need function values, closures and
|
||
allocation, and every element pays a chain of indirect calls. Rust has no transducers — it has iterators, which are
|
||
lazy but fuse into a single loop at compile time via monomorphisation and inlining, needing generics to do it. A
|
||
macro reaches the same destination with neither: `(map double xs)` expands to `(double x)` written straight into the
|
||
loop body, so the function name is *syntax* and never a value. **No intermediate collection at any step, no closure,
|
||
no generics, and nothing to inline.**
|
||
|
||
**It therefore does not need function values** and is independent of that work.
|
||
|
||
**What it gives up, and the author does not want it anyway:** you cannot build a transformation at runtime and pass it
|
||
around. That is transducers' actual selling point and it is close to useless in a game.
|
||
|
||
**Why the destination belongs in the form, and why this suits Flan better than `->>` would.** Every collecting
|
||
operation here allocates from an *explicit* allocator — that is a frozen rule in `spec-memory.md`. `->>` hides where
|
||
the result goes; `into` names it, so the macro knows the destination type, emits the right loop and the right
|
||
allocation, and the rule is honoured by construction. plan.org's `->>` threading over slices is the thing this
|
||
replaces for the collecting cases.
|
||
|
||
**Open, and worth settling when it is built:** whether reductions share the form. `(into xs 0 (map cost) (sum))`
|
||
reads oddly because zero is not a collection. A second macro with the same shape may be cleaner, so that the
|
||
destination is always honest about what it is.
|
||
|
||
Drop Clojure's `:eduction` branch — that is the pass-around case, and the one part that would need runtime machinery.
|
||
|
||
**Done.** See *`into`, which fuses at compile time because it is a macro* in [`docs/BUILT.md`](docs/BUILT.md). It is a prelude
|
||
`defmacro` over a plain `defn` that walks the transforms in reverse, and all four of the macro limits bit without
|
||
blocking anything: the three refusals are names nothing defines, `into-wrap` uses only special forms so
|
||
`Macro.reduce` does not drop it, the quasiquotes are all single-level, and `into` lives in the prelude because a
|
||
macro is not importable.
|
||
|
||
**The open question is settled: reductions do not share the form.** The reason the destination sits in `into` at all
|
||
is that the destination *is* the allocation, which is what makes `spec-memory.md`'s explicit-allocator rule true by
|
||
construction. A seed is not an allocation, so `(into xs 0 (map cost) (sum))` would be a second form wearing the same
|
||
spelling and the destination would stop being honest about what it is. A reducing macro of the same shape is a
|
||
separate form the day something wants one.
|
||
|
||
Two things the design did not anticipate, both written up there. **A source that is already a name is used as it is**
|
||
rather than bound — a `(Vec T)` is move-only, so binding it would take the caller's ownership for a read, and a fixed
|
||
array would be copied once per `into`; a source that is anything else is still bound once, which is what a call
|
||
needs. And **an owning temporary as the source leaks**, because the macro binds it to a name the caller cannot reach
|
||
and cannot know whether the type owns anything. A call in that position should borrow. `drop` is what would close
|
||
this, and it does not exist.
|
||
|
||
## ~~Queued: `loop`/`recur` (the return type is done)~~ — **landed**
|
||
|
||
~~**1. A `defn` must always state its return type, and unit is written `()`.**~~ **Done.** See *The return type is
|
||
the slot, and unit is `()`* in [`docs/BUILT.md`](docs/BUILT.md).
|
||
|
||
The slot after the parameters is unconditionally a type, the pre-pass that collected a file's type names is gone
|
||
along with `is_type_form`, `qualified_type`, `types_in`, `declared_types` and `prelude_types`, and `Parse.decl` no
|
||
longer takes a set of names at all. `(defn f [] f65 0.0)` now says *unknown type f65 — did you mean f64?* instead of
|
||
*unknown name*. `()` is the only spelling of unit: `Unit` is refused with a message naming it, and `Types.to_string`
|
||
prints `()` too, because that printer prints what a person would write for every other type it knows.
|
||
|
||
Two things the plan did not anticipate. `(defn f [] ())` — a return type and an empty body — is a shape the optional
|
||
slot could not produce, and it needed its own arm. And dropping `prelude_types` removes one of the two reasons
|
||
`Macro.reduce` may only drop `defn`s: the memoised set that a bootstrap build could have poisoned no longer exists,
|
||
so what is left is the plain one, that the surviving functions still mention those types.
|
||
|
||
**The sweep is `tools/unit-return.py`**, kept rather than thrown away, because the lanes that branched before this
|
||
wrote Flan in the old spelling and their files want the same pass at merge:
|
||
|
||
```
|
||
python3 tools/unit-return.py .
|
||
python3 tools/unit-return.py --in-strings test/test_flan.ml test/test_acceptance.ml \
|
||
test/test_session.ml emacs/test-flan-dev.el emacs/test-flan-mode.el
|
||
python3 tools/unit-return.py --raw-ml lib/prelude.ml
|
||
python3 tools/unit-return.py --in-html web/index.html
|
||
```
|
||
|
||
`-v` logs every `defn` it saw and what it decided; `--check` changes nothing. It is re-runnable, and on this tree it
|
||
reports exactly six sites, all in `test_flan.ml`, which spell the refused forms *on purpose* so the refusals can be
|
||
tested. Read the diff of every non-`.flan` file — docs/BUILT.md lists what the script can and cannot see.
|
||
|
||
~~**2. `loop` and `recur`.**~~ **Done.** See *`loop` and `recur`, and why `recur` is better than tail calls and not
|
||
only cheaper* in [`docs/BUILT.md`](docs/BUILT.md). `emit.ml` is untouched: a loop is a `let`, a `While` whose condition is
|
||
`true`, and two jumps, and the barrier question `recur` asks is the one labelled `break` already answered.
|
||
|
||
Three things the plan did not anticipate, each written up there. **Tail position is a permission that is withdrawn**
|
||
rather than a pre-pass over the `Ast` — `ctx.tail` is read and cleared at the top of `check`, exactly as `defer_ok`
|
||
is, and handed back only by the three forms that pass a tail through, so nothing has to enumerate the forms that do
|
||
not. **`loop` is itself a barrier** for `break` and `continue`, which is a restriction added rather than inherited: a
|
||
loop answers with the value of its body, so a jump out of one has no value to give, and therefore `loop` also takes
|
||
no label. And **the move tracker had to be told about the loop's own names**, which are bound before the loop entry is
|
||
pushed and would otherwise have tripped the "moves a value bound outside the loop" rule on the ordinary case.
|
||
|
||
Still not given, and still out of scope: **mutual recursion between two functions.** That needs real tail calls. The
|
||
refusal for a `recur` outside any loop says so by name.
|
||
|
||
## The next batch, in order
|
||
|
||
Agreed at the end of 2026-09-12. Ordered by priority, not by size. Items 1-3 and 5-6 want the compiler core and should
|
||
run one lane at a time; item 4 is disjoint and runs alongside any of them.
|
||
|
||
~~1. **Fix the one failing test** — `the frame of a superseded body answered with the new body's names`.~~ **Done**, and
|
||
the handoff's diagnosis was wrong. Nothing was dropping the number: the fingerprint was emitted into `%fninfo` and
|
||
never read back. `flan_dev.c` called the field `spare`, there was no accessor, the agent never snapshotted it, the
|
||
backtrace line never carried it, and `Dev.locals` compared slot counts and nothing else — four of the five
|
||
hand-offs were never written, and printing both sides of the comparison could not have found it because there was
|
||
no comparison. The mechanism was sound and stayed: it hashes slot *names* as well as types, so it does see a
|
||
rename. See docs/BUILT.md, "Locals of a stopped frame".
|
||
|
||
2. ~~**The colon-to-dot change.**~~ **Done**, and `Map` is unblocked: `{:key value}` is free. The sweep is
|
||
`tools/colon-to-dot.py`, kept rather than thrown away, because the lanes that branched before it wrote Flan in the
|
||
old spelling and their files want the same pass at merge — `python3 tools/colon-to-dot.py .` over the tree, and
|
||
`--in-strings` for a `test/*.ml` that embeds Flan.
|
||
|
||
3. ~~**`Map`, and the `defer` relaxation.**~~ **Both done.** See *`(Map K V)`, which is Odin's map* and *`defer` may be
|
||
written in a `let`* in [`docs/BUILT.md`](docs/BUILT.md). The `defer` relaxation **amends `spec-memory.md`**, which said
|
||
`(defer (free v))` for a `let`-bound `v` was not expressible; it is now. `Map` restricts one thing the spec does not:
|
||
a fixed array is a key only when its elements compare bytewise, so an array of structs or of strings is refused by
|
||
name. The measured answer to "is this another Python dict" is six times quicker cache-resident and *slower* at a
|
||
million entries, and the second half is the interesting one — see below.
|
||
|
||
4. **The Emacs batch. Disjoint from the compiler, so it runs in parallel with anything above.** Globals in the break
|
||
buffer; the buffer opening itself when the program stops; ~~the indentation rewrite with `clojure-mode` as the
|
||
reference~~; ~~`#_`~~; hex, binary and addresses on primitives in the inspector. **The indentation rewrite and `#_`
|
||
are done.** The indenter is ported from `clojure-mode`'s source rather than derived from it — `flan-mode` still
|
||
requires nothing outside stock Emacs — and it aligns a binding vector name-under-name, which is the bug that cost
|
||
friction on every keystroke. `defn` parameter lists and `restart-case` clause parameters were the same shape and
|
||
came with it. What remains in this batch is the break buffer and the inspector, and they are independent.
|
||
|
||
5. ~~**Union values**~~, then the macro expander, then `Result`/`try`. Promoted above `Handle` on the author's call —
|
||
macros are the thing most worth wanting, and unions are the only thing between here and them.
|
||
|
||
**Union values are done.** See *Unions, and the tag they carry* in [`docs/BUILT.md`](docs/BUILT.md). The diagnosis was right:
|
||
`Option` is a two-case union wearing a special coat, so `Tast.arm`'s `acase` and `binds` already were union shape
|
||
and `check_match` grew a second subject rather than a second path. A union is `Types.Named` exactly as a struct is,
|
||
so every path that merely carries a type learned nothing.
|
||
|
||
What the spec did not settle and this lane did: the tag is an `i32` and the payload a blob aligned to the widest
|
||
member of any case, so `%"U" = type { i32, [k x iA] }` is C's `struct { int tag; union {...} u; }` byte for byte —
|
||
checked against clang's answer for the same declaration. A value is `(U.C {.field value ...})` and construction is
|
||
**qualified**; a pattern is bare `(C x y)` and resolves against the scrutinee. Tags are declaration order from
|
||
zero, so **case order is part of a union's contract**: a zeroed union is the first declared case. A non-exhaustive
|
||
match is **refused**, never defaulted.
|
||
|
||
What is left for the macro lane, and it is one thing: **`load.ml:312` refuses an imported union outright**, so a
|
||
union is file-local. That is not a blocker for `Form` — the prelude is parsed and prepended into the same flat
|
||
namespace before `collect` runs, so a `defunion Form` in `prelude.ml` is an ordinary same-file declaration and
|
||
needs no import and no `load.ml` change. Verified by declaring one there and matching it from a program.
|
||
|
||
**Macros landed on top of this** and needed no `load.ml` change for `Form`, exactly as this said. See
|
||
[`docs/BUILT.md`](docs/BUILT.md), "Macros: the compiler dlopens the program", and the short list of what is left of them
|
||
below.
|
||
|
||
Macros are what buy `with-drawing` and `with-mode-2d` over raylib's begin/end pairs, the hiccup DSL if a JS backend
|
||
ever happens, and the removal of special forms from the compiler.
|
||
|
||
**`Result`/`try`** follows, being another union.
|
||
|
||
**Generics are deliberately NOT here** — and function values landing has *sharpened* the case rather than made it,
|
||
which is the useful update. `Vec` and `Map` needed none, being type-erased. Function values needed none. What
|
||
needs them is now concrete and small: the prelude's `map!`/`filter`/`reduce`/`sort-by!` are **two copies each**,
|
||
i32 and f32, differing in nothing but the element type; `map-keys`/`map-values` cannot be written at all because
|
||
a `defn` must name its types and `(defn map-keys [m {K V}] (Vec K))` has no `K`; and a `map` from `[i32]` to
|
||
`[f32]` would be one copy per ordered pair. A user-written allocator is *not* on this list any more — it wants
|
||
a C-shaped callback and somewhere to put a `flan_allocator`, neither of which is a type parameter.
|
||
|
||
6. ~~**`Handle` and the pool.**~~ **Built.** A reference to something that can die, that reports that it died rather
|
||
than silently resolving to whatever reused the slot. See [`docs/BUILT.md`](docs/BUILT.md), "`(Handle T)` and the pool, which
|
||
is what a stale reference answers with". A handle is one `i64` — slot index low, generation high — so it copies,
|
||
zeroes and compares like an integer and owns nothing; a live slot's generation is odd, which makes a zeroed handle
|
||
resolve to nothing rather than to slot 0; and a generation that would wrap retires its slot instead, because "rare"
|
||
is not an answer when the failure is the silent wrong one the type exists to prevent.
|
||
|
||
What the spec did not settle and this lane did, beyond those: `resolve` answers `(Option (Ptr T))` and not
|
||
`(Option T)` — the spec's own worked example is annotated that way, for the reason written a line above it, that a
|
||
pattern binding binds a value and a copy cannot be written back. `(len p)` is the *slot high-water* and `(live p)`
|
||
is the live count, in that direction, so a loop bounded by `len` cannot silently skip a live entry. A slot is
|
||
recycled by `(release p h)` on the owner and never by `free`, because a handle owns nothing and consuming one copy
|
||
would say nothing about the others — so `spec-memory.md`'s two release points are untouched.
|
||
|
||
**Two amendments to the frozen spec, both deferrals**: `.field` and `at` do not auto-deref a handle, and `deref` is
|
||
not overloaded on one. Neither can answer "gone", which is the whole job; the spec's own example resolves first and
|
||
matches, and that is the half that is right.
|
||
|
||
**Still open**: a `(Ptr T)` from `resolve` dies on any `insert` that grows the pool, the same explicit contract a
|
||
slice has against `push`. Chunked never-moving storage is the fix and it costs code. And a pool passed to a helper
|
||
is consumed, because there is no borrowing parameter — a pre-existing `Vec` gap, not a pool one.
|
||
|
||
7. ~~**`break` and `continue`, with loop labels.**~~ **Built.** Labels are Odin's in the head position, both blockers
|
||
are answered, and the refusals name the construct they refuse for. See docs/BUILT.md, "`break` and `continue`, and the
|
||
rule that replaced a blanket refusal". What was settled in conversation before it was built, kept:
|
||
|
||
**Labels, Odin-style but in the head position.** A keyword names a loop and `break` takes it:
|
||
|
||
```
|
||
(while :outer (< i n)
|
||
(while (< j m)
|
||
(when (hit? i j) (break :outer))))
|
||
```
|
||
|
||
A keyword there is unambiguous because a loop condition is never one. It is **not a goto**: control can only leave
|
||
a loop it is already inside, which is what keeps it safe and is the same restriction Odin's labelled `break` has.
|
||
|
||
The two known blockers stand and must be answered: `check.ml`'s `in_frames` rule refuses `return` inside
|
||
`handler-bind`/`restart-case` **because `return` always crosses**, while `break` crosses only sometimes — a loop
|
||
wholly inside a `restart-case` body has a legitimate local break — so that blanket refusal has to become a
|
||
loop-depth-relative-to-frame-entry rule. And `continue` forces a `Tast.While` signature change to carry a latch,
|
||
because `check_dotimes` folds the step into the body and a `continue` branching to the header would skip it and
|
||
hang.
|
||
|
||
8. ~~**Errors: a structured value with spans and notes, and more than one per compile.**~~ **Built.** See
|
||
*An error is a value, and there is more than one of them* in [`docs/BUILT.md`](docs/BUILT.md). `Loc.Error` carries a
|
||
`diag` — a stable `kind`, a *span*, `notes` that each have their own span and severity, and the macro expansion
|
||
the error came out of — and `flan check`/`flan build` print the source line with the offending span underlined,
|
||
in the GNU format `compilation-mode` already parses. No editor work was needed and none was done.
|
||
|
||
What made it cheap, and is worth knowing before anything else is retrofitted onto locations: **the span went into
|
||
`Loc.t` itself**, as an exclusive end defaulting to the start. A location nobody widened is a zero-width span at a
|
||
point, so every one of the ~260 refusal sites kept its meaning, only the reader had to learn to fill the end in,
|
||
and `Form`, `Ast` and `Tast` were not touched. **Macro provenance went the same way** — a `macro : string option`
|
||
on the location — because `Expand.unmarshal` already stamps the call site onto every node a macro produces, so the
|
||
tag travels to the checker for free.
|
||
|
||
**Three things deliberately not built, so they do not read as oversights:**
|
||
|
||
- **The reader does not collect.** There is no resynchronising a paren stream — after an unclosed bracket nothing
|
||
knows whether the next `)` closes this form or the one above it. First error, stop.
|
||
- **Pass one of the checker does not collect either.** Signatures are a foundation: a declaration pass one could
|
||
not make sense of leaves a hole that pass two reports once per mention, and thirty "unknown name" lines under
|
||
one wrong signature are the same error thirty times. Pass two — bodies, where the volume is — collects per
|
||
declaration.
|
||
- **There are not a hundred kinds.** The reader's fourteen have them and the checker's have them where a test
|
||
asserts on one; `check.ml` alone has 163 refusal sites and minting an id for each is a sweep nothing reads.
|
||
- **`Load` and `Shim` do not collect.** They sit between the two collecting phases and still stop at the first
|
||
refusal, for pass one's reason: an import that could not be resolved leaves a hole the checker would report
|
||
once per use.
|
||
|
||
**One claim checked rather than assumed,** and it is weaker than it first reads: `compile.el` groups `note` with
|
||
`info` at level 0, and `compilation-skip-threshold` defaults to 1, so `next-error` walks the **errors** with no
|
||
configuration — that part holds — but steps over the notes unless the threshold is set to 0. The notes are still
|
||
parsed, coloured and clickable. Labelling them `warning:` would make them navigable and is refused: a note is not
|
||
a warning.
|
||
|
||
**What the daemon sees, which the brief asked to be worked out and stated:** the single-diagnostic exception is
|
||
still the single-diagnostic exception. `Session.eval` and the daemon check one form, keep catching `Loc.Error`,
|
||
and take a location and a message out with `Loc.summary`; `dev.ml` and `session.ml` needed nothing but the
|
||
pattern rewrite. The list is a second exception, `Loc.Errors`, raised only by `Parse.program_all` /
|
||
`Check.program_all` — **separate names rather than a `~keep_going` flag**, so a list cannot reach a handler that
|
||
does not name it without somebody editing the session.
|
||
|
||
**Left for later, small and independent:** notes on the type-mismatch errors, which are the most common class and
|
||
want the *parameter's* declaration as the second place — `env.fns` stores types and not locations today, so that
|
||
is a small change to what `collect` records. And a checker error on macro-produced code names the macro but has no
|
||
separate location to point at, because the expansion has no source of its own; the note lands on the call site
|
||
beside the error, which tells the reader the code being refused is not the code they wrote and no more than that.
|
||
|
||
8b. ~~**The old entry, kept for its one extra fact.**~~ **Subsumed by 8, and it was right about the tooling:** no
|
||
editor work was needed and none was done. Flycheck and a structured JSON report stay refused for the reason it
|
||
gave — the workflow is compile-at-the-end, not live linting.
|
||
|
||
9. **Signature generations and stale-caller warnings.** The biggest remaining hole in "you never restart the program" —
|
||
a changed signature is still refused rather than versioned. Last because it is the largest and nothing else waits on
|
||
it.
|
||
|
||
Deliberately not scheduled: the JS backend and header-based C interop, both large and neither blocking current work; a
|
||
debug tracking allocator, which is the leak safety net and a good candidate whenever it is wanted.
|
||
|
||
## Decided in discussion — the array constructor and the module system
|
||
|
||
**`(array 4 rl/Vector2)` makes a fixed array; `[4 T]` stays the type syntax. Built** — see docs/BUILT.md, "`(array 4 rl/Vector2)`, and the one position with no type slot". The problem it solved: a `let`
|
||
binding takes no type, so `(let [pts [4 rl/Vector2]] ...)` reads `[4 rl/Vector2]` as a two-element array *literal* and
|
||
fails with *unknown name rl/Vector2*. It cost 32 hand-written `Vector2`s in one raylib example.
|
||
|
||
`[4 T]` is not a special syntax — it is the ordinary type syntax and already works everywhere a type is expected:
|
||
`(defvar points [4 rl/Vector2] ...)`, `(defn draw [pts [4 rl/Vector2]] ...)`. A `let` binding is the single position
|
||
with no type slot, which is the whole of the bug.
|
||
|
||
`(zeroed [4 rl/Vector2])` was proposed first and rejected on how it reads: in argument position the bracket form is
|
||
unambiguous to the *parser*, but it still looks like a two-element vector to a person. `(array 4 rl/Vector2)` says what
|
||
it does with the count and the type as plain arguments. `zeroed` keeps its existing job — an empty thing of whatever
|
||
type the destination wants — and `array` is the one that is told.
|
||
|
||
**The module system stays as it is: the directory name is the module name.** No `package foo` line at the top of each
|
||
file. Confirmed against Odin, which requires the declaration despite having the same one-package-per-directory rule —
|
||
`package os` appears in 85 files, all of them in `core/os` — so the line is ceremony that buys only the ability to
|
||
disagree with the directory name.
|
||
|
||
What the rule already gives, and what was checked in conversation: **several files in one directory are one module**,
|
||
which is the case directory-as-package exists for; **a loose file is a module of one**, so several modules can sit at
|
||
the same filesystem level without a directory each; and two modules cannot share a directory, which is also true of
|
||
Odin.
|
||
|
||
**Acyclic imports are kept deliberately, not inherited by accident.** Odin forbids import cycles and so should this:
|
||
a definite package order is what the macro expander will need later, since every `defmacro` must be compiled before
|
||
anything that calls it. Nested import paths not being real nesting — Odin's `core:math/bits` is a separate package
|
||
rather than a submodule of `math`, with no re-exporting — was reviewed and accepted as fine.
|
||
|
||
**A package importing a package has landed**, so a project is no longer an entry file plus one flat layer of
|
||
libraries. Four things were settled doing it:
|
||
|
||
- **A name imported *through* a package keeps the inner alias.** If `area/` imports `shape`, the type is `shape/Box`
|
||
in the finished program and never `area/shape/Box`. This is forced rather than chosen: a directory reached along two
|
||
routes has to arrive under one set of names, or the checker sees every declaration twice and two copies of one
|
||
struct fail to unify. It is also what makes the dedupe coherent, and what keeps a qualified name the resolvable
|
||
identity the `layout` op and the break loop depend on.
|
||
- **The same directory under two aliases is refused**, including when one of the two aliases is a package's own and
|
||
pages away from the other. That is the price of the rule above and the refusal names both aliases.
|
||
- **A diamond loads its bottom once**, keyed by the real path.
|
||
- **A ring is refused and named** — `a -> b -> c -> a`, not "there is a cycle". Tolerating one was the earlier
|
||
behaviour and looked like it worked; what it cost is a definite package order, which is the thing the macro expander
|
||
needs, since every `defmacro` must be compiled before anything that calls it.
|
||
|
||
`Load.t.pkgs` now comes back in topological order, dependencies first. The *declaration* list is deliberately not
|
||
sorted and does not need to be — `check.ml` collects every top-level name before it checks any body.
|
||
|
||
**The expander did not end up reading that order**, and it is worth saying so rather than leaving the paragraphs above
|
||
to imply otherwise. Macros are collected from the prelude and from the file being compiled; a `defmacro` in a package
|
||
is refused by name, because reaching one means resolving that package's own imports over `Form`s before `Load` runs.
|
||
The order is there and correct and is what package-level macros will read on the day they exist; nothing reads it
|
||
today.
|
||
|
||
**Still missing: package visibility.** `rl/get-color-raw` is callable. The blocker is surface syntax, not `load.ml`:
|
||
`exported` and the refusal machinery already exist and take a second rule in one line, but there is no way for a
|
||
package to *mark* a name private, and adding one means a parser change.
|
||
|
||
## Decided in discussion — three more, two now built
|
||
|
||
~~**A watch window, ported from the author's Clojure one.**~~ **Built.** See `docs/BUILT.md`, "The watch window, and why
|
||
it is the only listing that is pushed", and `emacs/MANUAL.md` under "Looking at values". Three of the original's
|
||
decisions were kept unchanged — the program decides what is shown, the request is async, and the paint is
|
||
`replace-buffer-contents` so point and scroll survive every tick.
|
||
|
||
**The design written here was superseded, and the correction is the interesting part.** This entry said the answer to
|
||
an expensive eval was to **compile the watch thunk once and re-invoke it cheaply per tick**. That is the right
|
||
instinct about the cost and it is still a *poll*, and a poll has a defect that caching cannot fix: it cannot answer
|
||
while the program is **stopped**. A thunk runs at a frame boundary and a stopped program has no more frame boundaries
|
||
— which is exactly the moment you most want to see what the last frame held. So the direction was reversed instead:
|
||
the program calls into a table from inside its own loop and Emacs reads the table, which is memory rather than an
|
||
evaluation. That also made the values update at *frame* rate rather than at the timer's, which the compile-once poll
|
||
could not have done at any price.
|
||
|
||
**What was not built, deliberately: the `(watch "hp" hp)` form.** Scalars work today through `declare-c` against four
|
||
runtime entry points, which needs no compiler change at all. A struct or a slice needs a compile-time walk over its
|
||
type — one arm in `check.ml` beside `print`, which `docs/BUILT.md` writes out in full — and that file is held by another
|
||
lane, so it was left alone rather than reached into.
|
||
|
||
~~**Ghost text is gated on that same arm**, which is the finding worth keeping.~~ **Wrong, and ghost text is built.**
|
||
The reasoning was that values shown inline need a *place*, that nothing in the table has one — `(watch-i64 "ticks"
|
||
ticks)` says what the value is called, not where it was written — and that carrying a source location means the
|
||
caller supplies it, which means a generated call site. Every step of that is still true **of the table**, and the
|
||
conclusion did not follow: the call site is in the *buffer*, and the name in the table is the string literal in it,
|
||
so the anchor is searched for rather than reported. Nothing new is asked of the daemon. The two open questions
|
||
`flan-watch.el` recorded are answered rather than solved — overlays are replaced wholesale every repaint, so an edit
|
||
has nothing to invalidate, and a watch in a loop shows the last value written exactly as the buffer does, because
|
||
every better answer is the query UI this design exists to avoid. See `docs/BUILT.md`, "Ghost text finds its anchor in the
|
||
buffer, not in the table".
|
||
|
||
~~**The inspector gets a second way to start: an address and a type.**~~ **Built.** See `docs/BUILT.md`, "Two ways to root
|
||
a walk, and why neither subsumes the other". It went in as a frame and a slot *index* rather than an address and a
|
||
type — the daemon holds both and an index is the thing the listing can hand back, while an address is not something an
|
||
editor should be holding. The one prediction that did not survive contact: `l` crossing between the two modes was
|
||
listed as a cost and is not one, because a stack entry carries its own root and a mixed stack cannot be built.
|
||
|
||
**Structural typing requires identical layout — same fields, same types, same order.** Settled by the author, and it
|
||
makes the feature simple rather than hard: structural compatibility becomes "the same memory", which costs nothing at
|
||
run time and needs no copy, no reordering and no adaptor. The motivating case is `{.x 1.0 .y 1.0}` and that order is
|
||
natural anyway.
|
||
|
||
**Flexible field order waits for classes, deliberately.** A class has an implementation-defined representation, so the
|
||
compiler owns the layout and field order stops being observable — any order can match. That is the right place to pay
|
||
for flexibility, because a class already carries identity and metadata, and a `Vector2` should pay for neither. See the
|
||
`defclass` entry: `Handle` was the gate, and it is built now.
|
||
|
||
Note what this settles from the earlier discussion: writability was the question that decided layout, and requiring
|
||
identical layout answers it — fields are writable on the ordinary terms, by value a copy and through a `(Ptr T)` the
|
||
original, with no special case.
|
||
|
||
## Blocked and unfinished
|
||
|
||
Everything below was found, decided or half-built and then stopped. Each says what blocks it. Nothing here is a
|
||
vague intention — if it is listed, someone has already established it is real.
|
||
|
||
### Unblocked now, and ranked
|
||
|
||
**0. Signature generations and stale-caller warnings — milestone 7's unfinished half.**
|
||
Promoted here on the author's correction, and `session.ml:146` already says the same thing at the refusal itself. A
|
||
changed signature is refused today and **that is a placeholder, not the design**. plan.org's open decision #6 says what
|
||
should happen: a signature change makes a new version of the function, new callers resolve it, existing callers and any
|
||
stored `Fn` value stay safely on the old one, and the session *warns* at each tracked stale caller site. Milestone 7
|
||
names it outright — "signature generations and stale-caller warnings".
|
||
|
||
**The thesis of this project is that you never restart the program.** Every refusal that ends in "restart to change it"
|
||
is a hole in that, and this is the biggest one. It needs three things that do not exist: function versions, a
|
||
trampoline per version, and caller tracking good enough to name the sites. The cell already gives the indirection; what
|
||
is missing is that a cell holds one bare pointer with no signature, so there is nowhere to put a second version.
|
||
|
||
A changed **struct layout** is the genuinely hard case and plan.org still specifies it as a rejection — storage already
|
||
allocated has the old shape and a new body reads its fields at the wrong offsets. Managed classes are the planned way
|
||
through, with an explicit migration at a frame boundary. Do not conflate the two: one is unbuilt, the other is decided.
|
||
|
||
|
||
**From porting ten raylib examples** — the first code the language was pushed by that it was not designed around.
|
||
Ranked by how often they were hit, top two first because they are walls rather than conveniences:
|
||
|
||
1. ~~No number reaches `draw-text`.~~ **Fixed** by `(string b)`.
|
||
2. ~~An enum parameter cannot be driven by a loop variable.~~ **Fixed** by explicit conversions in both directions:
|
||
`(i32 k)` takes an enum to its integer, `(GamepadAxis n)` takes an integer to an enum. Neither is an instruction —
|
||
an enum is an i32 at run time and `emit.ml`'s `cast` already reduced one to that before choosing an opcode — so the
|
||
change is a guard in `check.ml`'s cast arm and nothing in the backend. The rule the refusals came from is
|
||
deliberately *not* relaxed: a bare integer still does not fit an enum parameter, so `:spcae` is still an error at
|
||
the call site. The rule was "an integer must not arrive silently", and a written `(GamepadAxis i)` is not silent.
|
||
The other escape stays closed too — one `declare-c` per C function — and no longer needs to be open.
|
||
- **A value that is no declared member is allowed**, deliberately. raylib's gesture is a bitfield and an OR of
|
||
flags is a legal `Gesture` that is no single member; and `session.ml`'s printer already falls through to the
|
||
number for an out-of-range enum, on purpose, so refusing to construct one while agreeing to print it would be
|
||
incoherent. An `Option` would make every site unwrap for no safety bought, and a literal-only refusal would
|
||
catch nothing, because the bitfield case is a run-time value.
|
||
- **Only an integer converts *to* an enum.** Not a float, and not another enum — a cross-enum hop goes through
|
||
`(i32 x)` so both ends are written down. Enum → any numeric is always allowed: lossless to i32 by construction,
|
||
and a narrower target truncates by the rule every int→int cast already follows.
|
||
- **The comparisons needed nothing else.** `(> (i32 g) 255)` checks because `binary` takes the non-literal side
|
||
first; `binary` was deliberately left ignorant of enums, since teaching it would be the implicit conversion this
|
||
avoids.
|
||
- **A bit-set type later builds on this rather than replacing it.** It would be its own type with its own
|
||
operations and would still want a named escape to the underlying integer for the FFI, spelled the same way. If
|
||
`Gesture` becomes one, the `(i32 g)` calls stay valid and only the range tests migrate to a membership test.
|
||
- One parse fix came with it: `defenum` names were not in `parse.ml`'s type set, so a local enum could not be a
|
||
function's return type. They are in it now under a key of their own, admitted as a bare symbol and never as a
|
||
list head — because `(Key n)` is a *value* now, and putting `Key` in `types` would make a body starting with one
|
||
be eaten as a return type.
|
||
3. ~~**`break` is not implemented.**~~ **Built, with `continue` and loop labels.** Both blockers are answered: the
|
||
`in_frames` rule became a relative one rather than a blanket one, and `Tast.While` grew a latch. See docs/BUILT.md,
|
||
"`break` and `continue`, and the rule that replaced a blanket refusal".
|
||
4. ~~**A `let` binding takes no type annotation**~~ — still true, and **no longer the blocker it was**: `(array 4
|
||
rl/Vector2)` is built and is the answer to the case that raised it. The reasoning below is kept because it is what
|
||
chose between the three surfaces, and the first of them is not what was taken — see docs/BUILT.md, "`(array 4
|
||
rl/Vector2)`, and the one position with no type slot". The original entry:
|
||
|
||
A fixed array is either a top-level `defvar` or a literal with
|
||
every element spelled out. `(let [pts [4 rl/Vector2]] …)` parses as a two-element array literal and fails with
|
||
*unknown name rl/Vector2*. Cost: 32 hand-written `Vector2`s in one example. **Looked at and stopped — it is a
|
||
grammar question, not a missing feature.** Everything under the surface is already there: `Ast.binding` carries a
|
||
`bty`, `load.ml` renames through it, and `check.ml:723` consumes it as the `want` for the value. Only the way it
|
||
is written is open, and the parser says so where it refuses (`parse.ml:366`): `let` is a flat list of pairs, so it
|
||
cannot disambiguate by *count* the way `defvar` and `defconst` do — those read `[n t v]` as three arguments to a
|
||
form, and there is no such boundary between one pair and the next. Three surfaces, in the order they are worth
|
||
considering:
|
||
- **`(zeroed [4 rl/Vector2])` — `zeroed` takes its type as an argument.** Recommended. It is one extra branch in
|
||
the arity-0 `zeroed` case in `check.ml`, no parser change, no ambiguity, and it answers the actual complaint,
|
||
which is not "locals cannot be annotated" but "there is nothing here to infer *from*". It also reads as what it
|
||
does: the value is a zeroed thing of that type, not a name that has been told what it is.
|
||
- **A marker between the name and the type**, `(let [pts :- [4 rl/Vector2] …] …)` or similar. Unambiguous, and it
|
||
buys a general annotation rather than one form's escape hatch. The cost is a new piece of syntax in the binding
|
||
vector, which is the one place this language has kept looking exactly like Clojure's.
|
||
- **Bare `(let [pts [4 rl/Vector2] …])`.** The obvious spelling and the one that cannot work: `[4 rl/Vector2]` is
|
||
a well-formed two-element array literal, and telling the two apart needs types in the parser, which there are
|
||
none of by design.
|
||
Note that plan.org's rule is "annotate function signatures, infer locals", so the general annotation is a
|
||
deliberate absence and not an oversight — which is the other reason the `zeroed` route is the smaller answer.
|
||
5. ~~**Arithmetic is strictly binary** — *+ takes 2 arguments, given 5*.~~ **Fixed.** `+ - * /`, `min`/`max` and
|
||
`bit-and`/`bit-or`/`bit-xor` fold left over two operands or more. `%` and the shifts stay at two, and one operand
|
||
is refused with the form to write instead — there is no unary minus and no reciprocal.
|
||
6. ~~**No `sin`/`cos`/`abs` for floats.**~~ **Fixed.** `sin-f32` and `cos-f32` are `declare`s in the prelude now,
|
||
with the caveat written beside them: IEEE-754 makes `sqrt` correctly rounded and requires nothing of the kind for
|
||
`sinf`, so these are the one place in the prelude where native and wasm32 may disagree bit for bit. Float `abs` is
|
||
not wrapped, for the reason integer `abs` is not — it is `(max x (- 0.0 x))` over two builtins.
|
||
|
||
**A `string` cannot be returned from C at all**, which is what makes `GetGamepadName` unbindable: *a string only
|
||
crosses as a parameter — a C function that returns one returns something Flan has no owner for*. Same rule refuses
|
||
`TextFormat`, which is also variadic and so has no honest signature.
|
||
|
||
**The negative result is worth as much.** None of the gaps expected blocked anything — no generics, no allocator, no
|
||
`Vec`/`Map`, no escaping closures, and function-scoped `defer` never came up. Input-and-draw over fixed-size state is
|
||
the shape the language already has. Three constructs unexercised anywhere else in the repo worked first try: a fixed
|
||
array with a struct element, a 2-D struct array, and `[N string]` as both `defconst` and mutable `defvar`.
|
||
|
||
### The web target: what it does not reach yet
|
||
|
||
`flan build --target=web` works, a raylib example builds unchanged and `test/test_web.ml` is green — see docs/BUILT.md,
|
||
"The browser is the third target", for the mechanism and why asyncify rather than `emscripten_set_main_loop`. Four
|
||
things it does not cover.
|
||
|
||
~~**1. `sand.flan` has no web build, and the cause is one missing `#include`.**~~ **Built. It opens.** See docs/BUILT.md,
|
||
"sand.flan in a browser", for the whole of it. Three summary lines, because the diagnosis below was right about the
|
||
structure and wrong about the cause:
|
||
|
||
- The `#include` was never the fix. **The agent is a socket server and a browser has no sockets**, so an agent that
|
||
compiles there is an agent that can never accept a connection. `vendor/agent/flan_agent.web.c` is three no-ops, and
|
||
`Build` selects it over `flan_agent.c` on `--target=web` and nowhere else.
|
||
- **Refusing `vendor:agent` on web was the honest-looking option and is ruled out by arithmetic.** There is no
|
||
conditional compilation, `sand.flan` calls `agent/start` unconditionally, `Reach` cannot prune a package something
|
||
reachable calls into — so a refusal means the flagship program does not build for the browser at all. A refusal is
|
||
only honest when the caller has a way to not ask. This does **not** reverse decision 2 above: `barf`'s no-op loses a
|
||
file the program believed it wrote, and there is nothing for the agent to lose because `--dev` is already refused by
|
||
name on every wasm target. The argument is written out at the top of `flan_agent.web.c`.
|
||
- **A package's `.c` files can now be addressed to a target**, by a tag in the name before the extension, and a tagged
|
||
file *replaces* the untagged file of the same base name on that target. This is the C-source half of the
|
||
`@native`/`@wasi`/`@web` link-line mechanism decision 2 pointed at for per-package target isolation.
|
||
|
||
The brush is `(embed "brush.png")` decoded through a new `LoadImageFromMemory` binding. `load-texture` and
|
||
`load-image` now have no call site anywhere in this repository — deliberately, because a path-based load is the one
|
||
shape the browser cannot have, and said here so it is not read later as an accident.
|
||
|
||
The original entry follows.
|
||
|
||
**1. `sand.flan` has no web build, and the cause is one missing `#include`.** `vendor/agent/flan_agent.c` does not
|
||
compile under emcc: *variable has incomplete type 'struct timeval'* at line 426, because emscripten's headers do not
|
||
pull `<sys/time.h>` in transitively the way glibc's do. `sand.flan`'s `main` calls `(agent/start ...)`
|
||
unconditionally, so `Reach` cannot prune the package, so the flagship program stops at that error — even without
|
||
`--dev`. Beneath the include is a structural fact worth deciding rather than patching around: **the agent is a socket
|
||
server and the browser has no sockets**, which is the same family as the `--dev` refusal. So the two fixes are not
|
||
equivalent — add the include and the agent compiles into a web build that can never accept a connection, or refuse
|
||
`vendor:agent` by name on a web target the way `--dev` is refused. The second is the honest one. Neither was taken
|
||
here: `vendor/agent/` belonged to another lane this session.
|
||
|
||
~~**2. Assets are two questions and only one of them is about emscripten.**~~ **Answered by the embed above, and the
|
||
answer was the third option neither half here considered: make it a compiler feature and neither question arises.** The
|
||
hard half below is exactly right about the problem — the file that needs the asset is structurally the one file that
|
||
cannot declare it — and the conclusion drawn from it, that the fix must be a link channel or a new declaration, was
|
||
the wrong one. `(embed "brush.png")` needs no channel, because there is nothing to tell the linker. What is *not* done
|
||
is `sand.flan` itself: `(rl/load-texture "brush.png")` takes a path and raylib opens it, so pointing raylib at embedded
|
||
bytes needs `LoadTextureFromImage` over `LoadImageFromMemory`, which is a raylib binding question and not this one.
|
||
The original text follows. `sand.flan` does `(rl/load-texture "brush.png")` against a bare relative path.
|
||
|
||
- The easy half: a bare relative path has no meaning on a target with no filesystem. emscripten's answer is
|
||
`--embed-file` or `--preload-file` into MEMFS, and both are *linker arguments*, so they are already expressible as an
|
||
`@web` line in a package's `link` file. No new mechanism is needed for a package.
|
||
- The hard half, and the actual design question: **the file that needs the asset is structurally the one file that
|
||
cannot declare it.** `Load` hands out `lflags` only for a directory package (`one_file` → `[]`), and `main` is not
|
||
exported, so a program can never be a package. The program doing the `load-texture` therefore has no link channel at
|
||
all. Answering this means either giving a single-file program a way to carry build arguments, or making assets their
|
||
own declaration rather than a linker flag. No flag was invented for it here.
|
||
|
||
**3. Nothing has been opened in a browser.** *Still true, and now it is the only thing left between here and
|
||
"someone played with it".* `sand.flan` builds for the web, the module carries asyncify, raylib's GL imports and
|
||
brush.png's own bytes whole, and `node sand.js` gets as far as `glfwInit` before dying on `window is not defined` —
|
||
which proves the module is live and proves nothing about the canvas. docs/BUILT.md carries the exact commands to serve and
|
||
open it, and the list of what only a human will discover: whether it paints, whether the audio round trip through
|
||
MEMFS survives, and the canvas size. The `until` loop never exits on the web, so none of `main`'s `defer`s run —
|
||
expected, and worth knowing before reading anything into it.
|
||
|
||
The original entry follows.
|
||
|
||
**3. Nothing has been opened in a browser.** The test is headless and permanently so: it asserts the artifact's shape,
|
||
the `asyncify_start_unwind` export and the `glViewport` import, and that node runs the emitted JS. Whether the canvas
|
||
actually paints is unverified by anything in CI, and a human should look once.
|
||
|
||
**4. Unmeasured and untested.** Asyncify's cost is quoted from emscripten's documentation (roughly a doubling of code
|
||
size) and not measured here, and no frame time on web has been taken at all. raylib's audio and any use of threads on
|
||
the web target are untried. And a **wasi** build that reaches raylib now fails on undefined symbols rather than on a
|
||
missing `-l:libraylib.so.550`, because that line is tagged `@native` — the same error one step later, and a worse
|
||
message.
|
||
|
||
### `break`, and why it was not built — **it is built now**
|
||
|
||
Kept as written, because everything in it held and the two blockers at the end are the two things the build had to
|
||
rule on. Both are ruled on in docs/BUILT.md, "`break` and `continue`, and the rule that replaced a blanket refusal":
|
||
the `in_frames` precedent was replaced by a barrier on the loop stack, which refuses a *crossing* rather than
|
||
everything, and `Tast.While` grew the latch. The original note:
|
||
|
||
Settled, so the next attempt is cheap rather than a rediscovery:
|
||
|
||
- **`dotimes` gets it free** — it desugars to `Tast.While`, so one implementation covers both loop forms.
|
||
- **`defer` is a non-question.** It is function-scoped, `break` does not leave the function, nothing fires. No refusal
|
||
needed and no interaction to design.
|
||
- **Type it `Never`**, as `exit` and `return` already are.
|
||
- **`pads` is the structural model.** `emit_while` already makes an `endloop` label; break is a push/pop of that around
|
||
the body plus a `br`. `return` is a direct terminator with no context threading, so there is nothing else to mirror.
|
||
|
||
What stopped it, and neither is small:
|
||
|
||
- **`check.ml`'s `in_frames` rule does not extend.** It refuses `return` inside `handler-bind`/`restart-case` because
|
||
those frames are popped on the way out, and that refusal is blanket because `return` *always* crosses. `break`
|
||
crosses only sometimes — a loop wholly inside a `restart-case` body has a legitimate local break — so the precedent
|
||
has to be replaced by a loop-depth-relative-to-frame-entry rule nobody has ruled on.
|
||
- **`continue` forces a `Tast.While` signature change.** `check_dotimes` folds the step into the body as
|
||
`While (cond, body @ [step])`, so a `continue` branching to the header skips the increment and hangs. It needs a
|
||
latch — `While of expr * expr list * expr list` — across `check.ml` and `emit.ml`. plan.org settles break and
|
||
continue as one item and `parse.ml` refuses them in one case, so building break against today's `While` is exactly
|
||
the thing that would have to be undone.
|
||
|
||
plan.org's single line on it (831) names a `for` the language does not have and gives no mechanism.
|
||
|
||
|
||
1. ~~**Allocators, then `Vec` and `Map`.**~~ **Steps 1, 2 and 3 are done** — the allocator, the arena, `(Vec T)`,
|
||
`StorageExhausted` and `retry`. `Map` is step 4 and is what is left of this item. See *Allocators, `(Vec T)` and
|
||
`StorageExhausted`* in [`docs/BUILT.md`](docs/BUILT.md) for the shape, the three amendments to a frozen `spec-memory.md` and
|
||
the one addition. The claim below held: **`Vec` does not need generics** — that was wrong and is worth un-learning: Odin's containers are compiler
|
||
builtins over a *type-erased* runtime (`base/runtime/dynamic_array_internal.odin`), where `$T` appears only in thin
|
||
wrappers producing `size_of`/`align_of` at the call site, and per-key hash and equality are compiler-emitted
|
||
procedures passed as a runtime argument (`Map_Info`, `base/runtime/core.odin:369`). That runtime is what
|
||
`spec-memory.md` specifies.
|
||
|
||
**The four questions that used to sit here are answered**, in `spec-memory.md`'s "Allocators" section, which is
|
||
frozen along with the rest of that file: when storage is released, the `drop` hook, alignment, and allocation
|
||
failure. Read them there rather than in a second copy here. The one consequence the build order below turns on is
|
||
that no allocating operation returns an error — a failure signals `StorageExhausted` under a `retry` restart — so
|
||
`push` and `put` are `()`, `clone` returns the container, and no signature grows a `Result`. One question is left
|
||
open in that section on purpose; it does not block the build.
|
||
|
||
2. **The editor half of a typed restart.** The language half is in (see "Landed"): `(use-value [v i32] ...)` and
|
||
`(invoke-restart 'use-value 21)` work, and a mismatch is refused at run time with both signatures in the message.
|
||
What is missing is the half only an editor can do — the leverage SBCL lacks. `eval` already compiles and runs an
|
||
expression inside the live program and the daemon already holds the struct layouts, so "ask the human, type-check
|
||
the answer, hand it over" is a short hop, and it is the one path the runtime today *refuses*: a restart with
|
||
parameters taken from the break loop traps, because `flan_break_resume` and `flan_restart_take` aim the channel at
|
||
a frame and have nothing to fill its buffer with. What it needs, end to end:
|
||
- the frame already carries the arity and the signature as a string — `flan_restart_arity` and `flan_restart_sig`
|
||
beside `flan_restart_name`, the same walk, so `restarts` can say what each one takes;
|
||
- `:restarts` on the wire carries the signature per entry, so the minibuffer can show `use-value (i32)` rather
|
||
than a bare name, and `restart-at` grows an `:args` form — a list of *expressions*, since the answer is a Flan
|
||
expression and there is already something that compiles one;
|
||
- the daemon compiles each argument against the declared type with the session's layouts (the same path `C-x C-e`
|
||
takes), refuses it there if it does not fit, and otherwise writes the values into the frame's buffer and marks it
|
||
filled before aiming the channel. That last store is what `flan_restart_take` cannot do today and is the whole
|
||
of the remaining work; the marking exists so this cannot be forgotten silently.
|
||
|
||
3. **`handler-case`.** Not a convenience — it is the fix for the loudest gotcha in `conditions.org`. A handler closes
|
||
over nothing *only because* a `handler-bind` clause runs at the signal point; a `handler-case` clause runs in the
|
||
establishing frame, which is ordinary in-frame code exactly like a `restart-case` clause. SBCL's is `handler-bind`
|
||
plus a transfer and nothing more (`src/code/error.lisp:196-268`). Every piece exists.
|
||
|
||
### `Vec` and `Map` — the order to build them in
|
||
|
||
**Steps 1, 2 and 3 are built; 4 to 8 are what is left.** The reasoning is kept because it is what the remaining steps
|
||
rest on, and because the escape it describes was tested rather than assumed — see *Allocators, `(Vec T)` and
|
||
`StorageExhausted`* in [`docs/BUILT.md`](docs/BUILT.md).
|
||
|
||
The dependency nobody had written down, and the reason it looked worse than it is. `spec-memory.md` defines an
|
||
allocator as "a procedure plus an opaque data pointer" — a function value. `check.ml` refuses function values four
|
||
ways, and all four say milestone 5: a written `(Fn ...)` annotation (`Ast.Tfn`), a written `fn` literal (`Ast.Fn`), a
|
||
`defn`'s name used as a value, and calling anything other than a named function. `(Result T E)` was still refused
|
||
beside those as milestone 6, as `(Map K V)`, `(Handle T)` and `(Vec T)` were when this was written; only `Result` is
|
||
now. Read straight off those lines,
|
||
milestone 6's allocators need milestone 5's function values and the work doubles.
|
||
|
||
**The escape is real and the work did not double** — this is the claim the built thing confirms. All four refusals are about *surface syntax*, and a value the
|
||
compiler builds that no surface form names trips none of them. The compiler already does exactly this, twice:
|
||
|
||
- A `handler-bind` clause is lowered to a function whose address goes into a `flan_handler` and is called back through
|
||
`h->fn(condition, xfer)` (`runtime/flan_rt.c:38` and `:66`). `check.ml` builds that body as its own `Tast.fn`
|
||
(`:619`, `:654`), not as an `Ast.Fn`, so line 458 never sees it, and no Flan type names the result.
|
||
- In a dev build, `emit.ml`'s `call` loads a pointer out of an indirection cell and calls through it
|
||
(`lib/emit.ml:781`–`793`). That is the indirect call line 1023 refuses in source, emitted routinely.
|
||
|
||
It is also what `spec-memory.md` already assumes for `Map`: the hash and equality pair is compiler-emitted and passed
|
||
as a runtime argument. Odin's `Map_Info` is two contextless `proc` fields (`base/runtime/core.odin:369`), and Odin's
|
||
`Allocator` is a `procedure` plus a `data: rawptr` (`:422`) — the same shape, reached the same way. If the hash pair is
|
||
expressible with no function type in the surface language, so is the allocator's procedure.
|
||
|
||
So: **`Allocator` is a builtin opaque type, the way `string` is a builtin ptr+len.** It is a `Types.t` case with no
|
||
user-writable constructor. Its procedure is an ordinary top-level function resolved to a symbol at the emit site, and
|
||
`vec-new`, `push`, `put`, `clone`, `free` and `free-all` are named calls, which `check_call` already routes through
|
||
`named_call` (`check.ml:1021`). **The built-in allocators need nothing from milestone 5.**
|
||
|
||
What *does* need milestone 5 is a **user-written** allocator: the moment a program says "here is my proc, make an
|
||
`Allocator` from it", it needs a `defn`'s name in value position, which is `check.ml:571` verbatim. That is a real
|
||
limit and not a fatal one — Odin ships arena, general-purpose, stack, pool and scratch in its own std, and most
|
||
programs write none. Ship the built-in set; user allocators arrive with function values.
|
||
|
||
**5 and 6 interleave rather than nest.** plan.org orders generics and macros (5) before allocators and containers (6),
|
||
and that order cannot hold: the macro expander is blocked on `Form` being a Flan union and union *values* are milestone
|
||
6 (see "Macros" below). Conditions and restarts, also listed under 6, are already three steps of four. The milestone
|
||
numbers are a topological hint, not a sequence. Take 6's container half first, 5's generics half second, and 5's
|
||
expander last, on 6's unions.
|
||
|
||
~~1. **`Allocator` and the arena.**~~ **Done.** The builtin opaque type, the four operations with `size` and `align`,
|
||
the capability set read off the allocator value, `with-allocator`, `context/allocator`, `context/temp` and the epoch
|
||
counter. `free-all` was decided as retain-capacity with `arena-destroy` beside it; the context is a dynamic variable
|
||
rather than a literal calling-convention parameter; both are stated as amendments in `docs/BUILT.md`. A user-written
|
||
allocator is refused by name with milestone 5 as the reason.
|
||
|
||
~~2. **`(Vec T)`**~~ **Done**, over the type-erased runtime, with `push`, `reserve`, `at`, `len`, `as-slice`, `free` and
|
||
`clone`, and with move-only enforced by a dead set that unions at an `if` or a `match` join. `at` and `len` were
|
||
extended rather than duplicated. The header is six words in *every* build, not four in release — a layout that
|
||
changes with a build flag can disagree silently across the reload boundary — and that is the third amendment.
|
||
Ownership is not transitive yet, so a struct field of `Vec` type, a global `Vec` and a `(Vec (Vec T))` are each
|
||
refused where they are declared, naming `drop` as what they wait on.
|
||
|
||
The note below still stands and is now the only thing between `Vec` and the accumulation pattern: **capture does not
|
||
exist at all.** Nothing about it changed.
|
||
|
||
~~3. **`StorageExhausted` and `retry`**~~ **Done, with step 2 and not after**, exactly for the reason given. It is a
|
||
`while` around a `restart-case` around the attempt, built in the checker out of nodes that already existed, so the
|
||
backend learned nothing about allocation. `test/programs/exhausted.flan` exhausts an allocator for real and takes the
|
||
restart; `exhausted-unhandled.flan` is the same failure with nothing handling it.
|
||
|
||
~~4. **`(Map K V)`**~~ **Done**, following Odin: open-addressed Robin Hood hashing at a 75% load factor, cache-line
|
||
cell packing, and pointer-width integers through the probe loop. `map-new`, `put`, `get`, `has-key?`, and `len`,
|
||
`reserve`, `clone` and `free` extended rather than duplicated. Two departures from Odin, both deliberate: no
|
||
tombstones, because the spec defers removal, which deletes the backward-shift loop entirely; and no capacity tagged
|
||
into the data pointer, because this header has room and tagging would make correctness depend on an alignment that
|
||
is only requested. `Tast.FnAddr` carries the emitted hash and equality pair and is not a function value — the same
|
||
escape the allocator used.
|
||
5. **`drop`.** The hook, the transitive move-only and non-`clone`able rules, and the refusal to construct a
|
||
`drop`-carrying value against an allocator without `can-free`. It is additive — no type in the repo has a hook today
|
||
— but the `can-free` refusal has to land with the construction path it guards, before any arena-allocated container
|
||
of a user struct is trusted.
|
||
6. **`(Result T E)` and `try`, then the rest of union values.** Unions are what `Form` needs, and `Form` is what the
|
||
macro expander needs.
|
||
7. **Generics and monomorphisation**, then function values. User-written allocators and escaping closures both fall out
|
||
of the second.
|
||
8. **The macro expander**, last, on 6's unions.
|
||
|
||
**What is genuinely unsettled.**
|
||
|
||
- **The Map is slower than CPython's dict at a million entries** (1.41s against 1.16s on the same workload), while
|
||
being six times quicker cache-resident (21ns against 132ns per lookup at 10k entries). Both are memory-bound at the
|
||
larger size and this layout waits longer: keys, values and hashes are three separate runs, so a lookup that misses
|
||
everything costs three cache misses where a compact dict costs two, and the hash run is a full eight bytes a slot.
|
||
Cell packing buys probe locality, which is a win while the hash run is resident and a loss once nothing is. One byte
|
||
of metadata a slot — the Swiss-table arrangement — is the known answer and is not built. Worth measuring before
|
||
building: the crossover is somewhere between 10k and 1M and nobody has found it.
|
||
- **What is left at 18ns cache-resident is the type erasure itself** — one non-inlinable call into the runtime and two
|
||
non-inlinable indirect calls to the hash and equality pair. That is the trade `spec-memory.md` chose on purpose, and
|
||
monomorphisation is what would buy it back. It is a reason to want generics, not a reason to regret the choice.
|
||
- **A fixed array of structs or of strings is not a map key**, which is narrower than `spec-memory.md`'s key set. It
|
||
needs the per-element walk a struct key gets, driven by a loop rather than a field list. Refused by name rather than
|
||
written untested; a struct holding the array works today.
|
||
- **Map removal is not built**, which is what keeps the implementation free of tombstones and of Odin's backward-shift
|
||
loop. The spec defers it deliberately. When it arrives, that loop is the cost.
|
||
|
||
- `spec-memory.md`'s "Open: catching a use-after-release statically" is still open, and it is now open with evidence
|
||
available for the first time: the epoch trap is built and `test/programs/stale-region.flan` is the case it catches.
|
||
What the spec says would settle it — real Flan programs using arenas, to show whether the escapes that actually occur
|
||
are lexical — is now *producible*, because there is a `Vec` to write them with. That is the next thing to look at,
|
||
not the next thing to build.
|
||
- ~~**The operation table may be one operation short.**~~ **Decided.** `free-all` is retain-capacity and
|
||
`arena-destroy` hands the pages back — two names rather than the mode parameter, so the table the spec froze at four
|
||
operations did not grow. `docs/BUILT.md` states it as the amendment it is.
|
||
- **The `Vec` header is six words in release too, and should not stay that way.** The 32-byte layout the spec fixes is
|
||
blocked on one thing: a redefinition module is built by `llc` and `ld` against a host built separately, and nothing
|
||
makes the two agree on a struct size. Give the reload path a way to carry the build flags and this falls out.
|
||
- **The generation word has no reader.** It is bumped on every reallocation as specified, and the stale-slice trap it
|
||
exists for needs a slice that can carry the Vec's identity — a slice is ptr+len. Either slices grow a word in a dev
|
||
build or the trap does not exist; today it does not.
|
||
- **The allocator grew a budget** (`alloc-budget` / `set-alloc-budget`), which `spec-memory.md` does not have. It is
|
||
there because `retry` is only answerable by a handler that can make the *same* request succeed, and for a fixed
|
||
backing store that handler is the one that raises the ceiling — releasing the region the container lives in
|
||
invalidates the container. Worth folding into the spec or replacing with a growable arena.
|
||
- Escaping closures are still deferred (`spec-memory.md`, "Function values", case 3), and a user-written allocator is
|
||
not one — its procedure is a top-level `defn` with no captured environment. The two should not be conflated when
|
||
function values arrive.
|
||
|
||
### Bugs found and not yet fixed
|
||
|
||
- **Two citations in `spec-memory.md`'s Allocators section do not land where they say.** Checked against Odin
|
||
`819fdc7a8` and Carp `ea121b5a`, every other one is exact — `Map_Info` at `base/runtime/core.odin:369`,
|
||
`Allocator_Proc` at `:422`, the arena answering `.Free` with `.Mode_Not_Implemented` at
|
||
`core/mem/allocators.odin:307`–`308`, `#optional_allocator_error` on `append_elem` at
|
||
`base/runtime/core_builtin.odin:767`, and `// TODO(bill): Better error handling for failed reservation` at
|
||
`base/runtime/dynamic_array_internal.odin:107` and `:128`. The two that miss: `Map_Cell_Info` is at `core.odin:351`,
|
||
not `:350`; and `check.ml:1670` is the FFI `Declare` arm, not the `defer` registration — the claim it is offered for,
|
||
that a top-level `defer` is checked in a scope holding only parameters and globals, is true and lives in `check_fn`
|
||
at `check.ml:1800`–`1812`. `check.ml:505` is the `defer` refusal exactly as cited, and the Carp citations are right:
|
||
`getDropFunc` is `Memory.hs:804`, the drop-before-delete emit is `Emit.hs:1044`, and `docs/Drop.md` says outright
|
||
that `A.drop` "will be run ... when the `let` scope ends".
|
||
|
||
- ~~`web/examples/breakdemo.out` is stale and `check.sh` fails on it.~~ **Fixed.** Commit `4a6a8fa` made the break
|
||
banner number its restarts and the `.out` was never repinned. Nothing had to drive the socket in the end:
|
||
`check.sh` already builds this one `--dev` and runs it under `timeout 5`, keeping what it printed before it
|
||
stopped, so the repin was the `.out` plus the two prose copies of the banner — `web/index.html` and `docs/BUILT.md` —
|
||
and a sentence on the page saying what the numbers are for, since a restart is taken by position.
|
||
|
||
- ~~A shadowed restart is offered and cannot be taken.~~ **Fixed.** A restart is taken by *position* now:
|
||
`(:op "restart-at" :index N :name NAME)` on the daemon, `restart-at N NAME` on the agent, and a numbered
|
||
`completing-read` in `C-c C-b`. `:name` is a receipt, not the lookup — it is checked against the name the snapshot
|
||
holds at that index and refused if the two have drifted, so a bare integer can be wrong out loud. `restart <name>`
|
||
survives for a raw socket and is now defined as `restart-at` on the first index offering the name, so the two verbs
|
||
cannot disagree. `break.flan` grew the shadowed pair and asserts 900, which is the only value in that file no by-name
|
||
lookup can produce. The C&R buffer still marks the shadowed row by name and could now offer it instead — small, and
|
||
not done here.
|
||
|
||
- ~~A restart chosen at a break inside a thunk is accepted, announced, and silently not taken.~~ **Fixed by refusing
|
||
it, with the reason.** Not by the depth NEXT.md proposed: recording the restart-stack depth on *entering the break
|
||
loop* counts the frames a `restart-case` inside the thunk pushed before it erred, and those are above the boundary
|
||
and work. The boundary is where it is made — `restart_floor` is set to `flan_restart_count()` around `j.call()` in
|
||
`flan_agent_poll`, saved and restored so thunks nest — and the outermost `floor` entries of the snapshot are marked
|
||
unreachable. They are listed and marked rather than hidden, refused by the listener before the reply, and carried to
|
||
the editor as `:unreachable (2 3)`. `test_dev.ml` breaks a stopped program a second time from inside `C-x C-e` and
|
||
asserts both halves: index 2 refused, index 0 taken.
|
||
|
||
- ~~Restart names are served from a stack that is being mutated.~~ **Fixed, and it was a precondition rather than a
|
||
separate bug.** Index-based resume is wrong by construction against a moving stack: unlike a name, an index carries
|
||
no evidence of what it meant. The agent copies the list on entering `break_loop` — names into its own buffer, frames
|
||
as the addresses a transfer carries — one snapshot per nested break, and every verb answers from it. Caps are
|
||
`SNAP_MAX` 64 restarts and `SNAP_NAMES` 4096 bytes; past either, the listing says how many it did not show. Neither
|
||
cap has a test; the 4K result cap that shared that blind spot now does.
|
||
|
||
- **A snapshot generation has no test, and the window is a race.** A choice is validated against the snapshot on top
|
||
when the request arrives and resolved against the snapshot on top when the game thread next looks. Between those,
|
||
an evaluation the break loop is running can error and push a break of its own, whose loop would otherwise reach
|
||
[chosen_ready] first and take *its* index 2 for the one someone chose from the outer list. Each snapshot now carries
|
||
a generation, a choice is stamped with the one it was validated against, and a loop claims only what is addressed to
|
||
it — a mismatch is left set rather than discarded, because the listener already answered ok for it. Depth would not
|
||
do: an outer break resuming and a new one starting reuses the number. None of this is tested, because arranging the
|
||
window means landing a request inside a two-millisecond poll from outside the process. It wants a hook the test can
|
||
drive, not a sleep.
|
||
|
||
- ~~The job ring has no fullness check.~~ **Fixed by refusing, at the sender.** Dropping loses a reload the sender was
|
||
told was ok; blocking stalls the accept loop, which serves connections inline, so a program that had stopped polling
|
||
would also stop answering `status` and `abort`. The refusal happens before the `dlopen`, so a module there is no room
|
||
for is never relocated and no handle is taken for it. `programs/agent-queue.flan` blocks on stdin so the window is
|
||
held open by the test rather than by a timer: 64 queued, the 65th refused with a reason, 64 installed when it finally
|
||
polls.
|
||
|
||
- ~~`flan_dev_result_get` is not the seqlock its comment claims.~~ **Fixed by making it one**, rather than by writing
|
||
the honest comment — what it guaranteed was nothing, and the daemon has no other way to read a result. The counter is
|
||
odd while a value is being written, `flan_dev_result_read` copies into the caller's buffer and checks the counter
|
||
either side of the copy, and a reader that loses the race reports the last complete generation and no bytes. The
|
||
count handed out is the number of complete values, so `lib/dev.ml`'s "has it moved" still means what it meant. The
|
||
race itself has no test, for the same reason the snapshot generation above has none.
|
||
|
||
- ~~Smaller: `exit(134)` from the break loop with the listener inside `dlopen`; a `dlopen` handle leaked when a module
|
||
has no installer.~~ **Both fixed.** `exit` runs the atexit chain and the ELF destructors, which want the loader lock
|
||
the listener may be holding — a program asked to abort would hang instead of dying; `_exit`, with the streams flushed
|
||
by hand. The leak was the handle *value* and not the mapping: a module with no installer published nothing, so
|
||
nothing can point into it, and it is closed. The deadlock is read rather than tested; the exit status is tested.
|
||
- **`rt_die` in `flan_rt.c` still calls `exit(134)`**, which is the shape just fixed in the break loop: a trap on the
|
||
game thread runs the atexit chain and the ELF destructors, which want the loader lock the agent's listener thread may
|
||
be holding inside `dlopen`, so a program that should die could hang. Found while fixing the break loop and not fixed
|
||
with it — `rt_die` is the non-dev path too, where there is no listener and nothing to deadlock against, so whether it
|
||
should be `_exit` unconditionally or only under `--dev` is a decision rather than a typo.
|
||
|
||
- ~~**`(A {.x 1})` on a union variant says "unknown struct A"**~~ **Fixed** with union values. `env` now carries a
|
||
case table keyed both by the full spelling `U.C`, which is how a value of it is written, and by the bare `C`, which
|
||
is how a mistake spells a constructor; the bare entry exists only to say *"A is a case of the union U, not a struct
|
||
— a union value names both, as `(U.A {.field value ...})`"*. Two unions may share a case name and that is not
|
||
refused: construction is qualified and a pattern resolves against the scrutinee, so both are unambiguous.
|
||
|
||
### Test blind spots, from a mutation pass
|
||
|
||
Sixty mutations, nineteen left the whole suite green. The severe cluster was closed first (`cleanup.flan`,
|
||
`signedness.flan`); the rest are closed now. Every one below was re-planted, watched leave the suite green, and then
|
||
watched fail against the new test before the mutation was reverted — a test nobody saw fail is not evidence.
|
||
|
||
- **`Reach`'s walk of index expressions, `addr` places and `restart-case` clause bodies.** `programs/reach-walk.flan`
|
||
calls three functions from three places that are each the only route to them. The failure is not a wrong answer:
|
||
the function is not emitted and the program stops linking, so the case catches the build exception rather than
|
||
comparing output. The `addr` case goes through a `deref` place deliberately, so the index case cannot stand in for
|
||
it.
|
||
- **`flan_dev_global`'s size-change guard.** `programs/reload-v5.flan` is v4 with `extra` as an `i32`, loaded on top
|
||
of v3 in a host run of its own, because what it does is abort. The message is asserted next to the exit status: a
|
||
process that died for another reason is not this guard firing.
|
||
- **A local shadowing an imported name.** `programs/shadow-pkg.flan` binds locals over its own constant and var;
|
||
`pkg-shadow.flan` prints four numbers that separate the expression renamer from the place renamer. Nothing refuses
|
||
a renamer that qualifies through a binding — it reads the top-level name instead and runs — so only the number says
|
||
so.
|
||
- **The 4K result cap and the registry overflow guard.** `test/dev_limits.c`, a second C main beside `reload_host.c`,
|
||
drives them directly: neither has a Flan spelling and no corpus program reaches either. One process per mode — the
|
||
name table never shrinks and the overflow case aborts.
|
||
- **The reader's unknown string escape, and `+5`.** Rows in the reader table, with the escapes it *does* know
|
||
asserted on their decoded bytes rather than through `Form.to_string`, which escapes them again and would compare
|
||
the source with itself.
|
||
- **The hang.** A reader branch that forgets to advance loops for ever, and `dune test` waits as long as it is left
|
||
to; in CI that is a job the runner kills with nothing named. `test/watchdog.ml` arms an alarm on every test binary
|
||
— generous, because an alarm that fires on a slow machine is a flake and a flake is how a watchdog gets deleted —
|
||
and a five-second one around every read in `test_flan`. The first read that does not return wedges the rest, so a
|
||
looping reader costs five seconds and names the row instead of never finishing.
|
||
|
||
What is still open here: the mutation pass has not been re-run since, so the count of nineteen is the old one. The
|
||
sanitized sweep (`@sanitize`) is under the same watchdog but has never been observed to fire it, and so is the
|
||
memcheck sweep (`@valgrind`), whose alarm is looser at 5400s because memcheck is 20-50x on execution.
|
||
|
||
### Asked for by the editor lanes
|
||
|
||
- **`(:op "condition")` → the stopped program's condition, rendered.** Two steps: `break_loop` currently does
|
||
`(void)condition;` and *discards the pointer*, so stash it beside `condition_name`; then the daemon builds a render
|
||
thunk aimed at that address, which is `Session.render` rooted at a `Ptr` instead of an expression. **The second step
|
||
now exists** — `Session.render_locals` is exactly that thunk, rooted at an address the program supplies — so what is
|
||
left is the first: keep the pointer, and give the agent a verb that hands it back. The type is already known: it is
|
||
the `condition_name` the break loop reports, which `layout` already resolves.
|
||
- **The type identity is settled, and it is the qualified name** — `layout` is in, see docs/BUILT.md. `Load` qualifies
|
||
every declaration at import, so the names in `Tast.structs` are a flat namespace where two packages' `Missing` are
|
||
`a/Missing` and `b/Missing`; a bare name is refused with the candidates rather than resolved. `condition` inherits
|
||
it for free: the string the break loop already reports *is* that name, because `Emit.struct_name_of` writes
|
||
`Types.Named` into `flan_error`. It is still open for **locals**, where DWARF gives a name and the name a debugger
|
||
reads is not qualified by anything.
|
||
- ~~**`(:op "backtrace")` is blocked** on frame metadata.~~ **Built**, and not out of DWARF: decision 3's shadow
|
||
stack carries the name and the location on the frame itself, so a backtrace needs no debug information at all. See
|
||
docs/BUILT.md, "The shadow stack, and `backtrace`", for what it costs. **Locals landed with it** — the pointer-rooted
|
||
render thunk turned out to be `Render.render` over a `Deref` of a slot's address, and one new arm in the backend.
|
||
See "Locals of a stopped frame" for the four things it refuses. Restart source locations and arity are still blocked — `flan_restart` carries `prev`, `name_id`, `name` and `namelen`, so both need a new
|
||
field in the frame, which means the compiler emitting it.
|
||
|
||
### One line away
|
||
|
||
- **`match` over enums.** Fully desugarable, wanted, and blocked only by `Ast.pattern` needing a keyword case, which
|
||
`load.ml` matches exhaustively.
|
||
- **`Build.executable` returns only `out`**, so the daemon recovers the host `.ll` by recomputing `Build.workdir ()`.
|
||
- **A `!DILexicalBlock` per `Let`.** Not one line, but the one thing left in the DWARF work: every `!DILocalVariable`
|
||
is currently scoped to the subprogram, so inside `(let [v 22] …)` nested in `(let [v 11] …)` lldb still answers
|
||
`p v` with **11**. The `~2` suffix makes both *visible*, which is not the same as making the answer right. It needs
|
||
block structure the typed IR does not carry, and the `llvm.dbg.declare`s moved out of the entry block.
|
||
|
||
### Deferred with a reason
|
||
|
||
- **Writing through a string literal** — see Sharp edges. Needs provenance, which is open decision #3.
|
||
- **`cstring` as a type.** Odin has no `string → cstring` conversion at all; it pays the same copy our shim already
|
||
makes. The one thing it buys is the *return* direction, and nothing in `vendor/raylib` returns a string.
|
||
- **`rune`.** Odin's is a 4-byte integer distinguished by a flag, so `i32` is the same thing. Non-ASCII text is
|
||
blocked on font loading, not on the string layer — and fonts are now bound.
|
||
- **Macro expansion.** The reader and the declaration are in. Running a macro means compiling it and `dlopen`ing it
|
||
into the compiler, which is the reload primitive pointed at ourselves — but a macro is `[Form] -> Form`, so `Form`
|
||
has to be a Flan union whose layout the compiler and the loaded macro agree on, and union *values* are milestone 6.
|
||
|
||
### Documents that contradict the code
|
||
|
||
- **`plan.org`'s jank #947 citation is wrong in its mechanism.** jank does not relink (it calls through vars, which
|
||
are already indirection cells) and never unloads (`remove_symbol` has no callers). The real cause was a
|
||
process-teardown race. We are safe from the repro — because we compile out of process, not because of cells. A
|
||
normative document citing the wrong mechanism protects the wrong invariant.
|
||
- **`plan.org` still lists open decision #7 as open** and the interpreter as a backend. It was settled the other way;
|
||
`NEXT.md` records the consequences as "already applied" to `plan.org`, and they never were.
|
||
- **nREPL's `eval` does carry `file`, `line` and `column`** — jank reads all three. The choice of s-expressions still
|
||
stands on its other grounds; the stated reason does not.
|
||
|
||
## Sharp edges
|
||
|
||
- **Two formatted numbers cannot be held at once.** `flan_i64_to_bytes`, `flan_f64_to_bytes` and `flan_u64_to_bytes`
|
||
all write into one `static char scratch[64]` — "rendered text lives here until the next call", flan_rt.c:184 — and
|
||
`(string b)` does not copy. So
|
||
|
||
```
|
||
(let [a (string (i64->bytes 11))
|
||
b (string (i64->bytes 22))]
|
||
(print a) (print " ") (println b)) ; => 22 22
|
||
```
|
||
|
||
`a` is 11 and prints 22. No crash and no diagnostic. This is not new — the `[u8]` already aliased — but a `string`
|
||
reads as more value-like and invites exactly this. Format, draw, measure, then format the next one; `digits.flan`
|
||
sequences itself strictly for this reason. `rl/draw-text` is safe because the shim's `flan_shim_cstr` copies out of
|
||
ptr+len before the call.
|
||
|
||
**The prelude now has the shape that does not have this problem**, and it is the reason that shape exists.
|
||
`append-i64!` and `append-f64!` copy out of the scratch buffer into a `(Vec u8)` before returning, so a builder
|
||
holds as many rendered numbers as it likes, and `format-f64` answers a `Vec` rather than a view. The hazard is
|
||
unchanged for anyone calling `i64->bytes` directly — nothing was taken away — but a caller assembling a line of
|
||
text has a way not to meet it.
|
||
|
||
- **Writing through a string literal is undefined, and the two build modes
|
||
disagree about how.** `(let [s (bytes "Hi")] (set (at s 0) \h))` stores into
|
||
a `private unnamed_addr constant`. At `-O0` that is a store to read-only
|
||
memory and the program takes SIGSEGV; at `-O2` LLVM deletes it as undefined
|
||
and the program prints `Hi` and exits 0. Same source, and which way it fails
|
||
depends on a flag — the worst shape available, and worse than either outcome
|
||
alone.
|
||
|
||
Nothing refuses it. `bytes` turns a `string` into a `[u8]`, the language lets
|
||
you write through a slice, and by then nothing records that the bytes came
|
||
from a constant. The honest fix is provenance — knowing a slice's origin —
|
||
which is plan.org open decision #3 and deliberately deferred. A cheaper one
|
||
that is *not* a fix: emitting literals as mutable globals only moves which
|
||
flag misbehaves, and costs their read-only placement.
|
||
|
||
Found by the string lane while deciding whether `lower-ascii` should mutate
|
||
in place. It ships the copying version for exactly this reason, and that is
|
||
the rule to follow until provenance exists: **a function over a `string` must
|
||
not write through it.**
|
||
|
||
|
||
Most of these are edges the language keeps and you should know about. Two — the top-level namespace and the shift count,
|
||
both found by review after milestone 4 — were bugs that reached LLVM or ran wrong, and are **fixed**; each says so. They
|
||
stay written down because each one is now a rule the checker enforces, and a later change could quietly drop it.
|
||
|
||
- **An index converts from a narrower integer and never from a wider one.** `(at colors current-color)` with a `u32`
|
||
index works — anything above 2³¹ truncates to a negative `i32` and the unsigned bounds check rejects it. An `i64` index
|
||
is refused with the reason: 2³²+5 truncates to 5 and would read the wrong element with no trap at all.
|
||
- **There is one top-level namespace, and `check.ml` now enforces it.** The environment's tables are per-kind — structs,
|
||
unions, aliases, enums, functions, externs and globals each have their own — so only a function was ever checked for a
|
||
duplicate. `(defn item …)` beside `(defvar item …)` type checked and then died in LLVM as `redefinition of function
|
||
'@flan.item'`, a message about an emitted symbol with no source location left, and two colliding *type* declarations
|
||
were not caught anywhere. One pass over `Ast.declared_name` now runs before every other collection pass and rejects the
|
||
second declaration of a name whatever kind either one is. `declared_name` lives in `ast.ml` because `Load` needs exactly
|
||
the same set — the names an import renames — and two copies of that list would drift.
|
||
- **A shift count is bounded, two different ways.** A shift by the operand's own width or more is *poison* in LLVM, not
|
||
a wrong number: `(defn main [] i32 (<< 1 32))` compiled at -O2 to a bare `retq`, returning an undefined value. A literal
|
||
count out of range is now rejected in `check.ml` — that is the typo case — and `emit.ml` masks a computed count to
|
||
`width - 1`, which is what the hardware does anyway and which LLVM folds away whenever the count is constant. The
|
||
prelude's rotate masks its own count; that is now redundant but harmless.
|
||
- **A `u64` literal is its 64-bit pattern**, so `0xcbf29ce484222325` is a real `u64` and not an error. The cost is that
|
||
a negative *decimal* literal is accepted as a `u64` too, because the reader records the value and not how it was
|
||
written. Narrower unsigned types keep the strict check, which is where a typo like `300` for a `u8` actually shows up.
|
||
- **A folded constant skips `check`.** `(defconst rows (/ h c))` is emitted from the folding pass's value, because a
|
||
global's initialiser has to be a compile-time constant and only that pass knows this one is. Its range check is
|
||
therefore its own call to `in_range`; there is a regression test.
|
||
- A `let` binding takes no type annotation, which is why `sand.flan` names its FNV constants instead of writing them
|
||
inline.
|
||
- `(defn f [] f65 0.0)` still says *unknown name* rather than *did you mean f64*: with a single body form the parser
|
||
cannot tell a return type from the first expression. Only the parameter position and `(Option …)` are unambiguous.
|
||
|
||
## Loose ends from milestone 4
|
||
|
||
None of them blocking: block-scoped `defer`; package visibility, so `rl/get-color-raw` is not callable; imported
|
||
unions.
|
||
|
||
A package importing a package was on this list and is off it. It loads, a diamond shares one copy of the bottom
|
||
package, the alias clash is refused through a chain as well as inside one file, and a ring is refused by name. What is
|
||
left of the item is visibility, which is listed above and needs a marker the parser does not have.
|
||
|
||
## Macros — landed; what is left of them
|
||
|
||
**The expander works and `unless` is a prelude `defmacro`.** How all of it fits together is in
|
||
[`docs/BUILT.md`](docs/BUILT.md), "Macros: the compiler dlopens the program" — the image format, the thunk ABI, why quasiquote
|
||
runs before the walk, the two different ways expansion fails to terminate, `-linkall`, and the three cost numbers.
|
||
What follows is only the part that is still missing.
|
||
|
||
- **Four special forms left**, and two of them are the hard ones. `until` and `cond` are free to move whenever
|
||
somebody wants them. `when` and `dotimes` are not: the prelude itself uses them 29 and 12 times, so moving either
|
||
makes the prelude depend on the macro that the macro module has to compile the prelude to get. Breaking that needs
|
||
either a prelude that stops using them, or a two-stage prelude where the macro module is built from a subset. The
|
||
first is a mechanical edit of `prelude.ml` and is probably the answer.
|
||
|
||
`cond` has its own snag, and it is the reason `unless` went first: `parse.ml` refuses `(cond a)` with "cond clause
|
||
has no body", and a macro cannot produce that (see the next item), so moving `cond` changes an existing test.
|
||
|
||
- **A macro has no error facility**, and this is the biggest gap. A macro runs inside the compiler; anything it
|
||
signals aborts the compile with no location. So the prelude's `unless` answers `(unless-takes-a-test-and-a-body)`
|
||
when it is handed fewer than two forms, and the report is "unknown name unless-takes-a-test-and-a-body" at the call
|
||
site — right place, wrong sentence. What a macro wants is a way to say *this is wrong and here is why*, reported at
|
||
the call site. The queued structured-error rewrite is where that belongs, and the call site's `Loc.t` is already
|
||
stamped onto everything a macro returns, so the location half is done.
|
||
|
||
- **Macros are not imported.** A `defmacro` in a package is refused by name in `load.ml`. Reaching one would mean
|
||
resolving that package's own imports over `Form`s, before `Load` runs — a second import resolver. `programs/pkg-macro.flan`.
|
||
|
||
- **A prelude macro may not call a macro.** The prelude is in every macro module by construction, so there is no
|
||
round it could be compiled in after something else. It would fail with an unknown name rather than with a reason,
|
||
which is worth fixing the day the prelude wants one.
|
||
|
||
- **A prelude *function* may not call a prelude macro either**, which is the neighbouring gap and was found by
|
||
walking into it. `Macro.program` runs over the file being compiled; the prelude arrives at the checker through
|
||
`Check.program`'s own prepend and is never handed to the expander at all. A `defmacro` is an ordinary `defn` taking
|
||
one `[Form]` by the time the checker sees it, so the call resolves to that and the report is "clamp takes 1
|
||
argument, given 3" — pointing at the prelude, about a call the author wrote as a macro use. `format-f64` writes
|
||
`(min 9 (max 0 prec))` in place of `(clamp prec 0 9)` because of it. The fix is not obviously cheap: expanding the
|
||
prelude means building a macro module to compile the prelude that the macro module is built from, which is the same
|
||
bootstrap the `when`/`dotimes` item above describes.
|
||
|
||
- **A quasiquote inside a quasiquote is refused.** Nothing counts nesting levels — not the reader, deliberately, and
|
||
not the desugaring. Only a macro that writes a macro wants one.
|
||
|
||
- **`gensym`'s counter restarts in a second module.** It lives in the loaded module, and a module is dlopened once per
|
||
compiler process, so it is process-wide in practice. The rounds already build more than one module for a program
|
||
whose macros call macros, and the fix that day is to seed the counter from the module's index.
|
||
|
||
- **The macro programs are not in the sanitizer sweep.** `test_sanitize.ml` runs an explicit list, not a glob, so
|
||
`macros.flan` and `macro-unless.flan` were not added to it by landing them. `dune build --root . @sanitize` is
|
||
clean as it stands; adding the two is a one-line edit in a file this lane did not own.
|
||
|
||
- **No `&rest` sugar.** A macro takes one parameter, the slice of forms at its call site, and `(len args)` is the
|
||
arity. That is deliberate — it is where variadics come from — but a `when` written against it reads worse than
|
||
`parse.ml`'s version did.
|
||
|
||
## Watch for
|
||
|
||
The rule that caught the two misparse bugs applies unchanged: **anything that binds a name, alters control flow, or is
|
||
not yet implemented must be recognised explicitly and rejected if unsupported.** `check.ml` rejects `Vec`, `Map`,
|
||
`Result`/`try`, union values, closures, quoted symbols, generics and function values *by name*, each with the milestone
|
||
it belongs to; `load.ml` rejects the package shapes it does not handle; and the FFI boundary rejects an aggregate. The
|
||
tests assert on the reason, not just on the failure.
|
||
|
||
## Untracked on purpose
|
||
|
||
`calc-me` and `sand`, the executables `flan build` drops beside their sources, are now in `.gitignore` — anchored
|
||
(`/calc-me`, `/sand`) so the patterns cannot match anything nested.
|
||
|
||
`old-ocaml/` — the pre-rewrite menhir/ocamllex frontend, kept as reference and excluded from the build by the root
|
||
`dune` file. Its contents are also in git history at `2c232dd`.
|
||
|
||
## Handoff: the shadow stack lane, stopped mid-repair
|
||
|
||
Two commits landed and are green: the shadow stack with `(:op "backtrace")`, and `(:op "locals" :frame N)`. See
|
||
docs/BUILT.md's two new sections for the design and the measurements. A third commit was **half-built and its own test
|
||
left red** on purpose; **it is finished now** — see the struck item 1 above — and the rest of this section is kept
|
||
because the parts of it that were true are still worth having.
|
||
|
||
**What is broken, exactly — and this paragraph was wrong; kept for what it cost.** It said `locals` compares the
|
||
frame on the stack against the body this session holds and the comparison is not firing, that every piece of the
|
||
fingerprint was written, and that one of five hand-offs was dropping the number. Four of the five were never written at
|
||
all: `Emit.fninfo` stored the fingerprint and nothing else touched it. The first step it recommended — printing both
|
||
sides of the comparison in `Dev.locals` — could not have worked, because `Dev.locals` had no comparison to print. The
|
||
lesson is the ordinary one: a lane that stops mid-repair should say which pieces it *ran*, not which it believes it
|
||
wrote.
|
||
|
||
**Not obvious from the diff.** Two things cost a day between them. The linked-list frame beat an array-with-a-stack-
|
||
pointer on both benchmarks, which is the opposite of what the escaping-alloca argument predicts, and the measurement
|
||
that first said otherwise was comparing a 40-frame binary with a 600-frame one; every number in docs/BUILT.md is now a
|
||
minimum of nine runs for that reason. And `redefinition`'s transient rule (`m.nstr = 0`) silently stops every module
|
||
carrying a string literal from ever being unloaded — the frame descriptors go through their own counter, `m.nfi`, for
|
||
that reason, and a locals thunk passes `~retains:false` because everything it emits is memcpy'd into the result buffer.
|
||
|
||
**No Emacs surface.** `backtrace` and `locals` are daemon ops; nothing in `emacs/` calls them yet. One command showing
|
||
the backtrace with the selected frame's locals is the whole of what is missing, and `flan-cnr.el`'s
|
||
fixture-driven shape is the model.
|