2089 lines
166 KiB
Markdown
2089 lines
166 KiB
Markdown
## Decided by the author, 2026-09-13: a type variable takes a `$` sigil
|
||
|
||
**This overrides plan.org**, which says under Types: *"Lowercase type names are variables, Capitalized are
|
||
concrete — no sigil."* That rule is withdrawn. Taken as provisional, to make progress rather than as a final
|
||
design, and expected to be revisited after some real use.
|
||
|
||
```lisp
|
||
(defn swap [xs [$t] i i32 j i32] () ...)
|
||
```
|
||
|
||
Three reasons, the third being the one that actually forced it.
|
||
|
||
1. **There is no binding site without a sigil.** Nothing distinguishes where a variable is introduced from where
|
||
it is used.
|
||
2. **Introducing one is invisible.** `check.ml:535-562` resolves builtins, aliases, structs, unions, enums, then
|
||
near-miss, and *then* treats any leftover lowercase name as a type variable. So a lowercase name nobody has
|
||
heard of silently becomes a type parameter, and a mistyped type makes the function **more** permissive instead
|
||
of failing. The `near_miss` guard at line 553 exists for exactly this hazard and only reaches names close to a
|
||
known one — the comment there says so. `[elem]` is not close to anything and would become generic silently.
|
||
3. **Brackets carry two opposite rules for the same lexical thing.** In `[n t]`, `check.ml:563` resolves `n`
|
||
against `env.consts` and *errors* if it is absent, while `t` becomes a *variable* if absent. Absence means
|
||
"mistake" on one side of the bracket and "new parameter" on the other.
|
||
|
||
The rule is also not the one plan.org states. It is not case-based: `i32`, `f32`, `bool` and `string` are all
|
||
lowercase and concrete. What the code actually does is **treat any unresolved lowercase name as a variable**,
|
||
which is an absence rule, and absence is a bad thing to give meaning to.
|
||
|
||
**Open, and deliberately not decided here:** whether a *use* after the binding is `$t` or bare `t`. Odin binds with
|
||
`$T` and uses bare `T`. Left to whoever builds it, to be reported with the reason.
|
||
|
||
**Also open: `$n` in length position**, Odin's `$N: int`, which is what would make a function generic over array
|
||
length rather than only element type. Not in the spike unless it falls out for free; the spike is to report what it
|
||
would cost.
|
||
|
||
## To discuss: five gaps the raylib examples hit and could not close
|
||
|
||
Found by the lane that ported `core-2d-camera`, `core-scissor-test`, `core-window-flags`,
|
||
`core-world-screen` and `core-window-should-close`. None blocked those five. All five blocked
|
||
something else, and each is a language or checker question rather than a missing binding, which is
|
||
why they are here and not in a binding list.
|
||
|
||
1. **A `(Ptr T)` returned from C cannot be indexed.** `indexed` in `lib/check.ml` accepts `Array`
|
||
and `Slice` only, so a C function answering `int*` is readable at element 0 through `deref` and
|
||
nowhere else. `LoadRandomSequence` is the case that hit it and `core_random_sequence` is
|
||
unportable until it moves. The question is what the answer should be: a length has to come from
|
||
somewhere before a pointer can become a slice, and C does not supply one. Possibly a
|
||
`(slice-from-ptr p n)` where the caller states the length and owns being right about it.
|
||
|
||
2. **An enum-typed `defstruct` field is refused by the layout check**, even when the representation
|
||
is identical. The message is `DISAGREES Camera3D: field projection is CameraProjection in the
|
||
defstruct and i32 (int)`. The check is comparing a Flan enum against the header's `int` and
|
||
calling that a disagreement, which it is not — an enum is an `i32`. Worked around with an
|
||
`rl/camera-projection` helper so the call site still takes a checked keyword. Related and
|
||
probably the same decision: a keyword resolves only where an enum type is expected, so
|
||
`:perspective` cannot be written into an `i32` field directly.
|
||
|
||
3. **The header check does not reach `defconst` or `defenum`.** `generate-c`'s claim is that every
|
||
`defstruct` and every hand-written `declare-c` agrees with the header, and that claim holds. It
|
||
says nothing about constants. **A wrong flag bit or a wrong enum member is silent** — which is
|
||
exactly the class of error the header read exists to catch, and the class hardest to see by
|
||
reading. This lane added 16 `ConfigFlags` constants and four enums by hand off the header, so the
|
||
values are right today and nothing would notice if they stopped being.
|
||
|
||
4. **raymath is `static inline`, so there is no symbol to bind.** `Clamp`, `Vector2Add`, `Remap` and
|
||
the rest exist only in the header. `declare-c` has nothing to name. rlgl's matrix stack is
|
||
unbound for a different reason. Together they make `core_2d_camera_mouse_zoom` unfaithful rather
|
||
than merely awkward, so it was skipped. The options are writing the arithmetic in Flan, which is
|
||
what the prelude would do anyway, or compiling a small C file that re-exports them as real
|
||
symbols.
|
||
|
||
5. **Four families are still refused by the importer for want of a `defstruct`.** `FilePathList`
|
||
(a `char**`, blocks `core_drop_files`), `Model`/`Mesh`/`Ray`/`BoundingBox` (the model and
|
||
3D-collision families), and **function-pointer parameters** (`SetTraceLogCallback`, which blocks
|
||
`core_custom_logging`, and the audio stream processors). The first three are ordinary widening —
|
||
write the `defstruct` and they import. The function-pointer one is not, and is the interesting
|
||
one: it is the callback direction of the FFI, which nothing has needed yet.
|
||
|
||
## Queued, 2026-09-13 (second session) — everything four lanes left behind
|
||
|
||
Four lanes ran in parallel worktrees and three landed; all are merged into `dev-loop` and `dune test` is green on
|
||
two consecutive full runs. Each lane wrote a handoff at the repository root, and those files are the detail — what
|
||
is below is the queue, not the reasoning. **Delete a handoff when its list is empty**, the same rule this file
|
||
follows.
|
||
|
||
**A note on the worktrees, because it nearly cost two lanes their work:** at least two were handed out at `2c232dd`,
|
||
roughly 485 commits stale, on a tree where `lib/dev.ml` does not exist. Both agents noticed and reset to the branch
|
||
tip themselves. A lane that did not notice would produce plausible work against the wrong tree. Check
|
||
`git log --oneline -1` before starting in a worktree.
|
||
|
||
### ~~From `HANDOFF-f2.md` — `pause` marking, the Emacs half~~
|
||
|
||
**Built.** All seven items landed: the `eval_expr` pause path, `flan-dev--eval` sending the position,
|
||
`flan-eval-defun` taking `C-u`, the overlay and its face, `test/programs/dev-pause.flan`, the `test_dev.ml` block and
|
||
the `emacs/test-flan-dev.el` one. `HANDOFF-f2.md` is deleted; the reasoning is in `BUILT.md`, "A breakpoint is a
|
||
function call, and the editor only says where".
|
||
|
||
### From `HANDOFF-f1.md` — the socket flake is fixed, and two things follow it
|
||
|
||
The fix is ordering, not timing: the test checked for `agent.sock` before completing any round-trip, and
|
||
`merged_serve` does not accept until its own 10s await on that path. Moving the existing `describe` request ahead of
|
||
the check took it from 2/8 failures to 0/10, and closed a second race that was burning the full 10s timeout.
|
||
`lib/dev.ml` was not touched.
|
||
|
||
1. ~~**A different flake, seen once and unexplained.** `the daemon never listened`~~ **Diagnosed and fixed.** The
|
||
hypothesis was right and it is not a race at all: the wait is not for a socket but for an llc-and-link of the
|
||
whole program before `flan dev` binds. That is ~600ms idle and was measured twice at **6.6s and 6.8s** with the
|
||
rest of the suite running beside it under dune's own parallelism, against a 5s (`test_dev.ml`) and 8s
|
||
(`test_emacs.ml`, `test_repl.ml`) `await`. The message reads like a bug in the daemon and is a build slower than
|
||
the timeout. All three now wait a minute — the watchdog is what bounds the run, and this only has to be long
|
||
enough that a failure means the daemon is not coming. `test_dev.ml` has a named `listening` for it so the reason
|
||
is written once for its thirteen daemons.
|
||
2. **Decide whether `merged_serve`'s 10s warning path deserves a test.** `lib/dev.ml:2322-2326` is exercised by
|
||
nothing now that the unlink race is closed.
|
||
3. **Confirm at a longer sweep if 0/10 is too thin, then delete this file's "One flaky test, measured rather than
|
||
suspected" section.** It was deliberately left standing — it should go when someone confirms the fix, not in the
|
||
change that made it.
|
||
|
||
Two things already tried and rejected, so they are not re-derived: wrapping the check in the file's `await` helper
|
||
makes the test observe the file at the *earliest* instant and so more likely to lose the race, and no timeout needed
|
||
lengthening.
|
||
|
||
### From `HANDOFF-f3.md` — the watch is built; two tests are not
|
||
|
||
`spy-num` landed and `PORTING.md` Tier 1 item 5 is closed. The `spy` half already existed — the finding that
|
||
reframed the task is that the table, the agent commands, the buffer and the ghost text were all there at `344e571`,
|
||
and only the numeric accumulator was missing. A slot keeps count, min, max, last and mean; the window is since the
|
||
editor's last tick rather than cumulative, because cumulative min/max reach the session's extremes within seconds of
|
||
play and then never move again. Reset is its own message and never a side effect of reading, so the reader never
|
||
writes the table.
|
||
|
||
1. **`emacs/test-flan-watch.el` — a `watch-num-i64` fixture site.** The regexp fix is the one change in that lane
|
||
with no regression test. `flan-watch-ghost-call-regexp` allowed a single hyphenated segment, so `watch-num-i64`
|
||
backtracked to failure and a numeric watch showed *no* inline value while looking normal in the buffer.
|
||
2. **`test/test_dev.ml` — assert the `n=0` render.** `watch_render_num` in `runtime/flan_dev.c` has a branch for a
|
||
slot that was never sampled and nothing reaches it.
|
||
|
||
### Not started: five more raylib core examples
|
||
|
||
Ten are in `examples/`. Five more were queued and the lane was stopped before it wrote anything, so there is nothing
|
||
to pick up — only the brief. Pick for what they exercise of the bindings rather than for how they look, since the
|
||
point of the corpus is finding gaps in `vendor/raylib/raylib.flan`. Upstream is
|
||
`vendor/raylib/web/raylib-5.5/examples/core/`. `PORTING.md` §1's rule binds: a raylib function on a real per-frame
|
||
path is hand-written and header-checked, not left to the opt-in `FLAN_RAYLIB_H` import, because the default build
|
||
has no `FLAN_RAYLIB_H` and still has to draw. An enum value has to be added by hand whatever happens — the importer
|
||
generates functions and only functions.
|
||
|
||
### Landed: frame rollback, `PORTING.md` Tier 1 item 6
|
||
|
||
`test/programs/frame-rollback.flan`, with three acceptance rows beside `bounds-condition.flan`'s (plain, `-O0`,
|
||
dev). Nothing was added to the language, which was the premise: `snapshot` is `(set grid-backup grid)` and
|
||
`(set world-backup world)`, `restore` is the same two the other way, and that is the whole of what `engine.clj`
|
||
spends an `IntGrid` walk on and `engine.lisp` spends `sb-mop:class-slots` on.
|
||
|
||
**The ordering against defers was the decision.** Restore lives in the `continue` clause, so it is the last write
|
||
on the abandoned path and needs no agreement with any defer that ran on the way out. The rejected alternative —
|
||
restore in a `defer` inside the frame function — is the silent one: a defer runs on the ordinary return path too,
|
||
so it rolls back the frames that *succeeded*, and nothing reports that as an error. The test pins the ordering with
|
||
a counter inside the snapshot and a counter outside it rather than in prose, and carries a negative control (the
|
||
same bad frame, a `continue` that only counts) because "state equals snapshot" passes trivially on a program that
|
||
wrote nothing.
|
||
|
||
One edge left in the file's header comment rather than here: the snapshot covers plain values only. A defer that
|
||
frees a resource the snapshot holds a pointer to means restore resurrects a dangling one.
|
||
|
||
### Landed: sand.flan is the game the other ports are
|
||
|
||
765 lines to 201, at parity with `lisp/sand.lisp`, `clojure/src/fnm/sand.clj` and `src/fnm/sand.jank` in
|
||
`~/Development/fnm`. Audio, the brush textures and the embedded PNG, the render-texture scene, the HUD font, the
|
||
camera, the world cursor, the HUD and the input-state read-out are gone — none is in any reference version. The
|
||
single-use helpers are inlined. **The physics is untouched on purpose**: the three references disagree there, CL
|
||
using `landing-col`/`move-grain` and Clojure and jank using `step-cell` with velocity parking, so parity does not
|
||
name a target, and `settle` is what the pinned hash covers. It still prints `15595743031174623232` at `-O2`, `-O0`
|
||
and as a dev build, which is what says the inlining changed nothing.
|
||
|
||
Interactively the controls are now `r` and left-mouse only. `test_web.ml`'s assertion that brush.png's bytes reach
|
||
the wasm module went with the embed: `web-files.flan` is web-built and *run* under node and asserts the embedded
|
||
bytes print, which is the same property checked harder.
|
||
|
||
## Queued: an idiomatic layer over the generated bindings
|
||
|
||
Thin Flan-shaped wrappers **over** the generated bindings, not instead of them. The generated set stays honest to C —
|
||
that is what makes it checkable against the header — and the layer is where a Flan-shaped API lives. Two of these
|
||
already exist by hand in `vendor/raylib/raylib.flan` and are the shape to copy: `collision-point-poly?` takes a slice
|
||
and `collision-lines` answers with an `Option`, each wrapping a `-raw` binding of the same name.
|
||
|
||
## ~~Queued: a restart is not a transaction, and the docs must say so~~ — **landed**
|
||
|
||
See [`BUILT.md`](BUILT.md), "A restart is not a transaction". **This entry was stale** — all three places already
|
||
carried the note when it was re-read: `conditions.org` under *Gotchas*, `spec-conditions.md` §5, and
|
||
`web/index.html`'s restart gotcha list, which was rewritten in place rather than gaining a second bullet beside the
|
||
existing one. Nothing was left to write.
|
||
|
||
What follows is the original entry, kept for the reasoning.
|
||
|
||
Raised by the author, and it is a real sharp edge rather than a gap. **If a frame mutates a global and then signals,
|
||
taking a `retry` re-runs the mutation.** Nothing rolls back. Common Lisp has exactly this property and offers no help
|
||
either — restarts are explicitly not transactional.
|
||
|
||
The discipline is that **the author chooses where the retry boundary is**: a `restart-case` at the top of a frame
|
||
re-runs everything including mutations already applied; one placed after the mutations re-runs only what follows. So
|
||
either put the restart before anything mutates, make the retried section idempotent, or snapshot what will be
|
||
re-applied.
|
||
|
||
This matters more here than in most Lisps because the intended use is a **game loop**, where the author's plan is to
|
||
skip a frame and carry on rather than die — exactly the case where a non-idempotent mutation bites.
|
||
|
||
## ~~Queued: a dev-build allocation registry~~ — **landed, in part; three of the six remain**
|
||
## Queued: a dev-build allocation registry — address to type
|
||
|
||
Built. The table is in `runtime/flan_dev.c`, the note is emitted by `check.ml` and dropped by `emit.ml` in a release
|
||
build, and the inspector reads it. `BUILT.md`'s *"An address answers with a type"* is the account of it; what follows
|
||
is only what is **not** there, so that the gap is a queue entry rather than a discovery.
|
||
|
||
**Landed:** items 1 and 2 — the inspector follows a live `(Ptr T)` and renders the pointee, and names what died at a
|
||
dead one (`<ptr dead: was Enemy, freed at step 15>`). The dead-marking covers the heap `free`, a heap resize's old
|
||
block, an arena `free-all` and `arena-destroy`.
|
||
|
||
**Left, and in this order:**
|
||
|
||
- **Item 3, point at any heap address.** The lookup is there and answers for any address; nothing exposes it as an
|
||
editor op. It wants a verb beside `inspect` that takes an address and a type rather than a frame and a slot, and the
|
||
registry's own answer for the type when none is given — which needs the recorded name resolved back to a
|
||
`Types.t`, and the table records a string.
|
||
- **Item 4, a breakdown by type**, and **item 5, leak attribution at exit.** Both are a walk over the table and a
|
||
group-by; `flan_dev_reg_count` is the whole of what exists. Neither is hard and neither has a reader yet, which is
|
||
why they were left rather than half-built.
|
||
- **The memcheck half of item 6.** The registry now knows an arena's `free-all` killed everything in the region, so a
|
||
later read through a pointer into it is *answerable*. Memcheck still says nothing, because nothing told it: the
|
||
pages stay mapped and `free-all` is an integer going to zero inside one allocation. Closing that is
|
||
`VALGRIND_MAKE_MEM_UNDEFINED` in `flan_arena_proc`, which `test/test_valgrind.ml` already names. **The two must not
|
||
be blurred** — the registry answer and the memcheck answer are different tools reaching different people.
|
||
- **A test that drives the inspector's pointer arm.** `test/programs/dev-ptr.flan` is the program and its header has
|
||
the two lines a session answers with; they were read off a running session **by hand**. The case belongs beside the
|
||
other `locals`/`inspect` cases in `test_dev.ml`, which was another lane's file. `programs/registry.flan` covers the
|
||
table itself from the acceptance table, in a dev build and a release one.
|
||
|
||
**What it does not cover, and does not need to:** stack locals and globals, which the shadow stack and the static type
|
||
table already answer by name. A stack address is deliberately not in the table, and a pointer to one still renders
|
||
`<ptr>`.
|
||
|
||
**Note on classes:** `defclass` instances will carry shape metadata by design, so they get identification for free and
|
||
do not need the registry. This is for plain structs, `Vec`, `Map` and pool storage.
|
||
|
||
**On cost, as built.** One insert per allocation, always on in a dev build, no opt-out — the author's instruction,
|
||
followed literally. Nothing was built per-region, no range recording, no per-allocator opt-out. Revisit only if a real
|
||
program shows a problem, and `BUILT.md` names the two places a release build is not quite free.
|
||
|
||
## Picked up first, 2026-09-13
|
||
|
||
Three things, in order. The first two are one line each and unblock a real game.
|
||
|
||
**1. `DrawTexturePro` is not bound — DONE.** It is `draw-texture-pro` in `vendor/raylib/raylib.flan` now, hand-written
|
||
beside `draw-texture-rec` and read off a raylib header rather than remembered. `image-from-image` and `window-ready?`
|
||
went in with it. The rule the gap exposed is written down in `BUILT.md` and `PORTING.md` §1: *a raylib function on a
|
||
game's per-frame path is hand-written and header-checked, not left to the opt-in import* — the import widens the
|
||
surface and must not be load-bearing, because the default build has no `FLAN_RAYLIB_H` and still has to draw.
|
||
|
||
**2. `Key` has no `left-shift` — DONE.** `left-shift 340`, and nothing else: `PORTING.md` §5 checked every other enum
|
||
value the game touches and they were all already right.
|
||
**1. `DrawTexturePro` — done, and not by a hand-written line.** It was the one true blocker for `siam-farmer`
|
||
(see `PORTING.md`: every tile in both implementations goes through it, and neither `DrawTextureRec` nor
|
||
`DrawTextureEx` substitutes). It was reachable only through the opt-in `FLAN_RAYLIB_H` import. The bindings are
|
||
committed now, so `rl/draw-texture-pro` is in `vendor/raylib/generated.flan` and a default build has it. **Nothing
|
||
should add it by hand** — a second `declare-c` for the same C symbol is refused for the whole program.
|
||
|
||
**2. `Key` has no `left-shift`.** Both implementations use shift+1..5 to pick the tilemap. One enum member, and
|
||
still a hand edit: the importer generates functions and only functions, so no `defenum` comes out of the header.
|
||
`vendor/raylib/raylib.flan` is where `Key` lives.
|
||
|
||
**3. An out-of-bounds index should signal a condition, not `exit(134)` — DONE.** A failed bounds check signals
|
||
`BoundsError` with `error`; `runtime/flan_rt.c`'s `flan_bounds_error`/`flan_slice_error` walk the handlers, then offer
|
||
the break loop, and tail into the old `flan_bounds_fail` message and status only if nothing answered. `Vec`'s two
|
||
checks are plumbed the same way, since `(at v i)` and `(at arr i)` are one form in the source.
|
||
|
||
**No restart is established at the failing index**, and `BUILT.md` has the argument: `retry` exists for allocation and
|
||
for files because those attempts are *repeatable*, and nothing a handler can do makes index 51 valid for a length-50
|
||
array. `use-value` for the index would cost every indexing operation a restart frame and buy a silently different
|
||
element. What answers a bad index is the restart the program already had — a frame loop's `continue`, `sand.flan`'s
|
||
shape — which is on the restart stack and on the break loop's list without anything being pushed at the site.
|
||
|
||
**Defer** had to be answered rather than inherited, since the `noreturn`-then-`unreachable` shape is what the old note
|
||
followed from: an answered bounds failure leaves through the function's unwind block, which is `return`'s path, so it
|
||
runs the defers; an unanswered one still runs none. `test/programs/bounds-condition.flan` counts them.
|
||
|
||
Two tests, because there are two paths. `bounds-condition.flan` is the *answered* half — a `handler-bind` taking
|
||
`continue` over five routes to a bad index, at `-O2`, `-O0` and as a dev build. `dev-break-bounds.flan` in
|
||
`test_dev.ml` is the half this was built for: nothing handles it, the break loop reports `BoundsError`, the layout for
|
||
that name resolves to `low`/`high`/`length`, the only restart on offer is the program's own `continue`, and taking it
|
||
resumes with the session intact.
|
||
|
||
**Still dying, deliberately:** a `Map`'s bounds check and `flan_vec_stale_fail`. The stale-allocator case is a
|
||
different kind of failure — the region the container lived in was released — and there is no frame to go back to that
|
||
would not read freed memory. The map path was left alone rather than converted half-way.
|
||
|
||
### What is left on `PORTING.md`'s list
|
||
|
||
Tier 0 is finished, and so is item 5. **The watch for a running program is done** — the `spy` half was already
|
||
built (a pushed table, the watch buffer, inline ghost text) and the `spy-num` half landed 2026-09-13: a hot-loop slot
|
||
keeps count/min/max/last/mean, the write path does no formatting, and the window is since the editor's last tick
|
||
rather than cumulative, which is a deliberate divergence from `watch.clj` argued in `BUILT.md`, "A hot loop keeps
|
||
five numbers, and the window is the editor's". **Item 6 landed the same day**:
|
||
`test/programs/frame-rollback.flan` is the worked example — `snapshot` at the top of the frame, `restore` in the
|
||
`continue` clause — and `bounds-condition.flan`'s half-written abandoned frame is the thought it finishes. That was
|
||
the last Tier 1 item anyone was going to move. Items 7 (`drop`), 8 (generics) and 9 (`(read-edn T bytes)`) are still
|
||
on that list and still deferred with reasons written beside each; none is a blocker for this game.
|
||
|
||
**What `PORTING.md` says NOT to build, with evidence:** escaping closures (one capture site, fixed by one parameter),
|
||
`Handle`/pools, `Result`/`try`, `handler-case`, `loop`/`recur` and tail calls, user allocators, structural typing —
|
||
**none has a customer in that code**. (`Handle` and the pool were built anyway, and on the other reason: they are the
|
||
gate on classes. The finding stands and is why they were built small — see [`BUILT.md`](BUILT.md). `loop`/`recur` was
|
||
built too, and the finding stands there as well: what it is not is **tail calls**, which are still not built and still
|
||
have no customer.) And **generics is not the blocker** there either: the element-changing maps are
|
||
five-line load-time loops. That last one hangs on a design decision the report states flatly — whether the game's
|
||
state holds fixed arrays or `Vec`s.
|
||
|
||
**raylib 6.0 is not urgent:** all seven struct layouts on the game's path and every enum value it touches are
|
||
byte-identical between 5.5 and the vendored 6.0.
|
||
|
||
### One flaky test, measured rather than suspected — **fixed, pending a longer sweep**
|
||
|
||
The ordering fix is in (`f80a7a9`): 2/8 failures before, 0/10 after. This section stays until someone confirms it at
|
||
a sweep wide enough to be worth the claim, then goes. See `HANDOFF-f1.md`. The original reading follows.
|
||
|
||
|
||
`test_dev.ml`'s first block fails about one run in four with *"the merged program never bound
|
||
…/agent.sock"*. **It is not new** — reproduced at `54027ca`, before any of today's work, at the same rate. It is a
|
||
startup race in the one-process daemon's socket bind, not a real failure, and it makes "the suite is green" a
|
||
statement that needs a second run to make. Worth fixing before it trains someone to re-run on red.
|
||
|
||
### Left mid-flight when the session ended
|
||
|
||
Both lanes committed their main work and died on trailing polish; both are merged and the suite is green.
|
||
|
||
- ~~**`pause` marking from Emacs was not built.**~~ **Built**, both halves. See `BUILT.md`, "A breakpoint is a
|
||
function call, and the editor only says where".
|
||
- ~~**Ghost text** for the watch window — values shown inline at the code they belong to — is noted and not designed.~~ **Built.** See `BUILT.md`, "Ghost text finds its anchor in the buffer, not in the table".
|
||
- `tools/unit-return.py` is re-runnable; run it over any `.flan` file a lane wrote before the conversion landed.
|
||
|
||
# Where this is
|
||
|
||
## Start here — next session
|
||
|
||
**Branch `dev-loop`, 199 commits, working tree clean, `dune test` green.**
|
||
|
||
**[`DISCUSS.md`](DISCUSS.md) is what has been *asked* and not answered** — open questions with the repo context that
|
||
bears on each, so an investigation starts from what exists. Nothing in it is a decision or a task; when one becomes
|
||
either, it moves here.
|
||
|
||
**`NEXT.md` is what is left. [`BUILT.md`](BUILT.md) is why the existing parts are the shape they are** — the reload
|
||
primitive, cells, the agent, the session, the daemon, the Emacs client, conditions, the FFI shim, the layout, and the
|
||
order it all got built in. This file was half build log until it was split; do not let it become one again. When a
|
||
track here finishes, its explanation moves there and its entry here goes away.
|
||
|
||
The dev loop works end to end: `flan dev program.flan`, then `C-c C-c`, `C-x C-e` and `C-c C-r` in Emacs against the
|
||
running process.
|
||
|
||
**Conditions are three steps of four.** `(error c)` is the diverging variant — a handler that returns normally has not
|
||
answered it, so only a transfer gets past. The break loop is in, editor half included: an unhandled `error` stops the
|
||
program on the frame that erred, the daemon annotates every reply with `:stopped`/`:condition`, and `C-c C-b` lists the
|
||
restarts and resumes into the choice. A restart is chosen **by position**, off a snapshot taken when the break was
|
||
entered, because a name resolves to the innermost frame offering it and the stopped thread's stack does not hold still.
|
||
Restarts below the evaluation a break is inside are listed, marked, and refused with the reason.
|
||
|
||
**Restarts take parameters now** — §3's other half. `(use-value [v i32] ...)` binds them, `(invoke-restart 'use-value 21)`
|
||
supplies them, and what a clause takes against what was given is checked at run time and refused with both spellings,
|
||
because a restart is found by name on a dynamic stack and neither end of a transfer can see the other. The one path
|
||
that cannot yet supply a value is the break loop, which is item 2 below and is where the interesting half is.
|
||
|
||
Still open: **`handler-case`**, which §"What this does not settle" leaves open as possibly a macro over `handler-bind`
|
||
plus a transfer. **`find-restart` and `compute-restarts` are blocked on a type, not on effort** — §4 gives them
|
||
`(Option Restart)` and a list, and there is no `Restart` type and no list to return one in. The minibuffer prompt never
|
||
needed them; it reads the snapshot over the agent's socket. And a `restart-case` clause should still carry a **report
|
||
string**: `use-placeholder` is what `invoke-restart` needs, not what a person reading a list needs. §3 says to settle
|
||
that *before* parameters and it was not settled — the field is cheap and the accessor is cheap, but the only consumer
|
||
is the break loop's listing, which lives in the agent and the daemon, so it would have shipped as a field nothing
|
||
read. It belongs with item 2, where the listing is being changed anyway.
|
||
|
||
Read SBCL for what restarts should *mean* and ignore how it moves control: it transfers with `block`/`return-from`,
|
||
which §6 rules out.
|
||
|
||
### Landed — macros run, and `unless` is not a special form any more
|
||
|
||
The expander is written and the exit criterion plan.org set for milestone 5 is met: a conditional sugar moved out of
|
||
`parse.ml` and into `prelude.ml` as a `defmacro`, with the corpus that was written against the special form
|
||
unchanged. Running `test/programs/macro-unless.flan` means the compiler built a shared object, `dlopen`ed it into
|
||
itself and called a Flan function to find out what `(unless c a b)` means.
|
||
|
||
The full explanation is in [`BUILT.md`](BUILT.md), "Macros: the compiler dlopens the program". Four things worth
|
||
knowing before touching any of it, because each cost something to find:
|
||
|
||
- **A call inside a quasiquote is output, not a compile-order dependency.** A macro body that *calls* another macro
|
||
needs it compiled first; a macro body that *quasiquotes* a call to one needs nothing, because the call is part of
|
||
what it answers and the answer is expanded again. The first cycle test written for this got that wrong and was not
|
||
a cycle at all. The two non-termination failures are therefore different and are refused differently: a ring is
|
||
named, a macro that does not settle is bounded.
|
||
- **Quasiquote is desugared before the walk**, and that is load-bearing rather than tidy — with the quasiquote still
|
||
standing, the walk expands the call inside it against the wrong arguments.
|
||
- **`lib/dune` passes `-linkall`.** `lib/macro.ml` installs itself into `Parse.expander` and nothing references it, so
|
||
the linker would otherwise drop it from `bin/main.exe`. Installing by hand is not viable: `session.ml` parses for
|
||
`C-c C-c`, and `test_session.ml` drives the session library in-process.
|
||
- **Two parser bugs fell out of it**, both in the rule that tells a return type from the first form of a body. The
|
||
prelude's types were not in the set that rule consults, so `Form` in return position was read as a body form; and
|
||
adding them plainly made `(defn f [] (Rune {.code 65}) (bar))` a function returning a `Rune` with a one-form body,
|
||
silently, in every file in the language. Both are pinned in test_flan.ml's return-type section.
|
||
|
||
Costs: a build that names no macro is unchanged at 50ms; one that calls a macro is 310ms cold and 70ms warm, the
|
||
difference being a cached `.so`; and a hello-world carries eight bytes of it, because `Reach.link` drops the rest.
|
||
|
||
### Landed — a C header is read, so a binding is checked instead of trusted
|
||
|
||
`lib/cimport.ml`, `lib/cjson.ml`, a `headers` file beside `link`. Full reasoning in `BUILT.md`, "The header is read
|
||
now"; DISCUSS.md item 6 is rewritten down to the two decisions left, both the author's.
|
||
|
||
The gap closed is the one `BUILT.md` recorded as *trusted*: `declare-c` generates the wrapper, the typedefs and the
|
||
prototype from one declaration, so they agree with each other by construction and only the library could disagree —
|
||
and nothing had a second opinion to disagree with. Now clang is asked for a JSON AST dump of the header (shelled out,
|
||
never libclang — the dependency plan.org rejected; Zig has since left it too, for Aro) and both halves are compared
|
||
against it.
|
||
|
||
**The evidence.** Against raylib 5.5, the version whose `.so` `vendor/raylib/link` names: **all 16 `defstruct`s and
|
||
all 172 hand-written `declare-c` agree exactly.** Against the 5.1-dev header also installed on this machine, ten real
|
||
differences — nine functions that version lacks and one that gained a parameter — so picking the wrong header is
|
||
loud. Both comparisons run at build time and stop the build; verified by permuting `Texture2D` and by putting `f64`
|
||
where raylib says `float`, which is the hazard `BUILT.md` names and says only a test can catch.
|
||
|
||
**Costs, measured, because they decide the remaining question.** Release build +4ms warm — `Reach.link` already drops
|
||
a wrapper nothing reachable calls, confirmed on the wasm32 case it exists for with 256 extra declarations in play.
|
||
Redefinition 31.0ms → 46.5ms. Dev build +333ms cold, once per session, since `Build.shared` compiles no C. Reading the
|
||
header is cached (64ms → 17ms), keyed like the object cache; the cache was built against a measurement, not a guess.
|
||
|
||
**Re-measured, and the 15.5ms was misattributed** — see BUILT.md, "Where that 15.5ms actually is". A `C-c C-c` reads
|
||
no header: `Session.eval` puts the forms through `Load`, and forms with no `(import …)` in them touch no package. The
|
||
15.5ms is `flan reload`'s, and `flan reload` is a fresh process — ~14.5ms of it is session startup and ~4ms of *that*
|
||
is the header. What a redefinition really pays for an imported package is **+3.6ms per eval** in `Check` and in
|
||
`Emit.redefinition` declaring 256 more siblings, and no cache touches that; it is the number to attack next. The
|
||
header is now cached in the session as well as on disk, so a repeat import (a `C-c C-k` of a buffer carrying its own
|
||
`import` line) costs nothing, and a header edited mid-session is not picked up until the session restarts — the same
|
||
rule a changed `.c` file follows.
|
||
|
||
**Opt-in on purpose.** `vendor/raylib/headers` is `?${FLAN_RAYLIB_H}`. "A build needs libraylib linkable and not
|
||
raylib-devel installed" is a property chosen deliberately, and requiring a header would take it from everyone to give
|
||
the check to whoever has one. Unset means off; set-and-wrong is an error naming the path.
|
||
|
||
Worth knowing before touching it:
|
||
|
||
- **The import is bounded by the package's own `defstruct`s**, not by a curated list. A function mentioning a struct
|
||
the package has not described is refused with that reason. Of raylib's 581 functions, 256 import, 153 are refused,
|
||
172 are already bound by hand and left alone. Widening the binding is a `defstruct`, not a list edit.
|
||
- **No `defstruct` is generated, and that is load-bearing.** Generate them and the header becomes the authority on
|
||
layout, and checking the package's layouts against it would be comparing the header with itself — which is exactly
|
||
why `BUILT.md` rejected a `_Static_assert` as circular. Keeping them hand-written is what makes the check a second
|
||
source.
|
||
- **A refusal is a demotion, not a drop** — Zig's `failDecl`, which `Load.refuse_hidden` already implemented for
|
||
`main`. `rl/get-gamepad-name` is a name that exists, cannot be had, and says why at the use site.
|
||
- **`declare-c` and `declare` are untouched and still win.** A C symbol the package binds by hand is not imported, so
|
||
the escape hatch is the override.
|
||
- **`test/headers/sample.h`** is the importer's table — one function per decision, committed, no raylib needed. The
|
||
raylib acceptance case skips without `FLAN_RAYLIB_H`; that one does not.
|
||
|
||
Two things that are *not* done, and are 6a and 6b in DISCUSS.md: whether the header stays a build-time read or becomes
|
||
a committed generator (`flan import-c` already prints the lines, so it costs nothing more to switch), and whether the
|
||
172 hand-written lines migrate. Neither is blocked on correctness. The argument for the first is weaker than it
|
||
looked — committing the generated lines would save ~4ms of session startup and none of the +3.6ms per redefinition,
|
||
since that cost is the 256 declarations existing at all and not where they came from; needing the header at every build — vendoring raylib.h or requiring raylib-devel — is the argument on the
|
||
second.
|
||
|
||
One smaller thing found and worth not re-deriving: an enum parameter imports as `i32`, because the header says
|
||
`KeyboardKey` and nothing tells the importer the package calls that `Key`. The ABI is identical, the face is worse,
|
||
and it is why `(rl/key-down? :space)` keeps its hand-written line.
|
||
|
||
### Landed 2026-09-12 — six tracks, one session
|
||
|
||
Six agents in parallel worktrees. Kept short on purpose; the reasoning that outlives the change is in `BUILT.md` or in
|
||
the commit that made it.
|
||
|
||
1. **`nth` removed**, an alias of `at` that was asymmetric — `check.ml` aliased them but `parse.ml` and
|
||
`place_of_expr` matched only `at`, so `(set (nth a i) x)` and `(addr (nth a i))` were refused while the `at` forms
|
||
worked.
|
||
2. **`println` and `print`**, compiler-provided and structural. `Session.render` was already the compile-time walk
|
||
plan.org asks for; it moved to `lib/render.ml` parameterised on an emitter and a slot allocator, so the REPL and
|
||
stdout share one copy. Found doing it: `field_addr` in `emit.ml` accepted only `Types.Named`, so a field of an
|
||
`Option` threw at emit time and **the walk's Option arm had never run** — the inspector would have failed on the
|
||
first `(Option T)` pointed at it. prelude.ml's claim that this had to wait for milestone 5 and generics was wrong,
|
||
and is gone: a printer selected per *concrete* type has nothing to dispatch on and no type variable in it.
|
||
3. **`restart-at`** — a restart is taken by position now. See "Start here".
|
||
4. **Names in DWARF.** `Tast.fn` carries `snames` beside `slots`, so a let-bound local is its own name under lldb
|
||
instead of `s0`; a slot the compiler invented keeps `s<index>`, because inventing a name puts a variable in the
|
||
debugger that is not in the file. Shadowing had to be decided rather than assumed: every `!DILocalVariable` is
|
||
scoped to the subprogram — the typed IR has no block structure to build a `!DILexicalBlock` from — so two slots
|
||
called `v` left lldb answering `p v` with the outer one while the body computed with the inner, and not listing the
|
||
inner at all. A repeat gets a `~2` suffix, unspellable in source. That is a way of not lying rather than a way of
|
||
being right; see "One line away". Also `flan dev --debug`, one flag for host and every redefinition module, off by
|
||
default because a debug build is an `-O0` build.
|
||
5. **Ten raylib core examples** in `examples/`, plus seven bindings and the colour palette. The gap list they produced
|
||
is under "Unblocked now, and ranked"; the top item, that no number could reach `draw-text`, is fixed — `(string b)`
|
||
reinterprets a `[u8]` as a `string`, which costs no instructions because they are already the same 16 bytes.
|
||
|
||
6. **The `print-*` family is gone.** `print` and `println` are the whole printing surface; ~500 call sites across 47
|
||
files rewrote, and `web/index.html` gained a `#printing` section, the first documentation either has had. Two pinned
|
||
outputs moved and both are corrections: `sand-headless`'s hash is `15595743031174623232` rather than
|
||
`-2851001042534928384` — the same 64 bits, printed unsigned now that `hash-grid`'s `u64` no longer goes through an
|
||
`(i64 …)` cast — and a trap column shifted because the call it names got shorter.
|
||
|
||
### Landed — `(Map K V)`, and `defer` in a `let`
|
||
|
||
Step 4 of the container build order, following Odin: open-addressed Robin Hood hashing at a 75% load factor,
|
||
cache-line cell packing, pointer-width integers through the probe loop. Two deliberate departures from Odin — no
|
||
tombstones, because `spec-memory.md` defers removal, which deletes the backward-shift loop entirely; and no capacity
|
||
tagged into the data pointer, because this header has room for it and tagging would make correctness depend on an
|
||
alignment that is only ever requested.
|
||
|
||
**One amendment to a frozen `spec-memory.md`, and it is the `defer` half**: the spec says under "When storage is
|
||
released" that `(defer (free v))` for a `let`-bound `v` is "not expressible today" and that no idiom may depend on it.
|
||
It is expressible now. A `let` at the top level of a function body has exactly the function's extent — a `let` is not a
|
||
frame here, and nothing is released at scope exit — so a `defer` in one always registers. A loop body and a branch stay
|
||
refused, by name, for the reason that does apply to them.
|
||
|
||
**One restriction the spec does not have**: a fixed array is a map key only when its elements compare bytewise, so an
|
||
array of structs or of strings is refused by name. A struct key holding the array works, because a struct key is walked
|
||
field by field.
|
||
|
||
The measured answer to the author's "is this another Python dict": **six times quicker cache-resident and slower at a
|
||
million entries**. Python's algorithm is fine — what makes it slow is a separately allocated refcounted object per key
|
||
and value, and hashing through calls that cannot be inlined. The second half of that result is the interesting one and
|
||
is written down rather than left out; see the unsettled list under the build order.
|
||
|
||
### Landed — the allocator, the arena, `(Vec T)`, `StorageExhausted`
|
||
|
||
The critical path, and the thing NEXT.md said was the only one standing between this and writing a game. Steps 1, 2 and
|
||
3 of the build order below are struck; `Map` is step 4 and is untouched. `Allocator` is a builtin opaque type and
|
||
needed nothing from milestone 5, which was the whole bet. Three amendments to a **frozen** `spec-memory.md`, made
|
||
deliberately and stated as amendments in [`BUILT.md`](BUILT.md): `free-all` is retain-capacity with `arena-destroy`
|
||
beside it; `context/allocator` is a dynamic variable rather than a literal calling-convention parameter; and the `Vec`
|
||
header is six words in every build rather than four in release. One addition the spec does not have: a budget on the
|
||
allocator, because `retry` needs a handler that can make the *same* request succeed.
|
||
|
||
### Landed — the runtime under a sanitizer
|
||
|
||
`--sanitize` is a build flag beside `--debug`; `dune build --root . @sanitize` builds twenty-eight programs twice, plain
|
||
and sanitized, and compares output and exit status. Its own alias and not `dune test`, because the sweep is about nine
|
||
minutes. The checked sweep is **clean**. How ASan and UBSan reach a language whose IR is written by hand, and why the
|
||
flag does not force `-O0` when `--debug` does, is in [`BUILT.md`](BUILT.md).
|
||
|
||
Two defects came out of it, both found by reading rather than by the tools, both fixed with a regression case:
|
||
`flan_bytes_to_i64`/`flan_bytes_to_f64` clamped a slice length with `(size_t)n` and so read 63 or 511 bytes off the end
|
||
of a negative-length slice; and the three `snprintf` shims published snprintf's return as a slice length, which is what
|
||
it *would* have written.
|
||
|
||
**What is left, and it is most of what the sweep was meant to settle:**
|
||
|
||
1. **UBSan sees no Flan code and no flag changes that.** Its checks are branches clang's C frontend emits inline, not a
|
||
pass, so shift UB (`(<< 1 32)`, see Sharp edges), alignment, and the f32→i32 cast on NaN or an infinity — the things
|
||
`floor-f32` guards by hand and nothing else does — are unreached. Either `Emit` grows those checks behind the flag,
|
||
which is a compiler feature of the same shape the bounds checks already have, or they belong to the checker. Not
|
||
decided. `test_sanitize` pins the current answer with a control that must *not* report, so a future clang changing
|
||
this is a test failure rather than a discovery.
|
||
2. **Three of the four named buffers now have evidence; one still does not.** Two lanes closed different pairs and
|
||
they combine. The 4K result cap and `condition_name[128]` are driven over the agent's socket from `test_agent.ml`
|
||
— a 5000-byte value comes back as 4096 ending in the ellipsis, a 198-character condition class comes back from
|
||
`status` as 127. The 4K cap and the **dev registry overflow guard** are also run directly by `test/dev_limits.c`,
|
||
a C main beside `reload_host.c`, one process per limit because the name table never shrinks and the overflow case
|
||
aborts. Only `SNAP_MAX`/`SNAP_NAMES` is still read rather than tested: sixty-five nested `restart-case`s are a lot
|
||
of program for a clamp. `escaped[ESCAPE_MAX]` was already covered, because `println.flan`
|
||
drives a 1100-character string through it on purpose — 1019 bytes out against a worst case of 1021 into 1024.
|
||
`scratch[SCRATCH]` never sees more than 20 characters of 64.
|
||
3. **Valgrind over the headless corpus, done.** `dune build --root . @valgrind` runs forty-seven programs under
|
||
memcheck, twelve of them again with `--no-bounds-checks`, in 88 seconds including the compiles. Clean. It needs no
|
||
instrumentation at all — memcheck works on the binary, so `Emit`'s hand-written IR arrives on the same footing as
|
||
clang's C, which is why it was reachable where MSan was not. The uninitialised read ASan is blind to is now a
|
||
control that must report: index 3 of a `Vec` with len 2 and cap 4, with `--track-origins` naming the
|
||
`aligned_alloc` in `flan_vec_push`. Two more controls pin a heap overrun and a padded `Map` key. `test/valgrind.supp`
|
||
holds **no suppressions** — nothing false came up to suppress. Details, and the measured fact that memcheck catches
|
||
0 of `bounds.flan`'s 6 cases where ASan catches 3, in [`BUILT.md`](BUILT.md).
|
||
|
||
**The hole it leaves, and it is the arena.** `free-all` is retain-capacity, so the pages stay and memcheck is never
|
||
told the storage died: round two of a reset arena reads a byte it never wrote, prints round one's value, and
|
||
nothing reports. Interior overruns are invisible for the same structural reason — an arena is one `malloc`, and the
|
||
`Map`'s `keys | values | hashes | scratch` is one allocation too, so "probe overrun at high load" is not clean, it
|
||
is not observable. Closing the arena half means `VALGRIND_MAKE_MEM_UNDEFINED` in `flan_arena_proc`, a `runtime/`
|
||
change nobody has made. And both positive controls had to be written by hand: no corpus program reaches an
|
||
observable uninitialised read, so the sweep is a regression net from here rather than an audit that found the
|
||
runtime sound.
|
||
|
||
Two things the sweep structurally cannot cover: raylib and libm are uninstrumented, so the windowed examples are noise;
|
||
and a redefinition module is built by `llc` and `ld` rather than clang, so the reload path carries no instrumentation
|
||
whatever the flag says.
|
||
|
||
### Managed classes are planned. Do not start them.
|
||
|
||
plan.org grew a `class` facility beside `struct`: identity, runtime shape metadata, an implementation-defined
|
||
representation, generic-function dispatch, and live schema change with an explicit migration at a frame boundary. Its
|
||
own last line is the rule — nothing until ordinary `struct`, `Handle` and reload semantics are working. **All three
|
||
now do**, `Handle` and the pool having landed, so this section is no longer "not yet" but "next, and deliberately not
|
||
started here". It is here so that a session reading plan.org cold does not take it as the next task. Three things
|
||
found while reviewing it, none of them in plan.org yet:
|
||
|
||
- **A generic function is a cell.** "A later module can add `(defmethod draw ((e Enemy)) ...)` without editing the
|
||
original" means every compiled call site of `draw` has to find the new method — which is the problem the indirection
|
||
cells already solve. A generic function is a cell whose body is a dispatch table and a reload extends the table. The
|
||
expensive half of classes is therefore already built and tested.
|
||
- **The pool is not one storage option among three.** `migrate-instances` has to *enumerate* live instances. A pool
|
||
behind generational `(Handle T)` gives that by construction; a world arena and an owned region do not obviously.
|
||
plan.org presents the three as a free choice and they are not. **The pool is built**, and `(len p)` with
|
||
`(pool-handle p i)` is that enumeration.
|
||
- **`Enemy@1` has to stay resolvable** for `migrate` to dispatch on it, so the session retains every layout version's
|
||
metadata for as long as any instance holds it. Same rule as "nothing is ever `dlclose`d", and worth stating as one.
|
||
|
||
### Open: can a condition be a class?
|
||
|
||
Unanswered, and it wants answering before `handler-case`, because it decides whether handler matching has one path or
|
||
two.
|
||
|
||
It would buy the thing conditions most lack: a **hierarchy**. §1 says flatly there is none, which is why nothing can say
|
||
"any condition" — no catch-all handler and nothing for a break loop to match on. Class inheritance gives it.
|
||
|
||
Three costs, one serious:
|
||
|
||
- **Signalling would allocate.** A struct condition is a stack value and `signal` takes its address; a class instance
|
||
needs a pool slot at the signal site. That is the failure path, sometimes the hot path, and sometimes the thing that
|
||
failed is allocation itself. plan.org also says no implicit allocation anywhere in the core.
|
||
- **§5's lifetime inverts.** Today the condition dies with the signalling frame and a handler that keeps it copies,
|
||
which is free for a value struct. A class instance survives the transfer — nicer, but now something owns and frees it.
|
||
- **Layout versions meet handler frames.** A struct condition cannot change layout; it is refused. A class can, and then
|
||
a frame pushed against `MyError@1` is on the stack while the signaller builds `MyError@2`.
|
||
|
||
The shape that probably wins is both: a struct condition stays exactly what it is — no allocation, matched by name hash,
|
||
dies with the frame — and a class condition is allocated, survives, and matches by walking its class chain. That is two
|
||
matching paths, which is the same bill the struct/class split already signs, so it is consistent rather than a new cost.
|
||
Either way it is an amendment to a **frozen** `spec-conditions.md`, not a gap in it.
|
||
|
||
|
||
**The dev loop is closed.** `C-c C-c` in Emacs recompiles the top-level form at point and installs it in a running
|
||
program, at that program's next frame boundary. Verified against sand: an unsaved buffer edit to `game-draw`, and 240
|
||
consecutive frames drew it.
|
||
|
||
Steps 1, 2 and 3 are done — see *The reload primitive* in `BUILT.md`. A list of top-level forms can be recompiled and installed
|
||
into a running process; call sites compiled before they existed follow them, and a `defn` or `defvar` the process was
|
||
never built with can be added and then redefined again. That is the whole of `C-c C-c`, minus an editor: sand.flan takes
|
||
a redefinition over a socket and installs it between frames.
|
||
|
||
What is left is the *session* — something that holds the checker environment between evaluations, tracks which names the
|
||
running process was built with, and speaks a protocol an editor can talk to.
|
||
|
||
Milestone 4 is done: **sand.flan builds, links raylib and runs**, and its simulation has a headless acceptance case that
|
||
runs on the `dune test` path at `-O0` and `-O2`. Milestones 2 and 3 are behind it (`calc-me.flan` compiles and runs; the
|
||
interpreter was dropped — open decision #7, settled — see "Why there is no interpreter" in `BUILT.md`).
|
||
|
||
```
|
||
reader ✅ → parse ✅ → load ✅ → check ✅ → emit ✅ → clang ✅
|
||
```
|
||
|
||
| File | What it does |
|
||
|---|---|
|
||
| `lib/loc.ml` | source locations + `Loc.Error`, the frontend's one exception |
|
||
| `lib/form.ml` | reader output: `Sym Kw Int Float Str Byte List Vec Map` |
|
||
| `lib/reader.ml` | hand-written S-expression reader, no menhir/ocamllex |
|
||
| `lib/ast.ml` | AST: `texpr`, `expr`, `place`, `pattern`, `decl` |
|
||
| `lib/parse.ml` | forms → AST; special forms, desugaring, declarations |
|
||
| `lib/load.ml` | **imports: a package directory → qualified declarations** |
|
||
| `lib/types.ml` | resolved types; structural equality, `Never` fits anywhere |
|
||
| `lib/tast.ml` | the typed IR the backend consumes |
|
||
| `lib/check.ml` | AST → typed IR; two passes, bidirectional |
|
||
| `lib/session.ml` | **a live program: what the process was built from, plus every change since** |
|
||
| `lib/wire.ml` | **the editor protocol: one s-expression per message, length framed** |
|
||
| `lib/agent.ml` | **the agent, called rather than connected to, when it is in this process** |
|
||
| `lib/dev.ml` | **`flan dev`: a session, an editor socket, and the program it is a thread inside** |
|
||
| `lib/prelude.ml` | printers + `rand-f32`, written in Flan |
|
||
| `lib/emit.ml` | typed IR → LLVM IR text |
|
||
| `lib/build.ml` | `.ll` + the shim + the packages' C → clang → executable |
|
||
| `runtime/flan_rt.c` | the host ABI: argv, stdout, exit, 4 conversions |
|
||
| `runtime/flan_dev.c` | **dev only: the by-name registry a run-time-new name needs** |
|
||
| `lib/shim.ml` | **`declare-c` -> the generated C that flattens a struct crossing** |
|
||
| `vendor/raylib/` | **the raylib package: `raylib.flan` and `link`, and no C at all** |
|
||
| `vendor/agent/` | **the dev agent: one verb table, a loader thread, install at a frame boundary** |
|
||
| `emacs/` | **`flan-mode.el`, `flan-dev.el`, `flan-repl.el`: the editor half of the dev loop** |
|
||
| `bin/main.ml` | `flan read \| parse \| check \| emit \| shim \| build \| run \| reload \| dev` |
|
||
| `test/test_flan.ml` | reader, parser and checker |
|
||
| `test/test_acceptance.ml` | expression/result pairs + whole programs + the traps |
|
||
| `test/test_reload.ml` | **the reload primitive: recompile one function, load it, call it** |
|
||
| `test/test_agent.ml` | **a running program taking a redefinition over a socket** |
|
||
| `test/test_session.ml` | **what a running process cannot be told, and recovering from a typo** |
|
||
| `test/test_dev.ml` | **the daemon, driven the way an editor drives it** |
|
||
| `test/test_repl.ml` | **`C-x C-e`: an expression evaluated inside a running program** |
|
||
| `test/programs/conditions.flan` | **`handler-bind` and `signal`, the accumulation case** |
|
||
| `conditions.org` | **a cheatsheet for driving conditions: what works, the exact refusals, the gotchas** |
|
||
| `conditions-play.flan` | **a program to poke at them with, built to be attached to by `flan dev`** |
|
||
| `test/programs/restarts.flan` | **`restart-case` and `invoke-restart`: the transfer, across two frames** |
|
||
| `test/test_emacs.ml` | **the client, driven against a real daemon and a real program** |
|
||
| `test/reload_host.c` | the C host that loads and installs two rebuilds, in one process |
|
||
| `test/wasm-run.mjs` | **a WASI host in twenty lines of `node:wasi`, so the table can run a wasm32 build** |
|
||
|
||
```
|
||
$ flan run calc-me.flan "1 + 2 * (3 - 0.5) / 2"
|
||
3.5
|
||
$ flan run test/programs/sand-headless.flan
|
||
15595743031174623232
|
||
$ flan run sand.flan # a window, 120 fps, hold space
|
||
```
|
||
|
||
## Decided 2026-09-12, by the author, and not yet built
|
||
|
||
Five questions were put and answered in one sitting. Each is a decision, not a preference — build against them, and
|
||
reopen one only with a reason rather than a taste.
|
||
|
||
~~**1. Assets are embedded at compile time, one file or one directory.**~~ **Built** — `(embed "p")`, `(embed "p" string)`, `(embed-dir "d")`. See BUILT.md, "Assets are baked in". Odin's answer, and the reason it is the right
|
||
one here: it is a *compiler* feature, so it needs no build flags, no linker arguments and no per-target packaging, and
|
||
it works identically on desktop and web. That matters more here than it does for Odin, because `Load` gives link flags
|
||
only to directory packages — the single file doing `(rl/load-texture "brush.png")` is structurally the one file with
|
||
no link channel, which is what stopped the web lane from inventing a flag. Embedding has no such hole. Odin's
|
||
`#load` and `#load_directory` are the model (`src/parser.cpp:853`, `src/checker.cpp:3594`). emscripten's
|
||
`--preload-file` stays available later for assets that should load lazily rather than be baked in; the `@web` link
|
||
line already carries it if wanted.
|
||
|
||
~~**2. Reading a file works everywhere; writing is desktop-only and signals on web.**~~ **Built** — `barf` on the web signals `FileError` with reason `file-unsupported`, and `test/test_web.ml` runs it under node rather than asserting the artifact's shape. See BUILT.md, "slurp, barf, and the two ways they fail". Odin stubs its whole file API on
|
||
js/wasm — every operation returns `.Unsupported`, and `core/os/file_js.odin`'s own comment says the stubs exist only
|
||
so importing `core:os` "panics cleanly". Take the restriction and not the mechanism. **Flan has no conditional
|
||
compilation** — nothing in `parse.ml` or `check.ml` reads the target — so "isolate this code to desktop" is not
|
||
expressible in source, and a build-time refusal would therefore be unusable. A **silent no-op is worse than either**:
|
||
it is how a save file disappears with nothing said. So `barf` on web signals a condition under a restart and the
|
||
program decides. This is the language having something Odin does not; use it. Per-package target isolation, if a
|
||
whole desktop-only package is ever wanted, is the `@native`/`@wasi`/`@web` link-line tagging the web lane built.
|
||
|
||
**3. Build the shadow stack.** ~~Not yet built.~~ **Built**, both halves — see BUILT.md. Kept here as the decision it
|
||
was, with the measurement it asked for: +33% on call-heavy code over globals for the frames, +61% with the slot table,
|
||
and 0.06% of a 60fps frame.
|
||
|
||
plan.org:591 has specified it in the dev-build column since the beginning and nothing
|
||
has ever built it. It is the route to `(:op "backtrace")` *and* to locals, together, and it is dev-only so a shipped
|
||
game pays nothing. Chosen over the DWARF route deliberately: DWARF still owes a `!DILexicalBlock` per `Let` before
|
||
`p v` under shadowing is even honest, and that buys locals in lldb rather than in the break loop. The author's reason
|
||
is the one to keep in view — **the more a break loop can show, the less often a real debugger is needed** — which
|
||
makes this a dev-loop feature, not a debugger feature.
|
||
|
||
**4. Conditions get a parent link, not class inheritance.** A condition type may name a parent where it is declared;
|
||
matching walks that static chain. This buys the hierarchy §1 of `spec-conditions.md` says there is none of — a
|
||
catch-all handler, "any file error" — at compile-time cost only. **It is deliberately not the class answer** that the
|
||
"Open: can a condition be a class?" section below weighs: a class condition allocates at the signal site, which is
|
||
the failure path and sometimes the thing that failed; it inverts §5's lifetime, so something must own and free it;
|
||
and it lets a condition's layout change while a handler frame stands against the old one. A parent link has none of
|
||
those costs and leaves the frozen model otherwise intact. Real inheritance stays possible later if a case demands it;
|
||
this closes nothing off. That section stays open for the record but is no longer the blocking question for
|
||
`handler-case`.
|
||
|
||
~~**5. File I/O — `slurp` and `barf` — is the next stdlib work**~~ **Built.** It was the next stdlib work, after `Vec`, because `slurp` returns a string whose
|
||
length is not known until the file is read and therefore cannot exist before an allocator does.
|
||
|
||
## Decided later the same day, and queued
|
||
|
||
~~**6. A field label is written with a dot, not a colon, and the colon is reserved for keys.**~~ **Done.**
|
||
`{.x 1.0 .y 2.0}` is struct construction and `{inner .field}` is destructuring; the old spelling is refused, and the
|
||
refusal names the new one. `:keys` kept its colon — it names no field, so leaving it alone is what lets the dot mean
|
||
exactly one thing. 681 labels across 45 `.flan` files including `vendor/`, plus 94 more in the Flan embedded in
|
||
`lib/prelude.ml` and the tests. `Map` is now free to take `{:key value}` without colliding with struct literals. See
|
||
BUILT.md, "The colon belongs to keys".
|
||
|
||
**What it left for the Emacs lane, both verified.** `render.ml` still *prints* a struct with colons, deliberately:
|
||
`emacs/flan-inspect.el:165` parses that output and hard-codes the colon when it reads a field out, so the printer
|
||
has to move in the same commit as its reader. That half is still open and belongs with whoever next opens the
|
||
inspector. ~~And `flan-mode.el:61` font-locks `:name` as a constant with nothing matching `.name`, so a field label
|
||
is now unfontified where it used to be coloured.~~ **The font-lock half is done:** a field is drawn as a constant in
|
||
both of the spellings that exist while the corpus moves, so `{.x 1}` and the accessor `(.x v)` read alike, and the
|
||
keyword rule stayed where it was because the colon still means an enum member and a map key.
|
||
|
||
**7. `Map` follows Odin's implementation.** Read `base/runtime/dynamic_map_internal.odin` before writing any of it;
|
||
the checkout is at `~/Repositories/Odin`. Three properties are the ones worth copying, and they are stated in its own
|
||
header comment:
|
||
|
||
- **Open-addressed Robin Hood hashing at a 75% load factor.** No buckets, no per-entry allocation, and probe
|
||
distances stay even because a later arrival steals a slot from an earlier one.
|
||
- **Cache-line-aligned `Map_Cell` packing**, so no single key or value ever straddles a cache line and a linear probe
|
||
walks memory in a cache-friendly order. This is the part a hand-rolled open-addressed map usually gets wrong.
|
||
- **`uintptr` throughout** for sizes, masks and offsets, to keep sign-extension and masking instructions out of the
|
||
probe loop.
|
||
|
||
Its static/dynamic split is the same type-erasure this project already committed to: `Map_Info` carries size,
|
||
alignment and offsets, and the compiler emits the hash and equality pair per key type. `spec-memory.md`'s
|
||
structural-key restriction holds this to the built-in key set, so there is no dispatch to design.
|
||
|
||
**Why this will not be Python's dict.** Worth recording because it is the question that prompted the decision.
|
||
Python's dict algorithm is fine; what makes it slow is that every key and value is a separately allocated, reference-
|
||
counted object, and hashing and comparison go through indirect calls that cannot be inlined. Flan stores raw bytes
|
||
and compiles the hash and comparison concretely at each use. That difference is most of the gap before any
|
||
algorithmic cleverness. **jank is not the model** — it is Clojure, so its maps are persistent with structural
|
||
sharing, which plan.org rules out by name because shared structure destroys the clear ownership that is the whole
|
||
reason there is no collector.
|
||
|
||
## Decided in discussion, queued
|
||
|
||
**Globals in the break buffer — built.** One section under the stack, holding the union of the globals every frame on
|
||
the current stack references, each entry annotated with the frames that touch it and ordered by the innermost one. It
|
||
is `(:op "globals")` in `dev.ml` and `flan-cnr--insert-globals` in the break buffer. See BUILT.md, "Globals of a
|
||
stopped stack". ~~The one hole left open — the redefinition check is a fingerprint over a body's *slots*, so a new body
|
||
that names different globals while binding the same locals is not caught.~~ **Closed**: `Reach.ref_fingerprint` is a
|
||
second fingerprint over the set of globals a body names, carried beside the slot one in `%fninfo` and checked the same
|
||
way, and such a frame is now refused by name. Kept separate from the slot fingerprint deliberately, so `locals` still
|
||
reads a frame whose locals are fine and whose global attribution is not.
|
||
|
||
**The break buffer opens by itself when the program stops.** Today a condition stops the program and the buffer appears
|
||
only when `C-c C-b` is typed. `flan-dev--absorb` already inspects every reply for `:stopped` and a poll covers the case
|
||
where no reply is pending, so the client already knows the moment it happens and already moves the mode line from it —
|
||
this is a hook at a point that exists, not new plumbing.
|
||
|
||
Three things to settle while building it: whether it takes focus or only displays; whether `(pause)` should always take
|
||
the window, being a deliberate stop rather than a failure; and what it does when the program stops while point is
|
||
mid-edit in another buffer.
|
||
|
||
**~~`Handle` and the pool are the real gate on classes, and they are buildable now.~~ Built.** plan.org's rule is
|
||
that nothing starts on managed classes "until ordinary `struct`, `Handle`, and reload semantics are working". All
|
||
three now hold: structs work fully; reload works with one known hole (a changed signature is refused rather than
|
||
versioned); and `(Handle T)` and `(Pool T)` exist, with the enumeration primitive `migrate-instances` was blocked on.
|
||
See [`BUILT.md`](BUILT.md), "`(Handle T)` and the pool, which is what a stale reference answers with".
|
||
|
||
It was not an incidental precondition. `migrate-instances` has to *enumerate* live instances, and a pool behind a
|
||
generational `(Handle T)` gives that by construction while a world arena and an owned region do not. plan.org presents
|
||
the three storage strategies as a free choice and they are not: handles are the one that makes migration possible.
|
||
`(len p)` plus `(pool-handle p i)` is that enumeration, and it is two entry points rather than an iteration protocol.
|
||
|
||
Already banked, and it means classes are less work than plan.org implies: **a generic function is an indirection cell**
|
||
whose body is a dispatch table, which a reload extends. That is the expensive half of method dispatch, and it is built
|
||
and tested.
|
||
|
||
**Resource cleanup: `defer` stays the answer. `drop` is not built, and `with-cleanup` is not either.** Reached by
|
||
working the case through rather than by preference, so the reasoning is worth keeping.
|
||
|
||
`drop` was specified in `spec-memory.md` this morning. **This amends it: the hook is deferred, not built.** Three things
|
||
decided against it. It runs code somewhere the reader is not looking, which is the C++ behaviour the author explicitly
|
||
does not want. It would not even cover the motivating case — `Image` and `Texture2D` are *raylib's* types, and attaching
|
||
a hook to a foreign type is its own unsolved design question. And its one real advantage, cascading through a container,
|
||
is the case `Handle` makes rare: entities holding handles hold numbers, not resources.
|
||
|
||
`with-cleanup` / `unwind-protect` was also put and rejected: awkward with several resources, and it reads worse than
|
||
what already exists. The raylib begin/end pairs that seemed to motivate it are a macro problem, not a primitive one —
|
||
`with-drawing` and `with-mode-2d` are three-line macros once the expander lands.
|
||
|
||
**What to build instead is small: relax where `defer` may be written.** It is refused today inside a `let`, a loop or a
|
||
branch. The loop and branch refusals are right — `defer` is a *compile-time* construct, the cleanup copied into every
|
||
exit path, so "maybe registered" is not expressible and a loop body would fire once at function exit instead of once per
|
||
iteration. But **a `let` at the top level of a function body has exactly the function's extent** and always registers,
|
||
so it is as safe as function scope and is refused for a reason that does not apply to it. Relaxing it gives:
|
||
|
||
```
|
||
(defn load-brush []
|
||
(let [sheet (rl/load-image-from-memory ".png" brush-bytes)]
|
||
(defer (rl/unload-image sheet))
|
||
(set brush (rl/load-texture-from-image sheet))
|
||
(rl/image-flip-horizontal (addr sheet))
|
||
(set brush-mirrored (rl/load-texture-from-image sheet))))
|
||
```
|
||
|
||
Several resources are several defers, released in reverse, visible in acquisition order. A container of resources is an
|
||
ordinary loop inside the defer body — the manual cascade, three lines, at one level of nesting.
|
||
|
||
**Odin, for the record**, has no destructors, no drop and no finalizers: `delete` frees container memory and nothing
|
||
else, and resource release is `defer` at the acquisition site. That idiom does not transfer directly only because
|
||
Odin's `defer` is block-scoped; the relaxation above recovers most of it.
|
||
|
||
**The safety net, and the better use of effort: a debug tracking allocator.** ASan's leak detection covers memory
|
||
*instrumented* code allocated — the Flan allocator, and it is already wired up and clean. It does **not** cover a leaked
|
||
texture, because that memory belongs to uninstrumented raylib, which is the same reason the sanitizer sweep treats the
|
||
windowed examples as noise. But every raylib call goes through a *generated* wrapper, so a dev build can count
|
||
acquisitions against releases at that boundary and report what is still held at exit, by name. No hook, no type
|
||
annotation, nothing running at a distance — it does not change how code is written, it reports when something was
|
||
forgotten.
|
||
|
||
## ~~Before the batch below: read `DISCUSS.md`'s "NEXT SESSION STARTS HERE"~~ — **answered, and being unwound**
|
||
|
||
The architectural question raised at the end of 2026-09-12 — **putting the compiler inside the running program's
|
||
process** — was researched (`DISCUSS.md` §14), then built: `flan dev` is one binary and one process, and the editor
|
||
socket did not move. See "One process" in [`BUILT.md`](BUILT.md).
|
||
|
||
What it reopened is now being deleted one piece at a time, each with its own green run:
|
||
|
||
**One of the three was transport. The other two were concurrency, and were only mistaken for transport because the
|
||
socket was in front of them.**
|
||
|
||
1. ~~**The internal socket, the line protocol, and `Dev.deliver`/`result`/`ask`.**~~ **Done** — a delivery is a direct
|
||
call into the agent's verb table. Measured: the transport was ~50µs of a 23ms redefinition, so the end-to-end
|
||
number did not move. Code generation is 19 of the 22 milliseconds, which is the number any backend argument has to
|
||
start from.
|
||
2. ~~**The 4K `RESULT_MAX` cap** in `runtime/flan_dev.c`.~~ **Refused, with half of it deleted** — it is not a
|
||
transport buffer. The agent's second copy of the number and the drift check between the two files were, and those
|
||
are gone. The bound itself is the buffer the *game thread* writes into, so a growable one means the frame thread
|
||
calling `realloc`, and it would break the seqlock — which is a protocol about torn contents and assumes the address
|
||
it copies from does not move. Removing it is a redesign of the read, and belongs with moving the read to a frame
|
||
boundary.
|
||
3. ~~**`flan_agent.c`'s snapshot copying and generation stamping.**~~ **Refused, in full** — it was never about two
|
||
address spaces, it is about two threads, and there are still two. The break loop polls, a thunk it runs is
|
||
arbitrary Flan that pushes and pops the live restart list and the shadow stack, and the generation stamp is what
|
||
keeps a nested break from claiming a choice made against the outer one. A pointer is meaningful to the compiler
|
||
now; the frame it points into is no more alive for that.
|
||
4. **The render-thunk-per-inspection design for locals and globals.** A redesign rather than a deletion, and its own
|
||
lane: it is what unblocks "the inspector can retain a value".
|
||
|
||
Still reopened and still undecided: the watch design (push was chosen partly because polling costs a compile), and
|
||
whether an in-process JIT or a hand-written backend is needed at all.
|
||
|
||
## ~~Queued: a second tier of the standard library, after macros~~ — **landed**
|
||
|
||
See [`BUILT.md`](BUILT.md), "The prelude's second tier". The diagnosis here was right and the prelude had 44
|
||
allocation-free functions because there was nothing to allocate from; there are 24 more now, and the "Refused, by
|
||
name" block at the foot of `prelude.ml` is down from eight entries to four, each with a *different* reason rather
|
||
than the one shared sentence.
|
||
|
||
What landed: `append!`/`append-i64!`/`append-f64!` (the builder), `concat`, `join`, `split` returning a
|
||
`(Vec [u8])`, `repeat-bytes`, `replace-bytes`, `to-lower`, `to-upper`, `slices-new`; `format-f64` with a precision;
|
||
`atan2-f32` and `pow-f32`; `clamp` as a `defmacro`; and the slice family at two more element types —
|
||
`sort-f32!`, `reverse-f32!`, `swap-f32!`, `min-f32`, `max-f32`, `sum-f32`, `bytes<?`, `swap-bytes!`, `sort-bytes!`.
|
||
Tests: `programs/strings.flan`, `programs/format.flan`, `programs/algorithms.flan`, `programs/math2.flan`.
|
||
|
||
`string-from-bytes`, refused in that block, turned out to already exist: `string` is a builtin and
|
||
`(string (as-slice v))` is the round trip.
|
||
|
||
**What could not be built, and why each one could not.** All four want a compiler or runtime change, and none of
|
||
them wants a language decision.
|
||
|
||
- ~~**`Map` keys and values.**~~ **Iteration is built** — `flan_map_next` and the `map-next!` builtin, exactly the
|
||
shape this described. See [`BUILT.md`](BUILT.md), "`map-next!`, the one thing a Map could not do".
|
||
`map-keys`/`map-values` as *prelude functions* stay refused, and the reason is now generics rather than the
|
||
iterator: a `defn` has to name its types and `(defn map-keys [m {K V}] (Vec K))` has no `K`. The loop is three
|
||
lines at the call site, where `K` is known.
|
||
|
||
- ~~**`map`, `filter`, `reduce`, and a sort taking a comparator.**~~ **All four are in the prelude.** The diagnosis
|
||
was right and is now evidenced: **function values, not generics** — they arrived with no generics at all. See
|
||
[`BUILT.md`](BUILT.md), "Function values, with no capture". They are one copy per element type (i32 and f32), which
|
||
is the half generics would remove, and a `map` that *changes* the element type is the one shape that did not come
|
||
with them — one copy per ordered pair of types rather than per type.
|
||
|
||
**Capture is not built and escaping closures stay deferred.** An `fn` is lifted into a function of its own and
|
||
handed nothing but its parameters; a reference to an enclosing local is refused by name. That is what keeps a
|
||
function value a bare code address with no environment, and it is the next thing to want if a callback needs
|
||
state — `spec-memory.md`'s cases 1 and 2 are still the design to build from.
|
||
|
||
- **`(vec-new [u8])` is refused**, so a `(Vec [u8])` can only be made where the *context* names the type.
|
||
`check.ml`'s `vec_new_elem` accepts a single bare symbol naming a type and nothing else, and a `let` has no type
|
||
annotation to say it the other way round — so `split` needs a one-line `(defn slices-new [] (Vec [u8]) (vec-new))`
|
||
standing in as the place where the type is said. The fix is to let `vec_new_elem` take a type *expression* rather
|
||
than a name, which is the same parser that already reads `[u8]` in a parameter list.
|
||
|
||
- **An array literal cannot say it is `[f32]`.** A float literal defaults to `f64`, an array literal has no context,
|
||
and a `let` has no annotation, so `[3.5 -1.0]` is an `[f64]` and every element in `programs/algorithms.flan` is
|
||
written `(f32 3.5)`. Same shape of gap as the one above and probably the same fix.
|
||
|
||
Two smaller findings, both written down beside the code that ran into them:
|
||
|
||
- ~~**The prelude is never macro-expanded.**~~ **Fixed, and the diagnosis above was wrong in both halves** — see
|
||
[`BUILT.md`](BUILT.md), "A prelude function may call a prelude macro". The prelude *does* reach the expander; the
|
||
arity error came from the `Check.program` *inside* `Macro.compile`, where expansion is off. It is a cycle and not
|
||
an ordering — a macro module is compiled from the prelude — so moving the prepend would have changed nothing. What
|
||
fixed it is `Macro.reduce`, which makes the prelude smaller for that one build, plus dropping the prelude's own
|
||
macros from the forms fed back as `extra`. `format-f64` is `(clamp prec 0 9)` now. "A prelude macro may not call a
|
||
macro" stands and names itself when violated.
|
||
|
||
- **A returned `Vec` is a move, and the dead set spans the function**, so an early `(return v)` on one branch kills
|
||
the binding for the `v` at the foot of another. `replace-bytes` guards its empty-needle case with an `if` rather
|
||
than a `when`/`return` for that reason. Probably correct as it stands — the analysis is not path-sensitive and
|
||
making it so is a real piece of work — but it is a shape that reads as though it should compile.
|
||
|
||
Already present and easy to miss: an **EDN parser**, at `vendor/edn/edn.flan`.
|
||
|
||
## Decided: the Clojure patterns we are deliberately not copying
|
||
|
||
This language borrows Clojure's shape and is **not** trying to be Clojure-compatible, so its known wrinkles are ours
|
||
to avoid rather than inherit. Four, with what to do instead.
|
||
|
||
**1. One argument-order rule, held everywhere.** Clojure's sequence functions take the collection *last*
|
||
(`(map f coll)`) and its collection functions take it *first* (`(assoc m k v)`). The split is deliberate there, and it
|
||
is why Clojure needs **two** threading macros instead of one. **Our rule: the thing being operated on comes first.**
|
||
That is already what the language does — `(at a i)`, `(len xs)`, `(push v x)`, `(as-slice v)` — and `into` follows it
|
||
with the source first. Hold it; do not ship two of anything to paper over a split.
|
||
|
||
**2. A membership test says which thing it tests.** Clojure's `contains?` checks *keys*, so `(contains? [1 2 3] 1)` is
|
||
true because index 1 exists — the most-cited confusion in the language, and there is no built-in for "is this value in
|
||
this list". Half of this is already right here: `has-key?` on a `Map` is named for what it does. If a value-membership
|
||
test is added for sequences, name it for values and never overload one name across both meanings.
|
||
|
||
**3. A predicate returns a boolean; a search returns what it found.** Clojure's `some` returns the *value*, so
|
||
`(some even? [1 2])` is `true` but `(some identity [nil false])` is `nil` — one name doing two jobs. Keep them apart:
|
||
a `?` name answers yes or no, a finder answers the thing or nothing, and neither pretends to be the other.
|
||
|
||
**4. Composition reads in the same direction as threading.** Clojure's `comp` is right-to-left while `->` is
|
||
left-to-right, so the two compose mentally in opposite directions. If anything here ever composes operations, it reads
|
||
left to right, the way `into` does.
|
||
|
||
Sources are community consensus rather than a specification; the `contains?` complaint is documented in *Getting
|
||
Clojure*. Recorded because these are cheap to honour now and expensive to unpick once a standard library depends on
|
||
them.
|
||
|
||
## ~~Queued: `into`, fused transformation without transducers~~ — **landed**
|
||
|
||
Decided in conversation. **Not transducers, and not Rust's iterators — a macro that fuses the chain at compile time.**
|
||
|
||
```
|
||
(into xs (vec-new i32) (map double) (filter even?))
|
||
```
|
||
|
||
Argument order is **source, destination, then any number of transforms**, matching the `into->` macro the author
|
||
already uses in Clojure (`from to xform & xforms`). It reads as a sentence — take this, put it there, doing these —
|
||
and the variadic transforms have to trail anyway, which is the mechanical reason they cannot sit in the middle.
|
||
Clojure's own `into` composes them into one `xform` first, which is why that macro exists at all.
|
||
|
||
**Why a macro and not transducers.** Transducers compose at *runtime*: they need function values, closures and
|
||
allocation, and every element pays a chain of indirect calls. Rust has no transducers — it has iterators, which are
|
||
lazy but fuse into a single loop at compile time via monomorphisation and inlining, needing generics to do it. A
|
||
macro reaches the same destination with neither: `(map double xs)` expands to `(double x)` written straight into the
|
||
loop body, so the function name is *syntax* and never a value. **No intermediate collection at any step, no closure,
|
||
no generics, and nothing to inline.**
|
||
|
||
**It therefore does not need function values** and is independent of that work.
|
||
|
||
**What it gives up, and the author does not want it anyway:** you cannot build a transformation at runtime and pass it
|
||
around. That is transducers' actual selling point and it is close to useless in a game.
|
||
|
||
**Why the destination belongs in the form, and why this suits Flan better than `->>` would.** Every collecting
|
||
operation here allocates from an *explicit* allocator — that is a frozen rule in `spec-memory.md`. `->>` hides where
|
||
the result goes; `into` names it, so the macro knows the destination type, emits the right loop and the right
|
||
allocation, and the rule is honoured by construction. plan.org's `->>` threading over slices is the thing this
|
||
replaces for the collecting cases.
|
||
|
||
**Open, and worth settling when it is built:** whether reductions share the form. `(into xs 0 (map cost) (sum))`
|
||
reads oddly because zero is not a collection. A second macro with the same shape may be cleaner, so that the
|
||
destination is always honest about what it is.
|
||
|
||
Drop Clojure's `:eduction` branch — that is the pass-around case, and the one part that would need runtime machinery.
|
||
|
||
**Done.** See *`into`, which fuses at compile time because it is a macro* in [`BUILT.md`](BUILT.md). It is a prelude
|
||
`defmacro` over a plain `defn` that walks the transforms in reverse, and all four of the macro limits bit without
|
||
blocking anything: the three refusals are names nothing defines, `into-wrap` uses only special forms so
|
||
`Macro.reduce` does not drop it, the quasiquotes are all single-level, and `into` lives in the prelude because a
|
||
macro is not importable.
|
||
|
||
**The open question is settled: reductions do not share the form.** The reason the destination sits in `into` at all
|
||
is that the destination *is* the allocation, which is what makes `spec-memory.md`'s explicit-allocator rule true by
|
||
construction. A seed is not an allocation, so `(into xs 0 (map cost) (sum))` would be a second form wearing the same
|
||
spelling and the destination would stop being honest about what it is. A reducing macro of the same shape is a
|
||
separate form the day something wants one.
|
||
|
||
Two things the design did not anticipate, both written up there. **A source that is already a name is used as it is**
|
||
rather than bound — a `(Vec T)` is move-only, so binding it would take the caller's ownership for a read, and a fixed
|
||
array would be copied once per `into`; a source that is anything else is still bound once, which is what a call
|
||
needs. And **an owning temporary as the source leaks**, because the macro binds it to a name the caller cannot reach
|
||
and cannot know whether the type owns anything. A call in that position should borrow. `drop` is what would close
|
||
this, and it does not exist.
|
||
|
||
## ~~Queued: `loop`/`recur` (the return type is done)~~ — **landed**
|
||
|
||
~~**1. A `defn` must always state its return type, and unit is written `()`.**~~ **Done.** See *The return type is
|
||
the slot, and unit is `()`* in [`BUILT.md`](BUILT.md).
|
||
|
||
The slot after the parameters is unconditionally a type, the pre-pass that collected a file's type names is gone
|
||
along with `is_type_form`, `qualified_type`, `types_in`, `declared_types` and `prelude_types`, and `Parse.decl` no
|
||
longer takes a set of names at all. `(defn f [] f65 0.0)` now says *unknown type f65 — did you mean f64?* instead of
|
||
*unknown name*. `()` is the only spelling of unit: `Unit` is refused with a message naming it, and `Types.to_string`
|
||
prints `()` too, because that printer prints what a person would write for every other type it knows.
|
||
|
||
Two things the plan did not anticipate. `(defn f [] ())` — a return type and an empty body — is a shape the optional
|
||
slot could not produce, and it needed its own arm. And dropping `prelude_types` removes one of the two reasons
|
||
`Macro.reduce` may only drop `defn`s: the memoised set that a bootstrap build could have poisoned no longer exists,
|
||
so what is left is the plain one, that the surviving functions still mention those types.
|
||
|
||
**The sweep is `tools/unit-return.py`**, kept rather than thrown away, because the lanes that branched before this
|
||
wrote Flan in the old spelling and their files want the same pass at merge:
|
||
|
||
```
|
||
python3 tools/unit-return.py .
|
||
python3 tools/unit-return.py --in-strings test/test_flan.ml test/test_acceptance.ml \
|
||
test/test_session.ml emacs/test-flan-dev.el emacs/test-flan-mode.el
|
||
python3 tools/unit-return.py --raw-ml lib/prelude.ml
|
||
python3 tools/unit-return.py --in-html web/index.html
|
||
```
|
||
|
||
`-v` logs every `defn` it saw and what it decided; `--check` changes nothing. It is re-runnable, and on this tree it
|
||
reports exactly six sites, all in `test_flan.ml`, which spell the refused forms *on purpose* so the refusals can be
|
||
tested. Read the diff of every non-`.flan` file — BUILT.md lists what the script can and cannot see.
|
||
|
||
~~**2. `loop` and `recur`.**~~ **Done.** See *`loop` and `recur`, and why `recur` is better than tail calls and not
|
||
only cheaper* in [`BUILT.md`](BUILT.md). `emit.ml` is untouched: a loop is a `let`, a `While` whose condition is
|
||
`true`, and two jumps, and the barrier question `recur` asks is the one labelled `break` already answered.
|
||
|
||
Three things the plan did not anticipate, each written up there. **Tail position is a permission that is withdrawn**
|
||
rather than a pre-pass over the `Ast` — `ctx.tail` is read and cleared at the top of `check`, exactly as `defer_ok`
|
||
is, and handed back only by the three forms that pass a tail through, so nothing has to enumerate the forms that do
|
||
not. **`loop` is itself a barrier** for `break` and `continue`, which is a restriction added rather than inherited: a
|
||
loop answers with the value of its body, so a jump out of one has no value to give, and therefore `loop` also takes
|
||
no label. And **the move tracker had to be told about the loop's own names**, which are bound before the loop entry is
|
||
pushed and would otherwise have tripped the "moves a value bound outside the loop" rule on the ordinary case.
|
||
|
||
Still not given, and still out of scope: **mutual recursion between two functions.** That needs real tail calls. The
|
||
refusal for a `recur` outside any loop says so by name.
|
||
|
||
## The next batch, in order
|
||
|
||
Agreed at the end of 2026-09-12. Ordered by priority, not by size. Items 1-3 and 5-6 want the compiler core and should
|
||
run one lane at a time; item 4 is disjoint and runs alongside any of them.
|
||
|
||
~~1. **Fix the one failing test** — `the frame of a superseded body answered with the new body's names`.~~ **Done**, and
|
||
the handoff's diagnosis was wrong. Nothing was dropping the number: the fingerprint was emitted into `%fninfo` and
|
||
never read back. `flan_dev.c` called the field `spare`, there was no accessor, the agent never snapshotted it, the
|
||
backtrace line never carried it, and `Dev.locals` compared slot counts and nothing else — four of the five
|
||
hand-offs were never written, and printing both sides of the comparison could not have found it because there was
|
||
no comparison. The mechanism was sound and stayed: it hashes slot *names* as well as types, so it does see a
|
||
rename. See BUILT.md, "Locals of a stopped frame".
|
||
|
||
2. ~~**The colon-to-dot change.**~~ **Done**, and `Map` is unblocked: `{:key value}` is free. The sweep is
|
||
`tools/colon-to-dot.py`, kept rather than thrown away, because the lanes that branched before it wrote Flan in the
|
||
old spelling and their files want the same pass at merge — `python3 tools/colon-to-dot.py .` over the tree, and
|
||
`--in-strings` for a `test/*.ml` that embeds Flan.
|
||
|
||
3. ~~**`Map`, and the `defer` relaxation.**~~ **Both done.** See *`(Map K V)`, which is Odin's map* and *`defer` may be
|
||
written in a `let`* in [`BUILT.md`](BUILT.md). The `defer` relaxation **amends `spec-memory.md`**, which said
|
||
`(defer (free v))` for a `let`-bound `v` was not expressible; it is now. `Map` restricts one thing the spec does not:
|
||
a fixed array is a key only when its elements compare bytewise, so an array of structs or of strings is refused by
|
||
name. The measured answer to "is this another Python dict" is six times quicker cache-resident and *slower* at a
|
||
million entries, and the second half is the interesting one — see below.
|
||
|
||
4. **The Emacs batch. Disjoint from the compiler, so it runs in parallel with anything above.** Globals in the break
|
||
buffer; the buffer opening itself when the program stops; ~~the indentation rewrite with `clojure-mode` as the
|
||
reference~~; ~~`#_`~~; hex, binary and addresses on primitives in the inspector. **The indentation rewrite and `#_`
|
||
are done.** The indenter is ported from `clojure-mode`'s source rather than derived from it — `flan-mode` still
|
||
requires nothing outside stock Emacs — and it aligns a binding vector name-under-name, which is the bug that cost
|
||
friction on every keystroke. `defn` parameter lists and `restart-case` clause parameters were the same shape and
|
||
came with it. What remains in this batch is the break buffer and the inspector, and they are independent.
|
||
|
||
5. ~~**Union values**~~, then the macro expander, then `Result`/`try`. Promoted above `Handle` on the author's call —
|
||
macros are the thing most worth wanting, and unions are the only thing between here and them.
|
||
|
||
**Union values are done.** See *Unions, and the tag they carry* in [`BUILT.md`](BUILT.md). The diagnosis was right:
|
||
`Option` is a two-case union wearing a special coat, so `Tast.arm`'s `acase` and `binds` already were union shape
|
||
and `check_match` grew a second subject rather than a second path. A union is `Types.Named` exactly as a struct is,
|
||
so every path that merely carries a type learned nothing.
|
||
|
||
What the spec did not settle and this lane did: the tag is an `i32` and the payload a blob aligned to the widest
|
||
member of any case, so `%"U" = type { i32, [k x iA] }` is C's `struct { int tag; union {...} u; }` byte for byte —
|
||
checked against clang's answer for the same declaration. A value is `(U.C {.field value ...})` and construction is
|
||
**qualified**; a pattern is bare `(C x y)` and resolves against the scrutinee. Tags are declaration order from
|
||
zero, so **case order is part of a union's contract**: a zeroed union is the first declared case. A non-exhaustive
|
||
match is **refused**, never defaulted.
|
||
|
||
What is left for the macro lane, and it is one thing: **`load.ml:312` refuses an imported union outright**, so a
|
||
union is file-local. That is not a blocker for `Form` — the prelude is parsed and prepended into the same flat
|
||
namespace before `collect` runs, so a `defunion Form` in `prelude.ml` is an ordinary same-file declaration and
|
||
needs no import and no `load.ml` change. Verified by declaring one there and matching it from a program.
|
||
|
||
**Macros landed on top of this** and needed no `load.ml` change for `Form`, exactly as this said. See
|
||
[`BUILT.md`](BUILT.md), "Macros: the compiler dlopens the program", and the short list of what is left of them
|
||
below.
|
||
|
||
Macros are what buy `with-drawing` and `with-mode-2d` over raylib's begin/end pairs, the hiccup DSL if a JS backend
|
||
ever happens, and the removal of special forms from the compiler.
|
||
|
||
**`Result`/`try`** follows, being another union.
|
||
|
||
**Generics are deliberately NOT here** — and function values landing has *sharpened* the case rather than made it,
|
||
which is the useful update. `Vec` and `Map` needed none, being type-erased. Function values needed none. What
|
||
needs them is now concrete and small: the prelude's `map!`/`filter`/`reduce`/`sort-by!` are **two copies each**,
|
||
i32 and f32, differing in nothing but the element type; `map-keys`/`map-values` cannot be written at all because
|
||
a `defn` must name its types and `(defn map-keys [m {K V}] (Vec K))` has no `K`; and a `map` from `[i32]` to
|
||
`[f32]` would be one copy per ordered pair. A user-written allocator is *not* on this list any more — it wants
|
||
a C-shaped callback and somewhere to put a `flan_allocator`, neither of which is a type parameter.
|
||
|
||
6. ~~**`Handle` and the pool.**~~ **Built.** A reference to something that can die, that reports that it died rather
|
||
than silently resolving to whatever reused the slot. See [`BUILT.md`](BUILT.md), "`(Handle T)` and the pool, which
|
||
is what a stale reference answers with". A handle is one `i64` — slot index low, generation high — so it copies,
|
||
zeroes and compares like an integer and owns nothing; a live slot's generation is odd, which makes a zeroed handle
|
||
resolve to nothing rather than to slot 0; and a generation that would wrap retires its slot instead, because "rare"
|
||
is not an answer when the failure is the silent wrong one the type exists to prevent.
|
||
|
||
What the spec did not settle and this lane did, beyond those: `resolve` answers `(Option (Ptr T))` and not
|
||
`(Option T)` — the spec's own worked example is annotated that way, for the reason written a line above it, that a
|
||
pattern binding binds a value and a copy cannot be written back. `(len p)` is the *slot high-water* and `(live p)`
|
||
is the live count, in that direction, so a loop bounded by `len` cannot silently skip a live entry. A slot is
|
||
recycled by `(release p h)` on the owner and never by `free`, because a handle owns nothing and consuming one copy
|
||
would say nothing about the others — so `spec-memory.md`'s two release points are untouched.
|
||
|
||
**Two amendments to the frozen spec, both deferrals**: `.field` and `at` do not auto-deref a handle, and `deref` is
|
||
not overloaded on one. Neither can answer "gone", which is the whole job; the spec's own example resolves first and
|
||
matches, and that is the half that is right.
|
||
|
||
**Still open**: a `(Ptr T)` from `resolve` dies on any `insert` that grows the pool, the same explicit contract a
|
||
slice has against `push`. Chunked never-moving storage is the fix and it costs code. And a pool passed to a helper
|
||
is consumed, because there is no borrowing parameter — a pre-existing `Vec` gap, not a pool one.
|
||
|
||
7. ~~**`break` and `continue`, with loop labels.**~~ **Built.** Labels are Odin's in the head position, both blockers
|
||
are answered, and the refusals name the construct they refuse for. See BUILT.md, "`break` and `continue`, and the
|
||
rule that replaced a blanket refusal". What was settled in conversation before it was built, kept:
|
||
|
||
**Labels, Odin-style but in the head position.** A keyword names a loop and `break` takes it:
|
||
|
||
```
|
||
(while :outer (< i n)
|
||
(while (< j m)
|
||
(when (hit? i j) (break :outer))))
|
||
```
|
||
|
||
A keyword there is unambiguous because a loop condition is never one. It is **not a goto**: control can only leave
|
||
a loop it is already inside, which is what keeps it safe and is the same restriction Odin's labelled `break` has.
|
||
|
||
The two known blockers stand and must be answered: `check.ml`'s `in_frames` rule refuses `return` inside
|
||
`handler-bind`/`restart-case` **because `return` always crosses**, while `break` crosses only sometimes — a loop
|
||
wholly inside a `restart-case` body has a legitimate local break — so that blanket refusal has to become a
|
||
loop-depth-relative-to-frame-entry rule. And `continue` forces a `Tast.While` signature change to carry a latch,
|
||
because `check_dotimes` folds the step into the body and a `continue` branching to the header would skip it and
|
||
hang.
|
||
|
||
8. ~~**Errors: a structured value with spans and notes, and more than one per compile.**~~ **Built.** See
|
||
*An error is a value, and there is more than one of them* in [`BUILT.md`](BUILT.md). `Loc.Error` carries a
|
||
`diag` — a stable `kind`, a *span*, `notes` that each have their own span and severity, and the macro expansion
|
||
the error came out of — and `flan check`/`flan build` print the source line with the offending span underlined,
|
||
in the GNU format `compilation-mode` already parses. No editor work was needed and none was done.
|
||
|
||
What made it cheap, and is worth knowing before anything else is retrofitted onto locations: **the span went into
|
||
`Loc.t` itself**, as an exclusive end defaulting to the start. A location nobody widened is a zero-width span at a
|
||
point, so every one of the ~260 refusal sites kept its meaning, only the reader had to learn to fill the end in,
|
||
and `Form`, `Ast` and `Tast` were not touched. **Macro provenance went the same way** — a `macro : string option`
|
||
on the location — because `Expand.unmarshal` already stamps the call site onto every node a macro produces, so the
|
||
tag travels to the checker for free.
|
||
|
||
**Three things deliberately not built, so they do not read as oversights:**
|
||
|
||
- **The reader does not collect.** There is no resynchronising a paren stream — after an unclosed bracket nothing
|
||
knows whether the next `)` closes this form or the one above it. First error, stop.
|
||
- **Pass one of the checker does not collect either.** Signatures are a foundation: a declaration pass one could
|
||
not make sense of leaves a hole that pass two reports once per mention, and thirty "unknown name" lines under
|
||
one wrong signature are the same error thirty times. Pass two — bodies, where the volume is — collects per
|
||
declaration.
|
||
- **There are not a hundred kinds.** The reader's fourteen have them and the checker's have them where a test
|
||
asserts on one; `check.ml` alone has 163 refusal sites and minting an id for each is a sweep nothing reads.
|
||
- **`Load` and `Shim` do not collect.** They sit between the two collecting phases and still stop at the first
|
||
refusal, for pass one's reason: an import that could not be resolved leaves a hole the checker would report
|
||
once per use.
|
||
|
||
**One claim checked rather than assumed,** and it is weaker than it first reads: `compile.el` groups `note` with
|
||
`info` at level 0, and `compilation-skip-threshold` defaults to 1, so `next-error` walks the **errors** with no
|
||
configuration — that part holds — but steps over the notes unless the threshold is set to 0. The notes are still
|
||
parsed, coloured and clickable. Labelling them `warning:` would make them navigable and is refused: a note is not
|
||
a warning.
|
||
|
||
**What the daemon sees, which the brief asked to be worked out and stated:** the single-diagnostic exception is
|
||
still the single-diagnostic exception. `Session.eval` and the daemon check one form, keep catching `Loc.Error`,
|
||
and take a location and a message out with `Loc.summary`; `dev.ml` and `session.ml` needed nothing but the
|
||
pattern rewrite. The list is a second exception, `Loc.Errors`, raised only by `Parse.program_all` /
|
||
`Check.program_all` — **separate names rather than a `~keep_going` flag**, so a list cannot reach a handler that
|
||
does not name it without somebody editing the session.
|
||
|
||
**Left for later, small and independent:** notes on the type-mismatch errors, which are the most common class and
|
||
want the *parameter's* declaration as the second place — `env.fns` stores types and not locations today, so that
|
||
is a small change to what `collect` records. And a checker error on macro-produced code names the macro but has no
|
||
separate location to point at, because the expansion has no source of its own; the note lands on the call site
|
||
beside the error, which tells the reader the code being refused is not the code they wrote and no more than that.
|
||
|
||
8b. ~~**The old entry, kept for its one extra fact.**~~ **Subsumed by 8, and it was right about the tooling:** no
|
||
editor work was needed and none was done. Flycheck and a structured JSON report stay refused for the reason it
|
||
gave — the workflow is compile-at-the-end, not live linting.
|
||
|
||
9. **Signature generations and stale-caller warnings.** The biggest remaining hole in "you never restart the program" —
|
||
a changed signature is still refused rather than versioned. Last because it is the largest and nothing else waits on
|
||
it.
|
||
|
||
Deliberately not scheduled: the JS backend and header-based C interop, both large and neither blocking current work; a
|
||
debug tracking allocator, which is the leak safety net and a good candidate whenever it is wanted.
|
||
|
||
## Decided in discussion — the array constructor and the module system
|
||
|
||
**`(array 4 rl/Vector2)` makes a fixed array; `[4 T]` stays the type syntax. Built** — see BUILT.md, "`(array 4 rl/Vector2)`, and the one position with no type slot". The problem it solved: a `let`
|
||
binding takes no type, so `(let [pts [4 rl/Vector2]] ...)` reads `[4 rl/Vector2]` as a two-element array *literal* and
|
||
fails with *unknown name rl/Vector2*. It cost 32 hand-written `Vector2`s in one raylib example.
|
||
|
||
`[4 T]` is not a special syntax — it is the ordinary type syntax and already works everywhere a type is expected:
|
||
`(defvar points [4 rl/Vector2] ...)`, `(defn draw [pts [4 rl/Vector2]] ...)`. A `let` binding is the single position
|
||
with no type slot, which is the whole of the bug.
|
||
|
||
`(zeroed [4 rl/Vector2])` was proposed first and rejected on how it reads: in argument position the bracket form is
|
||
unambiguous to the *parser*, but it still looks like a two-element vector to a person. `(array 4 rl/Vector2)` says what
|
||
it does with the count and the type as plain arguments. `zeroed` keeps its existing job — an empty thing of whatever
|
||
type the destination wants — and `array` is the one that is told.
|
||
|
||
**The module system stays as it is: the directory name is the module name.** No `package foo` line at the top of each
|
||
file. Confirmed against Odin, which requires the declaration despite having the same one-package-per-directory rule —
|
||
`package os` appears in 85 files, all of them in `core/os` — so the line is ceremony that buys only the ability to
|
||
disagree with the directory name.
|
||
|
||
What the rule already gives, and what was checked in conversation: **several files in one directory are one module**,
|
||
which is the case directory-as-package exists for; **a loose file is a module of one**, so several modules can sit at
|
||
the same filesystem level without a directory each; and two modules cannot share a directory, which is also true of
|
||
Odin.
|
||
|
||
**Acyclic imports are kept deliberately, not inherited by accident.** Odin forbids import cycles and so should this:
|
||
a definite package order is what the macro expander will need later, since every `defmacro` must be compiled before
|
||
anything that calls it. Nested import paths not being real nesting — Odin's `core:math/bits` is a separate package
|
||
rather than a submodule of `math`, with no re-exporting — was reviewed and accepted as fine.
|
||
|
||
**A package importing a package has landed**, so a project is no longer an entry file plus one flat layer of
|
||
libraries. Four things were settled doing it:
|
||
|
||
- **A name imported *through* a package keeps the inner alias.** If `area/` imports `shape`, the type is `shape/Box`
|
||
in the finished program and never `area/shape/Box`. This is forced rather than chosen: a directory reached along two
|
||
routes has to arrive under one set of names, or the checker sees every declaration twice and two copies of one
|
||
struct fail to unify. It is also what makes the dedupe coherent, and what keeps a qualified name the resolvable
|
||
identity the `layout` op and the break loop depend on.
|
||
- **The same directory under two aliases is refused**, including when one of the two aliases is a package's own and
|
||
pages away from the other. That is the price of the rule above and the refusal names both aliases.
|
||
- **A diamond loads its bottom once**, keyed by the real path.
|
||
- **A ring is refused and named** — `a -> b -> c -> a`, not "there is a cycle". Tolerating one was the earlier
|
||
behaviour and looked like it worked; what it cost is a definite package order, which is the thing the macro expander
|
||
needs, since every `defmacro` must be compiled before anything that calls it.
|
||
|
||
`Load.t.pkgs` now comes back in topological order, dependencies first. The *declaration* list is deliberately not
|
||
sorted and does not need to be — `check.ml` collects every top-level name before it checks any body.
|
||
|
||
**The expander did not end up reading that order**, and it is worth saying so rather than leaving the paragraphs above
|
||
to imply otherwise. Macros are collected from the prelude and from the file being compiled; a `defmacro` in a package
|
||
is refused by name, because reaching one means resolving that package's own imports over `Form`s before `Load` runs.
|
||
The order is there and correct and is what package-level macros will read on the day they exist; nothing reads it
|
||
today.
|
||
|
||
**Still missing: package visibility.** `rl/get-color-raw` is callable. The blocker is surface syntax, not `load.ml`:
|
||
`exported` and the refusal machinery already exist and take a second rule in one line, but there is no way for a
|
||
package to *mark* a name private, and adding one means a parser change.
|
||
|
||
## Decided in discussion — three more, two now built
|
||
|
||
~~**A watch window, ported from the author's Clojure one.**~~ **Built.** See `BUILT.md`, "The watch window, and why
|
||
it is the only listing that is pushed", and `emacs/MANUAL.md` under "Looking at values". Three of the original's
|
||
decisions were kept unchanged — the program decides what is shown, the request is async, and the paint is
|
||
`replace-buffer-contents` so point and scroll survive every tick.
|
||
|
||
**The design written here was superseded, and the correction is the interesting part.** This entry said the answer to
|
||
an expensive eval was to **compile the watch thunk once and re-invoke it cheaply per tick**. That is the right
|
||
instinct about the cost and it is still a *poll*, and a poll has a defect that caching cannot fix: it cannot answer
|
||
while the program is **stopped**. A thunk runs at a frame boundary and a stopped program has no more frame boundaries
|
||
— which is exactly the moment you most want to see what the last frame held. So the direction was reversed instead:
|
||
the program calls into a table from inside its own loop and Emacs reads the table, which is memory rather than an
|
||
evaluation. That also made the values update at *frame* rate rather than at the timer's, which the compile-once poll
|
||
could not have done at any price.
|
||
|
||
**What was not built, deliberately: the `(watch "hp" hp)` form.** Scalars work today through `declare-c` against four
|
||
runtime entry points, which needs no compiler change at all. A struct or a slice needs a compile-time walk over its
|
||
type — one arm in `check.ml` beside `print`, which `BUILT.md` writes out in full — and that file is held by another
|
||
lane, so it was left alone rather than reached into.
|
||
|
||
~~**Ghost text is gated on that same arm**, which is the finding worth keeping.~~ **Wrong, and ghost text is built.**
|
||
The reasoning was that values shown inline need a *place*, that nothing in the table has one — `(watch-i64 "ticks"
|
||
ticks)` says what the value is called, not where it was written — and that carrying a source location means the
|
||
caller supplies it, which means a generated call site. Every step of that is still true **of the table**, and the
|
||
conclusion did not follow: the call site is in the *buffer*, and the name in the table is the string literal in it,
|
||
so the anchor is searched for rather than reported. Nothing new is asked of the daemon. The two open questions
|
||
`flan-watch.el` recorded are answered rather than solved — overlays are replaced wholesale every repaint, so an edit
|
||
has nothing to invalidate, and a watch in a loop shows the last value written exactly as the buffer does, because
|
||
every better answer is the query UI this design exists to avoid. See `BUILT.md`, "Ghost text finds its anchor in the
|
||
buffer, not in the table".
|
||
|
||
~~**The inspector gets a second way to start: an address and a type.**~~ **Built.** See `BUILT.md`, "Two ways to root
|
||
a walk, and why neither subsumes the other". It went in as a frame and a slot *index* rather than an address and a
|
||
type — the daemon holds both and an index is the thing the listing can hand back, while an address is not something an
|
||
editor should be holding. The one prediction that did not survive contact: `l` crossing between the two modes was
|
||
listed as a cost and is not one, because a stack entry carries its own root and a mixed stack cannot be built.
|
||
|
||
**Structural typing requires identical layout — same fields, same types, same order.** Settled by the author, and it
|
||
makes the feature simple rather than hard: structural compatibility becomes "the same memory", which costs nothing at
|
||
run time and needs no copy, no reordering and no adaptor. The motivating case is `{.x 1.0 .y 1.0}` and that order is
|
||
natural anyway.
|
||
|
||
**Flexible field order waits for classes, deliberately.** A class has an implementation-defined representation, so the
|
||
compiler owns the layout and field order stops being observable — any order can match. That is the right place to pay
|
||
for flexibility, because a class already carries identity and metadata, and a `Vector2` should pay for neither. See the
|
||
`defclass` entry: `Handle` was the gate, and it is built now.
|
||
|
||
Note what this settles from the earlier discussion: writability was the question that decided layout, and requiring
|
||
identical layout answers it — fields are writable on the ordinary terms, by value a copy and through a `(Ptr T)` the
|
||
original, with no special case.
|
||
|
||
## Blocked and unfinished
|
||
|
||
Everything below was found, decided or half-built and then stopped. Each says what blocks it. Nothing here is a
|
||
vague intention — if it is listed, someone has already established it is real.
|
||
|
||
### Unblocked now, and ranked
|
||
|
||
**0. Signature generations and stale-caller warnings — milestone 7's unfinished half.**
|
||
Promoted here on the author's correction, and `session.ml:146` already says the same thing at the refusal itself. A
|
||
changed signature is refused today and **that is a placeholder, not the design**. plan.org's open decision #6 says what
|
||
should happen: a signature change makes a new version of the function, new callers resolve it, existing callers and any
|
||
stored `Fn` value stay safely on the old one, and the session *warns* at each tracked stale caller site. Milestone 7
|
||
names it outright — "signature generations and stale-caller warnings".
|
||
|
||
**The thesis of this project is that you never restart the program.** Every refusal that ends in "restart to change it"
|
||
is a hole in that, and this is the biggest one. It needs three things that do not exist: function versions, a
|
||
trampoline per version, and caller tracking good enough to name the sites. The cell already gives the indirection; what
|
||
is missing is that a cell holds one bare pointer with no signature, so there is nowhere to put a second version.
|
||
|
||
A changed **struct layout** is the genuinely hard case and plan.org still specifies it as a rejection — storage already
|
||
allocated has the old shape and a new body reads its fields at the wrong offsets. Managed classes are the planned way
|
||
through, with an explicit migration at a frame boundary. Do not conflate the two: one is unbuilt, the other is decided.
|
||
|
||
|
||
**From porting ten raylib examples** — the first code the language was pushed by that it was not designed around.
|
||
Ranked by how often they were hit, top two first because they are walls rather than conveniences:
|
||
|
||
1. ~~No number reaches `draw-text`.~~ **Fixed** by `(string b)`.
|
||
2. ~~An enum parameter cannot be driven by a loop variable.~~ **Fixed** by explicit conversions in both directions:
|
||
`(i32 k)` takes an enum to its integer, `(GamepadAxis n)` takes an integer to an enum. Neither is an instruction —
|
||
an enum is an i32 at run time and `emit.ml`'s `cast` already reduced one to that before choosing an opcode — so the
|
||
change is a guard in `check.ml`'s cast arm and nothing in the backend. The rule the refusals came from is
|
||
deliberately *not* relaxed: a bare integer still does not fit an enum parameter, so `:spcae` is still an error at
|
||
the call site. The rule was "an integer must not arrive silently", and a written `(GamepadAxis i)` is not silent.
|
||
The other escape stays closed too — one `declare-c` per C function — and no longer needs to be open.
|
||
- **A value that is no declared member is allowed**, deliberately. raylib's gesture is a bitfield and an OR of
|
||
flags is a legal `Gesture` that is no single member; and `session.ml`'s printer already falls through to the
|
||
number for an out-of-range enum, on purpose, so refusing to construct one while agreeing to print it would be
|
||
incoherent. An `Option` would make every site unwrap for no safety bought, and a literal-only refusal would
|
||
catch nothing, because the bitfield case is a run-time value.
|
||
- **Only an integer converts *to* an enum.** Not a float, and not another enum — a cross-enum hop goes through
|
||
`(i32 x)` so both ends are written down. Enum → any numeric is always allowed: lossless to i32 by construction,
|
||
and a narrower target truncates by the rule every int→int cast already follows.
|
||
- **The comparisons needed nothing else.** `(> (i32 g) 255)` checks because `binary` takes the non-literal side
|
||
first; `binary` was deliberately left ignorant of enums, since teaching it would be the implicit conversion this
|
||
avoids.
|
||
- **A bit-set type later builds on this rather than replacing it.** It would be its own type with its own
|
||
operations and would still want a named escape to the underlying integer for the FFI, spelled the same way. If
|
||
`Gesture` becomes one, the `(i32 g)` calls stay valid and only the range tests migrate to a membership test.
|
||
- One parse fix came with it: `defenum` names were not in `parse.ml`'s type set, so a local enum could not be a
|
||
function's return type. They are in it now under a key of their own, admitted as a bare symbol and never as a
|
||
list head — because `(Key n)` is a *value* now, and putting `Key` in `types` would make a body starting with one
|
||
be eaten as a return type.
|
||
3. ~~**`break` is not implemented.**~~ **Built, with `continue` and loop labels.** Both blockers are answered: the
|
||
`in_frames` rule became a relative one rather than a blanket one, and `Tast.While` grew a latch. See BUILT.md,
|
||
"`break` and `continue`, and the rule that replaced a blanket refusal".
|
||
4. ~~**A `let` binding takes no type annotation**~~ — still true, and **no longer the blocker it was**: `(array 4
|
||
rl/Vector2)` is built and is the answer to the case that raised it. The reasoning below is kept because it is what
|
||
chose between the three surfaces, and the first of them is not what was taken — see BUILT.md, "`(array 4
|
||
rl/Vector2)`, and the one position with no type slot". The original entry:
|
||
|
||
A fixed array is either a top-level `defvar` or a literal with
|
||
every element spelled out. `(let [pts [4 rl/Vector2]] …)` parses as a two-element array literal and fails with
|
||
*unknown name rl/Vector2*. Cost: 32 hand-written `Vector2`s in one example. **Looked at and stopped — it is a
|
||
grammar question, not a missing feature.** Everything under the surface is already there: `Ast.binding` carries a
|
||
`bty`, `load.ml` renames through it, and `check.ml:723` consumes it as the `want` for the value. Only the way it
|
||
is written is open, and the parser says so where it refuses (`parse.ml:366`): `let` is a flat list of pairs, so it
|
||
cannot disambiguate by *count* the way `defvar` and `defconst` do — those read `[n t v]` as three arguments to a
|
||
form, and there is no such boundary between one pair and the next. Three surfaces, in the order they are worth
|
||
considering:
|
||
- **`(zeroed [4 rl/Vector2])` — `zeroed` takes its type as an argument.** Recommended. It is one extra branch in
|
||
the arity-0 `zeroed` case in `check.ml`, no parser change, no ambiguity, and it answers the actual complaint,
|
||
which is not "locals cannot be annotated" but "there is nothing here to infer *from*". It also reads as what it
|
||
does: the value is a zeroed thing of that type, not a name that has been told what it is.
|
||
- **A marker between the name and the type**, `(let [pts :- [4 rl/Vector2] …] …)` or similar. Unambiguous, and it
|
||
buys a general annotation rather than one form's escape hatch. The cost is a new piece of syntax in the binding
|
||
vector, which is the one place this language has kept looking exactly like Clojure's.
|
||
- **Bare `(let [pts [4 rl/Vector2] …])`.** The obvious spelling and the one that cannot work: `[4 rl/Vector2]` is
|
||
a well-formed two-element array literal, and telling the two apart needs types in the parser, which there are
|
||
none of by design.
|
||
Note that plan.org's rule is "annotate function signatures, infer locals", so the general annotation is a
|
||
deliberate absence and not an oversight — which is the other reason the `zeroed` route is the smaller answer.
|
||
5. ~~**Arithmetic is strictly binary** — *+ takes 2 arguments, given 5*.~~ **Fixed.** `+ - * /`, `min`/`max` and
|
||
`bit-and`/`bit-or`/`bit-xor` fold left over two operands or more. `%` and the shifts stay at two, and one operand
|
||
is refused with the form to write instead — there is no unary minus and no reciprocal.
|
||
6. ~~**No `sin`/`cos`/`abs` for floats.**~~ **Fixed.** `sin-f32` and `cos-f32` are `declare`s in the prelude now,
|
||
with the caveat written beside them: IEEE-754 makes `sqrt` correctly rounded and requires nothing of the kind for
|
||
`sinf`, so these are the one place in the prelude where native and wasm32 may disagree bit for bit. Float `abs` is
|
||
not wrapped, for the reason integer `abs` is not — it is `(max x (- 0.0 x))` over two builtins.
|
||
|
||
**A `string` cannot be returned from C at all**, which is what makes `GetGamepadName` unbindable: *a string only
|
||
crosses as a parameter — a C function that returns one returns something Flan has no owner for*. Same rule refuses
|
||
`TextFormat`, which is also variadic and so has no honest signature.
|
||
|
||
**The negative result is worth as much.** None of the gaps expected blocked anything — no generics, no allocator, no
|
||
`Vec`/`Map`, no escaping closures, and function-scoped `defer` never came up. Input-and-draw over fixed-size state is
|
||
the shape the language already has. Three constructs unexercised anywhere else in the repo worked first try: a fixed
|
||
array with a struct element, a 2-D struct array, and `[N string]` as both `defconst` and mutable `defvar`.
|
||
|
||
### The web target: what it does not reach yet
|
||
|
||
`flan build --target=web` works, a raylib example builds unchanged and `test/test_web.ml` is green — see BUILT.md,
|
||
"The browser is the third target", for the mechanism and why asyncify rather than `emscripten_set_main_loop`. Four
|
||
things it does not cover.
|
||
|
||
~~**1. `sand.flan` has no web build, and the cause is one missing `#include`.**~~ **Built. It opens.** See BUILT.md,
|
||
"sand.flan in a browser", for the whole of it. Three summary lines, because the diagnosis below was right about the
|
||
structure and wrong about the cause:
|
||
|
||
- The `#include` was never the fix. **The agent is a socket server and a browser has no sockets**, so an agent that
|
||
compiles there is an agent that can never accept a connection. `vendor/agent/flan_agent.web.c` is three no-ops, and
|
||
`Build` selects it over `flan_agent.c` on `--target=web` and nowhere else.
|
||
- **Refusing `vendor:agent` on web was the honest-looking option and is ruled out by arithmetic.** There is no
|
||
conditional compilation, `sand.flan` calls `agent/start` unconditionally, `Reach` cannot prune a package something
|
||
reachable calls into — so a refusal means the flagship program does not build for the browser at all. A refusal is
|
||
only honest when the caller has a way to not ask. This does **not** reverse decision 2 above: `barf`'s no-op loses a
|
||
file the program believed it wrote, and there is nothing for the agent to lose because `--dev` is already refused by
|
||
name on every wasm target. The argument is written out at the top of `flan_agent.web.c`.
|
||
- **A package's `.c` files can now be addressed to a target**, by a tag in the name before the extension, and a tagged
|
||
file *replaces* the untagged file of the same base name on that target. This is the C-source half of the
|
||
`@native`/`@wasi`/`@web` link-line mechanism decision 2 pointed at for per-package target isolation.
|
||
|
||
The brush is `(embed "brush.png")` decoded through a new `LoadImageFromMemory` binding. `load-texture` and
|
||
`load-image` now have no call site anywhere in this repository — deliberately, because a path-based load is the one
|
||
shape the browser cannot have, and said here so it is not read later as an accident.
|
||
|
||
The original entry follows.
|
||
|
||
**1. `sand.flan` has no web build, and the cause is one missing `#include`.** `vendor/agent/flan_agent.c` does not
|
||
compile under emcc: *variable has incomplete type 'struct timeval'* at line 426, because emscripten's headers do not
|
||
pull `<sys/time.h>` in transitively the way glibc's do. `sand.flan`'s `main` calls `(agent/start ...)`
|
||
unconditionally, so `Reach` cannot prune the package, so the flagship program stops at that error — even without
|
||
`--dev`. Beneath the include is a structural fact worth deciding rather than patching around: **the agent is a socket
|
||
server and the browser has no sockets**, which is the same family as the `--dev` refusal. So the two fixes are not
|
||
equivalent — add the include and the agent compiles into a web build that can never accept a connection, or refuse
|
||
`vendor:agent` by name on a web target the way `--dev` is refused. The second is the honest one. Neither was taken
|
||
here: `vendor/agent/` belonged to another lane this session.
|
||
|
||
~~**2. Assets are two questions and only one of them is about emscripten.**~~ **Answered by the embed above, and the
|
||
answer was the third option neither half here considered: make it a compiler feature and neither question arises.** The
|
||
hard half below is exactly right about the problem — the file that needs the asset is structurally the one file that
|
||
cannot declare it — and the conclusion drawn from it, that the fix must be a link channel or a new declaration, was
|
||
the wrong one. `(embed "brush.png")` needs no channel, because there is nothing to tell the linker. What is *not* done
|
||
is `sand.flan` itself: `(rl/load-texture "brush.png")` takes a path and raylib opens it, so pointing raylib at embedded
|
||
bytes needs `LoadTextureFromImage` over `LoadImageFromMemory`, which is a raylib binding question and not this one.
|
||
The original text follows. `sand.flan` does `(rl/load-texture "brush.png")` against a bare relative path.
|
||
|
||
- The easy half: a bare relative path has no meaning on a target with no filesystem. emscripten's answer is
|
||
`--embed-file` or `--preload-file` into MEMFS, and both are *linker arguments*, so they are already expressible as an
|
||
`@web` line in a package's `link` file. No new mechanism is needed for a package.
|
||
- The hard half, and the actual design question: **the file that needs the asset is structurally the one file that
|
||
cannot declare it.** `Load` hands out `lflags` only for a directory package (`one_file` → `[]`), and `main` is not
|
||
exported, so a program can never be a package. The program doing the `load-texture` therefore has no link channel at
|
||
all. Answering this means either giving a single-file program a way to carry build arguments, or making assets their
|
||
own declaration rather than a linker flag. No flag was invented for it here.
|
||
|
||
**3. Nothing has been opened in a browser.** *Still true, and now it is the only thing left between here and
|
||
"someone played with it".* `sand.flan` builds for the web, the module carries asyncify, raylib's GL imports and
|
||
brush.png's own bytes whole, and `node sand.js` gets as far as `glfwInit` before dying on `window is not defined` —
|
||
which proves the module is live and proves nothing about the canvas. BUILT.md carries the exact commands to serve and
|
||
open it, and the list of what only a human will discover: whether it paints, whether the audio round trip through
|
||
MEMFS survives, and the canvas size. The `until` loop never exits on the web, so none of `main`'s `defer`s run —
|
||
expected, and worth knowing before reading anything into it.
|
||
|
||
The original entry follows.
|
||
|
||
**3. Nothing has been opened in a browser.** The test is headless and permanently so: it asserts the artifact's shape,
|
||
the `asyncify_start_unwind` export and the `glViewport` import, and that node runs the emitted JS. Whether the canvas
|
||
actually paints is unverified by anything in CI, and a human should look once.
|
||
|
||
**4. Unmeasured and untested.** Asyncify's cost is quoted from emscripten's documentation (roughly a doubling of code
|
||
size) and not measured here, and no frame time on web has been taken at all. raylib's audio and any use of threads on
|
||
the web target are untried. And a **wasi** build that reaches raylib now fails on undefined symbols rather than on a
|
||
missing `-l:libraylib.so.550`, because that line is tagged `@native` — the same error one step later, and a worse
|
||
message.
|
||
|
||
### `break`, and why it was not built — **it is built now**
|
||
|
||
Kept as written, because everything in it held and the two blockers at the end are the two things the build had to
|
||
rule on. Both are ruled on in BUILT.md, "`break` and `continue`, and the rule that replaced a blanket refusal":
|
||
the `in_frames` precedent was replaced by a barrier on the loop stack, which refuses a *crossing* rather than
|
||
everything, and `Tast.While` grew the latch. The original note:
|
||
|
||
Settled, so the next attempt is cheap rather than a rediscovery:
|
||
|
||
- **`dotimes` gets it free** — it desugars to `Tast.While`, so one implementation covers both loop forms.
|
||
- **`defer` is a non-question.** It is function-scoped, `break` does not leave the function, nothing fires. No refusal
|
||
needed and no interaction to design.
|
||
- **Type it `Never`**, as `exit` and `return` already are.
|
||
- **`pads` is the structural model.** `emit_while` already makes an `endloop` label; break is a push/pop of that around
|
||
the body plus a `br`. `return` is a direct terminator with no context threading, so there is nothing else to mirror.
|
||
|
||
What stopped it, and neither is small:
|
||
|
||
- **`check.ml`'s `in_frames` rule does not extend.** It refuses `return` inside `handler-bind`/`restart-case` because
|
||
those frames are popped on the way out, and that refusal is blanket because `return` *always* crosses. `break`
|
||
crosses only sometimes — a loop wholly inside a `restart-case` body has a legitimate local break — so the precedent
|
||
has to be replaced by a loop-depth-relative-to-frame-entry rule nobody has ruled on.
|
||
- **`continue` forces a `Tast.While` signature change.** `check_dotimes` folds the step into the body as
|
||
`While (cond, body @ [step])`, so a `continue` branching to the header skips the increment and hangs. It needs a
|
||
latch — `While of expr * expr list * expr list` — across `check.ml` and `emit.ml`. plan.org settles break and
|
||
continue as one item and `parse.ml` refuses them in one case, so building break against today's `While` is exactly
|
||
the thing that would have to be undone.
|
||
|
||
plan.org's single line on it (831) names a `for` the language does not have and gives no mechanism.
|
||
|
||
|
||
1. ~~**Allocators, then `Vec` and `Map`.**~~ **Steps 1, 2 and 3 are done** — the allocator, the arena, `(Vec T)`,
|
||
`StorageExhausted` and `retry`. `Map` is step 4 and is what is left of this item. See *Allocators, `(Vec T)` and
|
||
`StorageExhausted`* in [`BUILT.md`](BUILT.md) for the shape, the three amendments to a frozen `spec-memory.md` and
|
||
the one addition. The claim below held: **`Vec` does not need generics** — that was wrong and is worth un-learning: Odin's containers are compiler
|
||
builtins over a *type-erased* runtime (`base/runtime/dynamic_array_internal.odin`), where `$T` appears only in thin
|
||
wrappers producing `size_of`/`align_of` at the call site, and per-key hash and equality are compiler-emitted
|
||
procedures passed as a runtime argument (`Map_Info`, `base/runtime/core.odin:369`). That runtime is what
|
||
`spec-memory.md` specifies.
|
||
|
||
**The four questions that used to sit here are answered**, in `spec-memory.md`'s "Allocators" section, which is
|
||
frozen along with the rest of that file: when storage is released, the `drop` hook, alignment, and allocation
|
||
failure. Read them there rather than in a second copy here. The one consequence the build order below turns on is
|
||
that no allocating operation returns an error — a failure signals `StorageExhausted` under a `retry` restart — so
|
||
`push` and `put` are `()`, `clone` returns the container, and no signature grows a `Result`. One question is left
|
||
open in that section on purpose; it does not block the build.
|
||
|
||
2. **The editor half of a typed restart.** The language half is in (see "Landed"): `(use-value [v i32] ...)` and
|
||
`(invoke-restart 'use-value 21)` work, and a mismatch is refused at run time with both signatures in the message.
|
||
What is missing is the half only an editor can do — the leverage SBCL lacks. `eval` already compiles and runs an
|
||
expression inside the live program and the daemon already holds the struct layouts, so "ask the human, type-check
|
||
the answer, hand it over" is a short hop, and it is the one path the runtime today *refuses*: a restart with
|
||
parameters taken from the break loop traps, because `flan_break_resume` and `flan_restart_take` aim the channel at
|
||
a frame and have nothing to fill its buffer with. What it needs, end to end:
|
||
- the frame already carries the arity and the signature as a string — `flan_restart_arity` and `flan_restart_sig`
|
||
beside `flan_restart_name`, the same walk, so `restarts` can say what each one takes;
|
||
- `:restarts` on the wire carries the signature per entry, so the minibuffer can show `use-value (i32)` rather
|
||
than a bare name, and `restart-at` grows an `:args` form — a list of *expressions*, since the answer is a Flan
|
||
expression and there is already something that compiles one;
|
||
- the daemon compiles each argument against the declared type with the session's layouts (the same path `C-x C-e`
|
||
takes), refuses it there if it does not fit, and otherwise writes the values into the frame's buffer and marks it
|
||
filled before aiming the channel. That last store is what `flan_restart_take` cannot do today and is the whole
|
||
of the remaining work; the marking exists so this cannot be forgotten silently.
|
||
|
||
3. **`handler-case`.** Not a convenience — it is the fix for the loudest gotcha in `conditions.org`. A handler closes
|
||
over nothing *only because* a `handler-bind` clause runs at the signal point; a `handler-case` clause runs in the
|
||
establishing frame, which is ordinary in-frame code exactly like a `restart-case` clause. SBCL's is `handler-bind`
|
||
plus a transfer and nothing more (`src/code/error.lisp:196-268`). Every piece exists.
|
||
|
||
### `Vec` and `Map` — the order to build them in
|
||
|
||
**Steps 1, 2 and 3 are built; 4 to 8 are what is left.** The reasoning is kept because it is what the remaining steps
|
||
rest on, and because the escape it describes was tested rather than assumed — see *Allocators, `(Vec T)` and
|
||
`StorageExhausted`* in [`BUILT.md`](BUILT.md).
|
||
|
||
The dependency nobody had written down, and the reason it looked worse than it is. `spec-memory.md` defines an
|
||
allocator as "a procedure plus an opaque data pointer" — a function value. `check.ml` refuses function values four
|
||
ways, and all four say milestone 5: a written `(Fn ...)` annotation (`Ast.Tfn`), a written `fn` literal (`Ast.Fn`), a
|
||
`defn`'s name used as a value, and calling anything other than a named function. `(Result T E)` was still refused
|
||
beside those as milestone 6, as `(Map K V)`, `(Handle T)` and `(Vec T)` were when this was written; only `Result` is
|
||
now. Read straight off those lines,
|
||
milestone 6's allocators need milestone 5's function values and the work doubles.
|
||
|
||
**The escape is real and the work did not double** — this is the claim the built thing confirms. All four refusals are about *surface syntax*, and a value the
|
||
compiler builds that no surface form names trips none of them. The compiler already does exactly this, twice:
|
||
|
||
- A `handler-bind` clause is lowered to a function whose address goes into a `flan_handler` and is called back through
|
||
`h->fn(condition, xfer)` (`runtime/flan_rt.c:38` and `:66`). `check.ml` builds that body as its own `Tast.fn`
|
||
(`:619`, `:654`), not as an `Ast.Fn`, so line 458 never sees it, and no Flan type names the result.
|
||
- In a dev build, `emit.ml`'s `call` loads a pointer out of an indirection cell and calls through it
|
||
(`lib/emit.ml:781`–`793`). That is the indirect call line 1023 refuses in source, emitted routinely.
|
||
|
||
It is also what `spec-memory.md` already assumes for `Map`: the hash and equality pair is compiler-emitted and passed
|
||
as a runtime argument. Odin's `Map_Info` is two contextless `proc` fields (`base/runtime/core.odin:369`), and Odin's
|
||
`Allocator` is a `procedure` plus a `data: rawptr` (`:422`) — the same shape, reached the same way. If the hash pair is
|
||
expressible with no function type in the surface language, so is the allocator's procedure.
|
||
|
||
So: **`Allocator` is a builtin opaque type, the way `string` is a builtin ptr+len.** It is a `Types.t` case with no
|
||
user-writable constructor. Its procedure is an ordinary top-level function resolved to a symbol at the emit site, and
|
||
`vec-new`, `push`, `put`, `clone`, `free` and `free-all` are named calls, which `check_call` already routes through
|
||
`named_call` (`check.ml:1021`). **The built-in allocators need nothing from milestone 5.**
|
||
|
||
What *does* need milestone 5 is a **user-written** allocator: the moment a program says "here is my proc, make an
|
||
`Allocator` from it", it needs a `defn`'s name in value position, which is `check.ml:571` verbatim. That is a real
|
||
limit and not a fatal one — Odin ships arena, general-purpose, stack, pool and scratch in its own std, and most
|
||
programs write none. Ship the built-in set; user allocators arrive with function values.
|
||
|
||
**5 and 6 interleave rather than nest.** plan.org orders generics and macros (5) before allocators and containers (6),
|
||
and that order cannot hold: the macro expander is blocked on `Form` being a Flan union and union *values* are milestone
|
||
6 (see "Macros" below). Conditions and restarts, also listed under 6, are already three steps of four. The milestone
|
||
numbers are a topological hint, not a sequence. Take 6's container half first, 5's generics half second, and 5's
|
||
expander last, on 6's unions.
|
||
|
||
~~1. **`Allocator` and the arena.**~~ **Done.** The builtin opaque type, the four operations with `size` and `align`,
|
||
the capability set read off the allocator value, `with-allocator`, `context/allocator`, `context/temp` and the epoch
|
||
counter. `free-all` was decided as retain-capacity with `arena-destroy` beside it; the context is a dynamic variable
|
||
rather than a literal calling-convention parameter; both are stated as amendments in `BUILT.md`. A user-written
|
||
allocator is refused by name with milestone 5 as the reason.
|
||
|
||
~~2. **`(Vec T)`**~~ **Done**, over the type-erased runtime, with `push`, `reserve`, `at`, `len`, `as-slice`, `free` and
|
||
`clone`, and with move-only enforced by a dead set that unions at an `if` or a `match` join. `at` and `len` were
|
||
extended rather than duplicated. The header is six words in *every* build, not four in release — a layout that
|
||
changes with a build flag can disagree silently across the reload boundary — and that is the third amendment.
|
||
Ownership is not transitive yet, so a struct field of `Vec` type, a global `Vec` and a `(Vec (Vec T))` are each
|
||
refused where they are declared, naming `drop` as what they wait on.
|
||
|
||
The note below still stands and is now the only thing between `Vec` and the accumulation pattern: **capture does not
|
||
exist at all.** Nothing about it changed.
|
||
|
||
~~3. **`StorageExhausted` and `retry`**~~ **Done, with step 2 and not after**, exactly for the reason given. It is a
|
||
`while` around a `restart-case` around the attempt, built in the checker out of nodes that already existed, so the
|
||
backend learned nothing about allocation. `test/programs/exhausted.flan` exhausts an allocator for real and takes the
|
||
restart; `exhausted-unhandled.flan` is the same failure with nothing handling it.
|
||
|
||
~~4. **`(Map K V)`**~~ **Done**, following Odin: open-addressed Robin Hood hashing at a 75% load factor, cache-line
|
||
cell packing, and pointer-width integers through the probe loop. `map-new`, `put`, `get`, `has-key?`, and `len`,
|
||
`reserve`, `clone` and `free` extended rather than duplicated. Two departures from Odin, both deliberate: no
|
||
tombstones, because the spec defers removal, which deletes the backward-shift loop entirely; and no capacity tagged
|
||
into the data pointer, because this header has room and tagging would make correctness depend on an alignment that
|
||
is only requested. `Tast.FnAddr` carries the emitted hash and equality pair and is not a function value — the same
|
||
escape the allocator used.
|
||
5. **`drop`.** The hook, the transitive move-only and non-`clone`able rules, and the refusal to construct a
|
||
`drop`-carrying value against an allocator without `can-free`. It is additive — no type in the repo has a hook today
|
||
— but the `can-free` refusal has to land with the construction path it guards, before any arena-allocated container
|
||
of a user struct is trusted.
|
||
6. **`(Result T E)` and `try`, then the rest of union values.** Unions are what `Form` needs, and `Form` is what the
|
||
macro expander needs.
|
||
7. **Generics and monomorphisation**, then function values. User-written allocators and escaping closures both fall out
|
||
of the second.
|
||
8. **The macro expander**, last, on 6's unions.
|
||
|
||
**What is genuinely unsettled.**
|
||
|
||
- **The Map is slower than CPython's dict at a million entries** (1.41s against 1.16s on the same workload), while
|
||
being six times quicker cache-resident (21ns against 132ns per lookup at 10k entries). Both are memory-bound at the
|
||
larger size and this layout waits longer: keys, values and hashes are three separate runs, so a lookup that misses
|
||
everything costs three cache misses where a compact dict costs two, and the hash run is a full eight bytes a slot.
|
||
Cell packing buys probe locality, which is a win while the hash run is resident and a loss once nothing is. One byte
|
||
of metadata a slot — the Swiss-table arrangement — is the known answer and is not built. Worth measuring before
|
||
building: the crossover is somewhere between 10k and 1M and nobody has found it.
|
||
- **What is left at 18ns cache-resident is the type erasure itself** — one non-inlinable call into the runtime and two
|
||
non-inlinable indirect calls to the hash and equality pair. That is the trade `spec-memory.md` chose on purpose, and
|
||
monomorphisation is what would buy it back. It is a reason to want generics, not a reason to regret the choice.
|
||
- **A fixed array of structs or of strings is not a map key**, which is narrower than `spec-memory.md`'s key set. It
|
||
needs the per-element walk a struct key gets, driven by a loop rather than a field list. Refused by name rather than
|
||
written untested; a struct holding the array works today.
|
||
- **Map removal is not built**, which is what keeps the implementation free of tombstones and of Odin's backward-shift
|
||
loop. The spec defers it deliberately. When it arrives, that loop is the cost.
|
||
|
||
- `spec-memory.md`'s "Open: catching a use-after-release statically" is still open, and it is now open with evidence
|
||
available for the first time: the epoch trap is built and `test/programs/stale-region.flan` is the case it catches.
|
||
What the spec says would settle it — real Flan programs using arenas, to show whether the escapes that actually occur
|
||
are lexical — is now *producible*, because there is a `Vec` to write them with. That is the next thing to look at,
|
||
not the next thing to build.
|
||
- ~~**The operation table may be one operation short.**~~ **Decided.** `free-all` is retain-capacity and
|
||
`arena-destroy` hands the pages back — two names rather than the mode parameter, so the table the spec froze at four
|
||
operations did not grow. `BUILT.md` states it as the amendment it is.
|
||
- **The `Vec` header is six words in release too, and should not stay that way.** The 32-byte layout the spec fixes is
|
||
blocked on one thing: a redefinition module is built by `llc` and `ld` against a host built separately, and nothing
|
||
makes the two agree on a struct size. Give the reload path a way to carry the build flags and this falls out.
|
||
- **The generation word has no reader.** It is bumped on every reallocation as specified, and the stale-slice trap it
|
||
exists for needs a slice that can carry the Vec's identity — a slice is ptr+len. Either slices grow a word in a dev
|
||
build or the trap does not exist; today it does not.
|
||
- **The allocator grew a budget** (`alloc-budget` / `set-alloc-budget`), which `spec-memory.md` does not have. It is
|
||
there because `retry` is only answerable by a handler that can make the *same* request succeed, and for a fixed
|
||
backing store that handler is the one that raises the ceiling — releasing the region the container lives in
|
||
invalidates the container. Worth folding into the spec or replacing with a growable arena.
|
||
- Escaping closures are still deferred (`spec-memory.md`, "Function values", case 3), and a user-written allocator is
|
||
not one — its procedure is a top-level `defn` with no captured environment. The two should not be conflated when
|
||
function values arrive.
|
||
|
||
### Bugs found and not yet fixed
|
||
|
||
- **Two citations in `spec-memory.md`'s Allocators section do not land where they say.** Checked against Odin
|
||
`819fdc7a8` and Carp `ea121b5a`, every other one is exact — `Map_Info` at `base/runtime/core.odin:369`,
|
||
`Allocator_Proc` at `:422`, the arena answering `.Free` with `.Mode_Not_Implemented` at
|
||
`core/mem/allocators.odin:307`–`308`, `#optional_allocator_error` on `append_elem` at
|
||
`base/runtime/core_builtin.odin:767`, and `// TODO(bill): Better error handling for failed reservation` at
|
||
`base/runtime/dynamic_array_internal.odin:107` and `:128`. The two that miss: `Map_Cell_Info` is at `core.odin:351`,
|
||
not `:350`; and `check.ml:1670` is the FFI `Declare` arm, not the `defer` registration — the claim it is offered for,
|
||
that a top-level `defer` is checked in a scope holding only parameters and globals, is true and lives in `check_fn`
|
||
at `check.ml:1800`–`1812`. `check.ml:505` is the `defer` refusal exactly as cited, and the Carp citations are right:
|
||
`getDropFunc` is `Memory.hs:804`, the drop-before-delete emit is `Emit.hs:1044`, and `docs/Drop.md` says outright
|
||
that `A.drop` "will be run ... when the `let` scope ends".
|
||
|
||
- ~~`web/examples/breakdemo.out` is stale and `check.sh` fails on it.~~ **Fixed.** Commit `4a6a8fa` made the break
|
||
banner number its restarts and the `.out` was never repinned. Nothing had to drive the socket in the end:
|
||
`check.sh` already builds this one `--dev` and runs it under `timeout 5`, keeping what it printed before it
|
||
stopped, so the repin was the `.out` plus the two prose copies of the banner — `web/index.html` and `BUILT.md` —
|
||
and a sentence on the page saying what the numbers are for, since a restart is taken by position.
|
||
|
||
- ~~A shadowed restart is offered and cannot be taken.~~ **Fixed.** A restart is taken by *position* now:
|
||
`(:op "restart-at" :index N :name NAME)` on the daemon, `restart-at N NAME` on the agent, and a numbered
|
||
`completing-read` in `C-c C-b`. `:name` is a receipt, not the lookup — it is checked against the name the snapshot
|
||
holds at that index and refused if the two have drifted, so a bare integer can be wrong out loud. `restart <name>`
|
||
survives for a raw socket and is now defined as `restart-at` on the first index offering the name, so the two verbs
|
||
cannot disagree. `break.flan` grew the shadowed pair and asserts 900, which is the only value in that file no by-name
|
||
lookup can produce. The C&R buffer still marks the shadowed row by name and could now offer it instead — small, and
|
||
not done here.
|
||
|
||
- ~~A restart chosen at a break inside a thunk is accepted, announced, and silently not taken.~~ **Fixed by refusing
|
||
it, with the reason.** Not by the depth NEXT.md proposed: recording the restart-stack depth on *entering the break
|
||
loop* counts the frames a `restart-case` inside the thunk pushed before it erred, and those are above the boundary
|
||
and work. The boundary is where it is made — `restart_floor` is set to `flan_restart_count()` around `j.call()` in
|
||
`flan_agent_poll`, saved and restored so thunks nest — and the outermost `floor` entries of the snapshot are marked
|
||
unreachable. They are listed and marked rather than hidden, refused by the listener before the reply, and carried to
|
||
the editor as `:unreachable (2 3)`. `test_dev.ml` breaks a stopped program a second time from inside `C-x C-e` and
|
||
asserts both halves: index 2 refused, index 0 taken.
|
||
|
||
- ~~Restart names are served from a stack that is being mutated.~~ **Fixed, and it was a precondition rather than a
|
||
separate bug.** Index-based resume is wrong by construction against a moving stack: unlike a name, an index carries
|
||
no evidence of what it meant. The agent copies the list on entering `break_loop` — names into its own buffer, frames
|
||
as the addresses a transfer carries — one snapshot per nested break, and every verb answers from it. Caps are
|
||
`SNAP_MAX` 64 restarts and `SNAP_NAMES` 4096 bytes; past either, the listing says how many it did not show. Neither
|
||
cap has a test; the 4K result cap that shared that blind spot now does.
|
||
|
||
- **A snapshot generation has no test, and the window is a race.** A choice is validated against the snapshot on top
|
||
when the request arrives and resolved against the snapshot on top when the game thread next looks. Between those,
|
||
an evaluation the break loop is running can error and push a break of its own, whose loop would otherwise reach
|
||
[chosen_ready] first and take *its* index 2 for the one someone chose from the outer list. Each snapshot now carries
|
||
a generation, a choice is stamped with the one it was validated against, and a loop claims only what is addressed to
|
||
it — a mismatch is left set rather than discarded, because the listener already answered ok for it. Depth would not
|
||
do: an outer break resuming and a new one starting reuses the number. None of this is tested, because arranging the
|
||
window means landing a request inside a two-millisecond poll from outside the process. It wants a hook the test can
|
||
drive, not a sleep.
|
||
|
||
- ~~The job ring has no fullness check.~~ **Fixed by refusing, at the sender.** Dropping loses a reload the sender was
|
||
told was ok; blocking stalls the accept loop, which serves connections inline, so a program that had stopped polling
|
||
would also stop answering `status` and `abort`. The refusal happens before the `dlopen`, so a module there is no room
|
||
for is never relocated and no handle is taken for it. `programs/agent-queue.flan` blocks on stdin so the window is
|
||
held open by the test rather than by a timer: 64 queued, the 65th refused with a reason, 64 installed when it finally
|
||
polls.
|
||
|
||
- ~~`flan_dev_result_get` is not the seqlock its comment claims.~~ **Fixed by making it one**, rather than by writing
|
||
the honest comment — what it guaranteed was nothing, and the daemon has no other way to read a result. The counter is
|
||
odd while a value is being written, `flan_dev_result_read` copies into the caller's buffer and checks the counter
|
||
either side of the copy, and a reader that loses the race reports the last complete generation and no bytes. The
|
||
count handed out is the number of complete values, so `lib/dev.ml`'s "has it moved" still means what it meant. The
|
||
race itself has no test, for the same reason the snapshot generation above has none.
|
||
|
||
- ~~Smaller: `exit(134)` from the break loop with the listener inside `dlopen`; a `dlopen` handle leaked when a module
|
||
has no installer.~~ **Both fixed.** `exit` runs the atexit chain and the ELF destructors, which want the loader lock
|
||
the listener may be holding — a program asked to abort would hang instead of dying; `_exit`, with the streams flushed
|
||
by hand. The leak was the handle *value* and not the mapping: a module with no installer published nothing, so
|
||
nothing can point into it, and it is closed. The deadlock is read rather than tested; the exit status is tested.
|
||
- **`rt_die` in `flan_rt.c` still calls `exit(134)`**, which is the shape just fixed in the break loop: a trap on the
|
||
game thread runs the atexit chain and the ELF destructors, which want the loader lock the agent's listener thread may
|
||
be holding inside `dlopen`, so a program that should die could hang. Found while fixing the break loop and not fixed
|
||
with it — `rt_die` is the non-dev path too, where there is no listener and nothing to deadlock against, so whether it
|
||
should be `_exit` unconditionally or only under `--dev` is a decision rather than a typo.
|
||
|
||
- ~~**`(A {.x 1})` on a union variant says "unknown struct A"**~~ **Fixed** with union values. `env` now carries a
|
||
case table keyed both by the full spelling `U.C`, which is how a value of it is written, and by the bare `C`, which
|
||
is how a mistake spells a constructor; the bare entry exists only to say *"A is a case of the union U, not a struct
|
||
— a union value names both, as `(U.A {.field value ...})`"*. Two unions may share a case name and that is not
|
||
refused: construction is qualified and a pattern resolves against the scrutinee, so both are unambiguous.
|
||
|
||
### Test blind spots, from a mutation pass
|
||
|
||
Sixty mutations, nineteen left the whole suite green. The severe cluster was closed first (`cleanup.flan`,
|
||
`signedness.flan`); the rest are closed now. Every one below was re-planted, watched leave the suite green, and then
|
||
watched fail against the new test before the mutation was reverted — a test nobody saw fail is not evidence.
|
||
|
||
- **`Reach`'s walk of index expressions, `addr` places and `restart-case` clause bodies.** `programs/reach-walk.flan`
|
||
calls three functions from three places that are each the only route to them. The failure is not a wrong answer:
|
||
the function is not emitted and the program stops linking, so the case catches the build exception rather than
|
||
comparing output. The `addr` case goes through a `deref` place deliberately, so the index case cannot stand in for
|
||
it.
|
||
- **`flan_dev_global`'s size-change guard.** `programs/reload-v5.flan` is v4 with `extra` as an `i32`, loaded on top
|
||
of v3 in a host run of its own, because what it does is abort. The message is asserted next to the exit status: a
|
||
process that died for another reason is not this guard firing.
|
||
- **A local shadowing an imported name.** `programs/shadow-pkg.flan` binds locals over its own constant and var;
|
||
`pkg-shadow.flan` prints four numbers that separate the expression renamer from the place renamer. Nothing refuses
|
||
a renamer that qualifies through a binding — it reads the top-level name instead and runs — so only the number says
|
||
so.
|
||
- **The 4K result cap and the registry overflow guard.** `test/dev_limits.c`, a second C main beside `reload_host.c`,
|
||
drives them directly: neither has a Flan spelling and no corpus program reaches either. One process per mode — the
|
||
name table never shrinks and the overflow case aborts.
|
||
- **The reader's unknown string escape, and `+5`.** Rows in the reader table, with the escapes it *does* know
|
||
asserted on their decoded bytes rather than through `Form.to_string`, which escapes them again and would compare
|
||
the source with itself.
|
||
- **The hang.** A reader branch that forgets to advance loops for ever, and `dune test` waits as long as it is left
|
||
to; in CI that is a job the runner kills with nothing named. `test/watchdog.ml` arms an alarm on every test binary
|
||
— generous, because an alarm that fires on a slow machine is a flake and a flake is how a watchdog gets deleted —
|
||
and a five-second one around every read in `test_flan`. The first read that does not return wedges the rest, so a
|
||
looping reader costs five seconds and names the row instead of never finishing.
|
||
|
||
What is still open here: the mutation pass has not been re-run since, so the count of nineteen is the old one. The
|
||
sanitized sweep (`@sanitize`) is under the same watchdog but has never been observed to fire it, and so is the
|
||
memcheck sweep (`@valgrind`), whose alarm is looser at 5400s because memcheck is 20-50x on execution.
|
||
|
||
### Asked for by the editor lanes
|
||
|
||
- **`(:op "condition")` → the stopped program's condition, rendered.** Two steps: `break_loop` currently does
|
||
`(void)condition;` and *discards the pointer*, so stash it beside `condition_name`; then the daemon builds a render
|
||
thunk aimed at that address, which is `Session.render` rooted at a `Ptr` instead of an expression. **The second step
|
||
now exists** — `Session.render_locals` is exactly that thunk, rooted at an address the program supplies — so what is
|
||
left is the first: keep the pointer, and give the agent a verb that hands it back. The type is already known: it is
|
||
the `condition_name` the break loop reports, which `layout` already resolves.
|
||
- **The type identity is settled, and it is the qualified name** — `layout` is in, see BUILT.md. `Load` qualifies
|
||
every declaration at import, so the names in `Tast.structs` are a flat namespace where two packages' `Missing` are
|
||
`a/Missing` and `b/Missing`; a bare name is refused with the candidates rather than resolved. `condition` inherits
|
||
it for free: the string the break loop already reports *is* that name, because `Emit.struct_name_of` writes
|
||
`Types.Named` into `flan_error`. It is still open for **locals**, where DWARF gives a name and the name a debugger
|
||
reads is not qualified by anything.
|
||
- ~~**`(:op "backtrace")` is blocked** on frame metadata.~~ **Built**, and not out of DWARF: decision 3's shadow
|
||
stack carries the name and the location on the frame itself, so a backtrace needs no debug information at all. See
|
||
BUILT.md, "The shadow stack, and `backtrace`", for what it costs. **Locals landed with it** — the pointer-rooted
|
||
render thunk turned out to be `Render.render` over a `Deref` of a slot's address, and one new arm in the backend.
|
||
See "Locals of a stopped frame" for the four things it refuses. Restart source locations and arity are still blocked — `flan_restart` carries `prev`, `name_id`, `name` and `namelen`, so both need a new
|
||
field in the frame, which means the compiler emitting it.
|
||
|
||
### One line away
|
||
|
||
- **`match` over enums.** Fully desugarable, wanted, and blocked only by `Ast.pattern` needing a keyword case, which
|
||
`load.ml` matches exhaustively.
|
||
- **`Build.executable` returns only `out`**, so the daemon recovers the host `.ll` by recomputing `Build.workdir ()`.
|
||
- **A `!DILexicalBlock` per `Let`.** Not one line, but the one thing left in the DWARF work: every `!DILocalVariable`
|
||
is currently scoped to the subprogram, so inside `(let [v 22] …)` nested in `(let [v 11] …)` lldb still answers
|
||
`p v` with **11**. The `~2` suffix makes both *visible*, which is not the same as making the answer right. It needs
|
||
block structure the typed IR does not carry, and the `llvm.dbg.declare`s moved out of the entry block.
|
||
|
||
### Deferred with a reason
|
||
|
||
- **Writing through a string literal** — see Sharp edges. Needs provenance, which is open decision #3.
|
||
- **`cstring` as a type.** Odin has no `string → cstring` conversion at all; it pays the same copy our shim already
|
||
makes. The one thing it buys is the *return* direction, and nothing in `vendor/raylib` returns a string.
|
||
- **`rune`.** Odin's is a 4-byte integer distinguished by a flag, so `i32` is the same thing. Non-ASCII text is
|
||
blocked on font loading, not on the string layer — and fonts are now bound.
|
||
- **Macro expansion.** The reader and the declaration are in. Running a macro means compiling it and `dlopen`ing it
|
||
into the compiler, which is the reload primitive pointed at ourselves — but a macro is `[Form] -> Form`, so `Form`
|
||
has to be a Flan union whose layout the compiler and the loaded macro agree on, and union *values* are milestone 6.
|
||
|
||
### Documents that contradict the code
|
||
|
||
- **`plan.org`'s jank #947 citation is wrong in its mechanism.** jank does not relink (it calls through vars, which
|
||
are already indirection cells) and never unloads (`remove_symbol` has no callers). The real cause was a
|
||
process-teardown race. We are safe from the repro — because we compile out of process, not because of cells. A
|
||
normative document citing the wrong mechanism protects the wrong invariant.
|
||
- **`plan.org` still lists open decision #7 as open** and the interpreter as a backend. It was settled the other way;
|
||
`NEXT.md` records the consequences as "already applied" to `plan.org`, and they never were.
|
||
- **nREPL's `eval` does carry `file`, `line` and `column`** — jank reads all three. The choice of s-expressions still
|
||
stands on its other grounds; the stated reason does not.
|
||
|
||
## Sharp edges
|
||
|
||
- **Two formatted numbers cannot be held at once.** `flan_i64_to_bytes`, `flan_f64_to_bytes` and `flan_u64_to_bytes`
|
||
all write into one `static char scratch[64]` — "rendered text lives here until the next call", flan_rt.c:184 — and
|
||
`(string b)` does not copy. So
|
||
|
||
```
|
||
(let [a (string (i64->bytes 11))
|
||
b (string (i64->bytes 22))]
|
||
(print a) (print " ") (println b)) ; => 22 22
|
||
```
|
||
|
||
`a` is 11 and prints 22. No crash and no diagnostic. This is not new — the `[u8]` already aliased — but a `string`
|
||
reads as more value-like and invites exactly this. Format, draw, measure, then format the next one; `digits.flan`
|
||
sequences itself strictly for this reason. `rl/draw-text` is safe because the shim's `flan_shim_cstr` copies out of
|
||
ptr+len before the call.
|
||
|
||
**The prelude now has the shape that does not have this problem**, and it is the reason that shape exists.
|
||
`append-i64!` and `append-f64!` copy out of the scratch buffer into a `(Vec u8)` before returning, so a builder
|
||
holds as many rendered numbers as it likes, and `format-f64` answers a `Vec` rather than a view. The hazard is
|
||
unchanged for anyone calling `i64->bytes` directly — nothing was taken away — but a caller assembling a line of
|
||
text has a way not to meet it.
|
||
|
||
- **Writing through a string literal is undefined, and the two build modes
|
||
disagree about how.** `(let [s (bytes "Hi")] (set (at s 0) \h))` stores into
|
||
a `private unnamed_addr constant`. At `-O0` that is a store to read-only
|
||
memory and the program takes SIGSEGV; at `-O2` LLVM deletes it as undefined
|
||
and the program prints `Hi` and exits 0. Same source, and which way it fails
|
||
depends on a flag — the worst shape available, and worse than either outcome
|
||
alone.
|
||
|
||
Nothing refuses it. `bytes` turns a `string` into a `[u8]`, the language lets
|
||
you write through a slice, and by then nothing records that the bytes came
|
||
from a constant. The honest fix is provenance — knowing a slice's origin —
|
||
which is plan.org open decision #3 and deliberately deferred. A cheaper one
|
||
that is *not* a fix: emitting literals as mutable globals only moves which
|
||
flag misbehaves, and costs their read-only placement.
|
||
|
||
Found by the string lane while deciding whether `lower-ascii` should mutate
|
||
in place. It ships the copying version for exactly this reason, and that is
|
||
the rule to follow until provenance exists: **a function over a `string` must
|
||
not write through it.**
|
||
|
||
|
||
Most of these are edges the language keeps and you should know about. Two — the top-level namespace and the shift count,
|
||
both found by review after milestone 4 — were bugs that reached LLVM or ran wrong, and are **fixed**; each says so. They
|
||
stay written down because each one is now a rule the checker enforces, and a later change could quietly drop it.
|
||
|
||
- **An index converts from a narrower integer and never from a wider one.** `(at colors current-color)` with a `u32`
|
||
index works — anything above 2³¹ truncates to a negative `i32` and the unsigned bounds check rejects it. An `i64` index
|
||
is refused with the reason: 2³²+5 truncates to 5 and would read the wrong element with no trap at all.
|
||
- **There is one top-level namespace, and `check.ml` now enforces it.** The environment's tables are per-kind — structs,
|
||
unions, aliases, enums, functions, externs and globals each have their own — so only a function was ever checked for a
|
||
duplicate. `(defn item …)` beside `(defvar item …)` type checked and then died in LLVM as `redefinition of function
|
||
'@flan.item'`, a message about an emitted symbol with no source location left, and two colliding *type* declarations
|
||
were not caught anywhere. One pass over `Ast.declared_name` now runs before every other collection pass and rejects the
|
||
second declaration of a name whatever kind either one is. `declared_name` lives in `ast.ml` because `Load` needs exactly
|
||
the same set — the names an import renames — and two copies of that list would drift.
|
||
- **A shift count is bounded, two different ways.** A shift by the operand's own width or more is *poison* in LLVM, not
|
||
a wrong number: `(defn main [] i32 (<< 1 32))` compiled at -O2 to a bare `retq`, returning an undefined value. A literal
|
||
count out of range is now rejected in `check.ml` — that is the typo case — and `emit.ml` masks a computed count to
|
||
`width - 1`, which is what the hardware does anyway and which LLVM folds away whenever the count is constant. The
|
||
prelude's rotate masks its own count; that is now redundant but harmless.
|
||
- **A `u64` literal is its 64-bit pattern**, so `0xcbf29ce484222325` is a real `u64` and not an error. The cost is that
|
||
a negative *decimal* literal is accepted as a `u64` too, because the reader records the value and not how it was
|
||
written. Narrower unsigned types keep the strict check, which is where a typo like `300` for a `u8` actually shows up.
|
||
- **A folded constant skips `check`.** `(defconst rows (/ h c))` is emitted from the folding pass's value, because a
|
||
global's initialiser has to be a compile-time constant and only that pass knows this one is. Its range check is
|
||
therefore its own call to `in_range`; there is a regression test.
|
||
- A `let` binding takes no type annotation, which is why `sand.flan` names its FNV constants instead of writing them
|
||
inline.
|
||
- `(defn f [] f65 0.0)` still says *unknown name* rather than *did you mean f64*: with a single body form the parser
|
||
cannot tell a return type from the first expression. Only the parameter position and `(Option …)` are unambiguous.
|
||
|
||
## Loose ends from milestone 4
|
||
|
||
None of them blocking: block-scoped `defer`; package visibility, so `rl/get-color-raw` is not callable; imported
|
||
unions.
|
||
|
||
A package importing a package was on this list and is off it. It loads, a diamond shares one copy of the bottom
|
||
package, the alias clash is refused through a chain as well as inside one file, and a ring is refused by name. What is
|
||
left of the item is visibility, which is listed above and needs a marker the parser does not have.
|
||
|
||
## Macros — landed; what is left of them
|
||
|
||
**The expander works and `unless` is a prelude `defmacro`.** How all of it fits together is in
|
||
[`BUILT.md`](BUILT.md), "Macros: the compiler dlopens the program" — the image format, the thunk ABI, why quasiquote
|
||
runs before the walk, the two different ways expansion fails to terminate, `-linkall`, and the three cost numbers.
|
||
What follows is only the part that is still missing.
|
||
|
||
- **Four special forms left**, and two of them are the hard ones. `until` and `cond` are free to move whenever
|
||
somebody wants them. `when` and `dotimes` are not: the prelude itself uses them 29 and 12 times, so moving either
|
||
makes the prelude depend on the macro that the macro module has to compile the prelude to get. Breaking that needs
|
||
either a prelude that stops using them, or a two-stage prelude where the macro module is built from a subset. The
|
||
first is a mechanical edit of `prelude.ml` and is probably the answer.
|
||
|
||
`cond` has its own snag, and it is the reason `unless` went first: `parse.ml` refuses `(cond a)` with "cond clause
|
||
has no body", and a macro cannot produce that (see the next item), so moving `cond` changes an existing test.
|
||
|
||
- **A macro has no error facility**, and this is the biggest gap. A macro runs inside the compiler; anything it
|
||
signals aborts the compile with no location. So the prelude's `unless` answers `(unless-takes-a-test-and-a-body)`
|
||
when it is handed fewer than two forms, and the report is "unknown name unless-takes-a-test-and-a-body" at the call
|
||
site — right place, wrong sentence. What a macro wants is a way to say *this is wrong and here is why*, reported at
|
||
the call site. The queued structured-error rewrite is where that belongs, and the call site's `Loc.t` is already
|
||
stamped onto everything a macro returns, so the location half is done.
|
||
|
||
- **Macros are not imported.** A `defmacro` in a package is refused by name in `load.ml`. Reaching one would mean
|
||
resolving that package's own imports over `Form`s, before `Load` runs — a second import resolver. `programs/pkg-macro.flan`.
|
||
|
||
- **A prelude macro may not call a macro.** The prelude is in every macro module by construction, so there is no
|
||
round it could be compiled in after something else. It would fail with an unknown name rather than with a reason,
|
||
which is worth fixing the day the prelude wants one.
|
||
|
||
- **A prelude *function* may not call a prelude macro either**, which is the neighbouring gap and was found by
|
||
walking into it. `Macro.program` runs over the file being compiled; the prelude arrives at the checker through
|
||
`Check.program`'s own prepend and is never handed to the expander at all. A `defmacro` is an ordinary `defn` taking
|
||
one `[Form]` by the time the checker sees it, so the call resolves to that and the report is "clamp takes 1
|
||
argument, given 3" — pointing at the prelude, about a call the author wrote as a macro use. `format-f64` writes
|
||
`(min 9 (max 0 prec))` in place of `(clamp prec 0 9)` because of it. The fix is not obviously cheap: expanding the
|
||
prelude means building a macro module to compile the prelude that the macro module is built from, which is the same
|
||
bootstrap the `when`/`dotimes` item above describes.
|
||
|
||
- **A quasiquote inside a quasiquote is refused.** Nothing counts nesting levels — not the reader, deliberately, and
|
||
not the desugaring. Only a macro that writes a macro wants one.
|
||
|
||
- **`gensym`'s counter restarts in a second module.** It lives in the loaded module, and a module is dlopened once per
|
||
compiler process, so it is process-wide in practice. The rounds already build more than one module for a program
|
||
whose macros call macros, and the fix that day is to seed the counter from the module's index.
|
||
|
||
- **The macro programs are not in the sanitizer sweep.** `test_sanitize.ml` runs an explicit list, not a glob, so
|
||
`macros.flan` and `macro-unless.flan` were not added to it by landing them. `dune build --root . @sanitize` is
|
||
clean as it stands; adding the two is a one-line edit in a file this lane did not own.
|
||
|
||
- **No `&rest` sugar.** A macro takes one parameter, the slice of forms at its call site, and `(len args)` is the
|
||
arity. That is deliberate — it is where variadics come from — but a `when` written against it reads worse than
|
||
`parse.ml`'s version did.
|
||
|
||
## Watch for
|
||
|
||
The rule that caught the two misparse bugs applies unchanged: **anything that binds a name, alters control flow, or is
|
||
not yet implemented must be recognised explicitly and rejected if unsupported.** `check.ml` rejects `Vec`, `Map`,
|
||
`Result`/`try`, union values, closures, quoted symbols, generics and function values *by name*, each with the milestone
|
||
it belongs to; `load.ml` rejects the package shapes it does not handle; and the FFI boundary rejects an aggregate. The
|
||
tests assert on the reason, not just on the failure.
|
||
|
||
## Untracked on purpose
|
||
|
||
`calc-me` and `sand`, the executables `flan build` drops beside their sources, are now in `.gitignore` — anchored
|
||
(`/calc-me`, `/sand`) so the patterns cannot match anything nested.
|
||
|
||
`old-ocaml/` — the pre-rewrite menhir/ocamllex frontend, kept as reference and excluded from the build by the root
|
||
`dune` file. Its contents are also in git history at `2c232dd`.
|
||
|
||
## Handoff: the shadow stack lane, stopped mid-repair
|
||
|
||
Two commits landed and are green: the shadow stack with `(:op "backtrace")`, and `(:op "locals" :frame N)`. See
|
||
BUILT.md's two new sections for the design and the measurements. A third commit was **half-built and its own test
|
||
left red** on purpose; **it is finished now** — see the struck item 1 above — and the rest of this section is kept
|
||
because the parts of it that were true are still worth having.
|
||
|
||
**What is broken, exactly — and this paragraph was wrong; kept for what it cost.** It said `locals` compares the
|
||
frame on the stack against the body this session holds and the comparison is not firing, that every piece of the
|
||
fingerprint was written, and that one of five hand-offs was dropping the number. Four of the five were never written at
|
||
all: `Emit.fninfo` stored the fingerprint and nothing else touched it. The first step it recommended — printing both
|
||
sides of the comparison in `Dev.locals` — could not have worked, because `Dev.locals` had no comparison to print. The
|
||
lesson is the ordinary one: a lane that stops mid-repair should say which pieces it *ran*, not which it believes it
|
||
wrote.
|
||
|
||
**Not obvious from the diff.** Two things cost a day between them. The linked-list frame beat an array-with-a-stack-
|
||
pointer on both benchmarks, which is the opposite of what the escaping-alloca argument predicts, and the measurement
|
||
that first said otherwise was comparing a 40-frame binary with a 600-frame one; every number in BUILT.md is now a
|
||
minimum of nine runs for that reason. And `redefinition`'s transient rule (`m.nstr = 0`) silently stops every module
|
||
carrying a string literal from ever being unloaded — the frame descriptors go through their own counter, `m.nfi`, for
|
||
that reason, and a locals thunk passes `~retains:false` because everything it emits is memcpy'd into the result buffer.
|
||
|
||
**No Emacs surface.** `backtrace` and `locals` are daemon ops; nothing in `emacs/` calls them yet. One command showing
|
||
the backtrace with the selected frame's locals is the whole of what is missing, and `flan-cnr.el`'s
|
||
fixture-driven shape is the model.
|