NEXT.md was 1,738 lines and about half of it narrated work that was finished. BUILT.md takes that half — the reload primitive, cells, the agent, the session, the daemon, the Emacs client, conditions, the FFI shim, the layout, and the order it was all built in. Not deleted, because the reasons in it are load-bearing and would have to be derived again. NEXT.md keeps what is left, and says at the top what the two files are for, so it does not become a log a second time. The struck-through milestone checklist goes with the log; its one live item, the milestone 4 loose ends, stays. Start here was stale in three places and is rewritten: conditions are three steps of four, a restart is taken by position, and find-restart is blocked on a type rather than on effort. The raylib gap list and the reasons break was declined are written down where the next session will look for them, rather than living in a commit message.
577 lines
43 KiB
Markdown
577 lines
43 KiB
Markdown
# Where this is
|
||
|
||
## Start here — next session
|
||
|
||
**Branch `dev-loop`, 199 commits, working tree clean, `dune test` green.**
|
||
|
||
**`NEXT.md` is what is left. [`BUILT.md`](BUILT.md) is why the existing parts are the shape they are** — the reload
|
||
primitive, cells, the agent, the session, the daemon, the Emacs client, conditions, the FFI shim, the layout, and the
|
||
order it all got built in. This file was half build log until it was split; do not let it become one again. When a
|
||
track here finishes, its explanation moves there and its entry here goes away.
|
||
|
||
The dev loop works end to end: `flan dev program.flan`, then `C-c C-c`, `C-x C-e` and `C-c C-r` in Emacs against the
|
||
running process.
|
||
|
||
**Conditions are three steps of four.** `(error c)` is the diverging variant — a handler that returns normally has not
|
||
answered it, so only a transfer gets past. The break loop is in, editor half included: an unhandled `error` stops the
|
||
program on the frame that erred, the daemon annotates every reply with `:stopped`/`:condition`, and `C-c C-b` lists the
|
||
restarts and resumes into the choice. A restart is chosen **by position**, off a snapshot taken when the break was
|
||
entered, because a name resolves to the innermost frame offering it and the stopped thread's stack does not hold still.
|
||
Restarts below the evaluation a break is inside are listed, marked, and refused with the reason.
|
||
|
||
Still open from §3, each refused by name today: **restarts with parameters** (argument marshalling plus a runtime arity
|
||
check), and **`handler-case`**, which §"What this does not settle" leaves open as possibly a macro over `handler-bind`
|
||
plus a transfer. **`find-restart` and `compute-restarts` are blocked on a type, not on effort** — §4 gives them
|
||
`(Option Restart)` and a list, and there is no `Restart` type and no list to return one in. The minibuffer prompt never
|
||
needed them; it reads the snapshot over the agent's socket. And a `restart-case` clause should carry a **report
|
||
string** before any of this: `use-placeholder` is what `invoke-restart` needs, not what a person reading a list needs.
|
||
|
||
Read SBCL for what restarts should *mean* and ignore how it moves control: it transfers with `block`/`return-from`,
|
||
which §6 rules out.
|
||
|
||
### Landed 2026-09-12 — five tracks, one session
|
||
|
||
Five agents in parallel worktrees. Kept short on purpose; the reasoning that outlives the change is in `BUILT.md` or in
|
||
the commit that made it.
|
||
|
||
1. **`nth` removed**, an alias of `at` that was asymmetric — `check.ml` aliased them but `parse.ml` and
|
||
`place_of_expr` matched only `at`, so `(set (nth a i) x)` and `(addr (nth a i))` were refused while the `at` forms
|
||
worked.
|
||
2. **`println` and `print`**, compiler-provided and structural. `Session.render` was already the compile-time walk
|
||
plan.org asks for; it moved to `lib/render.ml` parameterised on an emitter and a slot allocator, so the REPL and
|
||
stdout share one copy. Found doing it: `field_addr` in `emit.ml` accepted only `Types.Named`, so a field of an
|
||
`Option` threw at emit time and **the walk's Option arm had never run** — the inspector would have failed on the
|
||
first `(Option T)` pointed at it. prelude.ml's claim that this had to wait for milestone 5 and generics was wrong,
|
||
and is gone: a printer selected per *concrete* type has nothing to dispatch on and no type variable in it.
|
||
3. **`restart-at`** — a restart is taken by position now. See "Start here".
|
||
4. **Names in DWARF.** `Tast.fn` carries `snames` beside `slots`, so a let-bound local is its own name under lldb
|
||
instead of `s0`; a slot the compiler invented keeps `s<index>`, because inventing a name puts a variable in the
|
||
debugger that is not in the file. Shadowing had to be decided rather than assumed: every `!DILocalVariable` is
|
||
scoped to the subprogram — the typed IR has no block structure to build a `!DILexicalBlock` from — so two slots
|
||
called `v` left lldb answering `p v` with the outer one while the body computed with the inner, and not listing the
|
||
inner at all. A repeat gets a `~2` suffix, unspellable in source. That is a way of not lying rather than a way of
|
||
being right; see "One line away". Also `flan dev --debug`, one flag for host and every redefinition module, off by
|
||
default because a debug build is an `-O0` build.
|
||
5. **Ten raylib core examples** in `examples/`, plus seven bindings and the colour palette. The gap list they produced
|
||
is under "Unblocked now, and ranked"; the top item, that no number could reach `draw-text`, is fixed — `(string b)`
|
||
reinterprets a `[u8]` as a `string`, which costs no instructions because they are already the same 16 bytes.
|
||
|
||
**In flight now:** retiring the `print-*` family, below.
|
||
|
||
### Queued — the runtime under a sanitizer
|
||
|
||
Nothing has ever run under ASan or UBSan on the test path. `grep -i 'sanitize\|asan\|valgrind'` over `lib/ bin/ test/
|
||
runtime/ vendor/` returns nothing at all. This is not a big project — the whole C surface is about 1,260 lines
|
||
(`flan_rt.c` 394, `flan_dev.c` 219, `flan_agent.c` 647) plus what `shim.ml` generates.
|
||
|
||
**The bugs are not in allocation.** There are four `malloc`/`calloc`/`strdup` sites in the entire runtime and every one
|
||
is allocate-once-never-free by design — `rt_args` says so in its own comment — so LeakSanitizer would mostly produce
|
||
suppressions. The risk is **fixed static buffers with bounds arithmetic**, and the ones with no coverage are already
|
||
written down: `scratch[SCRATCH]` and `escaped[ESCAPE_MAX]` in `flan_rt.c`, the 4K result cap, the registry overflow
|
||
guard, and `SNAP_MAX`/`SNAP_NAMES` from the restart snapshot.
|
||
|
||
There is evidence the sweep pays. Checking `flan_escape_bytes` by hand against lengths 0–1300 under ASan with a red
|
||
zone found the guard correct but its comment understating its own reserve by four bytes — worst output 1021 into 1024.
|
||
That was one buffer, found by looking.
|
||
|
||
The shape:
|
||
|
||
1. A `--sanitize` flag beside `--debug` in `lib/build.ml`, reaching both the clang run over the `.ll` and the runtime's
|
||
own C.
|
||
2. Run the existing corpus under it. `test/programs/` is about forty programs with pinned output — a second pass over
|
||
them is the cheapest coverage available here, and needs no new test written.
|
||
3. **UBSan is worth more than ASan**, with one exclusion that is not optional: arithmetic wraps by design, so
|
||
`-fno-sanitize=signed-integer-overflow` or every program trips on the first `+`. What is left is real — shift UB
|
||
(`(<< 1 32)` compiled to a bare `retq` at -O2, see Sharp edges), alignment, and the f32→i32 cast on NaN or an
|
||
infinity that `floor-f32` guards by hand and nothing else does.
|
||
4. The variant worth its own run: **ASan with `--no-bounds-checks`**. That asks whether the bounds checks are the only
|
||
thing between the language and corruption, which the checked build cannot ask.
|
||
|
||
Two limits, so nobody is surprised. raylib and libm are not instrumented, so the windowed examples are noise and the
|
||
headless corpus is the target — `sand-headless`, `values`, `machine`, `virtual-controls-headless`. And ASan does not
|
||
see uninitialised reads, which is where `zeroed` and struct padding live; that wants Valgrind as a slower second pass,
|
||
because MSan needs every dependency instrumented and raylib settles that.
|
||
|
||
### Queued — the `print-*` family goes
|
||
|
||
`print` and `println` are the whole printing surface. `print-str`, `print-i64`, `print-f64`, `print-bytes`,
|
||
`print-line` and `newline` are kept only until tracks 3 and 4 land, because an agent working in a tree that still has
|
||
them will write new fixtures that use them and the sweep would run twice.
|
||
|
||
`print` subsumes all of them, and is *better*: there is no implicit widening here, so `print-i64` forces an explicit
|
||
`(i64 x)` at every call site and `print` takes the `i32` as it is. `(print-line s)` is `(println s)`; `(newline)` is
|
||
`(println "")`.
|
||
|
||
About 470 call sites across ~60 files. Two places want reading rather than `sed`:
|
||
|
||
- `(print-i64 (i64 x))` where `x` is a `u64` changes its **answer**, not just its spelling — the old form widened
|
||
through the signed printer and printed `-1` for `0xFFFF…F`, and `print` routes a `u64` through its own shim. The
|
||
pinned `.out` files move, and each move is a thing to look at.
|
||
- `web/index.html` (55 sites) and `web/examples/` carry pinned output, so the prose changes with them. That is also
|
||
the moment to document `println` at all: it currently has **no mention on the language's own doc page**.
|
||
|
||
**`flan-cnr.el`'s stack pane is not blocked on DWARF**, and said so for two revisions. DWARF exists; `flan dev
|
||
--debug` now puts it in the host and every redefinition module. The real gap is that **nothing is attached to the
|
||
stopped program**: that buffer reaches it over the daemon's socket, and a socket cannot read another process's frames —
|
||
the break loop stopped itself, it is not being debugged. It wants an unwinder in the agent, or lldb on the same pid.
|
||
The refusal text now names that.
|
||
|
||
### Managed classes are planned. Do not start them.
|
||
|
||
plan.org grew a `class` facility beside `struct`: identity, runtime shape metadata, an implementation-defined
|
||
representation, generic-function dispatch, and live schema change with an explicit migration at a frame boundary. Its
|
||
own last line is the rule — nothing until ordinary `struct`, `Handle` and reload semantics are working. It is here so
|
||
that a session reading plan.org cold does not take it as the next task. Three things found while reviewing it, none of
|
||
them in plan.org yet:
|
||
|
||
- **A generic function is a cell.** "A later module can add `(defmethod draw ((e Enemy)) ...)` without editing the
|
||
original" means every compiled call site of `draw` has to find the new method — which is the problem the indirection
|
||
cells already solve. A generic function is a cell whose body is a dispatch table and a reload extends the table. The
|
||
expensive half of classes is therefore already built and tested.
|
||
- **The pool is not one storage option among three.** `migrate-instances` has to *enumerate* live instances. A pool
|
||
behind generational `(Handle T)` gives that by construction; a world arena and an owned region do not obviously.
|
||
plan.org presents the three as a free choice and they are not.
|
||
- **`Enemy@1` has to stay resolvable** for `migrate` to dispatch on it, so the session retains every layout version's
|
||
metadata for as long as any instance holds it. Same rule as "nothing is ever `dlclose`d", and worth stating as one.
|
||
|
||
### Open: can a condition be a class?
|
||
|
||
Unanswered, and it wants answering before `handler-case`, because it decides whether handler matching has one path or
|
||
two.
|
||
|
||
It would buy the thing conditions most lack: a **hierarchy**. §1 says flatly there is none, which is why nothing can say
|
||
"any condition" — no catch-all handler and nothing for a break loop to match on. Class inheritance gives it.
|
||
|
||
Three costs, one serious:
|
||
|
||
- **Signalling would allocate.** A struct condition is a stack value and `signal` takes its address; a class instance
|
||
needs a pool slot at the signal site. That is the failure path, sometimes the hot path, and sometimes the thing that
|
||
failed is allocation itself. plan.org also says no implicit allocation anywhere in the core.
|
||
- **§5's lifetime inverts.** Today the condition dies with the signalling frame and a handler that keeps it copies,
|
||
which is free for a value struct. A class instance survives the transfer — nicer, but now something owns and frees it.
|
||
- **Layout versions meet handler frames.** A struct condition cannot change layout; it is refused. A class can, and then
|
||
a frame pushed against `MyError@1` is on the stack while the signaller builds `MyError@2`.
|
||
|
||
The shape that probably wins is both: a struct condition stays exactly what it is — no allocation, matched by name hash,
|
||
dies with the frame — and a class condition is allocated, survives, and matches by walking its class chain. That is two
|
||
matching paths, which is the same bill the struct/class split already signs, so it is consistent rather than a new cost.
|
||
Either way it is an amendment to a **frozen** `spec-conditions.md`, not a gap in it.
|
||
|
||
|
||
**The dev loop is closed.** `C-c C-c` in Emacs recompiles the top-level form at point and installs it in a running
|
||
program, at that program's next frame boundary. Verified against sand: an unsaved buffer edit to `game-draw`, and 240
|
||
consecutive frames drew it.
|
||
|
||
Steps 1, 2 and 3 are done — see *The reload primitive* in `BUILT.md`. A list of top-level forms can be recompiled and installed
|
||
into a running process; call sites compiled before they existed follow them, and a `defn` or `defvar` the process was
|
||
never built with can be added and then redefined again. That is the whole of `C-c C-c`, minus an editor: sand.flan takes
|
||
a redefinition over a socket and installs it between frames.
|
||
|
||
What is left is the *session* — something that holds the checker environment between evaluations, tracks which names the
|
||
running process was built with, and speaks a protocol an editor can talk to.
|
||
|
||
Milestone 4 is done: **sand.flan builds, links raylib and runs**, and its simulation has a headless acceptance case that
|
||
runs on the `dune test` path at `-O0` and `-O2`. Milestones 2 and 3 are behind it (`calc-me.flan` compiles and runs; the
|
||
interpreter was dropped — open decision #7, settled — see "Why there is no interpreter" in `BUILT.md`).
|
||
|
||
```
|
||
reader ✅ → parse ✅ → load ✅ → check ✅ → emit ✅ → clang ✅
|
||
```
|
||
|
||
| File | What it does |
|
||
|---|---|
|
||
| `lib/loc.ml` | source locations + `Loc.Error`, the frontend's one exception |
|
||
| `lib/form.ml` | reader output: `Sym Kw Int Float Str Byte List Vec Map` |
|
||
| `lib/reader.ml` | hand-written S-expression reader, no menhir/ocamllex |
|
||
| `lib/ast.ml` | AST: `texpr`, `expr`, `place`, `pattern`, `decl` |
|
||
| `lib/parse.ml` | forms → AST; special forms, desugaring, declarations |
|
||
| `lib/load.ml` | **imports: a package directory → qualified declarations** |
|
||
| `lib/types.ml` | resolved types; structural equality, `Never` fits anywhere |
|
||
| `lib/tast.ml` | the typed IR the backend consumes |
|
||
| `lib/check.ml` | AST → typed IR; two passes, bidirectional |
|
||
| `lib/session.ml` | **a live program: what the process was built from, plus every change since** |
|
||
| `lib/wire.ml` | **the editor protocol: one s-expression per message, length framed** |
|
||
| `lib/dev.ml` | **`flan dev`: a session, the program running beside it, and a socket** |
|
||
| `lib/prelude.ml` | printers + `rand-f32`, written in Flan |
|
||
| `lib/emit.ml` | typed IR → LLVM IR text |
|
||
| `lib/build.ml` | `.ll` + the shim + the packages' C → clang → executable |
|
||
| `runtime/flan_rt.c` | the host ABI: argv, stdout, exit, 4 conversions |
|
||
| `runtime/flan_dev.c` | **dev only: the by-name registry a run-time-new name needs** |
|
||
| `lib/shim.ml` | **`declare-c` -> the generated C that flattens a struct crossing** |
|
||
| `vendor/raylib/` | **the raylib package: `raylib.flan` and `link`, and no C at all** |
|
||
| `vendor/agent/` | **the dev agent: a socket, a loader thread, install at a frame boundary** |
|
||
| `emacs/` | **`flan-mode.el`, `flan-dev.el`, `flan-repl.el`: the editor half of the dev loop** |
|
||
| `bin/main.ml` | `flan read \| parse \| check \| emit \| shim \| build \| run \| reload \| dev` |
|
||
| `test/test_flan.ml` | reader, parser and checker |
|
||
| `test/test_acceptance.ml` | expression/result pairs + whole programs + the traps |
|
||
| `test/test_reload.ml` | **the reload primitive: recompile one function, load it, call it** |
|
||
| `test/test_agent.ml` | **a running program taking a redefinition over a socket** |
|
||
| `test/test_session.ml` | **what a running process cannot be told, and recovering from a typo** |
|
||
| `test/test_dev.ml` | **the daemon, driven the way an editor drives it** |
|
||
| `test/test_repl.ml` | **`C-x C-e`: an expression evaluated inside a running program** |
|
||
| `test/programs/conditions.flan` | **`handler-bind` and `signal`, the accumulation case** |
|
||
| `conditions.org` | **a cheatsheet for driving conditions: what works, the exact refusals, the gotchas** |
|
||
| `conditions-play.flan` | **a program to poke at them with, built to be attached to by `flan dev`** |
|
||
| `test/programs/restarts.flan` | **`restart-case` and `invoke-restart`: the transfer, across two frames** |
|
||
| `test/test_emacs.ml` | **the client, driven against a real daemon and a real program** |
|
||
| `test/reload_host.c` | the C host that loads and installs two rebuilds, in one process |
|
||
| `test/wasm-run.mjs` | **a WASI host in twenty lines of `node:wasi`, so the table can run a wasm32 build** |
|
||
|
||
```
|
||
$ flan run calc-me.flan "1 + 2 * (3 - 0.5) / 2"
|
||
3.5
|
||
$ flan run test/programs/sand-headless.flan
|
||
-2851001042534928384
|
||
$ flan run sand.flan # a window, 120 fps, hold space
|
||
```
|
||
|
||
## Blocked and unfinished
|
||
|
||
Everything below was found, decided or half-built and then stopped. Each says what blocks it. Nothing here is a
|
||
vague intention — if it is listed, someone has already established it is real.
|
||
|
||
### Unblocked now, and ranked
|
||
|
||
**From porting ten raylib examples** — the first code the language was pushed by that it was not designed around.
|
||
Ranked by how often they were hit, top two first because they are walls rather than conveniences:
|
||
|
||
1. ~~No number reaches `draw-text`.~~ **Fixed** by `(string b)`.
|
||
2. **An enum parameter cannot be driven by a loop variable.** `(rl/get-gamepad-axis-movement pad i)` with an `i32`
|
||
index: *expected rl/GamepadAxis, found i32*, and the other direction refuses too — *i32 converts a number, found
|
||
rl/Gesture*. The obvious escape is closed as well: a second `declare-c` of the same symbol with an `i32` face gives
|
||
*one declare-c per C function, and another Flan name for it is a defn* — and a `defn` renames without retyping.
|
||
Two individually-correct rules composing into a wall; the caller writes a `cond` over the members instead. Cost:
|
||
every gamepad-axis loop, and four comparisons on the raw gesture bitfield.
|
||
3. **`break` is not implemented.** Declined deliberately rather than built — see below.
|
||
4. **A `let` binding takes no type annotation**, so a fixed array is either a top-level `defvar` or a literal with
|
||
every element spelled out. `(let [pts [4 rl/Vector2]] …)` parses as a two-element array literal and fails with
|
||
*unknown name rl/Vector2*. Cost: 32 hand-written `Vector2`s in one example.
|
||
5. **Arithmetic is strictly binary** — *+ takes 2 arguments, given 5*.
|
||
6. **No `sin`/`cos`/`abs` for floats.** The prelude has `sqrt-f32` and nothing else transcendental. Two `declare`
|
||
lines over libm, and `(max x (- 0.0 x))`; the examples each declare their own, which is a copy per file.
|
||
|
||
**A `string` cannot be returned from C at all**, which is what makes `GetGamepadName` unbindable: *a string only
|
||
crosses as a parameter — a C function that returns one returns something Flan has no owner for*. Same rule refuses
|
||
`TextFormat`, which is also variadic and so has no honest signature.
|
||
|
||
**The negative result is worth as much.** None of the gaps expected blocked anything — no generics, no allocator, no
|
||
`Vec`/`Map`, no escaping closures, and function-scoped `defer` never came up. Input-and-draw over fixed-size state is
|
||
the shape the language already has. Three constructs unexercised anywhere else in the repo worked first try: a fixed
|
||
array with a struct element, a 2-D struct array, and `[N string]` as both `defconst` and mutable `defvar`.
|
||
|
||
### `break`, and why it was not built
|
||
|
||
Settled, so the next attempt is cheap rather than a rediscovery:
|
||
|
||
- **`dotimes` gets it free** — it desugars to `Tast.While`, so one implementation covers both loop forms.
|
||
- **`defer` is a non-question.** It is function-scoped, `break` does not leave the function, nothing fires. No refusal
|
||
needed and no interaction to design.
|
||
- **Type it `Never`**, as `exit` and `return` already are.
|
||
- **`pads` is the structural model.** `emit_while` already makes an `endloop` label; break is a push/pop of that around
|
||
the body plus a `br`. `return` is a direct terminator with no context threading, so there is nothing else to mirror.
|
||
|
||
What stopped it, and neither is small:
|
||
|
||
- **`check.ml`'s `in_frames` rule does not extend.** It refuses `return` inside `handler-bind`/`restart-case` because
|
||
those frames are popped on the way out, and that refusal is blanket because `return` *always* crosses. `break`
|
||
crosses only sometimes — a loop wholly inside a `restart-case` body has a legitimate local break — so the precedent
|
||
has to be replaced by a loop-depth-relative-to-frame-entry rule nobody has ruled on.
|
||
- **`continue` forces a `Tast.While` signature change.** `check_dotimes` folds the step into the body as
|
||
`While (cond, body @ [step])`, so a `continue` branching to the header skips the increment and hangs. It needs a
|
||
latch — `While of expr * expr list * expr list` — across `check.ml` and `emit.ml`. plan.org settles break and
|
||
continue as one item and `parse.ml` refuses them in one case, so building break against today's `While` is exactly
|
||
the thing that would have to be undone.
|
||
|
||
plan.org's single line on it (831) names a `for` the language does not have and gives no mechanism.
|
||
|
||
|
||
1. **Allocators, then `Vec` and `Map`.** The critical path, and the only thing standing between this and writing a
|
||
game. **`Vec` does not need generics** — that was wrong and is worth un-learning: Odin's containers are compiler
|
||
builtins over a *type-erased* runtime (`base/runtime/dynamic_array_internal.odin`), where `$T` appears only in thin
|
||
wrappers producing `size_of`/`align_of` at the call site, and per-key hash and equality are compiler-emitted
|
||
procedures passed as a runtime argument (`src/llvm_backend.cpp`, `Map_Info`). That is exactly what
|
||
`spec-memory.md` already specifies. Before writing any of it, settle the four things the Odin and Carp studies
|
||
converged on, because all four are cheap now and expensive after:
|
||
- **When is storage released?** `spec-memory.md` never says. Odin's answer is `defer delete`, which Flan cannot
|
||
express — `defer` is function-scoped and refused in a `let`, a loop or a branch. Carp's answer is scope-end
|
||
frees, which it then could not reconcile with arenas and so has no allocator at all.
|
||
- **A `drop` hook** for a struct owning something that is not memory — a `Texture2D`, a socket, a file handle. Carp
|
||
shipped `delete` and then had to add a separate `drop` interface (`docs/Drop.md`). The hard part is ours alone:
|
||
what runs `drop` when the arena resets underneath the value.
|
||
- **`alignment`** appears nowhere in the design. Every Odin allocation carries it, and `#soa` and component-wise
|
||
fixed arrays want 16-byte alignment.
|
||
- **Allocation failure.** Unspecified. Odin returns an ignorable error, so a failed `append` silently appends
|
||
nothing. Flan has a better answer available for free: a `StorageExhausted` condition with a `retry` restart.
|
||
Decide which, because `push`/`put`/`clone`'s signatures depend on it.
|
||
|
||
2. **Typed restarts — `(use-value [v T] v)`.** The author's third TODO, and the most-wanted thing across every
|
||
comparative study. SBCL's report: restarts without parameters lose "the entire supply-a-value half of the standard
|
||
vocabulary", because `use-value` and `store-value` are the only two whose answer comes from outside the program.
|
||
Needs argument marshalling in `emit.ml` and §3's arity check in `check.ml`; both files are free now. The leverage
|
||
SBCL lacks: `eval` already compiles and runs an expression inside the live program, and the daemon already holds
|
||
the struct layouts, so "ask the human, type-check the answer, hand it over" is a short hop.
|
||
|
||
3. **`handler-case`.** Not a convenience — it is the fix for the loudest gotcha in `conditions.org`. A handler closes
|
||
over nothing *only because* a `handler-bind` clause runs at the signal point; a `handler-case` clause runs in the
|
||
establishing frame, which is ordinary in-frame code exactly like a `restart-case` clause. SBCL's is `handler-bind`
|
||
plus a transfer and nothing more (`src/code/error.lisp:196-268`). Every piece exists.
|
||
|
||
### Bugs found and not yet fixed
|
||
|
||
- ~~A shadowed restart is offered and cannot be taken.~~ **Fixed.** A restart is taken by *position* now:
|
||
`(:op "restart-at" :index N :name NAME)` on the daemon, `restart-at N NAME` on the agent, and a numbered
|
||
`completing-read` in `C-c C-b`. `:name` is a receipt, not the lookup — it is checked against the name the snapshot
|
||
holds at that index and refused if the two have drifted, so a bare integer can be wrong out loud. `restart <name>`
|
||
survives for a raw socket and is now defined as `restart-at` on the first index offering the name, so the two verbs
|
||
cannot disagree. `break.flan` grew the shadowed pair and asserts 900, which is the only value in that file no by-name
|
||
lookup can produce. The C&R buffer still marks the shadowed row by name and could now offer it instead — small, and
|
||
not done here.
|
||
|
||
- ~~A restart chosen at a break inside a thunk is accepted, announced, and silently not taken.~~ **Fixed by refusing
|
||
it, with the reason.** Not by the depth NEXT.md proposed: recording the restart-stack depth on *entering the break
|
||
loop* counts the frames a `restart-case` inside the thunk pushed before it erred, and those are above the boundary
|
||
and work. The boundary is where it is made — `restart_floor` is set to `flan_restart_count()` around `j.call()` in
|
||
`flan_agent_poll`, saved and restored so thunks nest — and the outermost `floor` entries of the snapshot are marked
|
||
unreachable. They are listed and marked rather than hidden, refused by the listener before the reply, and carried to
|
||
the editor as `:unreachable (2 3)`. `test_dev.ml` breaks a stopped program a second time from inside `C-x C-e` and
|
||
asserts both halves: index 2 refused, index 0 taken.
|
||
|
||
- ~~Restart names are served from a stack that is being mutated.~~ **Fixed, and it was a precondition rather than a
|
||
separate bug.** Index-based resume is wrong by construction against a moving stack: unlike a name, an index carries
|
||
no evidence of what it meant. The agent copies the list on entering `break_loop` — names into its own buffer, frames
|
||
as the addresses a transfer carries — one snapshot per nested break, and every verb answers from it. Caps are
|
||
`SNAP_MAX` 64 restarts and `SNAP_NAMES` 4096 bytes; past either, the listing says how many it did not show. Neither
|
||
cap has a test, same blind spot as the 4K result cap below.
|
||
|
||
- **A snapshot generation has no test, and the window is a race.** A choice is validated against the snapshot on top
|
||
when the request arrives and resolved against the snapshot on top when the game thread next looks. Between those,
|
||
an evaluation the break loop is running can error and push a break of its own, whose loop would otherwise reach
|
||
[chosen_ready] first and take *its* index 2 for the one someone chose from the outer list. Each snapshot now carries
|
||
a generation, a choice is stamped with the one it was validated against, and a loop claims only what is addressed to
|
||
it — a mismatch is left set rather than discarded, because the listener already answered ok for it. Depth would not
|
||
do: an outer break resuming and a new one starting reuses the number. None of this is tested, because arranging the
|
||
window means landing a request inside a two-millisecond poll from outside the process. It wants a hook the test can
|
||
drive, not a sleep.
|
||
|
||
- **The job ring has no fullness check**, and the comment describing its overflow is wrong. `publish` never consults
|
||
`tail`; past `QUEUE` entries it overwrites the slot the consumer is reading, and `job` is 24 non-atomic bytes.
|
||
Reachable from a program that goes a long time between `agent/poll` calls.
|
||
- **`flan_dev_result_get` is not the seqlock its comment claims** — it reads the generation first, then a non-atomic
|
||
length, then returns a bare pointer the caller sends later.
|
||
- Smaller: `exit(134)` from the break loop with the listener inside `dlopen`; a `dlopen` handle leaked when a module
|
||
has no installer.
|
||
- **`(A {:x 1})` on a union variant says "unknown struct A"** rather than the union refusal `check_struct` plainly
|
||
intends — `env` has no table of variant names. A diagnostics bug, not a backend death.
|
||
|
||
### Test blind spots, from a mutation pass
|
||
|
||
Sixty mutations, nineteen left the whole suite green. The severe cluster is closed (`cleanup.flan`,
|
||
`signedness.flan`); these are not:
|
||
|
||
- `Reach`'s walk of index expressions, `addr` places and `restart-case` clause bodies — each confirmed to prune a
|
||
function a valid program calls, so the build fails to link.
|
||
- `flan_dev_global`'s size-change guard — the layout-drift check, with no test that retypes a global across a reload.
|
||
- A local shadowing an imported name is qualified anyway.
|
||
- The 4K result cap and the registry overflow guard have **no coverage at all**, rather than a missing assertion.
|
||
- The reader accepts an unknown string escape; `+5` stops being a number.
|
||
- And a warning: a reader mutation makes the suite **hang** rather than fail. A green run is not the only outcome to
|
||
plan for in CI.
|
||
|
||
### Asked for by the editor lanes
|
||
|
||
- **`(:op "layout" :type T)` → the struct's fields and their types.** `Tast.structs` is held by the daemon at all
|
||
times because it owns the build, and **no running program is involved** — this is the cheapest real win on the
|
||
list, and the C&R buffer already draws its result.
|
||
- **`(:op "condition")` → the stopped program's condition, rendered.** Two steps: `break_loop` currently does
|
||
`(void)condition;` and *discards the pointer*, so stash it beside `condition_name`; then the daemon builds a render
|
||
thunk aimed at that address, which is `Session.render` rooted at a `Ptr` instead of an expression.
|
||
- **One thing to get right for both:** the type must be an identity the daemon can resolve to a `Tast` type, not a
|
||
bare class name. The hook is handed a string, and two packages both declaring `Missing` leave the daemon unable to
|
||
pick a layout. A qualified name or a type id. The same wrinkle bites locals later, because DWARF also gives a name.
|
||
- **`(:op "backtrace")` is blocked** on frame metadata — unlocked by the DWARF work, then a new agent verb. Locals are
|
||
blocked twice: DWARF for the frame layout, *and* the pointer-rooted render thunk. Restart source locations and
|
||
arity are blocked too — `flan_restart` carries `prev`, `name_id`, `name` and `namelen`, so both need a new field in
|
||
the frame, which means the compiler emitting it.
|
||
|
||
### One line away
|
||
|
||
- **`match` over enums.** Fully desugarable, wanted, and blocked only by `Ast.pattern` needing a keyword case, which
|
||
`load.ml` matches exhaustively.
|
||
- **`Build.executable` returns only `out`**, so the daemon recovers the host `.ll` by recomputing `Build.workdir ()`.
|
||
- **A `!DILexicalBlock` per `Let`.** Not one line, but the one thing left in the DWARF work: every `!DILocalVariable`
|
||
is currently scoped to the subprogram, so inside `(let [v 22] …)` nested in `(let [v 11] …)` lldb still answers
|
||
`p v` with **11**. The `~2` suffix makes both *visible*, which is not the same as making the answer right. It needs
|
||
block structure the typed IR does not carry, and the `llvm.dbg.declare`s moved out of the entry block.
|
||
|
||
### Deferred with a reason
|
||
|
||
- **Writing through a string literal** — see Sharp edges. Needs provenance, which is open decision #3.
|
||
- **`cstring` as a type.** Odin has no `string → cstring` conversion at all; it pays the same copy our shim already
|
||
makes. The one thing it buys is the *return* direction, and nothing in `vendor/raylib` returns a string.
|
||
- **`rune`.** Odin's is a 4-byte integer distinguished by a flag, so `i32` is the same thing. Non-ASCII text is
|
||
blocked on font loading, not on the string layer — and fonts are now bound.
|
||
- **Macro expansion.** The reader and the declaration are in. Running a macro means compiling it and `dlopen`ing it
|
||
into the compiler, which is the reload primitive pointed at ourselves — but a macro is `[Form] -> Form`, so `Form`
|
||
has to be a Flan union whose layout the compiler and the loaded macro agree on, and union *values* are milestone 6.
|
||
|
||
### Documents that contradict the code
|
||
|
||
- **`plan.org`'s jank #947 citation is wrong in its mechanism.** jank does not relink (it calls through vars, which
|
||
are already indirection cells) and never unloads (`remove_symbol` has no callers). The real cause was a
|
||
process-teardown race. We are safe from the repro — because we compile out of process, not because of cells. A
|
||
normative document citing the wrong mechanism protects the wrong invariant.
|
||
- **`plan.org` still lists open decision #7 as open** and the interpreter as a backend. It was settled the other way;
|
||
`NEXT.md` records the consequences as "already applied" to `plan.org`, and they never were.
|
||
- **nREPL's `eval` does carry `file`, `line` and `column`** — jank reads all three. The choice of s-expressions still
|
||
stands on its other grounds; the stated reason does not.
|
||
|
||
## Sharp edges
|
||
|
||
- **Two formatted numbers cannot be held at once.** `flan_i64_to_bytes`, `flan_f64_to_bytes` and `flan_u64_to_bytes`
|
||
all write into one `static char scratch[64]` — "rendered text lives here until the next call", flan_rt.c:184 — and
|
||
`(string b)` does not copy. So
|
||
|
||
```
|
||
(let [a (string (i64->bytes 11))
|
||
b (string (i64->bytes 22))]
|
||
(print a) (print " ") (println b)) ; => 22 22
|
||
```
|
||
|
||
`a` is 11 and prints 22. No crash and no diagnostic. This is not new — the `[u8]` already aliased — but a `string`
|
||
reads as more value-like and invites exactly this. Format, draw, measure, then format the next one; `digits.flan`
|
||
sequences itself strictly for this reason. `rl/draw-text` is safe because the shim's `flan_shim_cstr` copies out of
|
||
ptr+len before the call.
|
||
|
||
- **Writing through a string literal is undefined, and the two build modes
|
||
disagree about how.** `(let [s (bytes "Hi")] (set (at s 0) \h))` stores into
|
||
a `private unnamed_addr constant`. At `-O0` that is a store to read-only
|
||
memory and the program takes SIGSEGV; at `-O2` LLVM deletes it as undefined
|
||
and the program prints `Hi` and exits 0. Same source, and which way it fails
|
||
depends on a flag — the worst shape available, and worse than either outcome
|
||
alone.
|
||
|
||
Nothing refuses it. `bytes` turns a `string` into a `[u8]`, the language lets
|
||
you write through a slice, and by then nothing records that the bytes came
|
||
from a constant. The honest fix is provenance — knowing a slice's origin —
|
||
which is plan.org open decision #3 and deliberately deferred. A cheaper one
|
||
that is *not* a fix: emitting literals as mutable globals only moves which
|
||
flag misbehaves, and costs their read-only placement.
|
||
|
||
Found by the string lane while deciding whether `lower-ascii` should mutate
|
||
in place. It ships the copying version for exactly this reason, and that is
|
||
the rule to follow until provenance exists: **a function over a `string` must
|
||
not write through it.**
|
||
|
||
|
||
Most of these are edges the language keeps and you should know about. Two — the top-level namespace and the shift count,
|
||
both found by review after milestone 4 — were bugs that reached LLVM or ran wrong, and are **fixed**; each says so. They
|
||
stay written down because each one is now a rule the checker enforces, and a later change could quietly drop it.
|
||
|
||
- **An index converts from a narrower integer and never from a wider one.** `(at colors current-color)` with a `u32`
|
||
index works — anything above 2³¹ truncates to a negative `i32` and the unsigned bounds check rejects it. An `i64` index
|
||
is refused with the reason: 2³²+5 truncates to 5 and would read the wrong element with no trap at all.
|
||
- **There is one top-level namespace, and `check.ml` now enforces it.** The environment's tables are per-kind — structs,
|
||
unions, aliases, enums, functions, externs and globals each have their own — so only a function was ever checked for a
|
||
duplicate. `(defn item …)` beside `(defvar item …)` type checked and then died in LLVM as `redefinition of function
|
||
'@flan.item'`, a message about an emitted symbol with no source location left, and two colliding *type* declarations
|
||
were not caught anywhere. One pass over `Ast.declared_name` now runs before every other collection pass and rejects the
|
||
second declaration of a name whatever kind either one is. `declared_name` lives in `ast.ml` because `Load` needs exactly
|
||
the same set — the names an import renames — and two copies of that list would drift.
|
||
- **A shift count is bounded, two different ways.** A shift by the operand's own width or more is *poison* in LLVM, not
|
||
a wrong number: `(defn main [] i32 (<< 1 32))` compiled at -O2 to a bare `retq`, returning an undefined value. A literal
|
||
count out of range is now rejected in `check.ml` — that is the typo case — and `emit.ml` masks a computed count to
|
||
`width - 1`, which is what the hardware does anyway and which LLVM folds away whenever the count is constant. The
|
||
prelude's rotate masks its own count; that is now redundant but harmless.
|
||
- **A `u64` literal is its 64-bit pattern**, so `0xcbf29ce484222325` is a real `u64` and not an error. The cost is that
|
||
a negative *decimal* literal is accepted as a `u64` too, because the reader records the value and not how it was
|
||
written. Narrower unsigned types keep the strict check, which is where a typo like `300` for a `u8` actually shows up.
|
||
- **A folded constant skips `check`.** `(defconst rows (/ h c))` is emitted from the folding pass's value, because a
|
||
global's initialiser has to be a compile-time constant and only that pass knows this one is. Its range check is
|
||
therefore its own call to `in_range`; there is a regression test.
|
||
- A `let` binding takes no type annotation, which is why `sand.flan` names its FNV constants instead of writing them
|
||
inline.
|
||
- `(defn f [] f65 0.0)` still says *unknown name* rather than *did you mean f64*: with a single body form the parser
|
||
cannot tell a return type from the first expression. Only the parameter position and `(Option …)` are unambiguous.
|
||
|
||
## Loose ends from milestone 4
|
||
|
||
None of them blocking: block-scoped `defer`; package visibility, so `rl/get-color-raw` is not callable; a package
|
||
importing a package; imported unions.
|
||
|
||
## Macros — the reader and the declaration are in, the expander is not
|
||
|
||
The front half landed. What exists:
|
||
|
||
- **The reader** reads `` `x ``, `~x` and `~@x` as `(quasiquote x)`, `(unquote x)` and `(unquote-splicing x)`, exactly
|
||
as `'x` reads as `(quote x)`. It stays dumb: it does not count nesting levels, does not know whether an unquote is
|
||
inside a quasiquote, and attaches no meaning to the three names. Clojure's spelling, not Common Lisp's, because a comma
|
||
is whitespace in `is_delimiter` and every binding vector in the corpus relies on that. Backtick and tilde are delimiters
|
||
now, so `a~b` is two things.
|
||
- **`parse.ml` refuses all four by name.** `quasiquote` and `gensym` say expansion is not wired up; `unquote` and
|
||
`unquote-splicing` say they mean nothing outside a quasiquote, which is a mistake rather than a missing feature.
|
||
`(defmacro name [params] body ...)` at the top level is checked for shape and *then* refused — a malformed defmacro and
|
||
an unimplemented one get different reasons, so the shape rule is enforced before the feature exists.
|
||
|
||
Nothing is stored. There is deliberately no macro table and no `Ast.Defmacro`, because a table nothing reads is a place
|
||
for a design to rot, and the storage shape is the expander author's first decision, not a decision to inherit.
|
||
|
||
### How the expander should work
|
||
|
||
**There is no interpreter** (see "Why there is no interpreter" in `BUILT.md`) and there is not going to be one, so running a macro at
|
||
compile time means *compiling it and loading it into the compiler*. That machinery already exists and is measured:
|
||
`Emit.redefinition` → `Build.shared` → `dlopen` is ~19ms end to end, with the load itself at 0.04ms (see "The reload
|
||
primitive"). A macro is that pipeline pointed at the compiler's own process instead of the program's.
|
||
|
||
The shape it wants:
|
||
|
||
1. **A macro is a function `[Form] -> Form`.** Its parameters are forms and its result is a form, which means `Form.t`
|
||
has to exist on the Flan side — a `defunion` mirroring `lib/form.ml`, in the prelude, plus constructors and accessors.
|
||
That is the real work, and it is bigger than the expander itself: the compiler and the compiled macro have to agree on
|
||
the *layout* of a `Form`, not merely its shape, so whatever the checker does for unions has to be exact here. Until
|
||
unions are values this cannot start — `check.ml` puts union values and `match` on a union at **milestone 6**, so that is
|
||
milestone 6 work landing before milestone 5's.
|
||
2. **Expansion runs over `Form`, before `Parse`.** Not a pass over `Ast`: there is no `Ast.Defmacro` and `Parse` refuses
|
||
`defmacro` outright, so an `Ast`-level pass would have nothing to work with. That refusal is not a dead end, it is the
|
||
ordering — the expander runs first and `Parse` never sees a macro call at all. It is also the Clojure ordering, and the
|
||
reason a macro expanding to a special form is ordinary rather than a special case.
|
||
3. **Order matters and files do not have one.** Top-level names in a package are order-independent everywhere else
|
||
(`declared_types`, the constant fixpoint in `check.ml`). Macros cannot be: a macro must be compiled and loaded before a
|
||
call to it is expanded. Either collect every `defmacro` in a pre-pass and compile them as one module, or require
|
||
definition-before-use for macros specifically and say so in the error. The pre-pass is better and matches how the rest
|
||
of the frontend already behaves.
|
||
4. **A macro's own body may call macros**, so the pre-pass is a fixpoint, not a single sweep, and a cycle has to be
|
||
detected and named rather than looping.
|
||
5. **`gensym` is a runtime function of the compiler**, called by the loaded macro while it runs. It needs a counter that
|
||
lives in the compiler process and a name that cannot collide with a reader-produced symbol — the usual trick is a
|
||
character no symbol may contain, and this reader now has two new ones it could reserve. Hygiene is settled (plan.org,
|
||
open decision 2): deliberately non-hygienic, Common Lisp/Clojure style, explicit `gensym`, no `macrolet` until a
|
||
concrete use case appears.
|
||
6. **Quasiquote itself is a macro-shaped desugaring**, not a compiler feature: `` `(a ~b) `` becomes list-construction
|
||
over quoted pieces, with `~@` splicing. Written once, in the expander, over `Form`.
|
||
|
||
The four files this touches — `build.ml`, `check.ml`, `emit.ml`, `load.ml` — were owned by other lanes when the front
|
||
half landed, which is the only reason the expander is not here too.
|
||
|
||
### What would tell you it works
|
||
|
||
`when`, `unless`, `until`, `cond` and `dotimes` are special forms in `parse.ml` today, and plan.org milestone 5 says
|
||
they are special forms *only until macros land*. Moving one of them out of the compiler and into the prelude as a
|
||
`defmacro`, with the existing tests unchanged and still green, is the exit criterion — it proves expansion, quasiquote,
|
||
`gensym` and the ordering pre-pass at once, against a test suite written before any of them existed.
|
||
|
||
## Watch for
|
||
|
||
The rule that caught the two misparse bugs applies unchanged: **anything that binds a name, alters control flow, or is
|
||
not yet implemented must be recognised explicitly and rejected if unsupported.** `check.ml` rejects `Vec`, `Map`,
|
||
`Result`/`try`, union values, closures, quoted symbols, generics and function values *by name*, each with the milestone
|
||
it belongs to; `load.ml` rejects the package shapes it does not handle; and the FFI boundary rejects an aggregate. The
|
||
tests assert on the reason, not just on the failure.
|
||
|
||
## Untracked on purpose
|
||
|
||
`calc-me` and `sand`, the executables `flan build` drops beside their sources, are now in `.gitignore` — anchored
|
||
(`/calc-me`, `/sand`) so the patterns cannot match anything nested.
|
||
|
||
`old-ocaml/` — the pre-rewrite menhir/ocamllex frontend, kept as reference and excluded from the build by the root
|
||
`dune` file. Its contents are also in git history at `2c232dd`.
|