719 lines
57 KiB
Markdown
719 lines
57 KiB
Markdown
# Where this is
|
||
|
||
## Start here — next session
|
||
|
||
**Branch `dev-loop`, 199 commits, working tree clean, `dune test` green.**
|
||
|
||
**`NEXT.md` is what is left. [`BUILT.md`](BUILT.md) is why the existing parts are the shape they are** — the reload
|
||
primitive, cells, the agent, the session, the daemon, the Emacs client, conditions, the FFI shim, the layout, and the
|
||
order it all got built in. This file was half build log until it was split; do not let it become one again. When a
|
||
track here finishes, its explanation moves there and its entry here goes away.
|
||
|
||
The dev loop works end to end: `flan dev program.flan`, then `C-c C-c`, `C-x C-e` and `C-c C-r` in Emacs against the
|
||
running process.
|
||
|
||
**Conditions are three steps of four.** `(error c)` is the diverging variant — a handler that returns normally has not
|
||
answered it, so only a transfer gets past. The break loop is in, editor half included: an unhandled `error` stops the
|
||
program on the frame that erred, the daemon annotates every reply with `:stopped`/`:condition`, and `C-c C-b` lists the
|
||
restarts and resumes into the choice. A restart is chosen **by position**, off a snapshot taken when the break was
|
||
entered, because a name resolves to the innermost frame offering it and the stopped thread's stack does not hold still.
|
||
Restarts below the evaluation a break is inside are listed, marked, and refused with the reason.
|
||
|
||
Still open from §3, each refused by name today: **restarts with parameters** (argument marshalling plus a runtime arity
|
||
check), and **`handler-case`**, which §"What this does not settle" leaves open as possibly a macro over `handler-bind`
|
||
plus a transfer. **`find-restart` and `compute-restarts` are blocked on a type, not on effort** — §4 gives them
|
||
`(Option Restart)` and a list, and there is no `Restart` type and no list to return one in. The minibuffer prompt never
|
||
needed them; it reads the snapshot over the agent's socket. And a `restart-case` clause should carry a **report
|
||
string** before any of this: `use-placeholder` is what `invoke-restart` needs, not what a person reading a list needs.
|
||
|
||
Read SBCL for what restarts should *mean* and ignore how it moves control: it transfers with `block`/`return-from`,
|
||
which §6 rules out.
|
||
|
||
### Landed 2026-09-12 — six tracks, one session
|
||
|
||
Six agents in parallel worktrees. Kept short on purpose; the reasoning that outlives the change is in `BUILT.md` or in
|
||
the commit that made it.
|
||
|
||
1. **`nth` removed**, an alias of `at` that was asymmetric — `check.ml` aliased them but `parse.ml` and
|
||
`place_of_expr` matched only `at`, so `(set (nth a i) x)` and `(addr (nth a i))` were refused while the `at` forms
|
||
worked.
|
||
2. **`println` and `print`**, compiler-provided and structural. `Session.render` was already the compile-time walk
|
||
plan.org asks for; it moved to `lib/render.ml` parameterised on an emitter and a slot allocator, so the REPL and
|
||
stdout share one copy. Found doing it: `field_addr` in `emit.ml` accepted only `Types.Named`, so a field of an
|
||
`Option` threw at emit time and **the walk's Option arm had never run** — the inspector would have failed on the
|
||
first `(Option T)` pointed at it. prelude.ml's claim that this had to wait for milestone 5 and generics was wrong,
|
||
and is gone: a printer selected per *concrete* type has nothing to dispatch on and no type variable in it.
|
||
3. **`restart-at`** — a restart is taken by position now. See "Start here".
|
||
4. **Names in DWARF.** `Tast.fn` carries `snames` beside `slots`, so a let-bound local is its own name under lldb
|
||
instead of `s0`; a slot the compiler invented keeps `s<index>`, because inventing a name puts a variable in the
|
||
debugger that is not in the file. Shadowing had to be decided rather than assumed: every `!DILocalVariable` is
|
||
scoped to the subprogram — the typed IR has no block structure to build a `!DILexicalBlock` from — so two slots
|
||
called `v` left lldb answering `p v` with the outer one while the body computed with the inner, and not listing the
|
||
inner at all. A repeat gets a `~2` suffix, unspellable in source. That is a way of not lying rather than a way of
|
||
being right; see "One line away". Also `flan dev --debug`, one flag for host and every redefinition module, off by
|
||
default because a debug build is an `-O0` build.
|
||
5. **Ten raylib core examples** in `examples/`, plus seven bindings and the colour palette. The gap list they produced
|
||
is under "Unblocked now, and ranked"; the top item, that no number could reach `draw-text`, is fixed — `(string b)`
|
||
reinterprets a `[u8]` as a `string`, which costs no instructions because they are already the same 16 bytes.
|
||
|
||
6. **The `print-*` family is gone.** `print` and `println` are the whole printing surface; ~500 call sites across 47
|
||
files rewrote, and `web/index.html` gained a `#printing` section, the first documentation either has had. Two pinned
|
||
outputs moved and both are corrections: `sand-headless`'s hash is `15595743031174623232` rather than
|
||
`-2851001042534928384` — the same 64 bits, printed unsigned now that `hash-grid`'s `u64` no longer goes through an
|
||
`(i64 …)` cast — and a trap column shifted because the call it names got shorter.
|
||
|
||
### Landed — the runtime under a sanitizer
|
||
|
||
`--sanitize` is a build flag beside `--debug`; `dune build --root . @sanitize` builds twenty-eight programs twice, plain
|
||
and sanitized, and compares output and exit status. Its own alias and not `dune test`, because the sweep is about nine
|
||
minutes. The checked sweep is **clean**. How ASan and UBSan reach a language whose IR is written by hand, and why the
|
||
flag does not force `-O0` when `--debug` does, is in [`BUILT.md`](BUILT.md).
|
||
|
||
Two defects came out of it, both found by reading rather than by the tools, both fixed with a regression case:
|
||
`flan_bytes_to_i64`/`flan_bytes_to_f64` clamped a slice length with `(size_t)n` and so read 63 or 511 bytes off the end
|
||
of a negative-length slice; and the three `snprintf` shims published snprintf's return as a slice length, which is what
|
||
it *would* have written.
|
||
|
||
**What is left, and it is most of what the sweep was meant to settle:**
|
||
|
||
1. **UBSan sees no Flan code and no flag changes that.** Its checks are branches clang's C frontend emits inline, not a
|
||
pass, so shift UB (`(<< 1 32)`, see Sharp edges), alignment, and the f32→i32 cast on NaN or an infinity — the things
|
||
`floor-f32` guards by hand and nothing else does — are unreached. Either `Emit` grows those checks behind the flag,
|
||
which is a compiler feature of the same shape the bounds checks already have, or they belong to the checker. Not
|
||
decided. `test_sanitize` pins the current answer with a control that must *not* report, so a future clang changing
|
||
this is a test failure rather than a discovery.
|
||
2. **Four named buffers got no evidence at all.** The 4K result cap, the dev registry overflow guard,
|
||
`SNAP_MAX`/`SNAP_NAMES` and `condition_name[128]` are on the daemon and agent paths, which need a socket and are not
|
||
in the corpus. Their guards were read and are correct; that is reading, not testing. `escaped[ESCAPE_MAX]` is the one
|
||
that *is* covered, because `println.flan` drives a 1100-character string through it on purpose — 1019 bytes out
|
||
against a worst case of 1021 into 1024. `scratch[SCRATCH]` never sees more than 20 characters of 64.
|
||
3. **Valgrind over the headless corpus, not done.** ASan does not see uninitialised reads, which is where `zeroed` and
|
||
struct padding live. MSan is out: it needs every dependency instrumented and raylib settles that.
|
||
|
||
Two things the sweep structurally cannot cover: raylib and libm are uninstrumented, so the windowed examples are noise;
|
||
and a redefinition module is built by `llc` and `ld` rather than clang, so the reload path carries no instrumentation
|
||
whatever the flag says.
|
||
|
||
### Managed classes are planned. Do not start them.
|
||
|
||
plan.org grew a `class` facility beside `struct`: identity, runtime shape metadata, an implementation-defined
|
||
representation, generic-function dispatch, and live schema change with an explicit migration at a frame boundary. Its
|
||
own last line is the rule — nothing until ordinary `struct`, `Handle` and reload semantics are working. It is here so
|
||
that a session reading plan.org cold does not take it as the next task. Three things found while reviewing it, none of
|
||
them in plan.org yet:
|
||
|
||
- **A generic function is a cell.** "A later module can add `(defmethod draw ((e Enemy)) ...)` without editing the
|
||
original" means every compiled call site of `draw` has to find the new method — which is the problem the indirection
|
||
cells already solve. A generic function is a cell whose body is a dispatch table and a reload extends the table. The
|
||
expensive half of classes is therefore already built and tested.
|
||
- **The pool is not one storage option among three.** `migrate-instances` has to *enumerate* live instances. A pool
|
||
behind generational `(Handle T)` gives that by construction; a world arena and an owned region do not obviously.
|
||
plan.org presents the three as a free choice and they are not.
|
||
- **`Enemy@1` has to stay resolvable** for `migrate` to dispatch on it, so the session retains every layout version's
|
||
metadata for as long as any instance holds it. Same rule as "nothing is ever `dlclose`d", and worth stating as one.
|
||
|
||
### Open: can a condition be a class?
|
||
|
||
Unanswered, and it wants answering before `handler-case`, because it decides whether handler matching has one path or
|
||
two.
|
||
|
||
It would buy the thing conditions most lack: a **hierarchy**. §1 says flatly there is none, which is why nothing can say
|
||
"any condition" — no catch-all handler and nothing for a break loop to match on. Class inheritance gives it.
|
||
|
||
Three costs, one serious:
|
||
|
||
- **Signalling would allocate.** A struct condition is a stack value and `signal` takes its address; a class instance
|
||
needs a pool slot at the signal site. That is the failure path, sometimes the hot path, and sometimes the thing that
|
||
failed is allocation itself. plan.org also says no implicit allocation anywhere in the core.
|
||
- **§5's lifetime inverts.** Today the condition dies with the signalling frame and a handler that keeps it copies,
|
||
which is free for a value struct. A class instance survives the transfer — nicer, but now something owns and frees it.
|
||
- **Layout versions meet handler frames.** A struct condition cannot change layout; it is refused. A class can, and then
|
||
a frame pushed against `MyError@1` is on the stack while the signaller builds `MyError@2`.
|
||
|
||
The shape that probably wins is both: a struct condition stays exactly what it is — no allocation, matched by name hash,
|
||
dies with the frame — and a class condition is allocated, survives, and matches by walking its class chain. That is two
|
||
matching paths, which is the same bill the struct/class split already signs, so it is consistent rather than a new cost.
|
||
Either way it is an amendment to a **frozen** `spec-conditions.md`, not a gap in it.
|
||
|
||
|
||
**The dev loop is closed.** `C-c C-c` in Emacs recompiles the top-level form at point and installs it in a running
|
||
program, at that program's next frame boundary. Verified against sand: an unsaved buffer edit to `game-draw`, and 240
|
||
consecutive frames drew it.
|
||
|
||
Steps 1, 2 and 3 are done — see *The reload primitive* in `BUILT.md`. A list of top-level forms can be recompiled and installed
|
||
into a running process; call sites compiled before they existed follow them, and a `defn` or `defvar` the process was
|
||
never built with can be added and then redefined again. That is the whole of `C-c C-c`, minus an editor: sand.flan takes
|
||
a redefinition over a socket and installs it between frames.
|
||
|
||
What is left is the *session* — something that holds the checker environment between evaluations, tracks which names the
|
||
running process was built with, and speaks a protocol an editor can talk to.
|
||
|
||
Milestone 4 is done: **sand.flan builds, links raylib and runs**, and its simulation has a headless acceptance case that
|
||
runs on the `dune test` path at `-O0` and `-O2`. Milestones 2 and 3 are behind it (`calc-me.flan` compiles and runs; the
|
||
interpreter was dropped — open decision #7, settled — see "Why there is no interpreter" in `BUILT.md`).
|
||
|
||
```
|
||
reader ✅ → parse ✅ → load ✅ → check ✅ → emit ✅ → clang ✅
|
||
```
|
||
|
||
| File | What it does |
|
||
|---|---|
|
||
| `lib/loc.ml` | source locations + `Loc.Error`, the frontend's one exception |
|
||
| `lib/form.ml` | reader output: `Sym Kw Int Float Str Byte List Vec Map` |
|
||
| `lib/reader.ml` | hand-written S-expression reader, no menhir/ocamllex |
|
||
| `lib/ast.ml` | AST: `texpr`, `expr`, `place`, `pattern`, `decl` |
|
||
| `lib/parse.ml` | forms → AST; special forms, desugaring, declarations |
|
||
| `lib/load.ml` | **imports: a package directory → qualified declarations** |
|
||
| `lib/types.ml` | resolved types; structural equality, `Never` fits anywhere |
|
||
| `lib/tast.ml` | the typed IR the backend consumes |
|
||
| `lib/check.ml` | AST → typed IR; two passes, bidirectional |
|
||
| `lib/session.ml` | **a live program: what the process was built from, plus every change since** |
|
||
| `lib/wire.ml` | **the editor protocol: one s-expression per message, length framed** |
|
||
| `lib/dev.ml` | **`flan dev`: a session, the program running beside it, and a socket** |
|
||
| `lib/prelude.ml` | printers + `rand-f32`, written in Flan |
|
||
| `lib/emit.ml` | typed IR → LLVM IR text |
|
||
| `lib/build.ml` | `.ll` + the shim + the packages' C → clang → executable |
|
||
| `runtime/flan_rt.c` | the host ABI: argv, stdout, exit, 4 conversions |
|
||
| `runtime/flan_dev.c` | **dev only: the by-name registry a run-time-new name needs** |
|
||
| `lib/shim.ml` | **`declare-c` -> the generated C that flattens a struct crossing** |
|
||
| `vendor/raylib/` | **the raylib package: `raylib.flan` and `link`, and no C at all** |
|
||
| `vendor/agent/` | **the dev agent: a socket, a loader thread, install at a frame boundary** |
|
||
| `emacs/` | **`flan-mode.el`, `flan-dev.el`, `flan-repl.el`: the editor half of the dev loop** |
|
||
| `bin/main.ml` | `flan read \| parse \| check \| emit \| shim \| build \| run \| reload \| dev` |
|
||
| `test/test_flan.ml` | reader, parser and checker |
|
||
| `test/test_acceptance.ml` | expression/result pairs + whole programs + the traps |
|
||
| `test/test_reload.ml` | **the reload primitive: recompile one function, load it, call it** |
|
||
| `test/test_agent.ml` | **a running program taking a redefinition over a socket** |
|
||
| `test/test_session.ml` | **what a running process cannot be told, and recovering from a typo** |
|
||
| `test/test_dev.ml` | **the daemon, driven the way an editor drives it** |
|
||
| `test/test_repl.ml` | **`C-x C-e`: an expression evaluated inside a running program** |
|
||
| `test/programs/conditions.flan` | **`handler-bind` and `signal`, the accumulation case** |
|
||
| `conditions.org` | **a cheatsheet for driving conditions: what works, the exact refusals, the gotchas** |
|
||
| `conditions-play.flan` | **a program to poke at them with, built to be attached to by `flan dev`** |
|
||
| `test/programs/restarts.flan` | **`restart-case` and `invoke-restart`: the transfer, across two frames** |
|
||
| `test/test_emacs.ml` | **the client, driven against a real daemon and a real program** |
|
||
| `test/reload_host.c` | the C host that loads and installs two rebuilds, in one process |
|
||
| `test/wasm-run.mjs` | **a WASI host in twenty lines of `node:wasi`, so the table can run a wasm32 build** |
|
||
|
||
```
|
||
$ flan run calc-me.flan "1 + 2 * (3 - 0.5) / 2"
|
||
3.5
|
||
$ flan run test/programs/sand-headless.flan
|
||
15595743031174623232
|
||
$ flan run sand.flan # a window, 120 fps, hold space
|
||
```
|
||
|
||
## Blocked and unfinished
|
||
|
||
Everything below was found, decided or half-built and then stopped. Each says what blocks it. Nothing here is a
|
||
vague intention — if it is listed, someone has already established it is real.
|
||
|
||
### Unblocked now, and ranked
|
||
|
||
**0. Signature generations and stale-caller warnings — milestone 7's unfinished half.**
|
||
Promoted here on the author's correction, and `session.ml:146` already says the same thing at the refusal itself. A
|
||
changed signature is refused today and **that is a placeholder, not the design**. plan.org's open decision #6 says what
|
||
should happen: a signature change makes a new version of the function, new callers resolve it, existing callers and any
|
||
stored `Fn` value stay safely on the old one, and the session *warns* at each tracked stale caller site. Milestone 7
|
||
names it outright — "signature generations and stale-caller warnings".
|
||
|
||
**The thesis of this project is that you never restart the program.** Every refusal that ends in "restart to change it"
|
||
is a hole in that, and this is the biggest one. It needs three things that do not exist: function versions, a
|
||
trampoline per version, and caller tracking good enough to name the sites. The cell already gives the indirection; what
|
||
is missing is that a cell holds one bare pointer with no signature, so there is nowhere to put a second version.
|
||
|
||
A changed **struct layout** is the genuinely hard case and plan.org still specifies it as a rejection — storage already
|
||
allocated has the old shape and a new body reads its fields at the wrong offsets. Managed classes are the planned way
|
||
through, with an explicit migration at a frame boundary. Do not conflate the two: one is unbuilt, the other is decided.
|
||
|
||
|
||
**From porting ten raylib examples** — the first code the language was pushed by that it was not designed around.
|
||
Ranked by how often they were hit, top two first because they are walls rather than conveniences:
|
||
|
||
1. ~~No number reaches `draw-text`.~~ **Fixed** by `(string b)`.
|
||
2. ~~An enum parameter cannot be driven by a loop variable.~~ **Fixed** by explicit conversions in both directions:
|
||
`(i32 k)` takes an enum to its integer, `(GamepadAxis n)` takes an integer to an enum. Neither is an instruction —
|
||
an enum is an i32 at run time and `emit.ml`'s `cast` already reduced one to that before choosing an opcode — so the
|
||
change is a guard in `check.ml`'s cast arm and nothing in the backend. The rule the refusals came from is
|
||
deliberately *not* relaxed: a bare integer still does not fit an enum parameter, so `:spcae` is still an error at
|
||
the call site. The rule was "an integer must not arrive silently", and a written `(GamepadAxis i)` is not silent.
|
||
The other escape stays closed too — one `declare-c` per C function — and no longer needs to be open.
|
||
- **A value that is no declared member is allowed**, deliberately. raylib's gesture is a bitfield and an OR of
|
||
flags is a legal `Gesture` that is no single member; and `session.ml`'s printer already falls through to the
|
||
number for an out-of-range enum, on purpose, so refusing to construct one while agreeing to print it would be
|
||
incoherent. An `Option` would make every site unwrap for no safety bought, and a literal-only refusal would
|
||
catch nothing, because the bitfield case is a run-time value.
|
||
- **Only an integer converts *to* an enum.** Not a float, and not another enum — a cross-enum hop goes through
|
||
`(i32 x)` so both ends are written down. Enum → any numeric is always allowed: lossless to i32 by construction,
|
||
and a narrower target truncates by the rule every int→int cast already follows.
|
||
- **The comparisons needed nothing else.** `(> (i32 g) 255)` checks because `binary` takes the non-literal side
|
||
first; `binary` was deliberately left ignorant of enums, since teaching it would be the implicit conversion this
|
||
avoids.
|
||
- **A bit-set type later builds on this rather than replacing it.** It would be its own type with its own
|
||
operations and would still want a named escape to the underlying integer for the FFI, spelled the same way. If
|
||
`Gesture` becomes one, the `(i32 g)` calls stay valid and only the range tests migrate to a membership test.
|
||
- One parse fix came with it: `defenum` names were not in `parse.ml`'s type set, so a local enum could not be a
|
||
function's return type. They are in it now under a key of their own, admitted as a bare symbol and never as a
|
||
list head — because `(Key n)` is a *value* now, and putting `Key` in `types` would make a body starting with one
|
||
be eaten as a return type.
|
||
3. **`break` is not implemented.** Declined deliberately rather than built — see below.
|
||
4. **A `let` binding takes no type annotation**, so a fixed array is either a top-level `defvar` or a literal with
|
||
every element spelled out. `(let [pts [4 rl/Vector2]] …)` parses as a two-element array literal and fails with
|
||
*unknown name rl/Vector2*. Cost: 32 hand-written `Vector2`s in one example. **Looked at and stopped — it is a
|
||
grammar question, not a missing feature.** Everything under the surface is already there: `Ast.binding` carries a
|
||
`bty`, `load.ml` renames through it, and `check.ml:723` consumes it as the `want` for the value. Only the way it
|
||
is written is open, and the parser says so where it refuses (`parse.ml:366`): `let` is a flat list of pairs, so it
|
||
cannot disambiguate by *count* the way `defvar` and `defconst` do — those read `[n t v]` as three arguments to a
|
||
form, and there is no such boundary between one pair and the next. Three surfaces, in the order they are worth
|
||
considering:
|
||
- **`(zeroed [4 rl/Vector2])` — `zeroed` takes its type as an argument.** Recommended. It is one extra branch in
|
||
the arity-0 `zeroed` case in `check.ml`, no parser change, no ambiguity, and it answers the actual complaint,
|
||
which is not "locals cannot be annotated" but "there is nothing here to infer *from*". It also reads as what it
|
||
does: the value is a zeroed thing of that type, not a name that has been told what it is.
|
||
- **A marker between the name and the type**, `(let [pts :- [4 rl/Vector2] …] …)` or similar. Unambiguous, and it
|
||
buys a general annotation rather than one form's escape hatch. The cost is a new piece of syntax in the binding
|
||
vector, which is the one place this language has kept looking exactly like Clojure's.
|
||
- **Bare `(let [pts [4 rl/Vector2] …])`.** The obvious spelling and the one that cannot work: `[4 rl/Vector2]` is
|
||
a well-formed two-element array literal, and telling the two apart needs types in the parser, which there are
|
||
none of by design.
|
||
Note that plan.org's rule is "annotate function signatures, infer locals", so the general annotation is a
|
||
deliberate absence and not an oversight — which is the other reason the `zeroed` route is the smaller answer.
|
||
5. ~~**Arithmetic is strictly binary** — *+ takes 2 arguments, given 5*.~~ **Fixed.** `+ - * /`, `min`/`max` and
|
||
`bit-and`/`bit-or`/`bit-xor` fold left over two operands or more. `%` and the shifts stay at two, and one operand
|
||
is refused with the form to write instead — there is no unary minus and no reciprocal.
|
||
6. ~~**No `sin`/`cos`/`abs` for floats.**~~ **Fixed.** `sin-f32` and `cos-f32` are `declare`s in the prelude now,
|
||
with the caveat written beside them: IEEE-754 makes `sqrt` correctly rounded and requires nothing of the kind for
|
||
`sinf`, so these are the one place in the prelude where native and wasm32 may disagree bit for bit. Float `abs` is
|
||
not wrapped, for the reason integer `abs` is not — it is `(max x (- 0.0 x))` over two builtins.
|
||
|
||
**A `string` cannot be returned from C at all**, which is what makes `GetGamepadName` unbindable: *a string only
|
||
crosses as a parameter — a C function that returns one returns something Flan has no owner for*. Same rule refuses
|
||
`TextFormat`, which is also variadic and so has no honest signature.
|
||
|
||
**The negative result is worth as much.** None of the gaps expected blocked anything — no generics, no allocator, no
|
||
`Vec`/`Map`, no escaping closures, and function-scoped `defer` never came up. Input-and-draw over fixed-size state is
|
||
the shape the language already has. Three constructs unexercised anywhere else in the repo worked first try: a fixed
|
||
array with a struct element, a 2-D struct array, and `[N string]` as both `defconst` and mutable `defvar`.
|
||
|
||
### `break`, and why it was not built
|
||
|
||
Settled, so the next attempt is cheap rather than a rediscovery:
|
||
|
||
- **`dotimes` gets it free** — it desugars to `Tast.While`, so one implementation covers both loop forms.
|
||
- **`defer` is a non-question.** It is function-scoped, `break` does not leave the function, nothing fires. No refusal
|
||
needed and no interaction to design.
|
||
- **Type it `Never`**, as `exit` and `return` already are.
|
||
- **`pads` is the structural model.** `emit_while` already makes an `endloop` label; break is a push/pop of that around
|
||
the body plus a `br`. `return` is a direct terminator with no context threading, so there is nothing else to mirror.
|
||
|
||
What stopped it, and neither is small:
|
||
|
||
- **`check.ml`'s `in_frames` rule does not extend.** It refuses `return` inside `handler-bind`/`restart-case` because
|
||
those frames are popped on the way out, and that refusal is blanket because `return` *always* crosses. `break`
|
||
crosses only sometimes — a loop wholly inside a `restart-case` body has a legitimate local break — so the precedent
|
||
has to be replaced by a loop-depth-relative-to-frame-entry rule nobody has ruled on.
|
||
- **`continue` forces a `Tast.While` signature change.** `check_dotimes` folds the step into the body as
|
||
`While (cond, body @ [step])`, so a `continue` branching to the header skips the increment and hangs. It needs a
|
||
latch — `While of expr * expr list * expr list` — across `check.ml` and `emit.ml`. plan.org settles break and
|
||
continue as one item and `parse.ml` refuses them in one case, so building break against today's `While` is exactly
|
||
the thing that would have to be undone.
|
||
|
||
plan.org's single line on it (831) names a `for` the language does not have and gives no mechanism.
|
||
|
||
|
||
1. **Allocators, then `Vec` and `Map`.** The critical path, and the only thing standing between this and writing a
|
||
game. **`Vec` does not need generics** — that was wrong and is worth un-learning: Odin's containers are compiler
|
||
builtins over a *type-erased* runtime (`base/runtime/dynamic_array_internal.odin`), where `$T` appears only in thin
|
||
wrappers producing `size_of`/`align_of` at the call site, and per-key hash and equality are compiler-emitted
|
||
procedures passed as a runtime argument (`Map_Info`, `base/runtime/core.odin:369`). That runtime is what
|
||
`spec-memory.md` specifies.
|
||
|
||
**The four questions that used to sit here are answered**, in `spec-memory.md`'s "Allocators" section, which is
|
||
frozen along with the rest of that file: when storage is released, the `drop` hook, alignment, and allocation
|
||
failure. Read them there rather than in a second copy here. The one consequence the build order below turns on is
|
||
that no allocating operation returns an error — a failure signals `StorageExhausted` under a `retry` restart — so
|
||
`push` and `put` are `Unit`, `clone` returns the container, and no signature grows a `Result`. One question is left
|
||
open in that section on purpose; it does not block the build.
|
||
|
||
2. **Typed restarts — `(use-value [v T] v)`.** The author's third TODO, and the most-wanted thing across every
|
||
comparative study. SBCL's report: restarts without parameters lose "the entire supply-a-value half of the standard
|
||
vocabulary", because `use-value` and `store-value` are the only two whose answer comes from outside the program.
|
||
Needs argument marshalling in `emit.ml` and §3's arity check in `check.ml`; both files are free now. The leverage
|
||
SBCL lacks: `eval` already compiles and runs an expression inside the live program, and the daemon already holds
|
||
the struct layouts, so "ask the human, type-check the answer, hand it over" is a short hop.
|
||
|
||
3. **`handler-case`.** Not a convenience — it is the fix for the loudest gotcha in `conditions.org`. A handler closes
|
||
over nothing *only because* a `handler-bind` clause runs at the signal point; a `handler-case` clause runs in the
|
||
establishing frame, which is ordinary in-frame code exactly like a `restart-case` clause. SBCL's is `handler-bind`
|
||
plus a transfer and nothing more (`src/code/error.lisp:196-268`). Every piece exists.
|
||
|
||
### `Vec` and `Map` — the order to build them in
|
||
|
||
The dependency nobody had written down, and the reason it looked worse than it is. `spec-memory.md` defines an
|
||
allocator as "a procedure plus an opaque data pointer" — a function value. `check.ml` refuses function values four
|
||
ways, and all four say milestone 5: a written `(Fn ...)` annotation (`Ast.Tfn`, line 209), a written `fn` literal
|
||
(`Ast.Fn`, 458), a `defn`'s name used as a value (571), and calling anything other than a named function (1023).
|
||
`(Vec T)`, `(Map K V)`, `(Result T E)` and `(Handle T)` are refused at 215–218 as milestone 6. Read straight off those
|
||
lines, milestone 6's allocators need milestone 5's function values and the work doubles.
|
||
|
||
**The escape is real and the work does not double.** All four refusals are about *surface syntax*, and a value the
|
||
compiler builds that no surface form names trips none of them. The compiler already does exactly this, twice:
|
||
|
||
- A `handler-bind` clause is lowered to a function whose address goes into a `flan_handler` and is called back through
|
||
`h->fn(condition, xfer)` (`runtime/flan_rt.c:38` and `:66`). `check.ml` builds that body as its own `Tast.fn`
|
||
(`:619`, `:654`), not as an `Ast.Fn`, so line 458 never sees it, and no Flan type names the result.
|
||
- In a dev build, `emit.ml`'s `call` loads a pointer out of an indirection cell and calls through it
|
||
(`lib/emit.ml:781`–`793`). That is the indirect call line 1023 refuses in source, emitted routinely.
|
||
|
||
It is also what `spec-memory.md` already assumes for `Map`: the hash and equality pair is compiler-emitted and passed
|
||
as a runtime argument. Odin's `Map_Info` is two contextless `proc` fields (`base/runtime/core.odin:369`), and Odin's
|
||
`Allocator` is a `procedure` plus a `data: rawptr` (`:422`) — the same shape, reached the same way. If the hash pair is
|
||
expressible with no function type in the surface language, so is the allocator's procedure.
|
||
|
||
So: **`Allocator` is a builtin opaque type, the way `string` is a builtin ptr+len.** It is a `Types.t` case with no
|
||
user-writable constructor. Its procedure is an ordinary top-level function resolved to a symbol at the emit site, and
|
||
`vec-new`, `push`, `put`, `clone`, `free` and `free-all` are named calls, which `check_call` already routes through
|
||
`named_call` (`check.ml:1021`). **The built-in allocators need nothing from milestone 5.**
|
||
|
||
What *does* need milestone 5 is a **user-written** allocator: the moment a program says "here is my proc, make an
|
||
`Allocator` from it", it needs a `defn`'s name in value position, which is `check.ml:571` verbatim. That is a real
|
||
limit and not a fatal one — Odin ships arena, general-purpose, stack, pool and scratch in its own std, and most
|
||
programs write none. Ship the built-in set; user allocators arrive with function values.
|
||
|
||
**5 and 6 interleave rather than nest.** plan.org orders generics and macros (5) before allocators and containers (6),
|
||
and that order cannot hold: the macro expander is blocked on `Form` being a Flan union and union *values* are milestone
|
||
6 (see "Macros" below). Conditions and restarts, also listed under 6, are already three steps of four. The milestone
|
||
numbers are a topological hint, not a sequence. Take 6's container half first, 5's generics half second, and 5's
|
||
expander last, on 6's unions.
|
||
|
||
1. **`Allocator` and the arena.** The builtin type, the four operations (`alloc`, `resize`, `free`, `free-all`) with
|
||
`size` and `align` as parameters, the capability set with `can-free` in it, `with-allocator`, and the dev-build
|
||
epoch counter. No container yet. Odin reads its capability set back through the same procedure — `Query_Features`
|
||
returning an `Allocator_Mode_Set` (`core/mem/allocators.odin:315`) — and a field on the allocator value is the same
|
||
information without the round trip.
|
||
2. **`(Vec T)`**, over the type-erased runtime: `ptr + len + cap + allocator` in release, plus the generation word and
|
||
the epoch in dev. `push`, `reserve`, `at`, `len`, `as-slice`, `free`, `clone`. `size_of`/`align_of` are produced at
|
||
the call site, which here is simply the concrete call site, there being no generics yet.
|
||
|
||
**`Vec` alone does not buy the accumulation pattern, and it is worth knowing that before step 2 is scoped.**
|
||
`(fn [c] (push errors c) ...)` over an enclosing `(Vec ParseError)` is the pattern `handler-bind` exists for, and it
|
||
needs two further things. Capture does not exist at all: `check.ml`'s `captured` (`:176`) consults `ctx.outer` only
|
||
to raise a better refusal — "a handler cannot see %s ... Use a global, or pass it on the condition" — and `lookup`
|
||
(`:167`) never reads `outer`. That is `spec-memory.md`'s case 2, a non-escaping `fn` capturing by value into a stack
|
||
environment, and it is unbuilt. On top of it, the same spec says a captured `Vec` or `Map` is captured **by pointer,
|
||
not moved**, and every capturable type today is a value type, so the by-value/by-pointer split has never had to
|
||
exist in a capture path. Both are their own work and neither falls out of `Vec`. Step 2 delivers a container;
|
||
accumulating into one from a handler is a separate item and should be planned as one.
|
||
3. **`StorageExhausted` and `retry`, with step 2 and not after.** `restart-case` and the transfer channel exist, so
|
||
this is a compiler-emitted restart at each allocating site and little else. It goes in at the same time because the
|
||
signatures depend on it: retrofitting it later adds a transfer check to every call site of every allocating
|
||
operation, which is the whole point of having decided it now.
|
||
4. **`(Map K V)`** — flat open-addressed key and value arrays, with a compiler-emitted hash and equality pair per key
|
||
type passed as arguments. `spec-memory.md`'s structural-key restriction holds this to the built-in key set, so there
|
||
is no dispatch to design.
|
||
5. **`drop`.** The hook, the transitive move-only and non-`clone`able rules, and the refusal to construct a
|
||
`drop`-carrying value against an allocator without `can-free`. It is additive — no type in the repo has a hook today
|
||
— but the `can-free` refusal has to land with the construction path it guards, before any arena-allocated container
|
||
of a user struct is trusted.
|
||
6. **`(Result T E)` and `try`, then the rest of union values.** Unions are what `Form` needs, and `Form` is what the
|
||
macro expander needs.
|
||
7. **Generics and monomorphisation**, then function values. User-written allocators and escaping closures both fall out
|
||
of the second.
|
||
8. **The macro expander**, last, on 6's unions.
|
||
|
||
**What is genuinely unsettled, and none of it blocks step 1.**
|
||
|
||
- `spec-memory.md`'s "Open: catching a use-after-release statically" is open by decision, not by omission. It says the
|
||
static rule needs to know which allocator a construction used and that `with-allocator` plus `context/allocator` are
|
||
exactly what deny that knowledge; the shipping answer is the dev-build epoch trap, which is specified and buildable.
|
||
It also says what would settle it — real Flan programs using arenas, to show whether the escapes that occur are
|
||
lexical — and that is evidence this repo cannot produce until after step 2. **Build against the epoch trap.**
|
||
- **The operation table may be one operation short.** It has `free-all` and nothing else. Zig's `ArenaAllocator.reset`
|
||
takes a `ResetMode` of `free_all`, `retain_capacity` or `retain_with_limit`
|
||
(`lib/std/heap/arena_allocator.zig:57`–`69`), and for a frame arena reset every frame, retain-capacity is the normal
|
||
case and free-all is the unusual one — handing the pages back to the backing allocator only to ask for them again.
|
||
Odin's `arena_free_all` is retain-capacity in effect, because its arena is one fixed backing buffer and the call only
|
||
sets `offset = 0` (`core/mem/allocators.odin:287`). Flan should decide whether `free-all` means either of these or
|
||
takes the mode. It is an amendment to `spec-memory.md`, it is small, and it is better made before the arena is
|
||
written than after.
|
||
- Escaping closures are still deferred (`spec-memory.md`, "Function values", case 3), and a user-written allocator is
|
||
not one — its procedure is a top-level `defn` with no captured environment. The two should not be conflated when
|
||
function values arrive.
|
||
|
||
### Bugs found and not yet fixed
|
||
|
||
- **Two citations in `spec-memory.md`'s Allocators section do not land where they say.** Checked against Odin
|
||
`819fdc7a8` and Carp `ea121b5a`, every other one is exact — `Map_Info` at `base/runtime/core.odin:369`,
|
||
`Allocator_Proc` at `:422`, the arena answering `.Free` with `.Mode_Not_Implemented` at
|
||
`core/mem/allocators.odin:307`–`308`, `#optional_allocator_error` on `append_elem` at
|
||
`base/runtime/core_builtin.odin:767`, and `// TODO(bill): Better error handling for failed reservation` at
|
||
`base/runtime/dynamic_array_internal.odin:107` and `:128`. The two that miss: `Map_Cell_Info` is at `core.odin:351`,
|
||
not `:350`; and `check.ml:1670` is the FFI `Declare` arm, not the `defer` registration — the claim it is offered for,
|
||
that a top-level `defer` is checked in a scope holding only parameters and globals, is true and lives in `check_fn`
|
||
at `check.ml:1800`–`1812`. `check.ml:505` is the `defer` refusal exactly as cited, and the Carp citations are right:
|
||
`getDropFunc` is `Memory.hs:804`, the drop-before-delete emit is `Emit.hs:1044`, and `docs/Drop.md` says outright
|
||
that `A.drop` "will be run ... when the `let` scope ends".
|
||
|
||
- ~~`web/examples/breakdemo.out` is stale and `check.sh` fails on it.~~ **Fixed.** Commit `4a6a8fa` made the break
|
||
banner number its restarts and the `.out` was never repinned. Nothing had to drive the socket in the end:
|
||
`check.sh` already builds this one `--dev` and runs it under `timeout 5`, keeping what it printed before it
|
||
stopped, so the repin was the `.out` plus the two prose copies of the banner — `web/index.html` and `BUILT.md` —
|
||
and a sentence on the page saying what the numbers are for, since a restart is taken by position.
|
||
|
||
- ~~A shadowed restart is offered and cannot be taken.~~ **Fixed.** A restart is taken by *position* now:
|
||
`(:op "restart-at" :index N :name NAME)` on the daemon, `restart-at N NAME` on the agent, and a numbered
|
||
`completing-read` in `C-c C-b`. `:name` is a receipt, not the lookup — it is checked against the name the snapshot
|
||
holds at that index and refused if the two have drifted, so a bare integer can be wrong out loud. `restart <name>`
|
||
survives for a raw socket and is now defined as `restart-at` on the first index offering the name, so the two verbs
|
||
cannot disagree. `break.flan` grew the shadowed pair and asserts 900, which is the only value in that file no by-name
|
||
lookup can produce. The C&R buffer still marks the shadowed row by name and could now offer it instead — small, and
|
||
not done here.
|
||
|
||
- ~~A restart chosen at a break inside a thunk is accepted, announced, and silently not taken.~~ **Fixed by refusing
|
||
it, with the reason.** Not by the depth NEXT.md proposed: recording the restart-stack depth on *entering the break
|
||
loop* counts the frames a `restart-case` inside the thunk pushed before it erred, and those are above the boundary
|
||
and work. The boundary is where it is made — `restart_floor` is set to `flan_restart_count()` around `j.call()` in
|
||
`flan_agent_poll`, saved and restored so thunks nest — and the outermost `floor` entries of the snapshot are marked
|
||
unreachable. They are listed and marked rather than hidden, refused by the listener before the reply, and carried to
|
||
the editor as `:unreachable (2 3)`. `test_dev.ml` breaks a stopped program a second time from inside `C-x C-e` and
|
||
asserts both halves: index 2 refused, index 0 taken.
|
||
|
||
- ~~Restart names are served from a stack that is being mutated.~~ **Fixed, and it was a precondition rather than a
|
||
separate bug.** Index-based resume is wrong by construction against a moving stack: unlike a name, an index carries
|
||
no evidence of what it meant. The agent copies the list on entering `break_loop` — names into its own buffer, frames
|
||
as the addresses a transfer carries — one snapshot per nested break, and every verb answers from it. Caps are
|
||
`SNAP_MAX` 64 restarts and `SNAP_NAMES` 4096 bytes; past either, the listing says how many it did not show. Neither
|
||
cap has a test, same blind spot as the 4K result cap below.
|
||
|
||
- **A snapshot generation has no test, and the window is a race.** A choice is validated against the snapshot on top
|
||
when the request arrives and resolved against the snapshot on top when the game thread next looks. Between those,
|
||
an evaluation the break loop is running can error and push a break of its own, whose loop would otherwise reach
|
||
[chosen_ready] first and take *its* index 2 for the one someone chose from the outer list. Each snapshot now carries
|
||
a generation, a choice is stamped with the one it was validated against, and a loop claims only what is addressed to
|
||
it — a mismatch is left set rather than discarded, because the listener already answered ok for it. Depth would not
|
||
do: an outer break resuming and a new one starting reuses the number. None of this is tested, because arranging the
|
||
window means landing a request inside a two-millisecond poll from outside the process. It wants a hook the test can
|
||
drive, not a sleep.
|
||
|
||
- **The job ring has no fullness check**, and the comment describing its overflow is wrong. `publish` never consults
|
||
`tail`; past `QUEUE` entries it overwrites the slot the consumer is reading, and `job` is 24 non-atomic bytes.
|
||
Reachable from a program that goes a long time between `agent/poll` calls.
|
||
- **`flan_dev_result_get` is not the seqlock its comment claims** — it reads the generation first, then a non-atomic
|
||
length, then returns a bare pointer the caller sends later.
|
||
- Smaller: `exit(134)` from the break loop with the listener inside `dlopen`; a `dlopen` handle leaked when a module
|
||
has no installer.
|
||
- **`(A {:x 1})` on a union variant says "unknown struct A"** rather than the union refusal `check_struct` plainly
|
||
intends — `env` has no table of variant names. A diagnostics bug, not a backend death.
|
||
|
||
### Test blind spots, from a mutation pass
|
||
|
||
Sixty mutations, nineteen left the whole suite green. The severe cluster is closed (`cleanup.flan`,
|
||
`signedness.flan`); these are not:
|
||
|
||
- `Reach`'s walk of index expressions, `addr` places and `restart-case` clause bodies — each confirmed to prune a
|
||
function a valid program calls, so the build fails to link.
|
||
- `flan_dev_global`'s size-change guard — the layout-drift check, with no test that retypes a global across a reload.
|
||
- A local shadowing an imported name is qualified anyway.
|
||
- The 4K result cap and the registry overflow guard have **no coverage at all**, rather than a missing assertion.
|
||
- The reader accepts an unknown string escape; `+5` stops being a number.
|
||
- And a warning: a reader mutation makes the suite **hang** rather than fail. A green run is not the only outcome to
|
||
plan for in CI.
|
||
|
||
### Asked for by the editor lanes
|
||
|
||
- **`(:op "condition")` → the stopped program's condition, rendered.** Two steps: `break_loop` currently does
|
||
`(void)condition;` and *discards the pointer*, so stash it beside `condition_name`; then the daemon builds a render
|
||
thunk aimed at that address, which is `Session.render` rooted at a `Ptr` instead of an expression.
|
||
- **The type identity is settled, and it is the qualified name** — `layout` is in, see BUILT.md. `Load` qualifies
|
||
every declaration at import, so the names in `Tast.structs` are a flat namespace where two packages' `Missing` are
|
||
`a/Missing` and `b/Missing`; a bare name is refused with the candidates rather than resolved. `condition` inherits
|
||
it for free: the string the break loop already reports *is* that name, because `Emit.struct_name_of` writes
|
||
`Types.Named` into `flan_error`. It is still open for **locals**, where DWARF gives a name and the name a debugger
|
||
reads is not qualified by anything.
|
||
- **`(:op "backtrace")` is blocked** on frame metadata — unlocked by the DWARF work, then a new agent verb. Locals are
|
||
blocked twice: DWARF for the frame layout, *and* the pointer-rooted render thunk. Restart source locations and
|
||
arity are blocked too — `flan_restart` carries `prev`, `name_id`, `name` and `namelen`, so both need a new field in
|
||
the frame, which means the compiler emitting it.
|
||
|
||
### One line away
|
||
|
||
- **`match` over enums.** Fully desugarable, wanted, and blocked only by `Ast.pattern` needing a keyword case, which
|
||
`load.ml` matches exhaustively.
|
||
- **`Build.executable` returns only `out`**, so the daemon recovers the host `.ll` by recomputing `Build.workdir ()`.
|
||
- **A `!DILexicalBlock` per `Let`.** Not one line, but the one thing left in the DWARF work: every `!DILocalVariable`
|
||
is currently scoped to the subprogram, so inside `(let [v 22] …)` nested in `(let [v 11] …)` lldb still answers
|
||
`p v` with **11**. The `~2` suffix makes both *visible*, which is not the same as making the answer right. It needs
|
||
block structure the typed IR does not carry, and the `llvm.dbg.declare`s moved out of the entry block.
|
||
|
||
### Deferred with a reason
|
||
|
||
- **Writing through a string literal** — see Sharp edges. Needs provenance, which is open decision #3.
|
||
- **`cstring` as a type.** Odin has no `string → cstring` conversion at all; it pays the same copy our shim already
|
||
makes. The one thing it buys is the *return* direction, and nothing in `vendor/raylib` returns a string.
|
||
- **`rune`.** Odin's is a 4-byte integer distinguished by a flag, so `i32` is the same thing. Non-ASCII text is
|
||
blocked on font loading, not on the string layer — and fonts are now bound.
|
||
- **Macro expansion.** The reader and the declaration are in. Running a macro means compiling it and `dlopen`ing it
|
||
into the compiler, which is the reload primitive pointed at ourselves — but a macro is `[Form] -> Form`, so `Form`
|
||
has to be a Flan union whose layout the compiler and the loaded macro agree on, and union *values* are milestone 6.
|
||
|
||
### Documents that contradict the code
|
||
|
||
- **`plan.org`'s jank #947 citation is wrong in its mechanism.** jank does not relink (it calls through vars, which
|
||
are already indirection cells) and never unloads (`remove_symbol` has no callers). The real cause was a
|
||
process-teardown race. We are safe from the repro — because we compile out of process, not because of cells. A
|
||
normative document citing the wrong mechanism protects the wrong invariant.
|
||
- **`plan.org` still lists open decision #7 as open** and the interpreter as a backend. It was settled the other way;
|
||
`NEXT.md` records the consequences as "already applied" to `plan.org`, and they never were.
|
||
- **nREPL's `eval` does carry `file`, `line` and `column`** — jank reads all three. The choice of s-expressions still
|
||
stands on its other grounds; the stated reason does not.
|
||
|
||
## Sharp edges
|
||
|
||
- **Two formatted numbers cannot be held at once.** `flan_i64_to_bytes`, `flan_f64_to_bytes` and `flan_u64_to_bytes`
|
||
all write into one `static char scratch[64]` — "rendered text lives here until the next call", flan_rt.c:184 — and
|
||
`(string b)` does not copy. So
|
||
|
||
```
|
||
(let [a (string (i64->bytes 11))
|
||
b (string (i64->bytes 22))]
|
||
(print a) (print " ") (println b)) ; => 22 22
|
||
```
|
||
|
||
`a` is 11 and prints 22. No crash and no diagnostic. This is not new — the `[u8]` already aliased — but a `string`
|
||
reads as more value-like and invites exactly this. Format, draw, measure, then format the next one; `digits.flan`
|
||
sequences itself strictly for this reason. `rl/draw-text` is safe because the shim's `flan_shim_cstr` copies out of
|
||
ptr+len before the call.
|
||
|
||
- **Writing through a string literal is undefined, and the two build modes
|
||
disagree about how.** `(let [s (bytes "Hi")] (set (at s 0) \h))` stores into
|
||
a `private unnamed_addr constant`. At `-O0` that is a store to read-only
|
||
memory and the program takes SIGSEGV; at `-O2` LLVM deletes it as undefined
|
||
and the program prints `Hi` and exits 0. Same source, and which way it fails
|
||
depends on a flag — the worst shape available, and worse than either outcome
|
||
alone.
|
||
|
||
Nothing refuses it. `bytes` turns a `string` into a `[u8]`, the language lets
|
||
you write through a slice, and by then nothing records that the bytes came
|
||
from a constant. The honest fix is provenance — knowing a slice's origin —
|
||
which is plan.org open decision #3 and deliberately deferred. A cheaper one
|
||
that is *not* a fix: emitting literals as mutable globals only moves which
|
||
flag misbehaves, and costs their read-only placement.
|
||
|
||
Found by the string lane while deciding whether `lower-ascii` should mutate
|
||
in place. It ships the copying version for exactly this reason, and that is
|
||
the rule to follow until provenance exists: **a function over a `string` must
|
||
not write through it.**
|
||
|
||
|
||
Most of these are edges the language keeps and you should know about. Two — the top-level namespace and the shift count,
|
||
both found by review after milestone 4 — were bugs that reached LLVM or ran wrong, and are **fixed**; each says so. They
|
||
stay written down because each one is now a rule the checker enforces, and a later change could quietly drop it.
|
||
|
||
- **An index converts from a narrower integer and never from a wider one.** `(at colors current-color)` with a `u32`
|
||
index works — anything above 2³¹ truncates to a negative `i32` and the unsigned bounds check rejects it. An `i64` index
|
||
is refused with the reason: 2³²+5 truncates to 5 and would read the wrong element with no trap at all.
|
||
- **There is one top-level namespace, and `check.ml` now enforces it.** The environment's tables are per-kind — structs,
|
||
unions, aliases, enums, functions, externs and globals each have their own — so only a function was ever checked for a
|
||
duplicate. `(defn item …)` beside `(defvar item …)` type checked and then died in LLVM as `redefinition of function
|
||
'@flan.item'`, a message about an emitted symbol with no source location left, and two colliding *type* declarations
|
||
were not caught anywhere. One pass over `Ast.declared_name` now runs before every other collection pass and rejects the
|
||
second declaration of a name whatever kind either one is. `declared_name` lives in `ast.ml` because `Load` needs exactly
|
||
the same set — the names an import renames — and two copies of that list would drift.
|
||
- **A shift count is bounded, two different ways.** A shift by the operand's own width or more is *poison* in LLVM, not
|
||
a wrong number: `(defn main [] i32 (<< 1 32))` compiled at -O2 to a bare `retq`, returning an undefined value. A literal
|
||
count out of range is now rejected in `check.ml` — that is the typo case — and `emit.ml` masks a computed count to
|
||
`width - 1`, which is what the hardware does anyway and which LLVM folds away whenever the count is constant. The
|
||
prelude's rotate masks its own count; that is now redundant but harmless.
|
||
- **A `u64` literal is its 64-bit pattern**, so `0xcbf29ce484222325` is a real `u64` and not an error. The cost is that
|
||
a negative *decimal* literal is accepted as a `u64` too, because the reader records the value and not how it was
|
||
written. Narrower unsigned types keep the strict check, which is where a typo like `300` for a `u8` actually shows up.
|
||
- **A folded constant skips `check`.** `(defconst rows (/ h c))` is emitted from the folding pass's value, because a
|
||
global's initialiser has to be a compile-time constant and only that pass knows this one is. Its range check is
|
||
therefore its own call to `in_range`; there is a regression test.
|
||
- A `let` binding takes no type annotation, which is why `sand.flan` names its FNV constants instead of writing them
|
||
inline.
|
||
- `(defn f [] f65 0.0)` still says *unknown name* rather than *did you mean f64*: with a single body form the parser
|
||
cannot tell a return type from the first expression. Only the parameter position and `(Option …)` are unambiguous.
|
||
|
||
## Loose ends from milestone 4
|
||
|
||
None of them blocking: block-scoped `defer`; package visibility, so `rl/get-color-raw` is not callable; a package
|
||
importing a package; imported unions.
|
||
|
||
## Macros — the reader and the declaration are in, the expander is not
|
||
|
||
The front half landed. What exists:
|
||
|
||
- **The reader** reads `` `x ``, `~x` and `~@x` as `(quasiquote x)`, `(unquote x)` and `(unquote-splicing x)`, exactly
|
||
as `'x` reads as `(quote x)`. It stays dumb: it does not count nesting levels, does not know whether an unquote is
|
||
inside a quasiquote, and attaches no meaning to the three names. Clojure's spelling, not Common Lisp's, because a comma
|
||
is whitespace in `is_delimiter` and every binding vector in the corpus relies on that. Backtick and tilde are delimiters
|
||
now, so `a~b` is two things.
|
||
- **`parse.ml` refuses all four by name.** `quasiquote` and `gensym` say expansion is not wired up; `unquote` and
|
||
`unquote-splicing` say they mean nothing outside a quasiquote, which is a mistake rather than a missing feature.
|
||
`(defmacro name [params] body ...)` at the top level is checked for shape and *then* refused — a malformed defmacro and
|
||
an unimplemented one get different reasons, so the shape rule is enforced before the feature exists.
|
||
|
||
Nothing is stored. There is deliberately no macro table and no `Ast.Defmacro`, because a table nothing reads is a place
|
||
for a design to rot, and the storage shape is the expander author's first decision, not a decision to inherit.
|
||
|
||
### How the expander should work
|
||
|
||
**There is no interpreter** (see "Why there is no interpreter" in `BUILT.md`) and there is not going to be one, so running a macro at
|
||
compile time means *compiling it and loading it into the compiler*. That machinery already exists and is measured:
|
||
`Emit.redefinition` → `Build.shared` → `dlopen` is ~19ms end to end, with the load itself at 0.04ms (see "The reload
|
||
primitive"). A macro is that pipeline pointed at the compiler's own process instead of the program's.
|
||
|
||
The shape it wants:
|
||
|
||
1. **A macro is a function `[Form] -> Form`.** Its parameters are forms and its result is a form, which means `Form.t`
|
||
has to exist on the Flan side — a `defunion` mirroring `lib/form.ml`, in the prelude, plus constructors and accessors.
|
||
That is the real work, and it is bigger than the expander itself: the compiler and the compiled macro have to agree on
|
||
the *layout* of a `Form`, not merely its shape, so whatever the checker does for unions has to be exact here. Until
|
||
unions are values this cannot start — `check.ml` puts union values and `match` on a union at **milestone 6**, so that is
|
||
milestone 6 work landing before milestone 5's.
|
||
2. **Expansion runs over `Form`, before `Parse`.** Not a pass over `Ast`: there is no `Ast.Defmacro` and `Parse` refuses
|
||
`defmacro` outright, so an `Ast`-level pass would have nothing to work with. That refusal is not a dead end, it is the
|
||
ordering — the expander runs first and `Parse` never sees a macro call at all. It is also the Clojure ordering, and the
|
||
reason a macro expanding to a special form is ordinary rather than a special case.
|
||
3. **Order matters and files do not have one.** Top-level names in a package are order-independent everywhere else
|
||
(`declared_types`, the constant fixpoint in `check.ml`). Macros cannot be: a macro must be compiled and loaded before a
|
||
call to it is expanded. Either collect every `defmacro` in a pre-pass and compile them as one module, or require
|
||
definition-before-use for macros specifically and say so in the error. The pre-pass is better and matches how the rest
|
||
of the frontend already behaves.
|
||
4. **A macro's own body may call macros**, so the pre-pass is a fixpoint, not a single sweep, and a cycle has to be
|
||
detected and named rather than looping.
|
||
5. **`gensym` is a runtime function of the compiler**, called by the loaded macro while it runs. It needs a counter that
|
||
lives in the compiler process and a name that cannot collide with a reader-produced symbol — the usual trick is a
|
||
character no symbol may contain, and this reader now has two new ones it could reserve. Hygiene is settled (plan.org,
|
||
open decision 2): deliberately non-hygienic, Common Lisp/Clojure style, explicit `gensym`, no `macrolet` until a
|
||
concrete use case appears.
|
||
6. **Quasiquote itself is a macro-shaped desugaring**, not a compiler feature: `` `(a ~b) `` becomes list-construction
|
||
over quoted pieces, with `~@` splicing. Written once, in the expander, over `Form`.
|
||
|
||
The four files this touches — `build.ml`, `check.ml`, `emit.ml`, `load.ml` — were owned by other lanes when the front
|
||
half landed, which is the only reason the expander is not here too.
|
||
|
||
### What would tell you it works
|
||
|
||
`when`, `unless`, `until`, `cond` and `dotimes` are special forms in `parse.ml` today, and plan.org milestone 5 says
|
||
they are special forms *only until macros land*. Moving one of them out of the compiler and into the prelude as a
|
||
`defmacro`, with the existing tests unchanged and still green, is the exit criterion — it proves expansion, quasiquote,
|
||
`gensym` and the ordering pre-pass at once, against a test suite written before any of them existed.
|
||
|
||
## Watch for
|
||
|
||
The rule that caught the two misparse bugs applies unchanged: **anything that binds a name, alters control flow, or is
|
||
not yet implemented must be recognised explicitly and rejected if unsupported.** `check.ml` rejects `Vec`, `Map`,
|
||
`Result`/`try`, union values, closures, quoted symbols, generics and function values *by name*, each with the milestone
|
||
it belongs to; `load.ml` rejects the package shapes it does not handle; and the FFI boundary rejects an aggregate. The
|
||
tests assert on the reason, not just on the failure.
|
||
|
||
## Untracked on purpose
|
||
|
||
`calc-me` and `sand`, the executables `flan build` drops beside their sources, are now in `.gitignore` — anchored
|
||
(`/calc-me`, `/sand`) so the patterns cannot match anything nested.
|
||
|
||
`old-ocaml/` — the pre-rewrite menhir/ocamllex frontend, kept as reference and excluded from the build by the root
|
||
`dune` file. Its contents are also in git history at `2c232dd`.
|