Joseph Ferano
20fedd4ad8
C-x C-e rendered the scalars and refused the rest, which made it a calculator
rather than a REPL. The renderer is now a compile-time walk over the type,
emitting a piece at a time: structs, nested structs, fixed arrays, slices,
options, enums by name, and pointers as their shape. A raylib Color comes back
through the FFI as (rl/Color {:r 17 :g 34 :b 51 :a 68}).
Piecewise emission is what makes composites possible at all - a struct is its
fields with punctuation between them, and concatenating that in generated IR
would need an allocator the language does not have.
u64 now renders, in C, with %llu. It used to refuse because i64->bytes is
signed and it would otherwise come back as -1, but refusing a whole struct
because one field is a u64 is much worse than adding a runtime entry point.
Strings are quoted and escaped in C for the same reason: unescaped content does
not round-trip and reads as a framing bug rather than as the value it is.
An enum renders as :name, recovered from the checker's table as a chain of
comparisons, since members are erased to i32 before the backend sees them; a
value outside the declared members falls through to its number, which is what
you would want to see. A pointer is rendered and never followed - it is the
only thing that could make the walk cycle, and dereferencing one a REPL was
handed is not a safe thing to do on someone's behalf.
Three bounds, easy to conflate. depth and span bound the walk, so sand's
[100 [100 u32]] grid does not unroll into ten thousand render sites. The output
is bounded once in the runtime, since a slice renders through a loop the
compiler cannot bound, and one place enforcing it means no renderer carries a
budget.
emit.ml's cast now treats an enum as the i32 it is. Nothing in the surface
language produces that - a keyword resolves against its enum and never widens -
but the renderer needs an enum's number when it falls outside the members.
927 lines
51 KiB
Markdown
927 lines
51 KiB
Markdown
# Where this is
|
||
|
||
**The dev loop is closed.** `C-c C-c` in Emacs recompiles the top-level form
|
||
at point and installs it in a running program, at that program's next frame
|
||
boundary. Verified against sand: an unsaved buffer edit to `game-draw`, and 240
|
||
consecutive frames drew it.
|
||
|
||
Steps 1, 2 and 3 are done — see *The reload primitive* below. A list
|
||
of top-level forms can be recompiled and installed into a running process; call
|
||
sites compiled before they existed follow them, and a `defn` or `defvar` the
|
||
process was never built with can be added and then redefined again. That is the
|
||
whole of `C-c C-c`, minus an editor: sand.flan takes a redefinition over a
|
||
socket and installs it between frames.
|
||
|
||
What is left is the *session* — something that holds the checker environment
|
||
between evaluations, tracks which names the running process was built with, and
|
||
speaks a protocol an editor can talk to.
|
||
|
||
Milestone 4 is done: **sand.flan builds, links raylib and runs**, and its
|
||
simulation has a headless acceptance case that runs on the `dune test` path at
|
||
`-O0` and `-O2`. Milestones 2 and 3 are behind it (`calc-me.flan` compiles and
|
||
runs; the interpreter was dropped — open decision #7, settled, see below).
|
||
|
||
```
|
||
reader ✅ → parse ✅ → load ✅ → check ✅ → emit ✅ → clang ✅
|
||
```
|
||
|
||
| File | What it does |
|
||
|---|---|
|
||
| `lib/loc.ml` | source locations + `Loc.Error`, the frontend's one exception |
|
||
| `lib/form.ml` | reader output: `Sym Kw Int Float Str Byte List Vec Map` |
|
||
| `lib/reader.ml` | hand-written S-expression reader, no menhir/ocamllex |
|
||
| `lib/ast.ml` | AST: `texpr`, `expr`, `place`, `pattern`, `decl` |
|
||
| `lib/parse.ml` | forms → AST; special forms, desugaring, declarations |
|
||
| `lib/load.ml` | **imports: a package directory → qualified declarations** |
|
||
| `lib/types.ml` | resolved types; structural equality, `Never` fits anywhere |
|
||
| `lib/tast.ml` | the typed IR the backend consumes |
|
||
| `lib/check.ml` | AST → typed IR; two passes, bidirectional |
|
||
| `lib/session.ml` | **a live program: what the process was built from, plus every change since** |
|
||
| `lib/wire.ml` | **the editor protocol: one s-expression per message, length framed** |
|
||
| `lib/dev.ml` | **`flan dev`: a session, the program running beside it, and a socket** |
|
||
| `lib/prelude.ml` | printers + `rand-f32`, written in Flan |
|
||
| `lib/emit.ml` | typed IR → LLVM IR text |
|
||
| `lib/build.ml` | `.ll` + the shim + the packages' C → clang → executable |
|
||
| `runtime/flan_rt.c` | the host ABI: argv, stdout, exit, 4 conversions |
|
||
| `runtime/flan_dev.c` | **dev only: the by-name registry a run-time-new name needs** |
|
||
| `vendor/raylib/` | **the raylib package: `raylib.flan`, `shim.c`, `link`** |
|
||
| `vendor/agent/` | **the dev agent: a socket, a loader thread, install at a frame boundary** |
|
||
| `emacs/` | **`flan-mode.el` and `flan-dev.el`: the editor half of the dev loop** |
|
||
| `sand-sim/` | **the falling-sand simulation, with no raylib in it** |
|
||
| `bin/main.ml` | `flan read \| parse \| check \| emit \| build \| run \| reload \| dev` |
|
||
| `test/test_flan.ml` | reader, parser and checker |
|
||
| `test/test_acceptance.ml` | expression/result pairs + whole programs + the traps |
|
||
| `test/test_reload.ml` | **the reload primitive: recompile one function, load it, call it** |
|
||
| `test/test_agent.ml` | **a running program taking a redefinition over a socket** |
|
||
| `test/test_session.ml` | **what a running process cannot be told, and recovering from a typo** |
|
||
| `test/test_dev.ml` | **the daemon, driven the way an editor drives it** |
|
||
| `test/test_repl.ml` | **`C-x C-e`: an expression evaluated inside a running program** |
|
||
| `test/test_emacs.ml` | **the client, driven against a real daemon and a real program** |
|
||
| `test/reload_host.c` | the C host that loads and installs two rebuilds, in one process |
|
||
|
||
```
|
||
$ flan run calc-me.flan "1 + 2 * (3 - 0.5) / 2"
|
||
3.5
|
||
$ flan run test/programs/sand-headless.flan
|
||
2256461126764447066
|
||
$ flan run sand.flan # a window, 120 fps, hold space
|
||
```
|
||
|
||
## What milestone 4 added
|
||
|
||
**`dotimes`** desugars in `check.ml` to a `Let` plus a `While` — no new IR node.
|
||
The bound is evaluated once into a hidden slot before the loop, so a body that
|
||
changes it cannot change the trip count, and the loop variable is not
|
||
assignable, which makes the generated step its only writer.
|
||
|
||
**`defer`** is recognised in `check_fn` and nowhere else, because that is the
|
||
only place that knows a form is at the top level of a function body. Each one
|
||
is checked in place, then registered on the context; it emits nothing where it
|
||
stands. Function exit runs them innermost-first, and an explicit `return` runs
|
||
the ones registered *above* it — a defer written below a return has not
|
||
executed yet and must not fire. A trap runs none of them, which follows from
|
||
the bounds-check shape (`noreturn` then `unreachable`) rather than being a
|
||
separate decision.
|
||
|
||
`defer` inside a `let`, a loop or a branch is **rejected**, not accepted with
|
||
function scope. It would run once at function exit rather than once per
|
||
iteration, and that is the silent-wrongness class the rule below is about.
|
||
Block-scoped defer is real work and is not done.
|
||
|
||
**New builtins:** `zeroed` (takes its type from the place it is stored into),
|
||
`min`/`max` (each operand through a slot, so neither is evaluated twice),
|
||
`bit-and`/`bit-or`/`bit-xor`/`<<`/`>>` (integers only; `>>` is arithmetic on a
|
||
signed type and logical on an unsigned one), and `rand-f32`.
|
||
|
||
**`rand-f32` is in the prelude, in Flan** — PCG-XSH-RR 32 over a `u64` state.
|
||
It is not libc's, because a grid hash is only a regression test if the sequence
|
||
is byte-identical on native and wasm32 (plan.org, RNG is ours). `rand-seed`
|
||
sets the state. This is what the bitwise operators were added for.
|
||
|
||
**Enums and keywords.** `(defenum Name [member value ...])` gives a type that
|
||
is an `i32` at run time and its own type in the checker, so `:space` at a call
|
||
site resolves against the parameter's enum and a typo is an error there rather
|
||
than a wrong number later. A keyword means nothing where no enum is expected —
|
||
there is no keyword type to fall back on.
|
||
|
||
## Why the FFI goes through a C shim
|
||
|
||
The decision that shapes the whole raylib package. What clang generates for
|
||
raylib's own prototypes on x86-64:
|
||
|
||
```
|
||
Vector2 {float,float} → declare <2 x float> @GetMousePosition()
|
||
Color {u8,u8,u8,u8} → declare void @ClearBackground(i32)
|
||
Rectangle {4 × int} → declare { i64, i64 } @mkrect()
|
||
```
|
||
|
||
None of those is the struct's own LLVM type. A small aggregate's calling
|
||
convention is not part of its layout — it is a per-target classification the
|
||
*caller* has to reproduce, and x86-64, arm64 and wasm32 classify differently.
|
||
Putting that in `emit.ml` is three classifiers to write and then keep correct
|
||
forever, and a mistake shows up as `(.y m)` returning garbage rather than as a
|
||
link error.
|
||
|
||
So `vendor/raylib/shim.c` has one wrapper per binding, each one flattening the
|
||
aggregates: a struct returns through an out-pointer, a struct argument is
|
||
passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates
|
||
a copy. clang classifies all of it, per target, for free. `check.ml` enforces
|
||
the rule — an aggregate in a `declare` signature is rejected with the reason —
|
||
so the boundary cannot quietly acquire one. This is plan.org's "one narrow host
|
||
ABI, implemented twice", and `flan_rt.c` is the same pattern.
|
||
|
||
The price is a hand-written wrapper per raylib call. They are one-liners and
|
||
mechanical enough to generate if that ever becomes the bottleneck.
|
||
|
||
`raylib.flan` declares each `-raw` entry point and wraps it in an ordinary Flan
|
||
function just below, so the surface sand.flan sees is `(rl/get-mouse-position)`
|
||
returning a `Vector2`. Verified end to end, headless: `GetColor(0x11223344)`
|
||
comes back as `17 34 51 68`, four separate bytes — a `Color` is *not* the
|
||
little-endian reading of the packed integer, so an identity would have passed a
|
||
weaker test. That case is in the acceptance table, skipped if `libraylib` is
|
||
not installed.
|
||
|
||
The bindings are 18 calls: window (`init-window`, `close-window`,
|
||
`window-should-close?`, `set-target-fps`, `set-trace-log-level`), keyboard
|
||
(`key-pressed?`/`down?`/`released?`), mouse (`mouse-button-pressed?`/`down?`/
|
||
`released?`, `get-mouse-position`), `get-color`, and drawing (`begin-drawing`,
|
||
`end-drawing`, `draw-fps`, `clear-background`, `draw-rectangle`), plus the
|
||
`Key`, `MouseButton` and `TraceLogLevel` enums. Adding one is three lines: a
|
||
`declare`, an `extern` prototype, and a one-line wrapper.
|
||
|
||
No raylib headers are needed: `shim.c` declares the prototypes it uses, so the
|
||
build depends on the shared library being linkable and not on `raylib-devel`.
|
||
`vendor/raylib/link` carries `-l:libraylib.so.550` because Fedora ships the
|
||
runtime library without the `.so` symlink.
|
||
|
||
## Packages
|
||
|
||
`lib/load.ml` resolves `(import rl "vendor:raylib")` before the checker runs.
|
||
The directory is the package; `vendor:` is a collection, resolved by walking up
|
||
from the importing file until a directory of that name is found; a path with no
|
||
collection is relative to the importing file. Importing is a **rename**: every
|
||
top-level name the package declares becomes `alias/name`, and every use of one
|
||
— in a type, in a body, in a struct literal, in an *array length* — is
|
||
rewritten to match. Local bindings shadow. Nothing downstream knows a package
|
||
existed; the checker sees one flat list of declarations whose names contain a
|
||
slash.
|
||
|
||
A package may also carry the C it binds to: every `.c` file in the directory is
|
||
compiled into the build, and a file named `link` lists extra linker arguments.
|
||
|
||
This is not a module system yet. No visibility (hence `rl/get-color-raw` being
|
||
callable), no cycle detection, and a package cannot import another one.
|
||
|
||
## sand.flan is two programs
|
||
|
||
plan.org wants sand tested twice — interactive at 120 fps, and headless over N
|
||
frames with the grid hashed, the version CI runs on native *and* wasm32. Those
|
||
cannot be one binary: `Load` collects a package's C sources and linker
|
||
arguments unconditionally, so anything importing the raylib package links
|
||
libraylib on every target regardless of what its `main` does, and on wasm32
|
||
that link cannot succeed.
|
||
|
||
So the simulation moved to `sand-sim/`, which imports nothing. `sand.flan`
|
||
imports it as `sim/` and adds the window, the mouse and the drawing;
|
||
`test/programs/sand-headless.flan` imports it and adds a seed, four
|
||
deterministic clouds, 40 frames and an FNV-1a hash. One copy of the physics.
|
||
|
||
The headless case is what actually *verifies* milestone 4 — running the
|
||
interactive build only proves it enters its loop, because with no mouse input
|
||
the grid stays empty and `paint-at`, `settle` and `move-grain` never execute on
|
||
real data. Measured through the probe: 168 grains painted around row 4–8, still
|
||
168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.
|
||
|
||
**Three edits were made to sand.flan's own text**, and they are language
|
||
decisions rather than fixes:
|
||
|
||
- `(defconst gravity 0.05)` → `(defconst gravity f32 0.05)`. An untyped float
|
||
constant is `f64`, `velocity` is `[f32]`, and there is no implicit widening.
|
||
- `(defvar current-color u32)` → `i32`. It is an index into `colors`, and
|
||
`(len colors)` is an `i32`.
|
||
- The file was split as above, so its body now says `sim/rows` and so on.
|
||
|
||
`(defn main [])` is unchanged — the short form, as plan.org says.
|
||
|
||
Painting is on **hold left mouse button** rather than on space, since the mouse
|
||
bindings exist now. Space is still what cycles the colour, on release, which is
|
||
a leftover and probably wants to move to the right button or to a key press.
|
||
|
||
## Bounds checks — done at milestone 3
|
||
|
||
`at` and `slice` emit `icmp` → `br` → cold block → `call` → `unreachable`; a
|
||
failure names the source location. Three check sites: `at` on `[n T]` (static
|
||
bound, folded by LLVM for a literal index — and a literal that is out of bounds
|
||
never reaches emit, `check.ml` rejects it), `at` on a slice or string (runtime
|
||
len), and `slice` (two comparisons — `lo <= hi` is not redundant, without it a
|
||
reversed range yields a huge unsigned length). All comparisons unsigned.
|
||
`Build.opts.checks` is on by default and **not** tied to `opts.opt`, which is
|
||
what lets the acceptance table run the same programs at `-O0` and `-O2` with
|
||
identical checks. The flag is `--no-bounds-checks`.
|
||
|
||
The write path is its own case: `(set (at arr n) …)` lowers through
|
||
`place`/`Pindex`, not through `At`, so a refactor that split them would break
|
||
the write check silently. The test covers both.
|
||
|
||
Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs
|
||
at 0.11–0.12s checked against 0.12–0.13s unchecked. Indistinguishable.
|
||
|
||
## Why there is no interpreter
|
||
|
||
Open decision #7 is settled: **the compiled path is the only backend.** Both
|
||
arguments for a permanent interpreter had expired — the instrumentation step
|
||
debugger that wanted it is cut, and compiled redefinition measured at ~16ms,
|
||
perceptually instant for expression eval too. Milestone 3 did not need an
|
||
oracle either: the acceptance table is hand-written, so the table *is* the
|
||
oracle. Consequences already applied: milestone 2's "interpreted calls per
|
||
second" criterion is dropped, and the host ABI moved onto the critical path.
|
||
|
||
## The layout, which is the whole backend design
|
||
|
||
```
|
||
i8..i64 / u8..u64 i8..i64 signedness lives in the ops
|
||
f32 f64 float double
|
||
bool i1
|
||
an enum i32
|
||
[T] and string { ptr, i64 } ptr+len, non-owning
|
||
[n T] [n x T] inline, a value
|
||
(Ptr T) ptr opaque pointers
|
||
(Option T) { i8, T } tag 0 None, 1 Some
|
||
a struct a literal struct, declaration order
|
||
Unit and Never {}
|
||
```
|
||
|
||
No object headers anywhere, so a Flan struct is exactly its C struct and
|
||
nothing marshals. Two consequences carry the semantics:
|
||
|
||
- **Every slot is an `alloca`.** Reading a local is a `load`, assigning is a
|
||
`store`, and a `store` of an aggregate *is* the copy `spec-memory.md`
|
||
requires. `addr` of a local is then just the alloca, and `mem2reg` removes
|
||
the ones nobody addressed. `test/programs/values.flan` pins this down.
|
||
- **A place is a pointer, a value is a load from it.** `(set (.pos c) …)`
|
||
through a `(Ptr Cursor)` becomes a `getelementptr` on the pointer, not on a
|
||
copy. This is the split that would have made a tree-walker silently wrong.
|
||
|
||
Non-local exit is lowered explicitly: `return`, `some` and a failed bounds
|
||
check are branches, never platform unwinding, so wasm32 needs no exception
|
||
proposal.
|
||
|
||
## Sharp edges
|
||
|
||
Most of these are edges the language keeps and you should know about. Two —
|
||
the top-level namespace and the shift count, both found by review after
|
||
milestone 4 — were bugs that reached LLVM or ran wrong, and are **fixed**; each
|
||
says so. They stay written down because each one is now a rule the checker
|
||
enforces, and a later change could quietly drop it.
|
||
|
||
- **An index converts from a narrower integer and never from a wider one.**
|
||
`(nth colors current-color)` with a `u32` index works — anything above 2³¹
|
||
truncates to a negative `i32` and the unsigned bounds check rejects it. An
|
||
`i64` index is refused with the reason: 2³²+5 truncates to 5 and would read
|
||
the wrong element with no trap at all.
|
||
- **There is one top-level namespace, and `check.ml` now enforces it.** The
|
||
environment's tables are per-kind — structs, unions, aliases, enums,
|
||
functions, externs and globals each have their own — so only a function was
|
||
ever checked for a duplicate. `(defn item …)` beside `(defvar item …)` type
|
||
checked and then died in LLVM as `redefinition of function '@flan.item'`, a
|
||
message about an emitted symbol with no source location left, and two
|
||
colliding *type* declarations were not caught anywhere. One pass over
|
||
`Ast.declared_name` now runs before every other collection pass and rejects
|
||
the second declaration of a name whatever kind either one is. `declared_name`
|
||
lives in `ast.ml` because `Load` needs exactly the same set — the names an
|
||
import renames — and two copies of that list would drift.
|
||
- **A shift count is bounded, two different ways.** A shift by the operand's
|
||
own width or more is *poison* in LLVM, not a wrong number: `(defn main [] i32
|
||
(<< 1 32))` compiled at -O2 to a bare `retq`, returning an undefined value. A
|
||
literal count out of range is now rejected in `check.ml` — that is the typo
|
||
case — and `emit.ml` masks a computed count to `width - 1`, which is what the
|
||
hardware does anyway and which LLVM folds away whenever the count is
|
||
constant. The prelude's rotate masks its own count; that is now redundant but
|
||
harmless.
|
||
- **A `u64` literal is its 64-bit pattern**, so `0xcbf29ce484222325` is a real
|
||
`u64` and not an error. The cost is that a negative *decimal* literal is
|
||
accepted as a `u64` too, because the reader records the value and not how it
|
||
was written. Narrower unsigned types keep the strict check, which is where a
|
||
typo like `300` for a `u8` actually shows up.
|
||
- **A folded constant skips `check`.** `(defconst rows (/ h c))` is emitted
|
||
from the folding pass's value, because a global's initialiser has to be a
|
||
compile-time constant and only that pass knows this one is. Its range check
|
||
is therefore its own call to `in_range`; there is a regression test.
|
||
- A `let` binding takes no type annotation, which is why `sand-sim` names its
|
||
FNV constants instead of writing them inline.
|
||
- `(defn f [] f65 0.0)` still says *unknown name* rather than *did you mean
|
||
f64*: with a single body form the parser cannot tell a return type from the
|
||
first expression. Only the parameter position and `(Option …)` are
|
||
unambiguous.
|
||
|
||
## The reload primitive — dev loop steps 1 and 2, measured
|
||
|
||
`llc` → `ld -shared` → `dlopen` → call, with no protocol and no daemon.
|
||
`dune test` runs it: one function is recompiled into its own object and called
|
||
inside a process that is already running, twice, with a changed body the second
|
||
time.
|
||
|
||
| Step | Cost |
|
||
|---|---|
|
||
| `Emit.redefinition` | below the timer (<0.1ms) |
|
||
| `llc -O2 -filetype=obj` | 15–17ms |
|
||
| `ld -shared` | 3ms |
|
||
| `dlopen` + `dlsym` | **0.04ms** |
|
||
|
||
**~19ms end to end**, and the load itself is free. plan.org's 16ms was measured
|
||
with clang somewhere else; this is the number from this codebase. For contrast,
|
||
`clang -shared` on the same IR is 50ms — the driver is again most of the cost,
|
||
which is why the dev path skips it. `llc` and `clang` are both 20.1.8 here;
|
||
check that before trusting the `.ll`, since the driver absorbs IR the bare
|
||
tools reject.
|
||
|
||
`ld -shared` rather than `clang -shared` for a second reason: a shared object
|
||
is allowed undefined symbols, and that *is* the mechanism. What the new module
|
||
does **not** define is the whole design:
|
||
|
||
- **a global is `external`.** This settles the open question below in the only
|
||
direction that supports the demo: a redefinition can change a function's
|
||
body and can never re-initialise the program's data. Define the global and
|
||
the loaded object gets a second copy — sand's `grid` would reset on every
|
||
reload, and "edit the code, keep the sand" is the thesis.
|
||
- **every other function is a `declare`**, so a redefined `settle` calls the
|
||
host's `move-grain` rather than freezing a private copy of it.
|
||
- **no `main`.** This module is loaded, not started.
|
||
|
||
Its string constants still come along; omitting them is an undefined `@.str.N`
|
||
at link time, and it is easy to miss because a one-function module usually has
|
||
none. `Emit.signature` is now the single place a function's LLVM signature is
|
||
spelled, because a `define` here and a `declare` there drift the moment one of
|
||
them grows a case for `Unit` or for a slice parameter.
|
||
|
||
**`-rdynamic` is load-bearing.** A normal executable exports nothing: `nm -D
|
||
calc-me | grep 'flan\.'` is empty, so a loaded module's `declare`s would have
|
||
nothing to bind to. The test passes it through `lflags`, which keeps it a
|
||
property of the dev build rather than of every build. `dlsym` on `"flan.bump"`
|
||
works — a dot is legal in an ELF symbol.
|
||
|
||
Two things about the test are deliberate and are what make it prove anything:
|
||
both loads happen in **one process**, since two runs would pass while saying
|
||
nothing about an in-process swap; and the versions are **two paths**, since
|
||
`dlopen` caches by path and re-opening one would hand back the handle it
|
||
already had, so the check would lie. And `helper` is `(* x 2)` in one fixture
|
||
and `(* x 3)` in the other: the second body is dead text, since the module
|
||
declares `helper` rather than defining it, so the expected 1024 coming back
|
||
instead of 1036 is what proves the call landed on the host's copy. With the two
|
||
bodies identical nothing at run time would notice a module that grew its own.
|
||
|
||
String constants are emitted `private unnamed_addr`, so the module's own
|
||
`@.str.N` cannot be interposed by the host's — worth knowing, because with
|
||
external linkage a redefined function would silently print the *old* text and
|
||
nothing would fail at link time. The fixtures each print a literal so that path
|
||
is actually exercised.
|
||
|
||
### Cells — how a call site follows a redefinition
|
||
|
||
Loading a new body is not installing it. A call bound at link time cannot be
|
||
made to notice one, so **a dev build routes every Flan-to-Flan call through a
|
||
cell**: a mutable global holding the address of the function that is current.
|
||
|
||
```
|
||
@"flan.cell.bump" = global ptr @"flan.bump" ; the host defines it
|
||
%p = load ptr, ptr @"flan.cell.bump" ; every call site
|
||
%r = call i64 %p()
|
||
```
|
||
|
||
Redefinition is then one store. A redefinition module declares the cells
|
||
`external`, exactly like the globals, and exposes `flan_reload_install()` that
|
||
stores its own body into its own cell — cost **below a microsecond**, which is
|
||
what makes a frame-boundary swap a non-event.
|
||
|
||
The cell load is emitted *after* the arguments, so a redefinition landing
|
||
between two calls cannot land in the middle of one.
|
||
|
||
Four things about this that are not free choices:
|
||
|
||
- **`flan_reload_install` is a named function and not an ELF constructor.** A
|
||
constructor runs during `dlopen`, on whatever thread called it, mid-frame.
|
||
The agent has to choose when the store happens. Loading and installing are
|
||
separate on purpose.
|
||
- **A redefinition's own body is `hidden`.** Default visibility in a shared
|
||
object is interposable, and that applies to *taking the address* too: plain
|
||
`@"flan.bump"` inside the module resolves to the host's copy, so the
|
||
installer would publish the very function it was replacing and the reload
|
||
would appear to do nothing. There is a test on the linkage, because the
|
||
failure is silent.
|
||
- **This also fixes the self-call edge**, which the previous version of this
|
||
section listed as a sharp edge: a redefined function calling itself goes
|
||
through the cell like any other call, so it reaches the new body. v2 of the
|
||
fixture recurses on purpose, and would print the old body's text if it did
|
||
not.
|
||
- **`-rdynamic` is what exports the cells**, so it and cells are one flag:
|
||
`Build.opts.dev`, `flan build --dev`. This is the first time `opts` means
|
||
something semantic rather than an optimisation level.
|
||
|
||
LLVM cannot fold the indirection away — the cell is an external mutable global
|
||
— and a `--dev` build of calc-me keeps 46 indirect calls at `-O2`. The
|
||
acceptance table now runs `values`, `machine` and `sand-headless` as dev builds
|
||
as well; the sand hash is the case that matters, since it is the one result
|
||
that would notice a call reaching the wrong function.
|
||
|
||
### Names that did not exist when the process started
|
||
|
||
Editing a `defvar` or a `defn` is a symbol the host exports. *Adding* one is
|
||
not: there is no symbol to bind to and ELF cannot grow one. Those go through
|
||
`runtime/flan_dev.c`, which is two lookups and nothing else:
|
||
|
||
```
|
||
void **flan_dev_cell(const char *name); /* a new function's cell */
|
||
void *flan_dev_global(const char *name, uint64_t); /* a new global's storage */
|
||
```
|
||
|
||
Both are idempotent, so the second module to mention a name gets what the first
|
||
one got — which is the entire point. A new global's **declared initial value
|
||
travels with it**, as a constant the runtime copies on the allocation and
|
||
ignores on every call after: `calloc` alone is only right for ZII, and the
|
||
"ignores afterwards" half is where "a reload must not reset the program's
|
||
state" lives. Putting it in the allocation path rather than in a branch at the
|
||
call site means the rule cannot be got wrong at one of them. The compiler picks per name: a name the
|
||
host has is a symbol (one load at a call site), a name it lacks is a registry
|
||
lookup cached at install time in a module-local slot (two loads). So the common
|
||
case pays nothing for the general one.
|
||
|
||
**The unit is a list of top-level forms**, not one function — `Emit.redefinition
|
||
~fns`. `C-c C-c` passes one name, `C-c C-k` passes a file's worth, one code
|
||
path either way. It has to be: v3 of the fixture adds `extra` and uses it from
|
||
a redefined `bump`, and splitting that into two loads would leave a module
|
||
referring to storage that does not exist yet.
|
||
|
||
Four rules, each of which is a silent failure if broken:
|
||
|
||
- **Every lookup resolves before any body is published.** Publish first and a
|
||
caller reaches a function whose slots are still null. Not race-testable, so
|
||
it is asserted on the emitted `flan_reload_install`.
|
||
- **`flan_dev_global` refuses a size change.** The running process has already
|
||
laid that memory out; handing back the old allocation for a differently
|
||
shaped type means the new body reads fields at the wrong offsets and nothing
|
||
says so. This is the layout-drift rule's first enforcement point. Retyping a
|
||
var needs a restart.
|
||
- **Nothing is ever `dlclose`d.** A cell holds an address inside a module's
|
||
text; unloading it leaves every call site pointing at unmapped memory. That
|
||
is a constraint on the agent too.
|
||
- **The registry never moves.** A module holds a cell's address for as long as
|
||
it is loaded, so the table is fixed capacity with a loud failure rather than
|
||
growable.
|
||
|
||
The test that separates this from a plausible wrong version is **v4**, which
|
||
redefines `added` — a name v3 introduced at run time. v3's `bump` is already
|
||
installed and is not rebuilt, so it picks v4 up only if its call goes through a
|
||
*cell* both modules found by the same name. Had v3 cached the function's
|
||
address instead, every other assertion would still pass and the transcript
|
||
would read 246 instead of 432.
|
||
|
||
Sizes are spelled LLVM's way — `ptrtoint (ptr getelementptr (T, ptr null, i32
|
||
1) to i64)` — rather than by a layout calculator in OCaml that would have to
|
||
agree with LLVM's on every target.
|
||
|
||
### The agent — dev loop step 3
|
||
|
||
`vendor/agent/` is a package like any other: `agent.flan` declares three calls,
|
||
`flan_agent.c` implements them, `link` asks for `-lpthread`.
|
||
|
||
```
|
||
(agent/start path) listen on a unix socket; once, at startup
|
||
(agent/poll) install whatever has arrived; returns how many
|
||
(agent/wait ms) the same, but waits for something first
|
||
```
|
||
|
||
The split between them is the design. `dlopen` relocates a module and takes the
|
||
loader lock — milliseconds, unbounded — so it happens on the listener thread.
|
||
`flan_reload_install` is one store per function and must not land while a
|
||
redefined function is on the stack, so it happens on the game thread, at the
|
||
top of the frame, when the program asks. The two are connected by a
|
||
single-producer/single-consumer ring and two atomics; the game thread never
|
||
blocks on the loader.
|
||
|
||
`wait` exists for tests. A test that races the frame rate fails on a loaded
|
||
machine, so `test/programs/agent.flan` waits for the reload instead of sleeping
|
||
past it. It takes **two** reloads, which is the daemon's actual loop: the first
|
||
introduces a global the process was never built with, the second only reads it,
|
||
and the second can only answer 1007 if it found the storage the first one
|
||
allocated rather than a fresh zeroed copy. One reload would not have shown
|
||
that.
|
||
|
||
Two details found by running it:
|
||
|
||
- **stdout is line buffered**, set in `flan_rt_init`. The C default when
|
||
stdout is a file or a pipe is a 4K block, so a program running with a REPL
|
||
attached shows nothing until it exits — and a test driving one cannot see
|
||
its progress at all, which is how this was found.
|
||
- **The reply goes out before the module is queued.** The other way round, the
|
||
game thread can install and the program can exit between the two, and the
|
||
answer reaches the sender as a connection reset rather than as `ok`.
|
||
- **`ok` means queued, not installed.** The sender does not get to know when
|
||
the swap happened; only the program knows when it is between frames.
|
||
|
||
**sand.flan calls `agent/poll` at the top of its loop**, which is what step 3
|
||
was for. Verified: with sand running under Xvfb, `flan reload sand-probe.flan
|
||
game-draw` and one line on the socket, and 455 consecutive frames drew from a
|
||
body that did not exist when the process started. Building without `--dev` is
|
||
fine — there are no cells, so a module is refused on the listener thread and
|
||
the loop never notices.
|
||
|
||
`flan reload <file.flan> <fn>... [-o out.so] [--new name,...]` builds one
|
||
module the way the daemon will. `--new` is the names the host was *not* built
|
||
with; it is the one thing the command cannot work out for itself, and it is
|
||
exactly what the session will track automatically.
|
||
|
||
### The session
|
||
|
||
`lib/session.ml` is the program as a live thing: the declarations the running
|
||
process was built from, plus every change accepted since.
|
||
|
||
**Transactionality came for free and needed no machinery.** `Check.program`
|
||
builds a fresh environment from a declaration list on every call, so a form
|
||
that fails to check mutates nothing — the accumulated list is simply not
|
||
replaced. Re-checking the whole program each evaluation costs the entire
|
||
frontend, under 10ms, less than the `llc` that follows. There is a test for the
|
||
case that actually matters: a typo, then a good form, in the same session.
|
||
|
||
Two things the session knows that no single evaluation could:
|
||
|
||
- **Which names the running process was built with.** It comes from the
|
||
*checked* program, not from any accumulated AST, because `Check.program`
|
||
prepends the prelude and no AST contains it. Derive it from declarations and
|
||
`print-line` reads as new, gets a registry cell nobody publishes, and the
|
||
first call jumps to null with no diagnostic.
|
||
- **What that process's memory looks like.** Three changes are refused with a
|
||
reason rather than loaded:
|
||
|
||
| Change | What it would have broken |
|
||
|---|---|
|
||
| a function's signature | a cell is a bare `ptr`; every call site compiled before the change still passes the old arguments through it |
|
||
| a global's type | the storage exists and has a shape — reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve |
|
||
| a struct's fields | the values the process is holding have the old layout |
|
||
| a `defconst`'s value, **when the checker consumed it** | it is in the *shape* of the program — `(defconst rows (/ h c))` decides `grid`'s type before anything else resolves — so no store can reach it |
|
||
| a `defenum` member | `:space` is erased to an `i32` literal in the caller, so it is folded there too |
|
||
|
||
A `defvar`'s *initial value* is deliberately **not** in that table. Its
|
||
storage holds live state the program moved past long ago, and refusing to
|
||
change the initialiser would be refusing "edit the code, keep the sand". Same
|
||
`Tast.global` record as a `defconst`, opposite answers, told apart by
|
||
`gconst`. The enum comparison runs over declarations rather than the checked
|
||
program, because `Tast.program` carries no enums at all — they are erased to
|
||
`i32` in the checker, which is the same fact that makes them unreloadable.
|
||
|
||
Note what the checker catches on its own: change `helper`'s parameter type
|
||
and the *caller* fails to type check first, loudly. The session's rules only
|
||
get a turn on a change the checker accepts — one to a name nothing else in
|
||
the program uses, which is exactly where the silent version lives. The
|
||
fixtures carry an unused `defvar` and a C-called `defn` for that reason.
|
||
|
||
A `defconst` the checker never consumed is a different matter and **can** be
|
||
changed: it is only ever bytes in memory. A dev build emits every `defconst` as
|
||
a mutable `global` rather than a `constant` — so LLVM cannot fold a read of it
|
||
and a module can store into it — and a changed one is published at the frame
|
||
boundary exactly as a new function body is. That is how sand's `colors` gets
|
||
tuned live while `rows` stays refused. Release builds emit `constant` and get
|
||
all the folding back; `Tast.global.gfolded` is what tells the two apart, because
|
||
nothing downstream of the checker could.
|
||
|
||
**A form typed into a file that is imported as a package is qualified the way
|
||
the import qualified it.** `settle` in `sand-sim/sim.flan` becomes `sim/settle`,
|
||
and its call to `move-grain` becomes `sim/move-grain` — through `Load`'s own
|
||
`qualify_decl`, so the rule cannot drift from the one used at import time.
|
||
Without this the form spliced as a brand-new unrelated name: the evaluation
|
||
answered `ok`, and the running program went on calling the `sim/settle` it
|
||
already had. Since sand's simulation lives in a package, the one thing worth
|
||
tuning live was the one thing that silently did nothing.
|
||
|
||
It is derived from the file's path and **not sent by the editor**, which is
|
||
where this departs from CIDER's `ns` key: a Clojure namespace is declared in
|
||
the file, but a Flan alias is chosen by whatever imported the directory and is
|
||
written nowhere the editor can see. One directory imported under two aliases is
|
||
refused with the reason rather than resolved to either.
|
||
|
||
The accumulated list is the **post-`Load`** one, so an evaluated `(import …)`
|
||
is spliced as its expansion. Otherwise re-evaluating a file that imports
|
||
something appends a second import, `Load` expands it again, and the
|
||
duplicate-name pass rejects it. `C-c C-k` on sand.flan's own text is the test.
|
||
|
||
`flan reload <program.flan> <forms.flan>` is that path from the command line: a
|
||
session over the program the process was built from, and a file of the forms
|
||
that changed. Verified against a running sand under Xvfb — a one-form
|
||
`game-draw` and 910 consecutive frames drew it.
|
||
|
||
Two limits of that command specifically, neither of them true of sessions:
|
||
it builds a fresh session from source on every invocation, so if the program
|
||
file has been edited since the process launched, its idea of which names the
|
||
host has and what its memory looks like describes a binary that is not running.
|
||
And `Session.eval`'s `origin` defaults to `<eval>`, so an error in forms sent
|
||
without one reports positions in a file that does not exist — the daemon has to
|
||
pass the real buffer path, which is the same key CIDER's `eval` carries.
|
||
|
||
### The daemon — `flan dev`
|
||
|
||
`flan dev <program.flan>` holds one `Session`, builds the program, launches it,
|
||
and listens on `.flan-dev.sock` beside the source. What it adds over `flan
|
||
reload` is that the session *persists* — a `defvar` added by one evaluation is
|
||
part of what the next one is checked against — and that it **owns the build**,
|
||
which is what makes its layout rules describe the process that is actually
|
||
running rather than a guess about it.
|
||
|
||
**The protocol is s-expressions, not bencode.** nREPL was the plan and the
|
||
argument for it evaporated once the client became ours too: there is no CIDER
|
||
to be compatible with, `eval` is string-in/string-out with no slot for *which
|
||
form, from which file*, and Emacs already has `read` and `prin1`. So it is one
|
||
sexp per message — no parsing code on the editor side, and on this side the
|
||
parser is the language's own reader, where `:op` is already a keyword and a
|
||
payload of Flan source is already a string literal. Framing is a decimal byte
|
||
count and a newline, because the payload contains newlines. An nREPL front end
|
||
can sit on the same `Session` later; it should not have gated the editor.
|
||
|
||
```
|
||
(:op "describe") → (:status "ok" :fns (…) :globals (…) :alive t)
|
||
(:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0)
|
||
→ (:status "error" :message "…" :loc "/buf.flan:1:19")
|
||
(:op "close")
|
||
```
|
||
|
||
An evaluation that declares nothing to install — a declaration the program
|
||
already has, with no body and no new storage — is accepted and answered with
|
||
`:note "nothing to install"` rather than by shipping an empty module. Building
|
||
one anyway reports success for a change that cannot have taken effect, and
|
||
costs the program a reload it did not need.
|
||
|
||
`:file` is not decoration: `Session.eval`'s origin defaults to `<eval>`, so
|
||
without it every error an editor shows points into a file that does not exist.
|
||
|
||
Two things the daemon must not paper over, both of which would look like a
|
||
successful evaluation:
|
||
|
||
- **The agent socket is chosen by the daemon**, not by the program. A program's
|
||
source has to name some path — sand.flan says `/tmp/flan-sand.sock` — and the
|
||
daemon overrides it through `FLAN_AGENT_SOCKET` before spawning. Guessing
|
||
instead fails silently: the module compiles, is built, and nobody receives
|
||
it.
|
||
- **Delivery is checked.** `agent/start` returning 0 means a socket was bound,
|
||
not that anyone connected. A failed connect or a reply that is not `ok`
|
||
becomes an error the editor sees.
|
||
|
||
It waits for the program to bind before accepting an evaluation — one arriving
|
||
first would fail for a reason that reads like a compiler bug — and it accepts
|
||
with a timeout so that a program which has exited takes the daemon with it
|
||
rather than leaving an editor waiting on a socket nobody is serving.
|
||
|
||
### The Emacs client
|
||
|
||
`emacs/flan-mode.el` derives from `prog-mode` with `lisp-mode`'s syntax table,
|
||
which is most of the work: Flan is s-expressions, so sexp motion, paren
|
||
matching, `beginning-of-defun` and indentation are already right. What it adds
|
||
is Flan's own bracket syntax (`[` and `{` are brackets, not symbol characters —
|
||
every binding list and every type is written with them), the characters a Flan
|
||
name may contain (`-`, `?`, `/`, `.`), and its keywords.
|
||
|
||
`emacs/flan-dev.el` is the client. There is no parser in it, which is the point
|
||
of the protocol choice: `prin1` writes a request and `read` reads a reply.
|
||
|
||
| | |
|
||
|---|---|
|
||
| `C-c C-c` | the top-level form at point, recompiled and installed |
|
||
| `C-c C-k` | the whole buffer, as **one** module |
|
||
| `C-x C-e` | the expression before point, evaluated *in the running program* |
|
||
| `C-c C-z` / `C-c C-q` | connect (finds `.flan-dev.sock` upward) / disconnect |
|
||
| `C-c C-o` | the running program's own output, in `*flan-output*` |
|
||
| `C-c C-d` | what the running program currently defines |
|
||
|
||
`C-c C-k` sends one module rather than a form at a time on purpose: a `defvar`
|
||
and the function that uses it have to arrive in the same load, or the first
|
||
refers to storage that does not exist yet.
|
||
|
||
**Framing is in bytes and Emacs counts characters.** Every length goes through
|
||
`string-bytes` and the process is binary, or a single non-ASCII character in a
|
||
buffer puts the reply stream out of step by exactly as many bytes as the
|
||
payload has of them — a bug that would look like a corrupt protocol and appear
|
||
only for some users. `test/test_emacs.ml` drives the real client against a real
|
||
daemon for this reason: it is not the same claim as the daemon answering
|
||
correctly, and a mistake in the framing, in `beginning-of-defun` over Flan's
|
||
syntax table, or in the reply reader passes `test_dev.ml` and fails here.
|
||
|
||
An error comes back with a location and the client moves point to it when it is
|
||
this buffer.
|
||
|
||
**The program's stdout is a pipe into the daemon**, and whatever it printed
|
||
since the last reply rides along with the next one into `*flan-output*`. Having
|
||
it arrive *with* a reply rather than by a separate request is the point: the
|
||
output an evaluation itself caused is the output anyone wants to see. Draining
|
||
that pipe is a liveness requirement and not a nicety — a pipe nobody reads
|
||
fills at 64K and the next write blocks the program forever — so it is read from
|
||
the accept loop's `select`, not only when an editor asks, and the buffer is
|
||
capped so a program printing every frame cannot grow the daemon without
|
||
limit.
|
||
|
||
### `C-x C-e` — evaluating an expression
|
||
|
||
A different primitive from redefining a name, and the difference is the whole
|
||
design. There is no name to install a body into, so the expression is wrapped
|
||
in a function with nowhere to be called from; the module says *run this once*
|
||
by exporting `flan_reload_call`, and the agent calls it after the install — on
|
||
the game thread, at a frame boundary, so an expression reading the program's
|
||
state sees a point the program agrees is consistent.
|
||
|
||
**Nothing is marshalled back, because nothing could be.** A Flan value carries
|
||
no header, so no code at run time can say what it is. The compiler knows the
|
||
type and renders it *there*, in the thunk, into `flan_dev_result`. That is the
|
||
layout decision's bill, and it is why the printer set is small rather than
|
||
universal.
|
||
|
||
It does not go through stdout. Stdout belongs to the program, it is in the hot
|
||
path for anything that prints, and a dev-only feature must not put a branch in
|
||
it — so `flan_rt.c` is untouched and the value is read back over the agent's
|
||
socket. The read is safe without a handshake because `flan_dev_result` bumps a
|
||
generation counter last; the daemon waits for it to move rather than assuming
|
||
the program has reached a frame boundary.
|
||
|
||
The renderer is a **compile-time walk over the type**, emitting a piece at a
|
||
time through `flan_dev_emit`. Piecewise because a struct is its fields with
|
||
punctuation between them, and concatenating that in generated IR would need an
|
||
allocator the language does not have.
|
||
|
||
```
|
||
big 18446744073709551615
|
||
col :blue
|
||
(.pos b) (V {:x 1.5 :y 0})
|
||
b (Blob {:id 7 :name "sandy \"quoted\"" :pos (V {:x 1.5 :y 0}) :tags [ 0 42 0]})
|
||
(slice (.tags b) 0 3) [ 0 42 0]
|
||
(rl/get-color 0x11223344) (rl/Color {:r 17 :g 34 :b 51 :a 68})
|
||
sim/grid [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...]
|
||
```
|
||
|
||
Details that are decisions rather than formatting:
|
||
|
||
- **`u64` renders in C**, with `%llu`. The language's own `i64->bytes` is
|
||
signed, so it used to refuse rather than come back as `-1` — but refusing a
|
||
whole struct because one field is a `u64` is much worse, so the runtime got
|
||
an entry point instead.
|
||
- **Strings are quoted and escaped**, also in C. Unescaped content does not
|
||
round-trip and reads as a framing bug rather than as the value it is.
|
||
- **An enum renders as `:name`**, recovered from the checker's table as a chain
|
||
of comparisons, because members are erased to `i32` before the backend sees
|
||
them. A value outside the declared members falls through to its number, which
|
||
is exactly what you would want to see.
|
||
- **A pointer is never followed** — `<ptr>`. It is the only thing that could
|
||
make the walk cycle, and dereferencing one a REPL was handed is not a safe
|
||
thing to do on someone's behalf.
|
||
- **Three separate bounds**, easy to conflate. `depth` (4) and `span` (8) bound
|
||
the *walk*, so `[100 [100 u32]]` does not become ten thousand render sites in
|
||
one module. The *output* is bounded once in the runtime — `emit` truncates at
|
||
4K and `end` appends `...` — because a slice renders through a loop the
|
||
compiler cannot bound, and one place enforcing it means no renderer carries a
|
||
budget.
|
||
- A slice is the one case needing a runtime loop, and the slice goes into a
|
||
slot first so the expression it came from is not evaluated once per element.
|
||
|
||
What still refuses by name: `Map`, `Fn`, a type variable.
|
||
|
||
A caveat inherited from the language, not introduced here: `3.0` renders as
|
||
`3`, indistinguishable from the integer. `flan run calc-me.flan "1.5 * 2.0"`
|
||
has always said `3`.
|
||
|
||
An evaluation is **not** a declaration: the thunk is built against the program
|
||
and never spliced into it, so `describe` does not fill up with `eval/N` for
|
||
every expression ever typed.
|
||
|
||
**The module is unloaded afterwards**, which is the one case where that is
|
||
safe. The thunk is called directly by `flan_reload_call` rather than through a
|
||
cell, and it takes no registry slot — so once it has returned, nothing points
|
||
into its text and the value it produced has been copied out. It declares that
|
||
with `@flan_reload_transient` and the agent `dlclose`s it. Measured: sixteen
|
||
expression evaluations retain **zero** mappings, where each *redefinition*
|
||
retains three, permanently and correctly — a module that publishes a body
|
||
exists precisely to leave a pointer behind, and can never claim this.
|
||
|
||
Skipping the registry matters for more than tidiness: the table holds 4096
|
||
names and an expression evaluated in a loop would exhaust it.
|
||
|
||
The test that matters is the same expression twice: the fixture increments
|
||
`ticks` every frame, so two evaluations must disagree. A value computed in the
|
||
compiler, or read out of a copy of the program's state, would not.
|
||
|
||
### What is left
|
||
- **A REPL buffer.** `C-x C-e` echoes into the minibuffer; there is no prompt
|
||
to type at and no history.
|
||
- **Printers beyond the scalars.** See below — a struct, an `(Option T)` or a
|
||
slice of structs still refuses by name.
|
||
|
||
**Session identity is the daemon that owns the build.** A session's struct
|
||
layouts and global types have to describe the memory of the process it is
|
||
talking to, which is only guaranteed if it is the session that compiled the
|
||
running binary. Attaching to a process someone else built is not a thing to
|
||
support by default.
|
||
|
||
## Where build time goes
|
||
|
||
`flan build calc-me.flan` was ~160ms, and ~95% of it was clang. **The object
|
||
cache is in**, and it is now ~110ms:
|
||
|
||
| Step | Cost |
|
||
|---|---|
|
||
| frontend: read → parse → load → check → emit | <10ms, below the timer |
|
||
| `clang` on the `.ll` | 60ms — `llc` does the same codegen in **20ms** |
|
||
| `clang` on `flan_rt.c` | 40ms — **now cached, paid once** |
|
||
| link | 20ms |
|
||
|
||
Every C translation unit a build needs — the host shim and each package's shim
|
||
— goes through `Build.compile_c`, which compiles to a `.o` under
|
||
`$TMPDIR/flan-objcache` and reuses it. The key is a digest of the source text,
|
||
the compiler (its path, size and mtime, so an upgrade invalidates without
|
||
paying a `clang --version` subprocess per build), `opts.opt` and `opts.target`.
|
||
The opt level has to be in there: the acceptance table builds the same programs
|
||
at `-O0` and `-O2`, and an `-O2` object must not serve an `-O0` build. The
|
||
object is written to a temporary name and `rename`d into place, so two
|
||
concurrent builds cannot see a half-written one.
|
||
|
||
Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is
|
||
mostly linking libraylib and its `shim.c` was never the cost. The cache is
|
||
keyed by content, so it never needs invalidating by hand — `rm -rf` on the
|
||
directory is only ever a disk-space decision.
|
||
|
||
The other cheap win is still open: skip the clang driver for the `.ll` (`llc` +
|
||
link directly), worth another ~40ms. It is a subset of the dev path's
|
||
machinery. Check `llc`'s major version against clang's before relying on it —
|
||
the emitted IR text is currently absorbed by the driver behind
|
||
`-Wno-override-module`, and a version mismatch surfaces as IR parse errors.
|
||
|
||
**There is still no REPL.** Nothing does redefinition, `dlopen`, or nREPL.
|
||
`build` is the only way to run code.
|
||
|
||
## Next — the REPL is the priority
|
||
|
||
Decided in conversation: wasm32 can wait (it is believed to be a solved problem
|
||
once the builtins archive is in place), and **the dev loop is the thesis of the
|
||
project**, so it comes first. Staged so each step is runnable on its own —
|
||
the failure mode is building a daemon and a protocol before knowing the reload
|
||
primitive works.
|
||
|
||
1. ~~**The reload primitive, measured.**~~ **Done** — `Emit.redefinition`,
|
||
`Build.shared`, `test/reload_host.c`, ~19ms. See the section above.
|
||
2. ~~**Indirection cells.**~~ **Done** — `Build.opts.dev` / `flan build --dev`,
|
||
`flan_reload_install`, `runtime/flan_dev.c` for names introduced at run
|
||
time, and a fixture where an untouched call site follows the swap and a
|
||
run-time-added function is itself redefined. See the section above.
|
||
3. ~~**The agent, in C.**~~ **Done** — `vendor/agent/`, a listener thread that
|
||
loads and a game thread that installs, and sand.flan polling at the top of
|
||
its frame. See the section above.
|
||
4. ~~**The daemon**~~ and ~~**5. the Emacs client**~~ — **both done**, and the
|
||
protocol is s-expressions rather than nREPL's bencode; see the two sections
|
||
above for why that changed. An nREPL front end can sit on the same
|
||
`Session` if something else ever needs to talk to it.
|
||
|
||
**One decision left to settle before step 2**, because both change codegen and are
|
||
painful to retrofit:
|
||
|
||
- ~~**Do cells cover globals, or only functions?**~~ **Settled by step 1:
|
||
functions only.** A redefinition module declares every global `external`, so
|
||
globals live in the host and survive a reload — which is what "edit the code,
|
||
keep the sand" needs. The consequence to watch is the other half: adding a
|
||
`defvar` to a file cannot take effect on reload, and changing one's type is a
|
||
silent mismatch against storage the host already laid out. Nothing detects
|
||
that yet.
|
||
- **What is a redefinition unit — one function, or a file?** A file is much
|
||
easier to make correct and is what `load-file` wants anyway; one function is
|
||
what `C-c C-c` wants and is where the 16ms number comes from.
|
||
|
||
Deferred until after the dev loop:
|
||
|
||
6. **wasm32.** The user installed `wasi-libc-devel` and `wasi-libc-static`; the
|
||
sysroot is `/usr/wasm32-wasi` and `wasm-ld` is present. `clang
|
||
--target=wasm32-wasi --sysroot=/usr/wasm32-wasi` gets past the headers and
|
||
then **fails to link**: it wants
|
||
`lib/clang/20/lib/wasm32-unknown-wasi/libclang_rt.builtins.a`, which no
|
||
Fedora package provides (`dnf provides '*libclang_rt.builtins*wasm*'` finds
|
||
nothing). It has to come from a wasi-sdk release, dropped into clang's
|
||
resource directory. After that: teach `build.ml` `--sysroot`, and run the
|
||
acceptance table — `sand-headless.flan` included, which is exactly why it
|
||
does not import raylib — on both targets in CI.
|
||
Note plan.org has the *web* build linking raylib via emscripten, which
|
||
brings its own sysroot: wasi-sdk is right for the headless table, not
|
||
necessarily for the eventual game build.
|
||
7. **Loose ends from milestone 4**, none of them blocking: block-scoped
|
||
`defer`; package visibility, so `rl/get-color-raw` is not callable; a
|
||
package importing a package; imported unions.
|
||
|
||
## Watch for
|
||
|
||
The rule that caught the two misparse bugs applies unchanged: **anything that
|
||
binds a name, alters control flow, or is not yet implemented must be recognised
|
||
explicitly and rejected if unsupported.** `check.ml` rejects `Vec`, `Map`,
|
||
`Result`/`try`, union values, closures, quoted symbols, generics and function
|
||
values *by name*, each with the milestone it belongs to; `load.ml` rejects the
|
||
package shapes it does not handle; and the FFI boundary rejects an aggregate.
|
||
The tests assert on the reason, not just on the failure.
|
||
|
||
## Untracked on purpose
|
||
|
||
`calc-me` and `sand`, the executables `flan build` drops beside their sources,
|
||
are now in `.gitignore` — anchored (`/calc-me`, `/sand`) so the patterns cannot
|
||
also match `sand-sim/` or anything nested.
|
||
|
||
`old-ocaml/` — the pre-rewrite menhir/ocamllex frontend, kept as reference and
|
||
excluded from the build by the root `dune` file. Its contents are also in git
|
||
history at `2c232dd`.
|