308 lines
16 KiB
Markdown
308 lines
16 KiB
Markdown
# Where this is
|
||
|
||
Milestone 4 is done: **sand.flan builds, links raylib and runs**, and its
|
||
simulation has a headless acceptance case that runs on the `dune test` path at
|
||
`-O0` and `-O2`. Milestones 2 and 3 are behind it (`calc-me.flan` compiles and
|
||
runs; the interpreter was dropped — open decision #7, settled, see below).
|
||
|
||
```
|
||
reader ✅ → parse ✅ → load ✅ → check ✅ → emit ✅ → clang ✅
|
||
```
|
||
|
||
| File | What it does |
|
||
|---|---|
|
||
| `lib/loc.ml` | source locations + `Loc.Error`, the frontend's one exception |
|
||
| `lib/form.ml` | reader output: `Sym Kw Int Float Str Byte List Vec Map` |
|
||
| `lib/reader.ml` | hand-written S-expression reader, no menhir/ocamllex |
|
||
| `lib/ast.ml` | AST: `texpr`, `expr`, `place`, `pattern`, `decl` |
|
||
| `lib/parse.ml` | forms → AST; special forms, desugaring, declarations |
|
||
| `lib/load.ml` | **imports: a package directory → qualified declarations** |
|
||
| `lib/types.ml` | resolved types; structural equality, `Never` fits anywhere |
|
||
| `lib/tast.ml` | the typed IR the backend consumes |
|
||
| `lib/check.ml` | AST → typed IR; two passes, bidirectional |
|
||
| `lib/prelude.ml` | printers + `rand-f32`, written in Flan |
|
||
| `lib/emit.ml` | typed IR → LLVM IR text |
|
||
| `lib/build.ml` | `.ll` + the shim + the packages' C → clang → executable |
|
||
| `runtime/flan_rt.c` | the host ABI: argv, stdout, exit, 4 conversions |
|
||
| `vendor/raylib/` | **the raylib package: `raylib.flan`, `shim.c`, `link`** |
|
||
| `sand-sim/` | **the falling-sand simulation, with no raylib in it** |
|
||
| `bin/main.ml` | `flan read \| parse \| check \| emit \| build \| run` |
|
||
| `test/test_flan.ml` | reader, parser and checker |
|
||
| `test/test_acceptance.ml` | expression/result pairs + whole programs + the traps |
|
||
|
||
```
|
||
$ flan run calc-me.flan "1 + 2 * (3 - 0.5) / 2"
|
||
3.5
|
||
$ flan run test/programs/sand-headless.flan
|
||
2256461126764447066
|
||
$ flan run sand.flan # a window, 120 fps, hold space
|
||
```
|
||
|
||
## What milestone 4 added
|
||
|
||
**`dotimes`** desugars in `check.ml` to a `Let` plus a `While` — no new IR node.
|
||
The bound is evaluated once into a hidden slot before the loop, so a body that
|
||
changes it cannot change the trip count, and the loop variable is not
|
||
assignable, which makes the generated step its only writer.
|
||
|
||
**`defer`** is recognised in `check_fn` and nowhere else, because that is the
|
||
only place that knows a form is at the top level of a function body. Each one
|
||
is checked in place, then registered on the context; it emits nothing where it
|
||
stands. Function exit runs them innermost-first, and an explicit `return` runs
|
||
the ones registered *above* it — a defer written below a return has not
|
||
executed yet and must not fire. A trap runs none of them, which follows from
|
||
the bounds-check shape (`noreturn` then `unreachable`) rather than being a
|
||
separate decision.
|
||
|
||
`defer` inside a `let`, a loop or a branch is **rejected**, not accepted with
|
||
function scope. It would run once at function exit rather than once per
|
||
iteration, and that is the silent-wrongness class the rule below is about.
|
||
Block-scoped defer is real work and is not done.
|
||
|
||
**New builtins:** `zeroed` (takes its type from the place it is stored into),
|
||
`min`/`max` (each operand through a slot, so neither is evaluated twice),
|
||
`bit-and`/`bit-or`/`bit-xor`/`<<`/`>>` (integers only; `>>` is arithmetic on a
|
||
signed type and logical on an unsigned one), and `rand-f32`.
|
||
|
||
**`rand-f32` is in the prelude, in Flan** — PCG-XSH-RR 32 over a `u64` state.
|
||
It is not libc's, because a grid hash is only a regression test if the sequence
|
||
is byte-identical on native and wasm32 (plan.org, RNG is ours). `rand-seed`
|
||
sets the state. This is what the bitwise operators were added for.
|
||
|
||
**Enums and keywords.** `(defenum Name [member value ...])` gives a type that
|
||
is an `i32` at run time and its own type in the checker, so `:space` at a call
|
||
site resolves against the parameter's enum and a typo is an error there rather
|
||
than a wrong number later. A keyword means nothing where no enum is expected —
|
||
there is no keyword type to fall back on.
|
||
|
||
## Why the FFI goes through a C shim
|
||
|
||
The decision that shapes the whole raylib package. What clang generates for
|
||
raylib's own prototypes on x86-64:
|
||
|
||
```
|
||
Vector2 {float,float} → declare <2 x float> @GetMousePosition()
|
||
Color {u8,u8,u8,u8} → declare void @ClearBackground(i32)
|
||
Rectangle {4 × int} → declare { i64, i64 } @mkrect()
|
||
```
|
||
|
||
None of those is the struct's own LLVM type. A small aggregate's calling
|
||
convention is not part of its layout — it is a per-target classification the
|
||
*caller* has to reproduce, and x86-64, arm64 and wasm32 classify differently.
|
||
Putting that in `emit.ml` is three classifiers to write and then keep correct
|
||
forever, and a mistake shows up as `(.y m)` returning garbage rather than as a
|
||
link error.
|
||
|
||
So `vendor/raylib/shim.c` has one wrapper per binding, each one flattening the
|
||
aggregates: a struct returns through an out-pointer, a struct argument is
|
||
passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates
|
||
a copy. clang classifies all of it, per target, for free. `check.ml` enforces
|
||
the rule — an aggregate in a `declare` signature is rejected with the reason —
|
||
so the boundary cannot quietly acquire one. This is plan.org's "one narrow host
|
||
ABI, implemented twice", and `flan_rt.c` is the same pattern.
|
||
|
||
The price is a hand-written wrapper per raylib call. They are one-liners and
|
||
mechanical enough to generate if that ever becomes the bottleneck.
|
||
|
||
`raylib.flan` declares each `-raw` entry point and wraps it in an ordinary Flan
|
||
function just below, so the surface sand.flan sees is `(rl/get-mouse-position)`
|
||
returning a `Vector2`. Verified end to end, headless: `GetColor(0x11223344)`
|
||
comes back as `17 34 51 68`, four separate bytes — a `Color` is *not* the
|
||
little-endian reading of the packed integer, so an identity would have passed a
|
||
weaker test. That case is in the acceptance table, skipped if `libraylib` is
|
||
not installed.
|
||
|
||
No raylib headers are needed: `shim.c` declares the prototypes it uses, so the
|
||
build depends on the shared library being linkable and not on `raylib-devel`.
|
||
`vendor/raylib/link` carries `-l:libraylib.so.550` because Fedora ships the
|
||
runtime library without the `.so` symlink.
|
||
|
||
## Packages
|
||
|
||
`lib/load.ml` resolves `(import rl "vendor:raylib")` before the checker runs.
|
||
The directory is the package; `vendor:` is a collection, resolved by walking up
|
||
from the importing file until a directory of that name is found; a path with no
|
||
collection is relative to the importing file. Importing is a **rename**: every
|
||
top-level name the package declares becomes `alias/name`, and every use of one
|
||
— in a type, in a body, in a struct literal, in an *array length* — is
|
||
rewritten to match. Local bindings shadow. Nothing downstream knows a package
|
||
existed; the checker sees one flat list of declarations whose names contain a
|
||
slash.
|
||
|
||
A package may also carry the C it binds to: every `.c` file in the directory is
|
||
compiled into the build, and a file named `link` lists extra linker arguments.
|
||
|
||
This is not a module system yet. No visibility (hence `rl/get-color-raw` being
|
||
callable), no cycle detection, and a package cannot import another one.
|
||
|
||
## sand.flan is two programs
|
||
|
||
plan.org wants sand tested twice — interactive at 120 fps, and headless over N
|
||
frames with the grid hashed, the version CI runs on native *and* wasm32. Those
|
||
cannot be one binary: `Load` collects a package's C sources and linker
|
||
arguments unconditionally, so anything importing the raylib package links
|
||
libraylib on every target regardless of what its `main` does, and on wasm32
|
||
that link cannot succeed.
|
||
|
||
So the simulation moved to `sand-sim/`, which imports nothing. `sand.flan`
|
||
imports it as `sim/` and adds the window, the mouse and the drawing;
|
||
`test/programs/sand-headless.flan` imports it and adds a seed, four
|
||
deterministic clouds, 40 frames and an FNV-1a hash. One copy of the physics.
|
||
|
||
The headless case is what actually *verifies* milestone 4 — running the
|
||
interactive build only proves it enters its loop, because with no mouse input
|
||
the grid stays empty and `paint-at`, `settle` and `move-grain` never execute on
|
||
real data. Measured through the probe: 168 grains painted around row 4–8, still
|
||
168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.
|
||
|
||
**Three edits were made to sand.flan's own text**, and they are language
|
||
decisions rather than fixes:
|
||
|
||
- `(defconst gravity 0.05)` → `(defconst gravity f32 0.05)`. An untyped float
|
||
constant is `f64`, `velocity` is `[f32]`, and there is no implicit widening.
|
||
- `(defvar current-color u32)` → `i32`. It is an index into `colors`, and
|
||
`(len colors)` is an `i32`.
|
||
- The file was split as above, so its body now says `sim/rows` and so on.
|
||
|
||
`(defn main [])` is unchanged — the short form, as plan.org says.
|
||
|
||
## Bounds checks — done at milestone 3
|
||
|
||
`at` and `slice` emit `icmp` → `br` → cold block → `call` → `unreachable`; a
|
||
failure names the source location. Three check sites: `at` on `[n T]` (static
|
||
bound, folded by LLVM for a literal index — and a literal that is out of bounds
|
||
never reaches emit, `check.ml` rejects it), `at` on a slice or string (runtime
|
||
len), and `slice` (two comparisons — `lo <= hi` is not redundant, without it a
|
||
reversed range yields a huge unsigned length). All comparisons unsigned.
|
||
`Build.opts.checks` is on by default and **not** tied to `opts.opt`, which is
|
||
what lets the acceptance table run the same programs at `-O0` and `-O2` with
|
||
identical checks. The flag is `--no-bounds-checks`.
|
||
|
||
The write path is its own case: `(set (at arr n) …)` lowers through
|
||
`place`/`Pindex`, not through `At`, so a refactor that split them would break
|
||
the write check silently. The test covers both.
|
||
|
||
Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs
|
||
at 0.11–0.12s checked against 0.12–0.13s unchecked. Indistinguishable.
|
||
|
||
## Why there is no interpreter
|
||
|
||
Open decision #7 is settled: **the compiled path is the only backend.** Both
|
||
arguments for a permanent interpreter had expired — the instrumentation step
|
||
debugger that wanted it is cut, and compiled redefinition measured at ~16ms,
|
||
perceptually instant for expression eval too. Milestone 3 did not need an
|
||
oracle either: the acceptance table is hand-written, so the table *is* the
|
||
oracle. Consequences already applied: milestone 2's "interpreted calls per
|
||
second" criterion is dropped, and the host ABI moved onto the critical path.
|
||
|
||
## The layout, which is the whole backend design
|
||
|
||
```
|
||
i8..i64 / u8..u64 i8..i64 signedness lives in the ops
|
||
f32 f64 float double
|
||
bool i1
|
||
an enum i32
|
||
[T] and string { ptr, i64 } ptr+len, non-owning
|
||
[n T] [n x T] inline, a value
|
||
(Ptr T) ptr opaque pointers
|
||
(Option T) { i8, T } tag 0 None, 1 Some
|
||
a struct a literal struct, declaration order
|
||
Unit and Never {}
|
||
```
|
||
|
||
No object headers anywhere, so a Flan struct is exactly its C struct and
|
||
nothing marshals. Two consequences carry the semantics:
|
||
|
||
- **Every slot is an `alloca`.** Reading a local is a `load`, assigning is a
|
||
`store`, and a `store` of an aggregate *is* the copy `spec-memory.md`
|
||
requires. `addr` of a local is then just the alloca, and `mem2reg` removes
|
||
the ones nobody addressed. `test/programs/values.flan` pins this down.
|
||
- **A place is a pointer, a value is a load from it.** `(set (.pos c) …)`
|
||
through a `(Ptr Cursor)` becomes a `getelementptr` on the pointer, not on a
|
||
copy. This is the split that would have made a tree-walker silently wrong.
|
||
|
||
Non-local exit is lowered explicitly: `return`, `some` and a failed bounds
|
||
check are branches, never platform unwinding, so wasm32 needs no exception
|
||
proposal.
|
||
|
||
## Sharp edges found and left visible
|
||
|
||
- **An index converts from a narrower integer and never from a wider one.**
|
||
`(nth colors current-color)` with a `u32` index works — anything above 2³¹
|
||
truncates to a negative `i32` and the unsigned bounds check rejects it. An
|
||
`i64` index is refused with the reason: 2³²+5 truncates to 5 and would read
|
||
the wrong element with no trap at all.
|
||
- **A `u64` literal is its 64-bit pattern**, so `0xcbf29ce484222325` is a real
|
||
`u64` and not an error. The cost is that a negative *decimal* literal is
|
||
accepted as a `u64` too, because the reader records the value and not how it
|
||
was written. Narrower unsigned types keep the strict check, which is where a
|
||
typo like `300` for a `u8` actually shows up.
|
||
- **A folded constant skips `check`.** `(defconst rows (/ h c))` is emitted
|
||
from the folding pass's value, because a global's initialiser has to be a
|
||
compile-time constant and only that pass knows this one is. Its range check
|
||
is therefore its own call to `in_range`; there is a regression test.
|
||
- A `let` binding takes no type annotation, which is why `sand-sim` names its
|
||
FNV constants instead of writing them inline.
|
||
- `(defn f [] f65 0.0)` still says *unknown name* rather than *did you mean
|
||
f64*: with a single body form the parser cannot tell a return type from the
|
||
first expression. Only the parameter position and `(Option …)` are
|
||
unambiguous.
|
||
|
||
## Where build time goes
|
||
|
||
`flan build calc-me.flan` is ~140ms, and ~95% of it is clang:
|
||
|
||
| Step | Cost |
|
||
|---|---|
|
||
| frontend: read → parse → load → check → emit | <10ms, below the timer |
|
||
| `clang` on the `.ll` | 60ms — `llc` does the same codegen in **20ms** |
|
||
| `clang` on `flan_rt.c` | 40ms — recompiled every build, never changes |
|
||
| link | 20ms |
|
||
|
||
sand.flan additionally recompiles `shim.c` every build. Two cheap wins take the
|
||
base to ~60ms: cache `flan_rt.o` (and the packages' `.o`), and skip the clang
|
||
driver for the `.ll` (`llc` + link directly). Both are a subset of the dev
|
||
path's machinery, so doing them now is not wasted work.
|
||
|
||
**There is still no REPL.** Nothing does redefinition, `dlopen`, or nREPL.
|
||
`build` is the only way to run code.
|
||
|
||
## Next
|
||
|
||
1. **wasm32.** The user installed `wasi-libc-devel` and `wasi-libc-static`; the
|
||
sysroot is `/usr/wasm32-wasi` and `wasm-ld` is present. `clang
|
||
--target=wasm32-wasi --sysroot=/usr/wasm32-wasi` gets past the headers and
|
||
then **fails to link**: it wants
|
||
`lib/clang/20/lib/wasm32-unknown-wasi/libclang_rt.builtins.a`, which no
|
||
Fedora package provides (`dnf provides '*libclang_rt.builtins*wasm*'` finds
|
||
nothing). It has to come from a wasi-sdk release, dropped into clang's
|
||
resource directory. After that: teach `build.ml` `--sysroot`, and run the
|
||
acceptance table — `sand-headless.flan` included, which is exactly why it
|
||
does not import raylib — on both targets in CI.
|
||
Note plan.org has the *web* build linking raylib via emscripten, which
|
||
brings its own sysroot: wasi-sdk is right for the headless table, not
|
||
necessarily for the eventual game build.
|
||
2. **The dev path / REPL.** `llc` + `ld -shared` + `dlopen` ≈ 16ms, a compiler
|
||
daemon plus an in-game reload agent (plan.org, Dev architecture). There is
|
||
now a frame loop for it to not stutter, which was the reason to do it after
|
||
milestone 4.
|
||
3. **Loose ends from milestone 4**, none of them blocking: block-scoped
|
||
`defer`; package visibility, so `rl/get-color-raw` is not callable; a
|
||
package importing a package; imported unions.
|
||
|
||
## Watch for
|
||
|
||
The rule that caught the two misparse bugs applies unchanged: **anything that
|
||
binds a name, alters control flow, or is not yet implemented must be recognised
|
||
explicitly and rejected if unsupported.** `check.ml` rejects `Vec`, `Map`,
|
||
`Result`/`try`, union values, closures, quoted symbols, generics and function
|
||
values *by name*, each with the milestone it belongs to; `load.ml` rejects the
|
||
package shapes it does not handle; and the FFI boundary rejects an aggregate.
|
||
The tests assert on the reason, not just on the failure.
|
||
|
||
## Untracked on purpose
|
||
|
||
`old-ocaml/` — the pre-rewrite menhir/ocamllex frontend, kept as reference and
|
||
excluded from the build by the root `dune` file. Its contents are also in git
|
||
history at `2c232dd`.
|