253 lines
13 KiB
Markdown
253 lines
13 KiB
Markdown
# Where this is
|
||
|
||
Milestones 2 and 3 of `plan.org` were merged: the interpreter was dropped
|
||
(open decision #7, settled — see below) and the compiled path is the only
|
||
backend. **calc-me.flan compiles and runs.**
|
||
|
||
```
|
||
reader ✅ → parse ✅ → check ✅ → emit ✅ → clang ✅
|
||
```
|
||
|
||
| File | What it does |
|
||
|---|---|
|
||
| `lib/loc.ml` | source locations + `Loc.Error`, the frontend's one exception |
|
||
| `lib/form.ml` | reader output: `Sym Kw Int Float Str Byte List Vec Map` |
|
||
| `lib/reader.ml` | hand-written S-expression reader, no menhir/ocamllex |
|
||
| `lib/ast.ml` | AST: `texpr`, `expr`, `place`, `pattern`, `decl` |
|
||
| `lib/parse.ml` | forms → AST; special forms, desugaring, declarations |
|
||
| `lib/types.ml` | resolved types; structural equality, `Never` fits anywhere |
|
||
| `lib/tast.ml` | the typed IR the backend consumes |
|
||
| `lib/check.ml` | AST → typed IR; two passes, bidirectional |
|
||
| `lib/prelude.ml` | `print-str`/`print-f64`/`print-line`, written in Flan |
|
||
| `lib/emit.ml` | typed IR → LLVM IR text |
|
||
| `lib/build.ml` | `.ll` + the shim → clang → executable |
|
||
| `runtime/flan_rt.c` | the whole host ABI: argv, stdout, exit, 4 conversions |
|
||
| `bin/main.ml` | `flan read \| parse \| check \| emit \| build \| run` |
|
||
| `test/test_flan.ml` | reader, parser and checker |
|
||
| `test/test_acceptance.ml` | 20 expression/result pairs + 3 whole programs + the traps |
|
||
| `test/programs/*.flan` | the milestone-2 surface calc-me does not reach |
|
||
|
||
```
|
||
$ flan run calc-me.flan "1 + 2 * (3 - 0.5) / 2"
|
||
3.5
|
||
```
|
||
|
||
`dune build && dune test` is green, and the whole-program cases run at `-O2`
|
||
*and* `-O0` — `mem2reg` launders a sloppy alloca, so -O0 is what tests the IR
|
||
actually emitted. `flan emit` is byte-reproducible. `flan check sand.flan` fails on
|
||
`(import rl ...)`, which is milestone 4 — as it should.
|
||
|
||
## Why there is no interpreter
|
||
|
||
Open decision #7 is settled: **the compiled path is the only backend.** The two
|
||
arguments for a permanent interpreter had both already expired in `plan.org` —
|
||
the instrumentation step debugger that wanted it is cut, and compiled
|
||
redefinition measured at ~16ms, which is perceptually instant for expression
|
||
eval too. CCL and SBCL both do full interactive development without leaning on
|
||
an interpreter; what makes a live image work is a fast compiler callable at
|
||
runtime.
|
||
|
||
The remaining argument was that milestone 3 needs an oracle to check the
|
||
compiler against. It does not: the acceptance test is a hand-written table of
|
||
expression/result pairs, so the table *is* the oracle.
|
||
|
||
Consequences, both already applied: milestone 2's "measured interpreted calls
|
||
per second" exit criterion is dropped — milestone 4 runs on the compiled build
|
||
and nothing depended on that number — and the host ABI moved onto the critical
|
||
path, which is why `runtime/flan_rt.c` exists now rather than at milestone 3.
|
||
|
||
## The layout, which is the whole backend design
|
||
|
||
```
|
||
i8..i64 / u8..u64 i8..i64 signedness lives in the ops
|
||
f32 f64 float double
|
||
bool i1
|
||
[T] and string { ptr, i64 } ptr+len, non-owning
|
||
[n T] [n x T] inline, a value
|
||
(Ptr T) ptr opaque pointers
|
||
(Option T) { i8, T } tag 0 None, 1 Some
|
||
a struct a literal struct, declaration order
|
||
Unit and Never {}
|
||
```
|
||
|
||
No object headers anywhere, so a Flan struct is exactly its C struct and
|
||
nothing marshals. Two consequences carry the semantics:
|
||
|
||
- **Every slot is an `alloca`.** Reading a local is a `load`, assigning is a
|
||
`store`, and a `store` of an aggregate *is* the copy `spec-memory.md`
|
||
requires — value structs and fixed arrays copy, a slice copies only its view.
|
||
`addr` of a local is then just the alloca, and `mem2reg` removes the ones
|
||
nobody addressed. `test/programs/values.flan` pins this down: mutate the
|
||
original, the copy is unchanged.
|
||
- **A place is a pointer, a value is a load from it.** `(set (.pos c) …)`
|
||
through a `(Ptr Cursor)` becomes a `getelementptr` on the pointer, not on a
|
||
copy. This is the split that would have made a tree-walker silently wrong.
|
||
|
||
Non-local exit is lowered explicitly: `return` and `some` are branches to a
|
||
`ret`, never platform unwinding, so wasm32 needs no exception proposal.
|
||
|
||
## Bounds checks — done
|
||
|
||
`at` and `slice` no longer emit a bare `getelementptr`. A failure is a branch
|
||
to a `noreturn cold` call and then `unreachable` — the same explicit shape as
|
||
`return` and `some`, so wasm32 needs nothing extra for it either. The message
|
||
carries the source location, because `Tast.expr` keeps a `Loc.t` and a language
|
||
that threads locations through the whole frontend should not trap anonymously:
|
||
|
||
```
|
||
$ flan run test/programs/bounds.flan 2
|
||
test/programs/bounds.flan:25:29: slice [2 1) is out of bounds for length 5 (exit 134)
|
||
```
|
||
|
||
Three check sites, and the third is the one with the trap in it:
|
||
|
||
- **`at` on `[n T]`** — the bound is static, so LLVM folds the check away for a
|
||
literal index. A literal that is *out* of bounds never reaches emit at all:
|
||
`check.ml` rejects it, along with a negative literal index (wrong whatever
|
||
the target) and a literal `slice` range that runs backwards. Only literals —
|
||
a `defconst` is a global in the typed IR, not a folded constant, so
|
||
`(at a k)` stays a runtime trap. A slice bound may sit one past the end and
|
||
an index may not, which is the one place the two rules differ.
|
||
- **`at` on a slice or string** — the bound is the runtime len.
|
||
- **`slice`** — *two* comparisons, `lo <= hi` and `hi <= len`, both non-strict
|
||
because a slice ending at len (or an empty one at `lo = len`) is legal and
|
||
its one-past-the-end gep is defined. `lo <= hi` is not redundant: without it
|
||
a reversed range yields `hi - lo` as a huge unsigned length, which is a worse
|
||
hole than the missing check was.
|
||
|
||
All comparisons are unsigned. Indices are i32 sign-extended to i64 for the gep,
|
||
so a negative one arrives as a huge unsigned value and one test catches both
|
||
directions; the runtime still prints the signed value in the message.
|
||
|
||
`Build.opts.checks` is on by default and **is not tied to `opts.opt`** — dev
|
||
traps, release does not, and that is a release decision rather than an
|
||
optimisation one. Keeping them separate is what lets the acceptance table go on
|
||
running the same programs at `-O0` and `-O2` with identical checks. The CLI
|
||
flag is `--no-bounds-checks`, on `build` and `emit`.
|
||
|
||
The write path is its own case. `(set (at arr n) …)` lowers through
|
||
`place`/`Pindex`, not through `At`, so a refactor that split them would break
|
||
the write check silently — the test covers both.
|
||
|
||
`test/programs/bounds.flan` is one program with one case per argument, because
|
||
a trap ends the process. The acceptance test asserts the exit code, that the
|
||
message names the file, and the reason — but not line and column, so editing
|
||
the program does not break the test that reads it. It runs at both `-O0` and
|
||
`-O2`, and one more case checks the IR directly: `--no-bounds-checks` emits no
|
||
`call` to either failure function. (The two `declare`s stay in the header
|
||
unconditionally; LLVM drops the unused ones.)
|
||
|
||
## The IR, inspected
|
||
|
||
Verified by reading `flan emit` output rather than by trusting the tests:
|
||
|
||
- Every check site is `icmp` → `br` → cold block → `call` → `unreachable`. The
|
||
verifier accepts it; no dominance or phi problems.
|
||
- `-O2` folds the two literal-index checks in `bounds.flan` and keeps the other
|
||
seven, which is exactly the intent.
|
||
- **A redundant check is not eliminated, and `index_ty` is why.** calc-me's
|
||
`peek` already guards with `(< (.pos c) (len (.src c)))`, yet the bounds check
|
||
survives `-O2`. `len` truncates the i64 length to i32 and the index is a
|
||
*signed* i32, so the guard emits `icmp slt i32 %pos, (trunc %len)` while the
|
||
check emits `icmp ult i64 (sext %pos), %len`. LLVM cannot bridge those and is
|
||
right not to: the trunc loses bits above 2³¹, and `slt` does not imply
|
||
`pos >= 0`. Lengths as i64, or unsigned indices, would let the two merge —
|
||
but that is `index_ty`, a plan.org-level decision, so it is left alone.
|
||
- Cost, measured: a 50M-iteration serial dependency chain over a 1024-element
|
||
array, argv-seeded so nothing folds, runs at 0.11–0.12s checked against
|
||
0.12–0.13s unchecked. Indistinguishable. The branch predicts perfectly and
|
||
the loop is latency-bound.
|
||
- Cosmetic: the fail block is emitted *before* the continuation block, so at
|
||
`-O0` the cold path sits inline in the hot path. `cold` plus LLVM's block
|
||
placement fixes it at `-O2`; nothing fixes it at `-O0`.
|
||
|
||
## Where build time goes
|
||
|
||
`flan build calc-me.flan` is ~140ms, and ~95% of it is clang:
|
||
|
||
| Step | Cost |
|
||
|---|---|
|
||
| frontend: read → parse → check → emit | <10ms, below the timer |
|
||
| `clang` on the `.ll` | 60ms — `llc` does the same codegen in **20ms** |
|
||
| `clang` on `flan_rt.c` | 40ms — recompiled every build, and it never changes |
|
||
| link | 20ms |
|
||
|
||
Two cheap wins take it to ~60ms: cache `flan_rt.o`, and skip the clang driver
|
||
for the `.ll` (`llc` + link directly). This reproduces plan.org's own
|
||
measurement — the driver is the cost, not codegen — and it is a subset of the
|
||
dev path's machinery, so doing it now is not wasted work.
|
||
|
||
**There is still no REPL.** Nothing in `lib/` or `bin/` does redefinition,
|
||
`dlopen`, or nREPL; `build.ml`'s docstring describes the dev path and says
|
||
nothing at milestone 2 needs it yet. `build` is the only way to run code, so
|
||
its 140ms is what you actually pay.
|
||
|
||
## Diagnostics
|
||
|
||
A lowercase name is a type variable (plan.org, Types), which meant a mistyped
|
||
primitive — `f65` for `f64` — was reported as *unimplemented generics, see
|
||
plan.org*, sending you to the plan instead of to the character you mistyped.
|
||
`resolve_name` now tries `near_miss` first: one edit (substitution, insertion,
|
||
deletion, or a transposed pair) against the primitives, aliases, structs and
|
||
unions. Bounded at one edit, because two is a guess, and because a real
|
||
single-letter type variable like `t` must still reach the milestone-5 message.
|
||
|
||
```
|
||
(defn f [x f65]) unknown type f65 — did you mean f64?
|
||
(defn f [x stirng]) unknown type stirng — did you mean string?
|
||
(defn f [x t]) generic code over the type variable t … milestone 5
|
||
(defn f [x Widget]) unknown type Widget
|
||
```
|
||
|
||
`Types.primitive_names` exists now because the list had only ever been match
|
||
arms. Caveat found while testing: a bare `(defn f [] f65 0.0)` says *unknown
|
||
name* instead, because with a single body form the parser cannot tell a return
|
||
type from the first expression. Only the parameter position and `(Option …)`
|
||
are unambiguous.
|
||
|
||
## Next
|
||
|
||
Recommended order — start with milestone 4; wasm32 needs a system install only
|
||
you can authorize, and the REPL is worth more once there is a frame loop for it
|
||
to not stutter.
|
||
|
||
1. **Milestone 4 — sand.flan.** `dotimes`, `defer`, and typed raylib FFI with
|
||
keyword→enum coercion. Fixed 2-D arrays are done. `flan check sand.flan`
|
||
already fails on `(import rl ...)`, as it should. FFI is the part most likely
|
||
to expose layout bugs the calc-me surface cannot reach.
|
||
2. **wasm32.** The backend is there (`llc` lists `wasm32`) and
|
||
`Build.opts.target` already plumbs `--target`, but there is **no wasi
|
||
sysroot on this machine** — `clang --target=wasm32-wasi` cannot find
|
||
`stdio.h`. Install `wasi-sdk`/`wasi-libc` (`dnf search wasi` for the Fedora
|
||
package name), then run the same acceptance table on both targets in CI.
|
||
That is milestone 3's real remaining work. The bounds work above was written
|
||
to survive the port — no unwinding, and `exit(134)` rather than `abort()`,
|
||
so the same trap assertion should hold on wasm32 — but that is intent, not a
|
||
tested result: nothing here has ever been built for wasm32.
|
||
WASI is the syscall interface wasm has to import to get stdout, argv and
|
||
exit at all; `flan_rt.c` calls libc (`fwrite`, `snprintf`, `strtod`,
|
||
`malloc`), so it needs `wasi-libc`. The alternative is a second freestanding
|
||
shim that imports host functions directly and links no libc — which is what
|
||
"one narrow host ABI, implemented twice" points at, and the ABI is small
|
||
enough to make it plausible. Note plan.org has the *web* build linking
|
||
raylib via emscripten, which brings its own sysroot: wasi-sdk is right for
|
||
the headless acceptance table, not necessarily for the eventual game build.
|
||
3. **The dev path / REPL.** `llc` + `ld -shared` + `dlopen` ≈ 16ms, a compiler
|
||
daemon plus an in-game reload agent (plan.org, Dev architecture). The build
|
||
wins above are a down payment on this.
|
||
|
||
## Watch for
|
||
|
||
The rule that caught the two misparse bugs applies unchanged: **anything that
|
||
binds a name, alters control flow, or is not yet implemented must be recognised
|
||
explicitly and rejected if unsupported.** `check.ml` rejects `Vec`, `Map`,
|
||
`Result`/`try`, union values, closures, `dotimes`, `defer`, keywords at call
|
||
sites, imports, generics and function values *by name*, each with the milestone
|
||
it belongs to. The tests assert on the reason, not just on the failure.
|
||
|
||
## Untracked on purpose
|
||
|
||
`old-ocaml/` — the pre-rewrite menhir/ocamllex frontend, kept as reference and
|
||
excluded from the build by the root `dune` file. Its contents are also in git
|
||
history at `2c232dd`.
|