JosephFerano/flan
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'worktree-agent-af9064091c0602dc8' into dev-loop

Browse Source
This commit is contained in:
Joseph Ferano 2026-09-13 21:24:12 +07:00
commit e725a5aab4
9 changed files with 689 additions and 20 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

171
DISCUSS.md
View File

@ -1259,3 +1259,174 @@ that can run, runs, and prints what LLVM's build prints — down to stderr.
What is left in `x86.ml` is not conditions. It is a container return convention, a redefinition cell, two arithmetic
edge cases the language has not decided, and no debug info. None of those is the shape conditions were: each is a
known thing in a known place, and the guard is not underneath any of them.
## 18. The container return convention was never there, and the redefinition cell now is
Item 17's no-plan bucket opened with two correctness items and a loose end it was honest about. Both items are closed
and the loose end is the reason the first one closed the way it did rather than the way it was written down. **LLVM is
untouched and still the default and the release backend.** `--x86` is still off by default and still refused with
`--debug`, `--sanitize` and every wasm target; it is **no longer refused with `--dev`**, and question 3 is the argument
for that.
### Question 1 — the counts, and a corpus that moved
`spike/x86/survey.sh`, unchanged in what it compares: every program in `test/programs` and every probe in `spike/x86`,
built both ways with the same bounds-check setting, run, and diffed on **stdout, stderr and exit status**.
| | item 17 | measured here, before | after |
|---|---|---|---|
| **MATCH** | 89 | **93** | **97** |
| **DIFFER** | 0 | 0 | **0** |
| **refused by name** | 0 | **2** | **0** |
| skipped: does not compile | 25 | 28 | 28 |
| skipped: no `main` | 6 | 6 | 6 |
| skipped: never terminates | 2 | 2 | 2 |
**The baseline is not the one the brief quoted, and that is the first finding.** Item 17 measured 89/0/0; this lane
measured 93 MATCH and **2 refusals** before writing anything. The corpus moved underneath: another lane landed
`(slice-from-ptr p n)`, and it arrived as two refusals rather than one — `slice-from-ptr.flan` and `bounds.flan` both
stopped building through `--x86`. A backend that refuses by name does not rot quietly, but it does rot, and nothing
was watching. That is an argument for running the survey in CI rather than in a lane.
The form itself is nothing: a `Slice _` is `{ptr, i64}` here exactly as it is in `emit.ml`, so it is one store of the
pointer and one of the length and no new representation at all. The half worth writing down is that **the check has to
be signed**. There is nothing to compare the length against — only the caller knows what is behind that pointer — so
what is checked is that the promise is not absurd, and `check_slice`'s own compares are *unsigned*. A negative `i32`
sign-extended to 64 bits is a huge unsigned value that `jbe` waves straight through, and the result is a slice about
2^64 long that reads as a pass and faults somewhere else entirely.
Nothing in the corpus walks that path, because every length in `slice-from-ptr.flan` is a literal and a negative
literal is refused by `check.ml` before any code is emitted. `spike/x86/p7-slice-from-ptr.flan` takes the length as a
parameter and runs it through a `restart-case`, which puts the condition's `low`/`high`/`length` on stdout beside the
LLVM build's.
The other two of the four new MATCHes are this lane's own probes, below.
### Question 2 — why `flan_vec_as_slice` avoided the aggregate-return refusal
**It is the first of the two possibilities item 17 named: the refusal is narrower than it reads, and nothing is going
right by accident.**
`flan_vec_as_slice`'s Flan-level return type is `Unit`. `check.ml:3721` builds it as `rt loc Types.Unit`,
`flan_rt.c:1165` is `void flan_vec_as_slice(flan_vec *v, void *out, ...)`, and `emit.ml:2464` declares it `void`. So
`is_void rty` answers first and the `is_agg rty` test below it is never reached. The slice comes back through an
out-pointer the checker allocated, which is not one symbol's accident but the convention:
- **Every aggregate-valued runtime result crosses through an out-pointer.** Every other `rt` builder in `check.ml`
answers `Unit`, an `Int`, a `Ptr`, an `Alloc` or a `Handle`. `flan_pool_resolve` answers `Ptr elem` and the `Option`
is built in Flan; `Argv` is its own `Tast` node with its own out-pointer and never comes through here at all.
- **`crossable`**, the other user of the same code path, admits `String` and `Slice _` only as *a parameter* and
refuses an aggregate return from a `declare` outright.
**So there is no sret convention to build for `Rt`, and building one would have been worse than the refusal.** This is
the C boundary, where the header says the backend must match SysV rather than pick: a 16-byte slice comes back in
`rax:rdx`, not through the internal hidden-pointer convention. There is no classifier in the file, there is nothing to
test one against, and "untestable and wrong" is a bad trade against a line that costs nothing. The line stays as a
guard against those two rules changing, and now names which rules and what the work would actually be.
**Item 16's claim that the container runtime is unexercised went with it, and it was already stale when item 17
repeated it.** `Vec` and `Map` run through `vec.flan`, `vec-of-vec.flan`, `vec-in-struct.flan`, `maps.flan` and
`map-iter.flan`; `Pool` — which neither report checked — runs through `registry.flan`, `handles.flan`,
`generics.flan` and `pool-stale-region.flan`. All match, and they have since item 17's guard landed.
### Question 3 — the cell, and why `--x86 --dev` is no longer refused
`FnAddr (Fnval n)` emitted the symbol. It now reads the cell — and **so does every direct call**, which is the half
that matters and is what `emit.ml`'s `body_of` does: a redefinition is one store, and its whole purpose is to reach
call sites that already exist. What is emitted, all of it behind `dev`:
- **One cell per function** in `.data`, `.globl`, initialised to the body this build compiled. Spelled exactly as
`Emit.cellname` spells it, because that is the point of having one here — an LLVM-built redefinition module binds
`@"flan.cell.<n>" = external global ptr` against whatever built the host. `nm -D` over the two builds of the same
program gives identical sets of 68 cell symbols.
- **The cell load placed after the arguments.** `emit.ml` has that as a load-bearing comment: a redefinition landing
between two calls must not land in the middle of one. `CallPtr` stays the other way round, for `emit.ml`'s reason.
- **The `flan_dev_reg_enable` constructor**, which arms the allocation registry.
Not emitted: `Emit.cellptr`, the deeper spelling for a name the host was never built with. It cannot arise in a
whole-program build, where `known` is true of everything, and it belongs with the redefinition module that would
introduce such a name.
**The refusal is relaxed, and the argument is that `flan dev` never reaches that fork.** `--x86` is read in exactly one
place, `flan build`'s argument list; the daemon builds its host through `Build.executable` and its modules through
`Build.shared` without it, and there is no spelling that hands it one. So the flag now means what it says — a host
whose call sites are redefinable, built by this backend — and nothing claims the module that would redefine through
them exists.
### Question 4 — what made that believable, because the corpus cannot
Two measurements, and the second is the one that matters.
**The corpus with `--dev` on both sides: 97 MATCH, 0 DIFFER.** `SURVEY_FLAGS=--dev` is opt-in so the counts in question
1 stay the same measurement. Before the constructor was added it read **96/1**: `registry.flan` asks `(live? ...)` and
got four zeroes, because the allocation registry was never armed. That is the whole of what a dev host does
differently besides the cells, and it is worth saying that the corpus found it — one program out of 97 observes it.
**And the thing the corpus structurally cannot test.** A dev build starts with every cell pointing at the body this
build compiled, so it prints exactly what a release build prints *whether or not anything reads the cell*. The
property that makes the whole corpus a safe test of the cells is the property that makes it a useless one.
So `spike/x86/cells.sh` changes what a cell holds. It preloads a shared object whose constructor looks up
`flan.cell.twice` with `dlsym` — the cells are in `.dynsym` because a dev build is `-rdynamic` — and stores a different
body there. That is the one store a redefinition ends in, done from outside, with no compiler involved. Four builds,
and the two controls are half the test:
| | |
|---|---|
| `llvm --dev` | 22 22 — the cell is read |
| `x86 --dev` | 22 22 — **this lane's claim** |
| `llvm` | 42 42 — no cell; `dlsym` answers NULL |
| `x86` | 42 42 |
`spike/x86/p8-cell.flan` has both call shapes, because they are two different cases in both backends: a direct call by
name, and a function *value*, which is the one `FnAddr` that is not the symbol. The release rows are what say the
change came from the indirection and not from ordinary symbol interposition.
### Question 5 — the licence in the header now has an edge, and it is the cell
This is the thing the next lane inherits, and it is worth more than either item above.
`x86.ml`'s header licenses its own calling convention on the grounds that **a dev build is compiled entirely by this
backend and a release build entirely by LLVM, so the two never meet in one process.** That is what dissolved item 15's
sharpest obstacle and it is why there is no classifier in the file.
**Publishing a cell an LLVM-built module can store into is the first thing that could make it false.** The two
conventions agree on scalars and disagree on every aggregate — this backend passes each by pointer and returns one
through a hidden `sret`, LLVM classifies — so an `Emit.redefinition` module dlopened into an `--x86` host would be
correct exactly until the first redefined function took or returned a struct. `cells.sh` does not reach it, because
the body it installs is `(i64, void *) -> i64` and the conventions agree there.
Nothing in the toolchain does that today: `flan reload` and `flan dev` build host and module through LLVM together,
and neither accepts `--x86`. But the lane that wires this backend into the dev loop will be the one that does it, and
**the answer then is a redefinition emitter here, not a classifier.** Written into both `x86.ml`'s header and
`build.ml`'s refusal so that it is found before it is discovered.
### The honest no-plan bucket
- **A redefinition emitter**, which is question 5 and is new to this list: `Emit.redefinition` has no counterpart here,
so a `--x86 --dev` host has cells nothing in the toolchain can yet write.
- **`f64` → `i64` out of range**, and **`INT64_MIN / -1`**. Unchanged from items 15, 16 and 17: `idiv` raises `SIGFPE`
where LLVM says undefined. A language decision, not a backend one, and nobody has taken it in three reports.
- **`(uninit)` and `unreachable`** still differ from LLVM on purpose and are still written down only in `x86.ml`.
- **The `flan_transfer_fail` branch** — a defer starting a second transfer while the first unwinds. Emitted, refused
loudly, still untested.
- **`"defers on a transfer path nothing reaches"`** — still reached by no program in the corpus.
- **`flan_dev_reg_note` is not dropped in a release build here.** `emit.ml` drops the whole family when `dev` is off;
this emits real calls to a registry that is disabled, so they are no-ops that cost a call each. Correct, not free,
and `emit.ml`'s stated reason for the drop — an escaped alloca `mem2reg` would refuse — does not apply to a backend
with no `mem2reg`.
- **Debug information.** None. `--x86` and `--debug` together are still refused.
- **Code size and speed.** Still not measured, and the list of what to measure has not got shorter: a guard after every
call, a bounds check spending three frame temporaries, every intermediate in memory, `rep movsb` for a block copy —
and now an extra load at every call site in a dev build, which is the one item on this list that `emit.ml` pays too.
### The verdict
**Neither of the two correctness items was the shape it was written down as, and finding that out was most of the
work.** The container return convention did not exist to be built; the loose end item 17 flagged was the answer and
not a symptom, and one afternoon spent tracing it saved a SysV classifier nobody could have tested. The cell was real,
took thirty lines, and could not be tested by anything in the corpus — which is why the useful artefact from it is a
preloaded `dlsym` and not a program.
What is left is a redefinition emitter, two arithmetic edge cases the language still has not decided, and no debug
info. The backend is no longer the thing standing between here and the dev loop.

165
HANDOFF-x86-rt.md Normal file
View File

@ -0,0 +1,165 @@
# Handoff — the x86 backend's last two correctness items
Branch: `dev-loop`, worktree `agent-af9064091c0602dc8`. Three commits plus this one; nothing is half-written and
nothing is reverted. The long-form report is **DISCUSS.md item 18**; this file is the operational version.
## 1. The `flan_vec_as_slice` answer — got it, and it is complete
**`flan_vec_as_slice` never reaches the aggregate-return refusal because its Flan-level return type is `Unit`.**
Primary sources, all three agreeing:
- `lib/check.ml:3721` — `rt loc Types.Unit "flan_vec_as_slice"`
- `runtime/flan_rt.c:1165` — `void flan_vec_as_slice(flan_vec *v, void *out, int32_t lo, int32_t hi, ...)`
- `lib/emit.ml:2464` — `declare void @flan_vec_as_slice(ptr, ptr, i32, i32, i64, ptr, i64, ptr)`
In `lib/x86.ml`'s `call_native` the two tests are in this order:
```
if not (is_void rty) then begin
if is_agg rty then unsupported ...
```
`rty` is the *node's* type, which is `Unit`, so `is_void` answers first and `is_agg` is never evaluated. The slice
leaves through the `void *out` pointer the checker allocated with `fresh_slot`.
**So the refusal is narrower than it reads — the first of the two possibilities the brief named. Nothing is going
right by accident.** It is not one symbol's quirk either; two rules in `check.ml` make the line unreachable for
*every* caller of that code path:
1. **The `rt` out-pointer convention.** Every `rt loc <ty> ...` builder in `check.ml` answers `Unit`, an `Int`, a
`Ptr`, an `Alloc`, a `Handle` or `Int U64`. Enumerated exhaustively by grepping `Tast.Rt` construction sites —
there are 26 and none is aggregate-typed. `flan_pool_resolve` answers `Ptr elem` and the `Option` is built in Flan;
`Argv` is its own `Tast` node with its own out-pointer and does not come through `call_native` at all.
2. **`crossable`** (`lib/check.ml`, the `Ast.Declare` arm, ~line 5228), the other user of `call_native`, admits
`String`/`Slice _` only when `what = "a parameter"` and refuses an aggregate return from a `declare` outright.
**Conclusion, and it changed the work: do not build sret-for-`Rt`.** That path is the *C* boundary, where the header
says the backend must match SysV rather than pick its own. A 16-byte slice comes back in `rax:rdx`, not through this
backend's internal hidden-pointer convention, and there is no classifier in the file. Building one would have been
untestable (nothing in the language can produce a call that needs it) *and* wrong (wrong convention). The refusal
stays as a guard against those two rules changing, with the reasoning in a comment and a message that now names SysV
classification instead of reading like a missing feature.
**Consequence:** item 16's "`Vec`, `Map` and `Pool` have not been exercised at all" was already stale when item 17
repeated it. Verified with the survey: `Vec`/`Map` via `vec.flan`, `vec-of-vec.flan`, `vec-in-struct.flan`,
`maps.flan`, `map-iter.flan`; **`Pool` — which neither report checked** — via `registry.flan`, `handles.flan`,
`generics.flan`, `pool-stale-region.flan`. All MATCH.
## 2. Survey counts, measured
`spike/x86/survey.sh`, unchanged in what it compares (stdout + stderr + exit status, same bounds-check setting both
sides).
| | brief said | measured before | after |
|---|---|---|---|
| MATCH | 89 | **93** | **97** |
| DIFFER | 0 | 0 | **0** |
| refused by name | 0 | **2** | **0** |
| skip: does not compile / no main / forever | 25 / 6 / 2 | 28 / 6 / 2 | 28 / 6 / 2 |
**The brief's 89/0/0 baseline was stale.** Another lane landed `(slice-from-ptr p n)` after item 17, and it arrived as
two refusals — `slice-from-ptr.flan` and `bounds.flan` — both reported as `x86: primitive with 2 arguments`. Fixing
that is commit 1 and is reported separately so the "after" number is not misread as this lane's work.
Also measured, opt-in and new: `SURVEY_FLAGS=--dev spike/x86/survey.sh` → **97 MATCH / 0 DIFFER**.
## 3. What was built, file by file
All **working and verified**; nothing in this list is unverified or reverted.
| file | state | what |
|---|---|---|
| `lib/x86.ml` — `prim`, `Tast.SliceFromPtr` case | working | new. Two stores; the length check is **signed** (`cc_ge`), not `check_slice`'s unsigned compare |
| `lib/x86.ml` — `call_native` | working | refusal reworded + the two `check.ml` invariants written down. No behaviour change |
| `lib/x86.ml` — `csym` | working | new, `"flan.cell." ^ n` quoted; must stay byte-identical to `Emit.cellname` |
| `lib/x86.ml` — `lower`, `FnAddr (Fnval _)` | working | splits from `Flanfn`; loads the cell when `md.dev` |
| `lib/x86.ml` — `lower`, `Tast.Call` | working | passes `` `Cell `` instead of `` `Sym `` when `md.dev` |
| `lib/x86.ml` — `call_flan` | working | new `` `Cell `` target: `mov r11, [rip+cell]; call r11`, **after** `emit_args` |
| `lib/x86.ml` — `emit_cells` | working | new. `.data`, `.globl`, `.quad <body>`, one per `p.Tast.fns` |
| `lib/x86.ml` — `program` | working | now `~checks ?dev`; emits cells and the `flan_dev_reg_enable` ctor when `dev` |
| `lib/x86.ml` — `layout_ctx` | working | now `~checks ~dev`; `Emit.m.dev` is no longer hardcoded `false` |
| `lib/build.ml` | working | `--dev` removed from the `--x86` refusal list; `~dev:opts.dev` threaded to `X86.program` |
| `spike/x86/survey.sh` | working | `SURVEY_FLAGS`, given to **both** sides. Default unchanged |
| `spike/x86/p7-slice-from-ptr.flan` | working, MATCH | negative length through a parameter and a `restart-case` |
| `spike/x86/p8-cell.flan` | working, MATCH | direct call + function value, for `cells.sh` |
| `spike/x86/cell-override.c` | working | `dlsym("flan.cell.twice")` + store, in a constructor |
| `spike/x86/cells.sh` | working, 4/4 ok | the only test of the cell that can exist |
`lib/emit.ml` was **not** modified. No change to it was needed.
## 4. What did not work, with the errors
Nothing fought for an hour. Four short false starts, all mine and all one-line:
- `handler-bind` clause syntax guessed as an `fn` literal:
`spike/x86/p7-slice-from-ptr.flan:30:18: a handler-bind clause is (Type [name] body ...)`.
The form is `(BoundsError [c] body ...)`; `test/programs/bounds-condition.flan:112` is the model.
- `(defn show [name [u8] ...])` for a literal argument:
`spike/x86/p7-slice-from-ptr.flan:37:14: expected [u8], found string`. A string literal wants `string`, not `[u8]`,
even though they are the same two words at the machine level.
- `cmp_imm` takes `~dst` and an `int`, not `~reg` and an `Int64`.
- `cell-override.c`: `error: 'NULL' undeclared` — needs `<stddef.h>` beside `<dlfcn.h>`.
One environment note for the next lane, not a failure of this work: `dune test --root .` prints
`/usr/bin/ld: cannot open output file /tmp/build_*_dune/flan-devtest-robust.cache/flan-macros-*.so.*: Permission
denied` and `clang: error: linker command failed with exit code 1` twice. That is **inside the `dev-robust` fixture**,
which exists to prove a failed build leaves the session standing; the run still exits 0. Item 17's four raylib
fixtures did **not** fail here — `/tmp` had room throughout (6% used at start and at finish).
## 5. Was `Fnval`'s cell reached, and the `--dev` call
**Yes, reached and tested.** And the test is the interesting part, because *the corpus cannot do it*: a dev build
starts with every cell pointing at the body that build compiled, so it prints exactly what a release build prints
whether or not anything reads the cell. `spike/x86/cells.sh` preloads a `.so` whose constructor `dlsym`s
`flan.cell.twice` (the cells are in `.dynsym` — a dev build is `-rdynamic`) and stores a different body there. Four
builds; the two release rows are the control that says the effect is the indirection and not symbol interposition:
```
ok llvm --dev: 22 22
ok x86 --dev: 22 22
ok llvm : 42 42
ok x86 : 42 42
```
Also checked: `nm -D` over an LLVM `--dev` build and an x86 `--dev` build of the same program gives **identical sets of
68 `flan.cell.*` symbols**. That is the property the later lane depends on.
**My call: yes, relax `--x86` with `--dev`, and it is relaxed.** The argument is narrow and verified: `--x86` is read
in exactly one place, `flan build`'s argument list in `bin/main.ml`. `flan dev` and `flan reload` build host and
module through `Build.executable` / `Build.shared` with no `x86` field set, and there is no spelling that hands them
one. So the daemon is unchanged and cannot reach the new path. What `--x86 --dev` gives is **a host whose call sites
are redefinable**; what it does not give is anything in the toolchain that can write a cell, because
`Emit.redefinition` has no counterpart here.
**The one thing the next lane must read before it writes that counterpart** (now in `x86.ml`'s header and
`build.ml`'s refusal comment too): `x86.ml` licenses its own calling convention on the grounds that a dev build is
compiled entirely here and a release build entirely by LLVM, *so the two never meet in one process*. A cell an
LLVM-built module can store into is the first thing that can make that false. The two conventions agree on scalars and
disagree on **every aggregate** — here each goes by pointer with a hidden `sret`; LLVM classifies. An
`Emit.redefinition` module dlopened into an `--x86` host would be correct until the first redefined function took or
returned a struct. `cells.sh` does not reach it: the body it installs is `(i64, void *) -> i64`.
**The answer is a redefinition emitter here, not a classifier.**
## 6. What remains, in the order to do it
1. **A redefinition emitter in `lib/x86.ml`** — the counterpart to `Emit.redefinition`, producing a `.so`: cells as
`.globl` externs rather than definitions, bodies hidden, `Emit.cellptr`'s deeper spelling for names the host lacks,
the `flan_dev_cell` / `flan_dev_global` lookups, and a publish function. This is the only item that unblocks the
dev loop, and question 5 above is why it cannot be skipped by leaning on LLVM for modules.
2. **`flan_dev_reg_note` dropped in a release build** — in `prim`'s `Tast.Rt` dispatch, matching `emit.ml:1918`'s
`when (not f.md.dev) && ...` arm. Today `x86.ml` emits real calls into a disabled registry: correct, no-ops, one
call each. Cheap; a code-size item, not a correctness one.
3. **`f64` → `i64` out of range**, and **`INT64_MIN / -1`**, in `prim`'s `Tast.Cast` and `Div`/`Rem` arms. `idiv`
raises `SIGFPE` where LLVM says undefined. Blocked on a *language* decision, unchanged across items 15, 16, 17.
4. **The `flan_transfer_fail` branch** — a defer starting a second transfer while the first unwinds. Emitted in
`transfer_exit`, refused loudly, reached by no program. Needs a probe in `spike/x86`.
5. **`"defers on a transfer path nothing reaches"`** — the refusal in `emit_fn`. Same: exists so that if the reasoning
is wrong it says so, and no program reaches it.
6. **Debug information.** None; `--x86 --debug` still refused in `build.ml`.
7. **Code size and speed.** Still unmeasured, and the list grew: a guard after every call, three frame temporaries per
bounds check, every intermediate in memory, `rep movsb` block copies, and now an extra load per call site in a dev
build — which is the one item `emit.ml` pays too.
**Also worth doing and not a backend item: run `spike/x86/survey.sh` in CI.** The 2 refusals this lane found were a
month-old lane's new prim, and nothing noticed. A backend that refuses by name does not rot quietly, but it does rot.

33
lib/build.ml
View File

@ -732,12 +732,35 @@ let executable ?(opts = default) ?(csrcs = []) ?(lflags = []) ?(pnames = [])
does not do this: see [opts]. *)
let opts = if opts.debug then { opts with opt = "-O0" } else opts in
let tflags = target_flags opts in
if opts.x86 && (wasm_target opts || opts.dev || opts.debug || opts.sanitize)
(* [--dev] used to be in this list, for the indirection cells. It is not any
more: [x86.ml] emits a cell per function, spelled as [Emit.cellname]
spells it and exported the same way, and calls and function values read
it. What a `--x86 --dev` build still lacks is anything to *write* one —
[Emit.redefinition] has no x86 counterpart. That costs nothing here,
because [flan dev] never reaches this fork: [--x86] is read only by
[flan build], the daemon builds its host and its modules through this
function without it, and there is no spelling that hands it one. So the
flag means what it says — a host whose call sites are redefinable, built
by this backend — and the module that would redefine through them arrives
with the lane that writes it.
That lane inherits one thing this comment should say out loud rather than
leave for it to find. [x86.ml]'s header licenses its own calling
convention on the grounds that a dev build is compiled entirely by it and
a release build entirely by LLVM, so the two never meet in one process.
Publishing a cell an LLVM-built module can store into is the first thing
that could make that false: the two conventions agree on scalars and
disagree on every aggregate, so an [Emit.redefinition] module dlopened
into an [--x86] host would be correct until the first redefined function
took or returned a struct. Nothing in the toolchain does that today —
[flan reload] and [flan dev] both build host and module through LLVM —
and the fix when something does is to emit the module through this
backend too, not to grow a classifier. *)
if opts.x86 && (wasm_target opts || opts.debug || opts.sanitize)
then
failwith
"--x86 is the native dev backend on its own: it emits no DWARF, has no \
indirection cells for a REPL to redefine through, and there is no \
sanitizer pass over hand-written assembly";
"--x86 is the native dev backend on its own: it emits no DWARF and \
there is no sanitizer pass over hand-written assembly";
let dir = workdir () in
(* The one fork in this function. The x86 backend hands clang an assembly
file where LLVM hands it IR text; clang takes either on its command line,
@ -748,7 +771,7 @@ let executable ?(opts = default) ?(csrcs = []) ?(lflags = []) ?(pnames = [])
(Filename.basename out ^ if opts.x86 then ".s" else ".ll")
in
write ll
(if opts.x86 then X86.program ~checks:opts.checks p
(if opts.x86 then X86.program ~checks:opts.checks ~dev:opts.dev p
else
Emit.program ~checks:opts.checks ~dev:opts.dev ~debug:opts.debug ~pnames
~sanitize:opts.sanitize p);

165
lib/x86.ml
View File

@ -20,6 +20,16 @@
this backend runs. So the convention is ours to pick, and we pick the
simplest one that exists:
{b The licence has an edge now, and it is the indirection cells below.} A
cell is a mutable global an out-of-process redefinition can store into, and
the module doing the storing is built by [Emit.redefinition], which is
LLVM. The two conventions agree on scalars and disagree on every
aggregate, so an LLVM-built module dlopened into a build made here would be
correct exactly until a redefined function took or returned a struct.
Nothing in the toolchain does that today — [flan reload] and [flan dev]
build host and module through LLVM together — and the answer when
something does is a redefinition emitter {e here}, not a classifier.
- {b Scalars} — integers, [bool], pointers, enums, handles, allocators,
function pointers — go in SysV's integer registers [rdi rsi rdx rcx r8
r9], then right-to-left on the stack. [bool] is one byte, zero-extended.
@ -319,7 +329,7 @@ let xorps b ~dst = rex b ~w:false ~r:dst ~x:0 ~m:dst; u8 b 0x0f; u8 b 0x57; modr
(* [Emit.m] carries the struct and union tables [Emit.lay] reads. Built here
rather than imported so that this module adds no line to [emit.ml]: the
record has no signature hiding it and every field it needs is inert. *)
let layout_ctx ~checks (p : Tast.program) : Emit.m =
let layout_ctx ~checks ~dev (p : Tast.program) : Emit.m =
let structs = Hashtbl.create 16 and unions = Hashtbl.create 16 in
List.iter (fun (s : Tast.structure) -> Hashtbl.replace structs s.Tast.sname s)
p.Tast.structs;
@ -327,7 +337,7 @@ let layout_ctx ~checks (p : Tast.program) : Emit.m =
p.Tast.unions;
{ Emit.out = Buffer.create 1; strs = Buffer.create 1; structs; unions;
globals = Hashtbl.create 1; externs = Hashtbl.create 1; checks;
dev = false; known = (fun _ -> true); dbg = None; sanitize = false;
dev; known = (fun _ -> true); dbg = None; sanitize = false;
nstr = 0; nfi = 0 }
let sizeof md t = fst (Emit.lay md t)
@ -365,6 +375,14 @@ let asm_sym s = "\"" ^ s ^ "\""
let fsym n = asm_sym ("flan." ^ n)
let gsym n = asm_sym ("flan." ^ n)
(* The indirection cell: a mutable global holding the address of the function
that is currently this name's body. Spelled exactly as [Emit.cellname]
spells it, because that is the whole point of having one here — a
redefinition module is still built by LLVM, and it binds
[@"flan.cell.<n>" = external global ptr] against whatever built the host.
Byte-for-byte or the link fails and the piece served nothing. *)
let csym n = asm_sym ("flan.cell." ^ n)
(* ── Function context ────────────────────────────────────────────────── *)
type fnctx = {
@ -919,12 +937,24 @@ and lower_at f (e : Tast.expr) (dst : loc) : unit =
let l = place f p in
addr_into f ~reg:rax l;
store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8
(* Not the indirection cell: this backend owns the whole build and nothing
is redefined into it, so a function's address is its symbol. When it
stops being true, [Fnval] is the case that grows a load. *)
| Tast.FnAddr (Tast.Flanfn n) | Tast.FnAddr (Tast.Fnval n) ->
(* The symbol itself, not a load from it: a function's address is a
link-time constant, and this is the spelling a lifted handler clause is
reached by. [emit.ml] says the same of [Flanfn]. *)
| Tast.FnAddr (Tast.Flanfn n) ->
lea f.b ~dst:rax ~mm:(Sym (fsym n, 0));
store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8
(* A function value someone wrote, which is the one [FnAddr] that is not the
symbol. In a release build there is nothing to redefine and it is the
symbol after all; in a dev build it is the cell's contents, so that a
value taken after a redefinition is the new body. What that does not give
— and [emit.ml] names it rather than papering over it with a trampoline —
is a value taken *before* a redefinition and called after it. Once the
address is in a slot there is nothing left to re-resolve. *)
| Tast.FnAddr (Tast.Fnval n) ->
if f.md.Emit.dev then
load_int f.b ~dst:rax ~mm:(Sym (csym n, 0)) ~size:8 ~signed:false
else lea f.b ~dst:rax ~mm:(Sym (fsym n, 0));
store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8
| Tast.FnAddr (Tast.Rtfn n) ->
lea f.b ~dst:rax ~mm:(Sym (n, 0));
store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8
@ -932,7 +962,12 @@ and lower_at f (e : Tast.expr) (dst : loc) : unit =
| Tast.Call (name, args) ->
(match Hashtbl.find_opt f.externs name with
| Some sym -> call_c f ~sym ~args ~rty:t dst
| None -> call_flan f ~target:(`Sym (fsym name)) ~args ~rty:t dst)
| None ->
(* A dev build calls through the cell so that a redefinition reaches
every existing call site; a release build names the symbol. *)
call_flan f
~target:(if f.md.Emit.dev then `Cell (csym name) else `Sym (fsym name))
~args ~rty:t dst)
| Tast.CallPtr (callee, args) ->
let c = eval f callee in
call_flan f ~target:(`Loc c) ~args ~rty:t dst
@ -1701,7 +1736,8 @@ and ret_loc f = if is_agg f.fret then Lp (f.sret_off, 0) else Lf f.retval
and call_flan f ~target ~args ~rty dst =
let vals = List.map (fun (a : Tast.expr) -> eval f a, a.Tast.ty) args in
let callee =
match target with `Sym s -> `Sym s | `Loc l -> `Loc (off_of l)
match target with
| `Sym s -> `Sym s | `Cell s -> `Cell s | `Loc l -> `Loc (off_of l)
in
let sret = (not (is_void rty)) && is_agg rty in
let head = if sret then [ Aptr dst ] else [] in
@ -1718,8 +1754,17 @@ and call_flan f ~target ~args ~rty dst =
the pointer we were handed, so one cell serves the whole chain. *)
let chan = [ Aint (Lf f.xfer_off, Types.Ptr Types.Unit) ] in
ignore (emit_args f (head @ body @ chan));
(* The cell is loaded *after* the arguments, and [emit.ml] has the same as a
load-bearing comment: a redefinition that lands between two calls still
must not land in the middle of one. [r11] is scratch and no argument
register, so this cannot disturb what [emit_args] just placed. [CallPtr]
is deliberately the other way round — the callee there is written first
and there is no cell to keep out of an argument list. *)
(match callee with
| `Sym s -> call_sym f.b s
| `Cell s ->
load_int f.b ~dst:r11 ~mm:(Sym (s, 0)) ~size:8 ~signed:false;
call_r f.b r11
| `Loc o ->
load_int f.b ~dst:r11 ~mm:(Frame o) ~size:8 ~signed:false;
call_r f.b r11);
@ -1776,7 +1821,29 @@ and call_native f ~sym ?(chan = false) ~(args : Tast.expr list) ~rty dst =
call_sym f.b sym;
if chan then guard f;
if not (is_void rty) then begin
if is_agg rty then unsupported "aggregate return from %s" sym;
(* Unreachable, and it is worth saying why rather than leaving it reading
like a gap in the backend. Nothing that crosses this boundary returns an
aggregate, by two rules that both live in [check.ml]:
- every aggregate-valued runtime result comes back through an
*out-pointer* the checker allocates, so the Flan-level return type is
[Unit] or a scalar. [flan_vec_as_slice] is the one that looks like a
counter-example and is not: [check.ml] builds it as [rt loc
Types.Unit] and [flan_rt.c] writes the two words through [void *out].
Every other [rt] builder in the file answers [Unit], an [Int], a
[Ptr], an [Alloc] or a [Handle].
- [crossable], which admits [String] and [Slice _] only as "a
parameter" and refuses an aggregate return from a [declare] outright.
So this is a guard against those two rules changing, and not a feature
waiting to be written. If one ever does change, the work it names is
*SysV classification* and not the internal convention in the header: C
returns a 16-byte slice in rax:rdx, and there is no classifier in this
file. Refusing is the honest answer until there is. *)
if is_agg rty then
unsupported
"%s returns %s by value, which needs SysV return classification this \
backend does not have" sym (Types.to_string rty);
store_loc f ~reg:(if is_float rty then xmm0 else rax) dst rty
end
@ -1900,6 +1967,41 @@ and prim f (e : Tast.expr) (p : Tast.prim) (args : Tast.expr list) dst =
load_loc f ~reg:rcx llo lo.Tast.ty;
sub_rr f.b ~dst:rax ~src:rcx;
store_int f.b ~src:rax ~mm:(lmem f (shift dst 8) ~scratch:r11) ~size:8
(* (slice-from-ptr p n): the two words a slice already is, with the pointer
the caller handed over and the length the caller promised. A [Slice _] is
{ptr, i64} here exactly as it is in [emit.ml], so there is no new
representation to build — one store of the pointer and one of the length.
The check is the *length itself* and not a range, because nothing here
knows how many elements live behind that pointer; only the caller does.
So what is checked is the half that can be — that the promise is not
absurd — and it is a *signed* test, which matters: [check_slice]'s
compares are unsigned, and a negative i32 sign-extended to 64 bits is a
huge unsigned value that [jbe] waves straight through.
It reuses [flan_slice_error] for [emit.ml]'s reason: the violated
condition is 0 <= n, which has the shape of a reversed slice, so the range
is reported as [0 n) against a length of 0. *)
| Tast.SliceFromPtr, [ p; n ] ->
let lp = eval f p in
let ln = eval f n in
if f.md.Emit.checks then
scoped f (fun () ->
let a = ptmp f and b = ptmp f and c = ptmp f in
xor_rr f.b ~dst:rax ~src:rax;
store_int f.b ~src:rax ~mm:(Frame a) ~size:8;
store_int f.b ~src:rax ~mm:(Frame c) ~size:8;
load_loc f ~reg:rax ln n.Tast.ty;
store_int f.b ~src:rax ~mm:(Frame b) ~size:8;
cmp_imm f.b ~dst:rax 0;
let ok = new_label f "inb" in
jcc_lbl f.b ~cc:cc_ge ok;
bounds_call f "flan_slice_error" e.Tast.loc [ a; b; c ];
lbl f.b ok);
load_loc f ~reg:rax lp p.Tast.ty;
store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8;
load_loc f ~reg:rax ln n.Tast.ty;
store_int f.b ~src:rax ~mm:(lmem f (shift dst 8) ~scratch:r11) ~size:8
(* string and [u8] are the same two words, so both directions are views and
not copies — the same non-instruction [emit.ml] emits. *)
| (Tast.Bytes | Tast.StrOfBytes), [ a ] -> lower f a dst
@ -2390,10 +2492,37 @@ let check_no_transfer (p : Tast.program) =
in
List.iter (fun (g : Tast.global) -> ex g.Tast.ginit) p.Tast.globals
(* One cell per function, initialised to the body this build compiled, and
[.globl] so that a redefinition module can bind to it. Nothing has been
redefined yet when the program starts, so a dev build begins by behaving
exactly like a release one — the indirection is the only difference, and
that is what makes the whole corpus a test of it.
[.data] and not [.bss]: the initialiser is a relocation against the body,
not a zero. Default visibility, because interposition is the point here;
only a redefined *body* is hidden, and this backend emits none.
What is not here is [Emit.cellptr] — the second, deeper spelling for a name
the host was never built with. It cannot arise in a whole-program build,
where [known] is true of everything, and it belongs with the redefinition
module that would introduce such a name. *)
let emit_cells (p : Tast.program) =
let out = Buffer.create 256 in
Buffer.add_string out "\t.data\n";
List.iter
(fun (fn : Tast.fn) ->
let c = csym fn.Tast.name in
Buffer.add_string out
(Printf.sprintf "\t.globl\t%s\n\t.align\t8\n\t.type\t%s, @object\n\
\t.size\t%s, 8\n%s:\n\t.quad\t%s\n"
c c c c (fsym fn.Tast.name)))
p.Tast.fns;
Buffer.contents out
(* A whole program as one assembly file. *)
let program ~checks (p : Tast.program) : string =
let program ~checks ?(dev = false) (p : Tast.program) : string =
check_no_transfer p;
let md = layout_ctx ~checks p in
let md = layout_ctx ~checks ~dev p in
let externs = Hashtbl.create 16 in
List.iter
(fun (e : Tast.extern) -> Hashtbl.replace externs e.Tast.ename e.Tast.esym)
@ -2422,9 +2551,19 @@ let program ~checks (p : Tast.program) : string =
Buffer.add_buffer out text;
(* The globals' initialiser runs before main, through the same constructor
slot [emit.ml] uses to arm the allocation registry. *)
(* [flan_dev_reg_enable] arms the allocation registry, and a dev build is the
only build that has one. A constructor rather than a line in [main] for
[emit.ml]'s reason: a [defvar] initialiser allocates before [main] runs,
and a note that arrived before the flag was set would be a block the table
never heard of. It is ordered before [init_sym] here for the same reason.
Leaving it out was the one visible difference between a `--x86 --dev`
build and an LLVM one over the whole corpus: [registry.flan] asks
[(live? ...)] and got four zeroes. *)
Buffer.add_string out
(Printf.sprintf "\t.section\t.init_array,\"aw\",@init_array\n\t.align\t8\n\
\t.quad\t%s\n\n" init_sym);
(Printf.sprintf "\t.section\t.init_array,\"aw\",@init_array\n\t.align\t8\n%s\
\t.quad\t%s\n\n"
(if dev then "\t.quad\tflan_dev_reg_enable\n" else "") init_sym);
if dev then Buffer.add_string out (emit_cells p);
Buffer.add_string out (emit_globals_data md p.Tast.globals);
Buffer.add_string out "\n\t.section\t.rodata\n";
Buffer.add_buffer out rodata;

31
spike/x86/cell-override.c Normal file
View File

@ -0,0 +1,31 @@
/* Redefine flan.cell.twice from outside the program, without a compiler.
*
* A dev build exports one cell per function -- a mutable global holding the
* address of the body that is current -- and it is -rdynamic, so the cell is
* in .dynsym and dlsym can find it by name. Storing a different function
* pointer there is the whole of what a redefinition does to an existing call
* site; the rest of the dev loop is about producing the new body, and none of
* that is needed to answer "does a call actually read the cell".
*
* A Flan function's signature is its parameters followed by the transfer
* channel, so this takes (i64, void *) where the Flan body takes (n i64). It
* never transfers, so it never writes through the channel.
*
* A release build has no cells, dlsym answers NULL, and this does nothing --
* which is the control: it shows the change below comes from the indirection
* and not from ordinary symbol interposition. See spike/x86/cells.sh.
*/
#define _GNU_SOURCE
#include <dlfcn.h>
#include <stddef.h>
#include <stdint.h>
static int64_t instead(int64_t n, void *xfer) {
(void)xfer;
return n + 1;
}
__attribute__((constructor)) static void install(void) {
void **cell = (void **)dlsym(RTLD_DEFAULT, "flan.cell.twice");
if (cell != NULL) *cell = (void *)instead;
}

59
spike/x86/cells.sh Executable file
View File

@ -0,0 +1,59 @@
#!/usr/bin/env bash
# Does a --x86 --dev build actually call through the indirection cell?
#
# survey.sh cannot answer this and no program can. A dev build begins with
# every cell pointing at the body this build compiled, so it prints exactly
# what a release build prints whether the call reads the cell or ignores it.
# The only way to tell is to change what a cell holds and see whether anything
# notices.
#
# So: cell-override.c is preloaded, and its constructor looks up
# flan.cell.twice with dlsym and stores a different body there. No compiler is
# involved and nothing is redefined in the language's sense -- this is just the
# one store a redefinition ends in, done from outside.
#
# Four builds, and the two controls are half the test:
#
# llvm --dev cell is read -> 22 22
# x86 --dev cell is read -> 22 22 (this lane's claim)
# llvm no cell -> 42 42 (dlsym answers NULL)
# x86 no cell -> 42 42
#
# The release rows are what say the change came from the indirection and not
# from ordinary symbol interposition.
set -u
here=$(cd "$(dirname "$0")" && pwd)
root=$(cd "$here/../.." && pwd)
cd "$root" || exit 1
dune build --root . bin/main.exe 2>&1 | head -30
flan=$root/_build/default/bin/main.exe
test -x "$flan" || { echo "build failed"; exit 1; }
out=$(mktemp -d); trap 'rm -rf "$out"' EXIT
cc -shared -fPIC -o "$out/override.so" "$here/cell-override.c" || exit 1
src=$here/p8-cell.flan
fail=0
run() { # run <label> <expected> <build flags...>
local label=$1 want=$2; shift 2
if ! "$flan" build "$src" "$@" -o "$out/p8" >"$out/build.err" 2>&1; then
echo "FAIL $label: build failed"; head -3 "$out/build.err"; fail=1; return
fi
local got
got=$(cd "$out" && LD_PRELOAD="$out/override.so" ./p8 | tr '\n' ' ')
got=${got% }
if [ "$got" = "$want" ]; then
echo "ok $label: $got"
else
echo "FAIL $label: expected '$want', got '$got'"; fail=1
fi
}
run "llvm --dev" "22 22" --dev
run "x86 --dev" "22 22" --dev --x86
run "llvm " "42 42"
run "x86 " "42 42" --x86
exit $fail

46
spike/x86/p7-slice-from-ptr.flan Normal file
View File

@ -0,0 +1,46 @@
;;;; (slice-from-ptr p n) with a length the checker cannot see.
;;;;
;;;; test/programs/slice-from-ptr.flan covers the form itself, but every length
;;;; in it is a literal, and a negative literal is refused by check.ml before
;;;; any code is emitted. So the run-time half of the check -- the one both
;;;; backends plant beside the form -- is walked by nothing in the corpus.
;;;;
;;;; The half that matters here is that the test is *signed*. check_slice's own
;;;; compares are unsigned, and a negative i32 sign-extended to 64 bits is a
;;;; huge unsigned value that an unsigned "hi <= len" waves straight through.
;;;; Getting that wrong yields a slice whose length is about 2^64, which reads
;;;; as a pass and segfaults somewhere else entirely.
;;;;
;;;; Both cases go through a restart-case, so what is compared is the message
;;;; on stderr as well as the fact that something was signalled.
(defvar a [4 i32])
(defn promised [n i32] i32
;; n is a parameter, so the checker has no literal to look at.
(restart-case
(let [s (slice-from-ptr (addr (at a 0)) n)]
(len s))
(give-up [] -1)))
(defn show [name string n i64] ()
(print name)
(print " ")
(println n))
(defn main [] ()
(dotimes [i 4]
(set (at a i) (* (+ i 1) 10)))
(handler-bind
[(BoundsError [c]
(show "low" (.low c))
(show "high" (.high c))
(show "length" (.length c))
(invoke-restart 'give-up))]
(println (promised 4)) ; 4 -- the truth
(println (promised 0)) ; 0 -- empty is not an error
(println (promised 2)) ; 2 -- shorter than the truth is legal
(println (promised -1)) ; -1 -- signalled, and the restart answered
(println (promised -1000000))))

26
spike/x86/p8-cell.flan Normal file
View File

@ -0,0 +1,26 @@
;;;; The indirection cell, from the outside.
;;;;
;;;; Nothing in the corpus can tell a build with cells from one without: a dev
;;;; build starts with every cell pointing at the body this build compiled, so
;;;; it prints what a release build prints, which is exactly the property that
;;;; makes the whole corpus a safe test of the cells and a useless test of
;;;; whether anything actually reads one.
;;;;
;;;; spike/x86/cells.sh is what reads one. It preloads a shared object whose
;;;; constructor looks up flan.cell.twice by name and stores a different body
;;;; into it, so a build that routes through the cell prints the new body's
;;;; answer and a build that calls the symbol prints the old one. Run against
;;;; the LLVM dev build and the x86 dev build, the two must agree; run against
;;;; a release build of either, nothing must change.
;;;;
;;;; Two call shapes, because they are two different cases in both backends:
;;;; a direct call by name, and a function *value* — Tast.FnAddr (Fnval _),
;;;; which is the one FnAddr that is not the symbol.
(defn twice [n i64] i64 (* n 2))
(defn apply1 [f (Fn [i64] i64) n i64] i64 (f n))
(defn main [] ()
(println (twice 21)) ; direct call
(println (apply1 twice 21))) ; through the value, taken here

13
spike/x86/survey.sh
View File

@ -49,6 +49,13 @@ forever="dev-loop dev-watch"
TIMEOUT=${TIMEOUT:-20}
# Extra flags, given to *both* sides. SURVEY_FLAGS=--dev is the one that has a
# use: a dev build with nothing yet redefined must behave exactly like a
# release one -- the indirection cell is the only difference -- so the whole
# corpus is a test of the cells, and of nothing else changing beside them.
# Off by default, so the counts above the line stay the same measurement.
read -r -a extra <<<"${SURVEY_FLAGS:-}"
declare -a match=() differ=() refused=() nox86=() skip=()
for src in "$root"/test/programs/*.flan "$root"/spike/x86/*.flan; do
@ -62,7 +69,8 @@ for src in "$root"/test/programs/*.flan "$root"/spike/x86/*.flan; do
# LLVM first. A program that does not compile at all, or has no main, is not
# this backend's business -- the frontend refused it either way.
if ! "$flan" build "$src" -o "$out/$name.llvm" >"$out/$name.llvm.err" 2>&1; then
if ! "$flan" build "$src" "${extra[@]}" -o "$out/$name.llvm" \
>"$out/$name.llvm.err" 2>&1; then
if grep -q "in function \`_start\|undefined reference to \`main\|crt1.o" "$out/$name.llvm.err"; then
skip+=("$name:no-main")
else
@ -71,7 +79,8 @@ for src in "$root"/test/programs/*.flan "$root"/spike/x86/*.flan; do
continue
fi
"$flan" build "$src" --x86 -o "$out/$name.x86" >"$out/$name.x86.err" 2>&1
"$flan" build "$src" --x86 "${extra[@]}" -o "$out/$name.x86" \
>"$out/$name.x86.err" 2>&1
rc=$?
if [ $rc = 3 ]; then
why=$(head -1 "$out/$name.x86.err" | sed 's/^x86: //')