Merge branch 'worktree-agent-a4778b00512de90d3' into dev-loop
This commit is contained in:
commit
8b79cae837
183
DISCUSS.md
183
DISCUSS.md
@ -913,3 +913,186 @@ language has, and that rule is worth having whether or not a backend is ever wri
|
||||
3. **Choose option 1 or option 2 from question 3**, deliberately. Everything else follows from it.
|
||||
4. **Only then**, and only if 3 says so, grow `spike/backend/x86.ml` from the node table in question 2 — floats
|
||||
first, because they gate most of the prelude, and conditions last, because they are the only row with no plan.
|
||||
|
||||
## 16. The dev backend, wired: a whole program compiles through `x86.ml` and runs, and conditions were never in the way
|
||||
|
||||
Item 15's step 4, taken. `lib/x86.ml` was 502 lines of encoder and frame model with nothing calling it. It now lowers a
|
||||
whole `Tast.program` to an assembly file, and `flan build --x86` hands that file to the same clang invocation the LLVM
|
||||
path uses, against the same runtime objects, the same generated shim and the same linker arguments. The flag is off by
|
||||
default and refused in combination with `--dev`, `--debug`, `--sanitize` and every wasm target. **LLVM stays the release
|
||||
backend and the default one; nothing on the existing path changed.** `dune test --root .` is green either side.
|
||||
|
||||
**41 programs out of `test/programs` build through it, and 40 of them print exactly what the LLVM build prints.** The
|
||||
41st is `bounds.flan`, and it diverges on purpose — see question 4.
|
||||
|
||||
### Question 1 — which program, and what did it actually cost
|
||||
|
||||
Measured with `spike/backend/hist.ml` before anything was written, over candidates, not guessed. The one picked is
|
||||
`spike/x86/p3-fizz.flan` — a `dotimes`, a call, an `if`, a remainder, two string literals and `print`/`println`:
|
||||
|
||||
```
|
||||
p3-fizz.flan: 2 reachable fns
|
||||
Call 1 Do 4 If 1 Int 15 Let 2 Local 7 Prim 14
|
||||
Set 1 Str 3 While 1 place/Plocal 1
|
||||
prim/Add 2 prim/Bytes 3 prim/Cast 1 prim/Eq 1
|
||||
prim/I64ToBytes 1 prim/Lt 1 prim/Rem 1 prim/WriteStdout 4
|
||||
```
|
||||
|
||||
Nineteen rows, two reachable functions after `Reach` prunes, and **no `Signal`, no `Handled`, no `RestartCase`, no
|
||||
`Make`, no `Field`, no allocator**. The same is true of a program that only loops and prints: zero of each.
|
||||
|
||||
That is worth stating flatly because the expectation going in was the opposite — that the smallest useful program
|
||||
carries one of each condition node, and that printing a single value drags in `Str`, `Make`, `Field` and `Call` because
|
||||
the prelude builds a slice to do it. **It does not.** Printing an integer is `Cast` → `I64ToBytes` → `WriteStdout`,
|
||||
three prims and no call at all; `flan_i64_to_bytes` renders into a static buffer in `flan_rt.c` and hands back a slice.
|
||||
Printing a string literal is `Str` → `Bytes` → `WriteStdout`, and `Bytes` is a non-instruction because a string and a
|
||||
`[u8]` are the same two words.
|
||||
|
||||
**What does drag conditions in is the bounds check and the allocator, and neither is a `Tast` node.** `check_at` and
|
||||
`check_slice` call `flan_bounds_error` with the transfer channel and then `guard`; `Rt flan_vec_at` takes the channel
|
||||
too. So it is `m.checks` and the container runtime that make a program need the condition machinery, not looping and
|
||||
not printing — and because both are *emit-time* constructs rather than IR nodes, `hist.ml` cannot see either. That is
|
||||
the correction to item 15's node table: the "no plan" row is not reached by writing a loop, it is reached by writing
|
||||
`(at a i)`.
|
||||
|
||||
### Question 2 — what runs
|
||||
|
||||
| program | what it is for |
|
||||
|---|---|
|
||||
| `spike/x86/p1-exit.flan` | `main` returns 0. The assembly path, the runtime link, a real executable. |
|
||||
| `spike/x86/p2-loop-print.flan` | a `dotimes` that prints — the smallest program that does something |
|
||||
| `spike/x86/p3-fizz.flan` | the measured target: a call, an `if`, `%`, two string literals |
|
||||
| `spike/x86/p4-convention.flan` | the *internal calling convention*, which p3 does not touch at all |
|
||||
| `spike/x86/p5-core.flan` | a global with an initialiser, recursion, `break`, `continue`, the bitwise family, unsigned shifts, both directions of every conversion |
|
||||
|
||||
p4 is the one that matters most, because item 15 named the internal aggregate convention as the sharpest obstacle in
|
||||
the whole report. It passes a struct by value, returns a struct by value, puts an `f32` through the SSE half, calls an
|
||||
eight-argument function so that two arguments go on the stack, and passes a slice — and it agrees with LLVM. **The
|
||||
obstacle really did dissolve the way the header claims**: a dev build is compiled entirely here and a release build
|
||||
entirely by LLVM, the two never meet in one process, so the convention is ours to pick. Every aggregate goes by
|
||||
pointer, an aggregate return is a hidden pointer in the first integer register returned in `rax`, and there is no
|
||||
classifier in the file. Nothing had to be discovered by disassembling clang.
|
||||
|
||||
Over `test/programs` (111 files):
|
||||
|
||||
| | n |
|
||||
|---|---|
|
||||
| built through `--x86` and **matched** the LLVM build's output and exit status | **40** |
|
||||
| built and diverged — `bounds.flan`, by design | 1 |
|
||||
| refused by name: a node this backend does not lower | 40 |
|
||||
| no `main` (package and library fixtures) | 6 |
|
||||
| do not compile at all (the checker-error fixtures) | 22 |
|
||||
| never terminate on their own (`dev-loop`, `dev-watch`) | 2 |
|
||||
|
||||
So of the 81 programs that compile, have a `main` and finish, **41 went through the hand-written backend and 40 were
|
||||
byte-identical in output.** That includes `edn.flan` — sixty lines of output from a hand-written EDN reader with
|
||||
unions, options, nested collections and a fixed-depth balance stack.
|
||||
|
||||
Proved by comparing output, never by reading bytes. The script builds both ways and diffs stdout and the exit status;
|
||||
`objdump` was used only after a program already had the wrong answer. Item 15 is right that this is the only honest
|
||||
order.
|
||||
|
||||
### Question 3 — the two bugs, and both are the shape item 15 predicted
|
||||
|
||||
**A `(set (.x (at pts 0)) 1.5)` wrote into a copy.** `lvalue` had no case for `At`, so it fell through to "evaluate it",
|
||||
and the store landed in a temporary while the array kept its zeros. `emit.ml` has this as `addr`'s own `At` case. One
|
||||
line. `array-ctor.flan` found it, and it found it as a segfault several statements later.
|
||||
|
||||
**A discarded value was stored over the return address.** This is the better one. A form whose value is thrown away was
|
||||
handed a sink, and the sink was spelled as an address — `rbp+0`. That is the saved `rbp`, and `rbp+8` is the return
|
||||
address, so a non-void form written in statement position stored straight over both; a 16-byte slice did it in one
|
||||
`rep movsb`. `edn.flan` crashed by jumping into `.rodata`, **several statements after the mistake and in a different
|
||||
function**, and the assembly at the jump read perfectly. The sink is now compared by identity and never used as an
|
||||
address: anything with a value that is handed it gets a frame temporary instead.
|
||||
|
||||
The second bug cannot exist on the LLVM path, and that is the general point. LLVM has no notion of "store this value
|
||||
nowhere" — an unused SSA value is simply unused. Every construct this backend has that LLVM does not is a place where
|
||||
a bug can live that the LLVM backend's own testing can never have covered.
|
||||
|
||||
Against that, the thing item 15 was most worried about did *not* happen: **nothing went wrong with the frame or the
|
||||
stack alignment.** `rsp` is written exactly twice — one rounded `sub` in the prologue that covers the temporaries and
|
||||
the outgoing-argument area together, and `leave` — so `rsp % 16 == 0` at every call site is a property of one
|
||||
subtraction rather than an invariant every case maintains. The spike's worst bug has no door to come in by, and p4
|
||||
calls an eight-argument function to prove it.
|
||||
|
||||
### Question 4 — where the two backends now differ, and it is three named places
|
||||
|
||||
Item 15's question 4 listed nine undefined cases. Three of them are now *real* divergences with a build on each side,
|
||||
and they should be written down before anyone uses this for anything.
|
||||
|
||||
| | LLVM | here |
|
||||
|---|---|---|
|
||||
| **A bounds violation** | `flan_bounds_error` signals; a `restart-case` can catch it; `bounds.flan` exits 134 | **no check at all**; `bounds.flan` exits 139 |
|
||||
| `(uninit)` | `poison`, and the optimiser may reason from it | whatever the stack slot held — stable garbage |
|
||||
| an exhausted `match`, a `noreturn` call | `unreachable`, undefined | `ud2` — a defined SIGILL at the instruction that fell through |
|
||||
|
||||
**The first is the one that matters, and it is not a footnote: `--x86` is silently a `--no-bounds-checks` build.** It is
|
||||
silent because it is not a decision the backend made — `check_at` signals, signalling needs the channel and the guard,
|
||||
and there is no guard here, so there is no check. `bounds.flan` is the direct evidence and `edn.flan`'s 33-brackets-
|
||||
against-a-32-deep-stack case is the second. Anyone reaching for this flag on a program that indexes anything should
|
||||
know that the trap is gone.
|
||||
|
||||
The other two are improvements and cost nothing. `ud2` in particular is two bytes and turns a class of miscompile into
|
||||
a crash with an address.
|
||||
|
||||
### Question 5 — conditions, which is still the row with no plan
|
||||
|
||||
**They were not reached, and converting that into a checked precondition is the most useful thing in this report.**
|
||||
|
||||
There is no transfer guard after a call here, no landing pad and no transfer exit. `emit.ml` emits a guard after *every*
|
||||
call; this emits none. What makes that sound is a whole-program argument rather than a hope: **if nothing in the
|
||||
reachable set can ever write the channel, no call can ever return with it set.** So `check_no_transfer` walks the
|
||||
linked program once per build and stops it — with the node's name and the function it is in — the moment it finds a
|
||||
`signal`, an `invoke-restart`, a `restart-case`, a `handler-bind`, a `with-allocator`, or the two `Rt` symbols whose
|
||||
bounds check signals.
|
||||
|
||||
That is what the 40 refusals are:
|
||||
|
||||
```
|
||||
27 restart-case 4 handler-bind 1 with-allocator
|
||||
7 signal 1 defers on the transfer path
|
||||
```
|
||||
|
||||
Forty programs refused by name rather than miscompiled, and one line of build output says which node and where. A
|
||||
backend that quietly omitted the guard would have compiled all forty and been wrong in a way no test distinguishes
|
||||
from a race.
|
||||
|
||||
What this does *not* do is measure what conditions cost. That is still unknown, and it is still the only row of item
|
||||
15's table with nothing behind it. What is now known is the shape of the bill: the guard is per call site, the pad is
|
||||
per `restart-case` activation, `fdefers` needs a second exit path that no form in `body` can reach, and **any function
|
||||
with `fdefers` at all is refused today** — which is most of the prelude's file and container code, and is why the
|
||||
programs that use a `Vec` are not in the 40.
|
||||
|
||||
### The honest no-plan bucket
|
||||
|
||||
Everything below is refused by name at build time, not silently wrong.
|
||||
|
||||
- **Conditions, entire** — the guard, the landing pad, `emit_restart_case`, `emit_with_alloc`, the transfer exit, and
|
||||
`fdefers` on it. Several hundred lines of `emit.ml` reimplemented from `spec-conditions.md` rather than ported.
|
||||
- **Bounds checks**, which are the same work: `check_at` and `check_slice` cannot exist without the guard.
|
||||
- **`Rt` with an aggregate return**, and with it most of the container runtime; `Vec`, `Map` and `Pool` have not been
|
||||
exercised at all.
|
||||
- **`Fnval`'s indirection cell.** `FnAddr (Fnval n)` emits the symbol, which is correct for a whole-program build and
|
||||
wrong the instant anything is redefined into it. This backend has no cells and no `--dev`; that is a deliberate
|
||||
restriction and not an oversight, but it is exactly item 15's question 5 waiting where it was left.
|
||||
- **`f64` → `i64` out of range**, and **`INT64_MIN / -1`**. `idiv` raises `SIGFPE` where LLVM says undefined, and
|
||||
`cvttsd2si` answers the integer-indefinite value. Unchanged from item 15: these want a language decision, not a
|
||||
backend.
|
||||
- **Debug information.** None. `--x86` and `--debug` together are refused.
|
||||
- **Code size and speed.** Not measured. Every value is in memory, every intermediate is a frame temporary, and a
|
||||
block copy is `rep movsb`; that is the trade the brief asks for and nobody has put a number on it.
|
||||
|
||||
### The verdict
|
||||
|
||||
**The wiring is done and it was the easy half. What is left is conditions, and the measurement moved them from "first
|
||||
obstacle" to "the only obstacle".**
|
||||
|
||||
The order item 15 recommended was floats first and conditions last. Floats turned out to be one afternoon's encodings
|
||||
and they are done. Conditions are still last and are now the *whole* remainder: they are what stands between 41
|
||||
programs and the corpus, they are what a bounds check is made of, and `check_no_transfer` is the line that says so out
|
||||
loud on every build until someone writes them.
|
||||
|
||||
Two things are worth doing before that, and both are cheap. Decide what a bounds violation means in a build with no
|
||||
handler — because "no check" is what it means today and nothing says so. And take item 15's question 4 seriously now
|
||||
that there are two backends to disagree: `(uninit)` and `unreachable` already differ, deliberately, and the difference
|
||||
is currently documented only in a comment in `x86.ml`.
|
||||
|
||||
15
bin/main.ml
15
bin/main.ml
@ -100,8 +100,16 @@ let sanitize_flag = "--sanitize"
|
||||
preference, and it goes away with the transport it drives. *)
|
||||
let two_process_flag = "--two-process"
|
||||
|
||||
(* The hand-written x86-64 backend (lib/x86.ml) instead of LLVM. The dev
|
||||
backend from DISCUSS.md item 15, off by default and named explicitly:
|
||||
LLVM stays the release path and the default one. It covers a subset of the
|
||||
IR and refuses the rest by name, so a build that succeeds is one it really
|
||||
compiled. *)
|
||||
let x86_flag = "--x86"
|
||||
|
||||
let flags =
|
||||
[ no_checks_flag; dev_flag; debug_flag; sanitize_flag; two_process_flag ]
|
||||
[ no_checks_flag; dev_flag; debug_flag; sanitize_flag; two_process_flag;
|
||||
x86_flag ]
|
||||
|
||||
(* [--target=wasm32-wasi] and [--target=web], the two cross targets. Unlike
|
||||
the flags above, a target
|
||||
@ -422,6 +430,7 @@ let () =
|
||||
let dev = List.mem dev_flag rest in
|
||||
let debug = List.mem debug_flag rest in
|
||||
let sanitize = List.mem sanitize_flag rest in
|
||||
let x86 = List.mem x86_flag rest in
|
||||
let target = target_of rest in
|
||||
let out =
|
||||
match List.filter (fun a -> not (is_flag a)) rest with
|
||||
@ -453,7 +462,7 @@ let () =
|
||||
let p, csrcs, lflags = Flan.Reach.link ~dev l p in
|
||||
ignore (Flan.Build.executable
|
||||
~opts:{ Flan.Build.default with checks; dev; debug; sanitize;
|
||||
target }
|
||||
target; x86 }
|
||||
~csrcs ~lflags ~pnames:(if debug then param_names l else [])
|
||||
p ~out))
|
||||
(* The daemon an editor talks to: one session, the program it belongs to
|
||||
@ -536,7 +545,7 @@ let () =
|
||||
\ flan import-c <header.h> [package.flan...] [clang flags...]\n\
|
||||
\ flan generate-c <package-dir>\n\
|
||||
\ flan build <file.flan> [-o out] [--no-bounds-checks] [--dev] \
|
||||
[--debug] [--sanitize] [--target=wasm32-wasi|web]\n\
|
||||
[--debug] [--sanitize] [--x86] [--target=wasm32-wasi|web]\n\
|
||||
\ flan run <file.flan> [args...]\n\
|
||||
\ flan reload <program.flan> <forms.flan> [-o out.so]\n\
|
||||
\ flan dev <program.flan> [-s socket]";
|
||||
|
||||
30
lib/build.ml
30
lib/build.ml
@ -147,6 +147,13 @@ type opts = {
|
||||
language's arithmetic means; without the exclusion every program
|
||||
trips on its first [+]. Nothing else is excluded. *)
|
||||
sanitize : bool;
|
||||
(* The dev backend: lower the typed IR to x86-64 assembly here instead of
|
||||
handing LLVM IR to clang. Off by default and off everywhere but the one
|
||||
flag that asks for it — LLVM stays the release backend and the default
|
||||
one. It refuses rather than degrades: a program holding a node [x86.ml]
|
||||
does not lower yet stops the build with that node's name, so a build that
|
||||
succeeds is one this backend really compiled. *)
|
||||
x86 : bool;
|
||||
}
|
||||
|
||||
(* Checks are deliberately independent of [opt]: the acceptance table runs the
|
||||
@ -155,7 +162,7 @@ type opts = {
|
||||
checks. Dropping them is a release decision, not an optimisation one. *)
|
||||
let default =
|
||||
{ target = None; opt = "-O2"; keep = false; checks = true; dev = false;
|
||||
debug = false; sanitize = false }
|
||||
debug = false; sanitize = false; x86 = false }
|
||||
|
||||
(* The flags that are neither [opt] nor the target, spelled once so that the
|
||||
compile command and the object-cache key cannot disagree. They did before:
|
||||
@ -725,11 +732,26 @@ let executable ?(opts = default) ?(csrcs = []) ?(lflags = []) ?(pnames = [])
|
||||
does not do this: see [opts]. *)
|
||||
let opts = if opts.debug then { opts with opt = "-O0" } else opts in
|
||||
let tflags = target_flags opts in
|
||||
if opts.x86 && (wasm_target opts || opts.dev || opts.debug || opts.sanitize)
|
||||
then
|
||||
failwith
|
||||
"--x86 is the native dev backend on its own: it emits no DWARF, has no \
|
||||
indirection cells for a REPL to redefine through, and there is no \
|
||||
sanitizer pass over hand-written assembly";
|
||||
let dir = workdir () in
|
||||
let ll = Filename.concat dir (Filename.basename out ^ ".ll") in
|
||||
(* The one fork in this function. The x86 backend hands clang an assembly
|
||||
file where LLVM hands it IR text; clang takes either on its command line,
|
||||
so everything past this point — the runtime objects, the shim, the
|
||||
package C, the linker arguments — is the same build. *)
|
||||
let ll =
|
||||
Filename.concat dir
|
||||
(Filename.basename out ^ if opts.x86 then ".s" else ".ll")
|
||||
in
|
||||
write ll
|
||||
(Emit.program ~checks:opts.checks ~dev:opts.dev ~debug:opts.debug ~pnames
|
||||
~sanitize:opts.sanitize p);
|
||||
(if opts.x86 then X86.program p
|
||||
else
|
||||
Emit.program ~checks:opts.checks ~dev:opts.dev ~debug:opts.debug ~pnames
|
||||
~sanitize:opts.sanitize p);
|
||||
(* [flan_dev.c] is compiled into every build, not only a dev one. Nothing in
|
||||
a release build calls into it — the compiler only emits a registry lookup
|
||||
for a name the host was not built with, which cannot arise without cells —
|
||||
|
||||
1274
lib/x86.ml
1274
lib/x86.ml
File diff suppressed because it is too large
Load Diff
2
spike/x86/p1-exit.flan
Normal file
2
spike/x86/p1-exit.flan
Normal file
@ -0,0 +1,2 @@
|
||||
(defn main [] i32
|
||||
0)
|
||||
5
spike/x86/p2-loop-print.flan
Normal file
5
spike/x86/p2-loop-print.flan
Normal file
@ -0,0 +1,5 @@
|
||||
(defn main [] i32
|
||||
(dotimes [i 5]
|
||||
(print i)
|
||||
(println ""))
|
||||
0)
|
||||
11
spike/x86/p3-fizz.flan
Normal file
11
spike/x86/p3-fizz.flan
Normal file
@ -0,0 +1,11 @@
|
||||
(defn fizz? [n i32] bool
|
||||
(= 0 (% n 3)))
|
||||
|
||||
(defn main [] i32
|
||||
(dotimes [i 15]
|
||||
(let [n (+ i 1)]
|
||||
(if (fizz? n)
|
||||
(print "fizz")
|
||||
(print n))
|
||||
(println "")))
|
||||
0)
|
||||
27
spike/x86/p4-convention.flan
Normal file
27
spike/x86/p4-convention.flan
Normal file
@ -0,0 +1,27 @@
|
||||
;; The internal calling convention, which the fizz program does not touch at
|
||||
;; all: an aggregate argument, an aggregate return, a float in the SSE half,
|
||||
;; and more integer arguments than there are registers for.
|
||||
|
||||
(defstruct V3 [x f32 y f32 z f32])
|
||||
|
||||
(defn scale [v V3 k f32] V3
|
||||
(V3 {.x (* (.x v) k) .y (* (.y v) k) .z (* (.z v) k)}))
|
||||
|
||||
(defn sum3 [v V3] f32
|
||||
(+ (+ (.x v) (.y v)) (.z v)))
|
||||
|
||||
(defn eight [a i64 b i64 c i64 d i64 e i64 f i64 g i64 h i64] i64
|
||||
(+ (+ (+ a b) (+ c d)) (+ (+ e f) (+ g h))))
|
||||
|
||||
(defn taglen [s [u8]] i64
|
||||
(i64 (len s)))
|
||||
|
||||
(defn main [] i32
|
||||
(let [v (V3 {.x 1.0 .y 2.0 .z 3.0})
|
||||
w (scale v 2.0)]
|
||||
(print (sum3 v)) (println "")
|
||||
(print (sum3 w)) (println "")
|
||||
(print (eight 1 2 3 4 5 6 7 8)) (println "")
|
||||
(print (taglen (bytes "hello"))) (println "")
|
||||
(print (.z w)) (println ""))
|
||||
0)
|
||||
46
spike/x86/p5-core.flan
Normal file
46
spike/x86/p5-core.flan
Normal file
@ -0,0 +1,46 @@
|
||||
;; The rest of the core: a global with an initialiser, recursion, break and
|
||||
;; continue, the bitwise family, unsigned arithmetic and shifts, and the
|
||||
;; conversions in both directions.
|
||||
|
||||
(defvar counter i64 0)
|
||||
|
||||
(defconst limit i32 6)
|
||||
|
||||
(defn fib [n i64] i64
|
||||
(if (< n 2) n (+ (fib (- n 1)) (fib (- n 2)))))
|
||||
|
||||
(defn main [] i32
|
||||
(print (fib 20)) (println "")
|
||||
|
||||
(let [i 0]
|
||||
(while (< i 100)
|
||||
(set i (+ i 1))
|
||||
(when (= i 7) (break)))
|
||||
(print i) (println ""))
|
||||
|
||||
(let [j 0 seen 0]
|
||||
(while (< j 10)
|
||||
(set j (+ j 1))
|
||||
(when (= (% j 2) 0) (continue))
|
||||
(set seen (+ seen j)))
|
||||
(print seen) (println ""))
|
||||
|
||||
(dotimes [k limit]
|
||||
(set counter (+ counter (i64 k))))
|
||||
(print counter) (println "")
|
||||
|
||||
(let [a (bit-xor (u32 0x0F0F0F0F) (u32 0xFFFFFFF))
|
||||
b (u32 0x0F0F0F0F)]
|
||||
(print (bit-or a b)) (println "")
|
||||
(print (bit-and a b)) (println "")
|
||||
(print (bit-xor a (u32 65535))) (println "")
|
||||
(print (>> a 4)) (println "")
|
||||
(print (<< b 4)) (println ""))
|
||||
|
||||
(let [x (i32 -9)]
|
||||
(print (/ x 2)) (println "")
|
||||
(print (% x 2)) (println "")
|
||||
(print (f64 x)) (println "")
|
||||
(print (i32 (f64 3.9))) (println "")
|
||||
(print (f32 1.5)) (println ""))
|
||||
0)
|
||||
Loading…
x
Reference in New Issue
Block a user