diff --git a/BUILT.md b/BUILT.md index 7723cd4..0200b33 100644 --- a/BUILT.md +++ b/BUILT.md @@ -132,6 +132,164 @@ No raylib headers are needed: the generated C declares the prototypes it uses, s library being linkable and not on `raylib-devel`. `vendor/raylib/link` carries `-l:libraylib.so.550` because Fedora ships the runtime library without the `.so` symlink. +### The header is read now — `headers`, `lib/cimport.ml` + +The section above ends by naming what the generator *trusts*: that the +`defstruct` matches the library's real struct, and that the `declare-c` +signature is the function's real signature. "No header is read, deliberately, +so nothing can check either." A header is read now, and both are checked. + +**The dependency, which is the crux, and which this project already answered +once.** Zig's old `@cImport` ran clang as a *library*. That is exactly the +dependency plan.org rejected in "Why LLVM IR as text": a version-pinned C++ +library breaks routinely on upgrade, a binary on PATH does not. So this shells +out for `clang -Xclang -ast-dump=json -fsyntax-only`, which is the same binary +every build already runs and adds nothing that is not already being paid for. +`lib/cjson.ml` is enough JSON to read that dump and no more, so it adds no opam +package either. + +A note worth recording, because it strengthens the argument rather than +weakening it: **Zig has since abandoned clang here too.** `translate_c.zig` is +gone; `lib/compiler/translate-c/` is built on Aro, a C frontend written in Zig. +Their reason was to ship a compiler containing no clang at all — the opposite +premise to this one, where `clang` on PATH *is* the toolchain assumption. Both +projects walked away from linking libclang; only the destination differs. + +**What is imported: functions, and only functions.** Not structs, not enums, +not macros. The bound on how much is not a curated list but the package's own +`defstruct`s — a function whose signature mentions a struct the package has not +described is refused with that reason, so `vendor/raylib` describing thirteen +structs is what makes the import thirteen structs wide, and describing a +fourteenth widens it. Of raylib 5.5's 581 functions, 256 import, 153 are +refused, and 172 are left alone because the package already binds them by hand. + +Not generating `defstruct`s is what makes the check possible at all. Generate +them and the header becomes the authority on layout, and comparing the +package's layouts against the header's would be comparing the header with +itself. A `_Static_assert` on `sizeof`/`offsetof` was rejected in the section +above as circular for exactly that reason; **this is not circular, because the +two sides have different authors.** It is the cheapest real closure of the +trusted-not-guaranteed gap. + +**Refusing by demotion, which is the one thing taken wholesale from Zig.** Zig's +translator never drops a declaration it cannot handle: `failDecl` binds the name +to a `@compileError` carrying the reason, so the name still exists, the program +still compiles, and asking for *that one name* fails at the use site with the +reason. A wholesale import has a hundred and fifty refusals and a caller cares +about the one they typed. Flan already had that mechanism — `Load.refuse_hidden`, +built for `main` — so `rl/get-gamepad-name` is not a name, and a program that +writes it is told *the return type is a string, and a string only crosses as a +parameter* rather than "unknown name". + +That is also the split `shim.ml` needed and did not have. It refuses through +`Loc.fail`, which is right when a human named one function and wrong for a +wholesale import, where one returned `const char *` would kill the header. Same +judgement, different disposition; a hand-written `declare-c` still hard-fails +and `shim.ml` is untouched. + +**Two C spellings mean something in a parameter that they mean nowhere else.** +`const char *` is a string going in, and the generator already knows how to hand +one over. `char *` without the const is very often a buffer the callee *writes*, +and handing it a NUL-terminated temporary would lose the writes with no +diagnostic anywhere — const is the only thing in the header that separates the +two, so it is what decides, and an out-buffer keeps a hand-written binding +saying `(Ptr u8)`. `long`, `size_t` and the rest are refused rather than +guessed, and for a reason specific to this project: it builds for x86-64, for +wasm32-wasi and for the browser, and `long` is 64 bits on the first and 32 on +the others, so a guess would be right for the target that gets tested and +silently wrong for two that do not. + +**Naming.** `rl/InitWindow` is `rl/init-window`. Reversibility is not a property +of the rule — the C symbol is stored verbatim in the declaration, so the wrapper +reads the library's spelling rather than reconstructing it. What the rule must +be is *injective over one header*, since two C functions arriving under one Flan +name would surface as a duplicate declaration about a name nobody wrote. A +boundary goes before an uppercase letter after a lowercase one, before an +uppercase letter between an uppercase and a lowercase, and before a digit after +a lowercase; nowhere else. So `SetTargetFPS` is `set-target-fps` and not +`set-target-f-p-s`, `BeginMode2D` is `begin-mode-2d`, `UnloadUTF8` is +`unload-utf8`. raylib's 581 names are injective under it. **When two do collide, +neither takes the name** — resolving by order would mean that moving two lines +in somebody else's header silently rebinds a name a program is already calling. +Both are refused, both say why, and the author binds the one they want with a +`declare-c`. + +**Where it runs.** `headers` beside `link` in the package directory, read the +same way: a path, any clang flags it needs, `${NAME}` expanded from the +environment. What comes back is ordinary `declare-c` declarations, generated +before the package's names are qualified, so they arrive as `rl/…` exactly like +the hand-written ones and nothing downstream can tell which is which. No new +form, no new `decl_kind`, no reader or parser change. A C symbol the package +already binds by hand is left alone, so `declare-c` stays the escape hatch and +stays the thing that wins. + +A leading `?` makes a line optional, and `vendor/raylib` uses it. "No raylib +headers are needed" is a real property — a build needs libraylib linkable, not +raylib-devel installed — and requiring a header would take it from everyone in +order to give the check to whoever has one. Unset `FLAN_RAYLIB_H` and the build +is exactly what it was; set it and every signature is checked. A path that is +*set and wrong* is an error naming it, because silently behaving as though +nobody had opted in is the difference between an opt-in and a trap. + +#### What the diff found + +The evidence the whole lane exists for. Against raylib **5.5** — the version +whose `.so` `link` names — **all 16 `defstruct`s and all 172 hand-written +`declare-c` agree exactly.** The half BUILT.md called trusted is now checked, +and it was right. + +That is only worth stating because the check has teeth. Against the **5.1-dev** +header installed in `/usr/local` it reports ten differences: nine functions that +version does not have (`CheckCollisionCircleLine`, the six `Is*Valid` renames, +`DrawRectangleRoundedLinesEx`) and `DrawRectangleRoundedLines`, which gained a +parameter. Picking the wrong header is therefore loud, which matters, because +the two headers are on the same machine and only one matches the linked library. + +Both comparisons run **at build time** and stop the build, not just in the tool. +Verified by breaking them: a permuted `Texture2D` fails naming the field that +moved, and `f64` where raylib says `float` fails naming the parameter — which is +the hazard the section above calls out by name and says only a test can catch. +The message points at the line in `raylib.flan`, not at the header. + +`flan import-c
[package.flan…]` prints what it would produce, what it +refused and why, and both comparisons, without building anything. That also +makes "generate once and commit the result" available for the cost of a +printer — explicit in the source, checked against reality, no header read at +build time. + +#### What it costs, measured + +The number that decides how much to import, because `reach.ml` was the reason to +think a wholesale import could be free. + +| | today (172 by hand) | + 256 imported | +|---|---|---| +| release build, cold | 0.298s | 0.312s | +| release build, warm | 0.078s | 0.082s | +| redefinition (`flan reload`) | 31.0ms | 46.5ms | +| dev build, cold | 0.649s | 0.982s | + +**`Reach.link` already drops a generated wrapper whose declaration nothing +reachable calls, and that is what makes the release column nearly flat.** +Confirmed on the case it exists for: a wasm32-wasi build of a program that +imports raylib and calls none of it still links without libraylib, with 256 +extra declarations in play. Dev builds are not pruned, on purpose, so one +compiles all 428 wrappers — once, at session start, since `Build.shared` is +llc + `ld -shared` and compiles no C. + +Reading the header is cached, and the cache earned itself against a measurement +rather than a guess: 64ms of a 72ms check, against 8ms for the whole program +without it. What is cached is the *extracted* signatures and not clang's JSON, +because the parse is half the cost — 30ms is clang writing 1.8 MB and the rest +is reading it. Keyed the way the object cache is keyed, on everything that could +change the answer: the header's path, size and mtime, the full flag list, and a +format version, since the value is marshalled. That takes the delta to 17ms. + +**The 15.5ms on redefinition is the real cost and it is the argument against +importing at build time**, on the branch where the dev loop is the priority. It +is the strongest case for the third option — generate from the header, commit +the result, regenerate when the library moves — and that decision is open. + ### What a headless FFI test can and cannot pin Worth knowing before writing another one, because two plausible tests in a row turned out to check nothing. diff --git a/DISCUSS.md b/DISCUSS.md index 581f0f0..de129d3 100644 --- a/DISCUSS.md +++ b/DISCUSS.md @@ -196,33 +196,62 @@ happens. **One dependency:** hiccup is a macro, and the macro expander is blocked on `Form` being a Flan union, which is union values. The backend can start before that; the DSL cannot. -## 6. C interop as seamless as Zig's +## 6. C interop as seamless as Zig's — built, with two decisions left -Today: `declare-c` names one C function per line and the compiler generates the wrapper, the typedefs and the flattened -declaration — 175 lines for raylib. **No header is ever read, deliberately**, which means nothing verifies that a -declaration matches the real signature. That is written down as trusted rather than guaranteed. +**The mechanism is in** (`lib/cimport.ml`, `lib/cjson.ml`, `vendor/raylib/headers`; BUILT.md, "The header +is read now"). Settled and not worth reopening: clang's JSON AST dump over a shelled-out `clang`, never +libclang — and Zig has since abandoned linking clang too, for Aro, which strengthens the argument rather +than weakening it. The import is bounded by the package's own `defstruct`s rather than by a curated list. +Refusals are demotions in Zig's sense: the name exists, cannot be had, and says why at the use site, which +`Load.refuse_hidden` already did for `main`. Names kebab by a rule that is injective over raylib's 581, and +reversibility is by storage — the C symbol is kept verbatim — so the rule never needs an inverse. Where two +names do collide, neither takes it. -The proposal: read the header, prefix a namespace, get `rl/InitWindow` for free, possibly kebab-cased to -`rl/init-window`. +**The evidence:** against raylib 5.5, all 16 `defstruct`s and all 172 hand-written `declare-c` agree +exactly. Against the 5.1-dev header also on this machine, ten real differences. Both comparisons stop the +build, and a permuted `Texture2D` or an `f64` for a `float` is caught by name. -**The dependency is the crux, and this project already answered the same question once.** Zig's `@cImport` runs clang as -a *library*. That is exactly the dependency rejected in plan.org's "Why LLVM IR as text": a version-pinned C++ library -breaks routinely on upgrade, while a binary on `PATH` does not. Linking libclang walks back into it. +What is left is two decisions, and both are the author's. -**The middle path that keeps the property:** clang will dump a parsed header as JSON from the command line -(`-Xclang -ast-dump=json`). Still only `clang` on `PATH`, still no library linkage, and it yields *real* signatures -instead of hand-transcribed ones — which closes the "trusted, not guaranteed" gap that is the strongest argument for -doing this at all. +### 6a. Does reading the header stay a build-time step, or become a code generator? -**The design question is how much to import.** Zig imports everything a header declares. For raylib that is several -hundred functions plus every struct and macro, nearly all unused. The current 175 lines are deliberate, and the -declaration site is also the checkpoint where the compiler *refuses* a signature it cannot safely flatten — an -aggregate return, a variadic, a `string` coming back. A wholesale import removes that checkpoint, or has to reproduce -it as a filter. Generating the list from the header while keeping it explicit in the source is a third option: generate -once, commit the result, regenerate when the library moves. +The cost, measured, with the wrappers pruned by `Reach` as they already were: -**Kebab-casing is separable and small**, with one constraint: it must be reversible, because the generated C wrapper -needs the library's own spelling. +| | today | + 256 imported | +|---|---|---| +| release build, warm | 0.078s | 0.082s | +| **redefinition** | **31.0ms** | **46.5ms** | +| dev build, cold | 0.649s | 0.982s | + +Release is nearly free and that question is answered. **The 15.5ms on redefinition is not nothing on the +branch where the dev loop is the priority** — it is a 50% increase on the number that lane exists to keep +small, and it buys a check of signatures that have not changed since the last build. + +So the third option from the original discussion is now the live one: `flan import-c` already prints +`declare-c` lines, so **generate from the header, commit the result, regenerate when raylib moves** costs +nothing more to build. Explicit in the source, checked against reality, no header read at build time, and +the check becomes a thing you run rather than a thing you pay for. Against it: a committed file goes stale +silently, which is the failure the whole lane exists to prevent, and "regenerate when the library moves" +is a discipline rather than a mechanism. + +A middle reading worth considering: keep the build-time check but run it only when *not* `--dev`, on the +grounds that a release build is where a wrong signature must not get through and a dev build is where +15.5ms is felt. That is the same shape as `Reach` not pruning dev builds, for a symmetric reason. + +### 6b. Do the 172 hand-written lines get migrated? + +The diff is clean, so nothing blocks it on correctness. What blocks it is that migration needs the header +present at *every* build, which means vendoring raylib.h into the repo or requiring `raylib-devel` — and +BUILT.md records "a build needs libraylib linkable and not raylib-devel installed" as a property that was +chosen on purpose. That is why `vendor/raylib/headers` is opt-in (`?${FLAN_RAYLIB_H}`) today and the +hand-written lines are untouched. + +Worth noting what migration would actually lose, since it is small but real: the hand-written names are +better than the rule's. `IsKeyPressed` is `key-pressed?` by hand and `is-key-pressed` by rule; +`CheckCollisionRecs` is `collision-recs?`. And an enum parameter imports as `i32`, because the header says +`KeyboardKey` and nothing tells the importer the package calls that `Key` — so `(rl/key-down? :space)` +would become an integer at the call site. A migration is therefore not a deletion; it is a deletion plus a +kept list of the lines whose face is deliberately nicer than the header's. ## 7. Watching variables diff --git a/NEXT.md b/NEXT.md index 091f38c..94017d9 100644 --- a/NEXT.md +++ b/NEXT.md @@ -40,6 +40,58 @@ read. It belongs with item 2, where the listing is being changed anyway. Read SBCL for what restarts should *mean* and ignore how it moves control: it transfers with `block`/`return-from`, which §6 rules out. +### Landed — a C header is read, so a binding is checked instead of trusted + +`lib/cimport.ml`, `lib/cjson.ml`, a `headers` file beside `link`. Full reasoning in `BUILT.md`, "The header is read +now"; DISCUSS.md item 6 is rewritten down to the two decisions left, both the author's. + +The gap closed is the one `BUILT.md` recorded as *trusted*: `declare-c` generates the wrapper, the typedefs and the +prototype from one declaration, so they agree with each other by construction and only the library could disagree — +and nothing had a second opinion to disagree with. Now clang is asked for a JSON AST dump of the header (shelled out, +never libclang — the dependency plan.org rejected; Zig has since left it too, for Aro) and both halves are compared +against it. + +**The evidence.** Against raylib 5.5, the version whose `.so` `vendor/raylib/link` names: **all 16 `defstruct`s and +all 172 hand-written `declare-c` agree exactly.** Against the 5.1-dev header also installed on this machine, ten real +differences — nine functions that version lacks and one that gained a parameter — so picking the wrong header is +loud. Both comparisons run at build time and stop the build; verified by permuting `Texture2D` and by putting `f64` +where raylib says `float`, which is the hazard `BUILT.md` names and says only a test can catch. + +**Costs, measured, because they decide the remaining question.** Release build +4ms warm — `Reach.link` already drops +a wrapper nothing reachable calls, confirmed on the wasm32 case it exists for with 256 extra declarations in play. +Redefinition 31.0ms → 46.5ms. Dev build +333ms cold, once per session, since `Build.shared` compiles no C. Reading the +header is cached (64ms → 17ms), keyed like the object cache; the cache was built against a measurement, not a guess. + +**Opt-in on purpose.** `vendor/raylib/headers` is `?${FLAN_RAYLIB_H}`. "A build needs libraylib linkable and not +raylib-devel installed" is a property chosen deliberately, and requiring a header would take it from everyone to give +the check to whoever has one. Unset means off; set-and-wrong is an error naming the path. + +Worth knowing before touching it: + +- **The import is bounded by the package's own `defstruct`s**, not by a curated list. A function mentioning a struct + the package has not described is refused with that reason. Of raylib's 581 functions, 256 import, 153 are refused, + 172 are already bound by hand and left alone. Widening the binding is a `defstruct`, not a list edit. +- **No `defstruct` is generated, and that is load-bearing.** Generate them and the header becomes the authority on + layout, and checking the package's layouts against it would be comparing the header with itself — which is exactly + why `BUILT.md` rejected a `_Static_assert` as circular. Keeping them hand-written is what makes the check a second + source. +- **A refusal is a demotion, not a drop** — Zig's `failDecl`, which `Load.refuse_hidden` already implemented for + `main`. `rl/get-gamepad-name` is a name that exists, cannot be had, and says why at the use site. +- **`declare-c` and `declare` are untouched and still win.** A C symbol the package binds by hand is not imported, so + the escape hatch is the override. +- **`test/headers/sample.h`** is the importer's table — one function per decision, committed, no raylib needed. The + raylib acceptance case skips without `FLAN_RAYLIB_H`; that one does not. + +Two things that are *not* done, and are 6a and 6b in DISCUSS.md: whether the header stays a build-time read or becomes +a committed generator (`flan import-c` already prints the lines, so it costs nothing more to switch), and whether the +172 hand-written lines migrate. Neither is blocked on correctness. The 15.5ms on redefinition is the argument for the +first; needing the header at every build — vendoring raylib.h or requiring raylib-devel — is the argument on the +second. + +One smaller thing found and worth not re-deriving: an enum parameter imports as `i32`, because the header says +`KeyboardKey` and nothing tells the importer the package calls that `Key`. The ABI is identical, the face is worse, +and it is why `(rl/key-down? :space)` keeps its hand-written line. + ### Landed 2026-09-12 — six tracks, one session Six agents in parallel worktrees. Kept short on purpose; the reasoning that outlives the change is in `BUILT.md` or in diff --git a/lib/shim.ml b/lib/shim.ml index b9bfbee..6ced73d 100644 --- a/lib/shim.ml +++ b/lib/shim.ml @@ -66,8 +66,14 @@ A [_Static_assert] on [sizeof] and [offsetof] was considered and left out: both sides of it would come from the same field list, so it would check this module's arithmetic against clang's and say nothing about the library. - What would convert the trusted half into a checked one is including the - real header when one is installed, and that is not built. *) + What converts the trusted half into a checked one is reading the real + header, and that is built: [Cimport] asks clang for a JSON dump of one + and compares both halves against it — every [defstruct] against the + header's record, and every [declare-c] against the header's signature. + Nothing in this module changed for it. The refusals below still raise, + which is right for a signature a human named; the importer makes the + same judgements and merely skips instead, since one returned + [const char *] must not kill a header of five hundred functions. *) let fail = Loc.fail