From 9d6784f2cdc4301615b75acfa507ed3c6d57eb7e Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Sat, 12 Sep 2026 16:17:30 +0700 Subject: [PATCH] Write down what was read, what was refused, and what it cost MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BUILT.md gains "The header is read now", directly under the section whose last paragraph promised that reading a header was what would convert the trusted half into a checked one and that it was not built. That sentence is replaced by a pointer to the one below it, in BUILT.md and in shim.ml's docstring both. It records the things worth not re-deriving: why the dump and not libclang (and that Zig left libclang too, which strengthens the argument rather than weakening it), why the import is bounded by the package's own defstructs, why generating defstructs would make the check circular in exactly the way a _Static_assert was rejected for, refusal-by-demotion from Zig's failDecl, the naming rule and what it must actually guarantee, and both const-vs-non-const char * and the target-varying widths. The diff and the costs are stated as measurements, with the table: 16 of 16 defstructs and 172 of 172 declare-c agree against 5.5, ten real differences against 5.1-dev, release +4ms warm, redefinition 31.0 -> 46.5ms. DISCUSS.md item 6 is rewritten rather than removed. The mechanism question is settled and is now in BUILT.md; what is left is narrower and is two decisions that are the author's — whether the header stays a build-time read or becomes a committed generator, and whether the 172 hand-written lines migrate. Both have the argument on each side written out, including what migration would lose: key-pressed? is a better name than is-key-pressed, and an enum parameter imports as i32 because nothing tells the importer the package calls KeyboardKey "Key". --- BUILT.md | 158 ++++++++++++++++++++++++++++++++++++++++++++++++++++ DISCUSS.md | 71 ++++++++++++++++------- NEXT.md | 52 +++++++++++++++++ lib/shim.ml | 10 +++- 4 files changed, 268 insertions(+), 23 deletions(-) diff --git a/BUILT.md b/BUILT.md index 7723cd4..0200b33 100644 --- a/BUILT.md +++ b/BUILT.md @@ -132,6 +132,164 @@ No raylib headers are needed: the generated C declares the prototypes it uses, s library being linkable and not on `raylib-devel`. `vendor/raylib/link` carries `-l:libraylib.so.550` because Fedora ships the runtime library without the `.so` symlink. +### The header is read now — `headers`, `lib/cimport.ml` + +The section above ends by naming what the generator *trusts*: that the +`defstruct` matches the library's real struct, and that the `declare-c` +signature is the function's real signature. "No header is read, deliberately, +so nothing can check either." A header is read now, and both are checked. + +**The dependency, which is the crux, and which this project already answered +once.** Zig's old `@cImport` ran clang as a *library*. That is exactly the +dependency plan.org rejected in "Why LLVM IR as text": a version-pinned C++ +library breaks routinely on upgrade, a binary on PATH does not. So this shells +out for `clang -Xclang -ast-dump=json -fsyntax-only`, which is the same binary +every build already runs and adds nothing that is not already being paid for. +`lib/cjson.ml` is enough JSON to read that dump and no more, so it adds no opam +package either. + +A note worth recording, because it strengthens the argument rather than +weakening it: **Zig has since abandoned clang here too.** `translate_c.zig` is +gone; `lib/compiler/translate-c/` is built on Aro, a C frontend written in Zig. +Their reason was to ship a compiler containing no clang at all — the opposite +premise to this one, where `clang` on PATH *is* the toolchain assumption. Both +projects walked away from linking libclang; only the destination differs. + +**What is imported: functions, and only functions.** Not structs, not enums, +not macros. The bound on how much is not a curated list but the package's own +`defstruct`s — a function whose signature mentions a struct the package has not +described is refused with that reason, so `vendor/raylib` describing thirteen +structs is what makes the import thirteen structs wide, and describing a +fourteenth widens it. Of raylib 5.5's 581 functions, 256 import, 153 are +refused, and 172 are left alone because the package already binds them by hand. + +Not generating `defstruct`s is what makes the check possible at all. Generate +them and the header becomes the authority on layout, and comparing the +package's layouts against the header's would be comparing the header with +itself. A `_Static_assert` on `sizeof`/`offsetof` was rejected in the section +above as circular for exactly that reason; **this is not circular, because the +two sides have different authors.** It is the cheapest real closure of the +trusted-not-guaranteed gap. + +**Refusing by demotion, which is the one thing taken wholesale from Zig.** Zig's +translator never drops a declaration it cannot handle: `failDecl` binds the name +to a `@compileError` carrying the reason, so the name still exists, the program +still compiles, and asking for *that one name* fails at the use site with the +reason. A wholesale import has a hundred and fifty refusals and a caller cares +about the one they typed. Flan already had that mechanism — `Load.refuse_hidden`, +built for `main` — so `rl/get-gamepad-name` is not a name, and a program that +writes it is told *the return type is a string, and a string only crosses as a +parameter* rather than "unknown name". + +That is also the split `shim.ml` needed and did not have. It refuses through +`Loc.fail`, which is right when a human named one function and wrong for a +wholesale import, where one returned `const char *` would kill the header. Same +judgement, different disposition; a hand-written `declare-c` still hard-fails +and `shim.ml` is untouched. + +**Two C spellings mean something in a parameter that they mean nowhere else.** +`const char *` is a string going in, and the generator already knows how to hand +one over. `char *` without the const is very often a buffer the callee *writes*, +and handing it a NUL-terminated temporary would lose the writes with no +diagnostic anywhere — const is the only thing in the header that separates the +two, so it is what decides, and an out-buffer keeps a hand-written binding +saying `(Ptr u8)`. `long`, `size_t` and the rest are refused rather than +guessed, and for a reason specific to this project: it builds for x86-64, for +wasm32-wasi and for the browser, and `long` is 64 bits on the first and 32 on +the others, so a guess would be right for the target that gets tested and +silently wrong for two that do not. + +**Naming.** `rl/InitWindow` is `rl/init-window`. Reversibility is not a property +of the rule — the C symbol is stored verbatim in the declaration, so the wrapper +reads the library's spelling rather than reconstructing it. What the rule must +be is *injective over one header*, since two C functions arriving under one Flan +name would surface as a duplicate declaration about a name nobody wrote. A +boundary goes before an uppercase letter after a lowercase one, before an +uppercase letter between an uppercase and a lowercase, and before a digit after +a lowercase; nowhere else. So `SetTargetFPS` is `set-target-fps` and not +`set-target-f-p-s`, `BeginMode2D` is `begin-mode-2d`, `UnloadUTF8` is +`unload-utf8`. raylib's 581 names are injective under it. **When two do collide, +neither takes the name** — resolving by order would mean that moving two lines +in somebody else's header silently rebinds a name a program is already calling. +Both are refused, both say why, and the author binds the one they want with a +`declare-c`. + +**Where it runs.** `headers` beside `link` in the package directory, read the +same way: a path, any clang flags it needs, `${NAME}` expanded from the +environment. What comes back is ordinary `declare-c` declarations, generated +before the package's names are qualified, so they arrive as `rl/…` exactly like +the hand-written ones and nothing downstream can tell which is which. No new +form, no new `decl_kind`, no reader or parser change. A C symbol the package +already binds by hand is left alone, so `declare-c` stays the escape hatch and +stays the thing that wins. + +A leading `?` makes a line optional, and `vendor/raylib` uses it. "No raylib +headers are needed" is a real property — a build needs libraylib linkable, not +raylib-devel installed — and requiring a header would take it from everyone in +order to give the check to whoever has one. Unset `FLAN_RAYLIB_H` and the build +is exactly what it was; set it and every signature is checked. A path that is +*set and wrong* is an error naming it, because silently behaving as though +nobody had opted in is the difference between an opt-in and a trap. + +#### What the diff found + +The evidence the whole lane exists for. Against raylib **5.5** — the version +whose `.so` `link` names — **all 16 `defstruct`s and all 172 hand-written +`declare-c` agree exactly.** The half BUILT.md called trusted is now checked, +and it was right. + +That is only worth stating because the check has teeth. Against the **5.1-dev** +header installed in `/usr/local` it reports ten differences: nine functions that +version does not have (`CheckCollisionCircleLine`, the six `Is*Valid` renames, +`DrawRectangleRoundedLinesEx`) and `DrawRectangleRoundedLines`, which gained a +parameter. Picking the wrong header is therefore loud, which matters, because +the two headers are on the same machine and only one matches the linked library. + +Both comparisons run **at build time** and stop the build, not just in the tool. +Verified by breaking them: a permuted `Texture2D` fails naming the field that +moved, and `f64` where raylib says `float` fails naming the parameter — which is +the hazard the section above calls out by name and says only a test can catch. +The message points at the line in `raylib.flan`, not at the header. + +`flan import-c
[package.flan…]` prints what it would produce, what it +refused and why, and both comparisons, without building anything. That also +makes "generate once and commit the result" available for the cost of a +printer — explicit in the source, checked against reality, no header read at +build time. + +#### What it costs, measured + +The number that decides how much to import, because `reach.ml` was the reason to +think a wholesale import could be free. + +| | today (172 by hand) | + 256 imported | +|---|---|---| +| release build, cold | 0.298s | 0.312s | +| release build, warm | 0.078s | 0.082s | +| redefinition (`flan reload`) | 31.0ms | 46.5ms | +| dev build, cold | 0.649s | 0.982s | + +**`Reach.link` already drops a generated wrapper whose declaration nothing +reachable calls, and that is what makes the release column nearly flat.** +Confirmed on the case it exists for: a wasm32-wasi build of a program that +imports raylib and calls none of it still links without libraylib, with 256 +extra declarations in play. Dev builds are not pruned, on purpose, so one +compiles all 428 wrappers — once, at session start, since `Build.shared` is +llc + `ld -shared` and compiles no C. + +Reading the header is cached, and the cache earned itself against a measurement +rather than a guess: 64ms of a 72ms check, against 8ms for the whole program +without it. What is cached is the *extracted* signatures and not clang's JSON, +because the parse is half the cost — 30ms is clang writing 1.8 MB and the rest +is reading it. Keyed the way the object cache is keyed, on everything that could +change the answer: the header's path, size and mtime, the full flag list, and a +format version, since the value is marshalled. That takes the delta to 17ms. + +**The 15.5ms on redefinition is the real cost and it is the argument against +importing at build time**, on the branch where the dev loop is the priority. It +is the strongest case for the third option — generate from the header, commit +the result, regenerate when the library moves — and that decision is open. + ### What a headless FFI test can and cannot pin Worth knowing before writing another one, because two plausible tests in a row turned out to check nothing. diff --git a/DISCUSS.md b/DISCUSS.md index 581f0f0..de129d3 100644 --- a/DISCUSS.md +++ b/DISCUSS.md @@ -196,33 +196,62 @@ happens. **One dependency:** hiccup is a macro, and the macro expander is blocked on `Form` being a Flan union, which is union values. The backend can start before that; the DSL cannot. -## 6. C interop as seamless as Zig's +## 6. C interop as seamless as Zig's — built, with two decisions left -Today: `declare-c` names one C function per line and the compiler generates the wrapper, the typedefs and the flattened -declaration — 175 lines for raylib. **No header is ever read, deliberately**, which means nothing verifies that a -declaration matches the real signature. That is written down as trusted rather than guaranteed. +**The mechanism is in** (`lib/cimport.ml`, `lib/cjson.ml`, `vendor/raylib/headers`; BUILT.md, "The header +is read now"). Settled and not worth reopening: clang's JSON AST dump over a shelled-out `clang`, never +libclang — and Zig has since abandoned linking clang too, for Aro, which strengthens the argument rather +than weakening it. The import is bounded by the package's own `defstruct`s rather than by a curated list. +Refusals are demotions in Zig's sense: the name exists, cannot be had, and says why at the use site, which +`Load.refuse_hidden` already did for `main`. Names kebab by a rule that is injective over raylib's 581, and +reversibility is by storage — the C symbol is kept verbatim — so the rule never needs an inverse. Where two +names do collide, neither takes it. -The proposal: read the header, prefix a namespace, get `rl/InitWindow` for free, possibly kebab-cased to -`rl/init-window`. +**The evidence:** against raylib 5.5, all 16 `defstruct`s and all 172 hand-written `declare-c` agree +exactly. Against the 5.1-dev header also on this machine, ten real differences. Both comparisons stop the +build, and a permuted `Texture2D` or an `f64` for a `float` is caught by name. -**The dependency is the crux, and this project already answered the same question once.** Zig's `@cImport` runs clang as -a *library*. That is exactly the dependency rejected in plan.org's "Why LLVM IR as text": a version-pinned C++ library -breaks routinely on upgrade, while a binary on `PATH` does not. Linking libclang walks back into it. +What is left is two decisions, and both are the author's. -**The middle path that keeps the property:** clang will dump a parsed header as JSON from the command line -(`-Xclang -ast-dump=json`). Still only `clang` on `PATH`, still no library linkage, and it yields *real* signatures -instead of hand-transcribed ones — which closes the "trusted, not guaranteed" gap that is the strongest argument for -doing this at all. +### 6a. Does reading the header stay a build-time step, or become a code generator? -**The design question is how much to import.** Zig imports everything a header declares. For raylib that is several -hundred functions plus every struct and macro, nearly all unused. The current 175 lines are deliberate, and the -declaration site is also the checkpoint where the compiler *refuses* a signature it cannot safely flatten — an -aggregate return, a variadic, a `string` coming back. A wholesale import removes that checkpoint, or has to reproduce -it as a filter. Generating the list from the header while keeping it explicit in the source is a third option: generate -once, commit the result, regenerate when the library moves. +The cost, measured, with the wrappers pruned by `Reach` as they already were: -**Kebab-casing is separable and small**, with one constraint: it must be reversible, because the generated C wrapper -needs the library's own spelling. +| | today | + 256 imported | +|---|---|---| +| release build, warm | 0.078s | 0.082s | +| **redefinition** | **31.0ms** | **46.5ms** | +| dev build, cold | 0.649s | 0.982s | + +Release is nearly free and that question is answered. **The 15.5ms on redefinition is not nothing on the +branch where the dev loop is the priority** — it is a 50% increase on the number that lane exists to keep +small, and it buys a check of signatures that have not changed since the last build. + +So the third option from the original discussion is now the live one: `flan import-c` already prints +`declare-c` lines, so **generate from the header, commit the result, regenerate when raylib moves** costs +nothing more to build. Explicit in the source, checked against reality, no header read at build time, and +the check becomes a thing you run rather than a thing you pay for. Against it: a committed file goes stale +silently, which is the failure the whole lane exists to prevent, and "regenerate when the library moves" +is a discipline rather than a mechanism. + +A middle reading worth considering: keep the build-time check but run it only when *not* `--dev`, on the +grounds that a release build is where a wrong signature must not get through and a dev build is where +15.5ms is felt. That is the same shape as `Reach` not pruning dev builds, for a symmetric reason. + +### 6b. Do the 172 hand-written lines get migrated? + +The diff is clean, so nothing blocks it on correctness. What blocks it is that migration needs the header +present at *every* build, which means vendoring raylib.h into the repo or requiring `raylib-devel` — and +BUILT.md records "a build needs libraylib linkable and not raylib-devel installed" as a property that was +chosen on purpose. That is why `vendor/raylib/headers` is opt-in (`?${FLAN_RAYLIB_H}`) today and the +hand-written lines are untouched. + +Worth noting what migration would actually lose, since it is small but real: the hand-written names are +better than the rule's. `IsKeyPressed` is `key-pressed?` by hand and `is-key-pressed` by rule; +`CheckCollisionRecs` is `collision-recs?`. And an enum parameter imports as `i32`, because the header says +`KeyboardKey` and nothing tells the importer the package calls that `Key` — so `(rl/key-down? :space)` +would become an integer at the call site. A migration is therefore not a deletion; it is a deletion plus a +kept list of the lines whose face is deliberately nicer than the header's. ## 7. Watching variables diff --git a/NEXT.md b/NEXT.md index 091f38c..94017d9 100644 --- a/NEXT.md +++ b/NEXT.md @@ -40,6 +40,58 @@ read. It belongs with item 2, where the listing is being changed anyway. Read SBCL for what restarts should *mean* and ignore how it moves control: it transfers with `block`/`return-from`, which §6 rules out. +### Landed — a C header is read, so a binding is checked instead of trusted + +`lib/cimport.ml`, `lib/cjson.ml`, a `headers` file beside `link`. Full reasoning in `BUILT.md`, "The header is read +now"; DISCUSS.md item 6 is rewritten down to the two decisions left, both the author's. + +The gap closed is the one `BUILT.md` recorded as *trusted*: `declare-c` generates the wrapper, the typedefs and the +prototype from one declaration, so they agree with each other by construction and only the library could disagree — +and nothing had a second opinion to disagree with. Now clang is asked for a JSON AST dump of the header (shelled out, +never libclang — the dependency plan.org rejected; Zig has since left it too, for Aro) and both halves are compared +against it. + +**The evidence.** Against raylib 5.5, the version whose `.so` `vendor/raylib/link` names: **all 16 `defstruct`s and +all 172 hand-written `declare-c` agree exactly.** Against the 5.1-dev header also installed on this machine, ten real +differences — nine functions that version lacks and one that gained a parameter — so picking the wrong header is +loud. Both comparisons run at build time and stop the build; verified by permuting `Texture2D` and by putting `f64` +where raylib says `float`, which is the hazard `BUILT.md` names and says only a test can catch. + +**Costs, measured, because they decide the remaining question.** Release build +4ms warm — `Reach.link` already drops +a wrapper nothing reachable calls, confirmed on the wasm32 case it exists for with 256 extra declarations in play. +Redefinition 31.0ms → 46.5ms. Dev build +333ms cold, once per session, since `Build.shared` compiles no C. Reading the +header is cached (64ms → 17ms), keyed like the object cache; the cache was built against a measurement, not a guess. + +**Opt-in on purpose.** `vendor/raylib/headers` is `?${FLAN_RAYLIB_H}`. "A build needs libraylib linkable and not +raylib-devel installed" is a property chosen deliberately, and requiring a header would take it from everyone to give +the check to whoever has one. Unset means off; set-and-wrong is an error naming the path. + +Worth knowing before touching it: + +- **The import is bounded by the package's own `defstruct`s**, not by a curated list. A function mentioning a struct + the package has not described is refused with that reason. Of raylib's 581 functions, 256 import, 153 are refused, + 172 are already bound by hand and left alone. Widening the binding is a `defstruct`, not a list edit. +- **No `defstruct` is generated, and that is load-bearing.** Generate them and the header becomes the authority on + layout, and checking the package's layouts against it would be comparing the header with itself — which is exactly + why `BUILT.md` rejected a `_Static_assert` as circular. Keeping them hand-written is what makes the check a second + source. +- **A refusal is a demotion, not a drop** — Zig's `failDecl`, which `Load.refuse_hidden` already implemented for + `main`. `rl/get-gamepad-name` is a name that exists, cannot be had, and says why at the use site. +- **`declare-c` and `declare` are untouched and still win.** A C symbol the package binds by hand is not imported, so + the escape hatch is the override. +- **`test/headers/sample.h`** is the importer's table — one function per decision, committed, no raylib needed. The + raylib acceptance case skips without `FLAN_RAYLIB_H`; that one does not. + +Two things that are *not* done, and are 6a and 6b in DISCUSS.md: whether the header stays a build-time read or becomes +a committed generator (`flan import-c` already prints the lines, so it costs nothing more to switch), and whether the +172 hand-written lines migrate. Neither is blocked on correctness. The 15.5ms on redefinition is the argument for the +first; needing the header at every build — vendoring raylib.h or requiring raylib-devel — is the argument on the +second. + +One smaller thing found and worth not re-deriving: an enum parameter imports as `i32`, because the header says +`KeyboardKey` and nothing tells the importer the package calls that `Key`. The ABI is identical, the face is worse, +and it is why `(rl/key-down? :space)` keeps its hand-written line. + ### Landed 2026-09-12 — six tracks, one session Six agents in parallel worktrees. Kept short on purpose; the reasoning that outlives the change is in `BUILT.md` or in diff --git a/lib/shim.ml b/lib/shim.ml index b9bfbee..6ced73d 100644 --- a/lib/shim.ml +++ b/lib/shim.ml @@ -66,8 +66,14 @@ A [_Static_assert] on [sizeof] and [offsetof] was considered and left out: both sides of it would come from the same field list, so it would check this module's arithmetic against clang's and say nothing about the library. - What would convert the trusted half into a checked one is including the - real header when one is installed, and that is not built. *) + What converts the trusted half into a checked one is reading the real + header, and that is built: [Cimport] asks clang for a JSON dump of one + and compares both halves against it — every [defstruct] against the + header's record, and every [declare-c] against the header's signature. + Nothing in this module changed for it. The refusals below still raise, + which is right for a signature a human named; the importer makes the + same judgements and merely skips instead, since one returned + [const char *] must not kill a header of five hundred functions. *) let fail = Loc.fail