diff --git a/DISCUSS.md b/DISCUSS.md index e39b469..da80d9a 100644 --- a/DISCUSS.md +++ b/DISCUSS.md @@ -559,3 +559,164 @@ way an SBCL executable can. The dev/release split is a build flag, not an archit **And one correction to keep:** OCaml's own wasm support is irrelevant to any of this. OCaml is the compiler's implementation language; Flan programs reach wasm through LLVM. The two only meet if the *compiler* should run in a browser, which is not a goal. + +## 14. The embedding spike, answered: OCaml 5.2 goes into a Flan dev build, and nothing objects + +Item 12's brief, run. **Feasible.** No obstacle was found that argues for porting the compiler, and the one expected to +be sharpest — signals — turned out not to exist on the platform measured. + +The apparatus is `spike/embed/`: three shell scripts and nine small sources, deliberately not a dune target, driving +`ocamlfind` and `clang` by hand against the `flan.cmxa` dune already builds. `bash spike/embed/run.sh` reproduces +everything below; `sig.sh`, `symbols.sh` and `merged.sh` each answer a question on their own. Nothing under `spike/` is +wired into the build, and `dune test --root .` is green either side of it. + +### The headline: one binary, and it compiles itself + +`spike/embed/merged.sh` builds a single executable out of, in one `clang` link: + +- the emitted Flan program (`test/programs/edn.flan`, `--dev`, `@main` renamed to `flan_program_main`), +- `runtime/flan_rt.c`, `runtime/flan_dev.c`, `vendor/agent/flan_agent.c`, +- and the entire OCaml compiler as one `-output-complete-obj` object. + +It runs. A C `main()` holds the main thread and runs the Flan program there; `caml_startup` happens on a `pthread` +beside it, next to where `flan_agent_start` already puts its listener. The compiler inside the binary then compiles +`edn.flan` — the source the program itself was built from — and emits 336,579 bytes of LLVM IR. That single result +answers questions 1 and 2 together; the ladder of smaller probes underneath it is support, not evidence in its own +right. + +Nothing is wired up. The two halves share an address space and do not speak to each other. That is the point: the +question was whether they *can*, not what they would say. + +### Question 1 — does OCaml link into a native binary here? + +Yes, and with less friction than expected. + +- `ocamlopt -output-complete-obj` is the one to use, not `-output-obj`: it bundles the runtime, so there is no hunt for + `libasmrun`. The final link needs `-lm -lpthread -ldl` and, on 5.2, **`-lzstd`** — 5.x's marshaller is compressed, + and the missing `ZSTD_*` symbols are the first thing a naive link fails on. That is the whole of the surprise. +- **dune is not in the way, because it does not have to be involved.** dune builds `lib/flan.cmxa` as it does today; + the `-output-complete-obj` step consumes that artifact afterwards. No dune rule had to change, and `lib/dune` and + `bin/dune` are untouched. A real merge would want a dune rule to drive that step, but the spike shows the artifact + boundary is clean, which is the part that could have failed. +- **The existing C stubs come through.** `lib/dynload_stubs.c` was taken verbatim from the unmerged `9e0ae3a` dlopen + branch and compiled into the same object; from inside the embedded runtime, `flan_mem_alloc`/`poke`/`peek` + round-trip correctly and `dlopen`+`dlsym` work. `-output-complete-obj` carries a `foreign_stubs`-shaped C file + through without special handling. +- **No symbol collides** (`symbols.sh`). Flan's own C — `flan_rt.c`, `flan_dev.c`, `flan_agent.c`, `dynload_stubs.c` — + defines 211 symbols; `libasmrun.a` defines 1,379; the intersection is empty, and the four Flan files do not collide + with each other either. Worth checking rather than assuming: those four are compiled into *two different processes* + today, and merging puts them in one link for the first time. + +### Question 2 — threads, and who owns the main loop + +**OCaml 5.2 is confirmed multicore**, and `Domain.recommended_domain_count ()` reports 16 here. A spawned domain does +real work in parallel with the main thread. The objection that would have been fatal is gone. + +The macOS shape works, and it was tested as the thing that matters rather than as "can OCaml use threads": + +- `caml_startup` can be called from a **C-created, non-main pthread**, while `main()` goes on to a loop it does not + leave. `harness4.c` runs a 298-frame mock game loop on the main thread that never once enters OCaml. +- **A second C thread — one the runtime never created, which is exactly the agent's listener — can call into OCaml** + after `caml_c_thread_register()`, bracketed by `caml_acquire_runtime_system`/`caml_release_runtime_system`. It + compiled the same program successfully from that thread. This is the specific capability the merged design needs + from `flan_agent.c`, and it exists. + +The spike could not test macOS. Nothing here is macOS-specific — the inversion is a portable pthread arrangement — but +it is a Linux measurement. + +### Question 3 — signals + +**On Linux/amd64, OCaml 5.2 installs no signal handlers at all.** Not SIGSEGV, not SIGINT, not SIGFPE, not SIGPIPE, +nothing. The expected conflict does not exist. + +`sig.sh` sweeps fifteen signals from inside the runtime at four moments — before `caml_startup`, at module init, from +inside a spawned domain, and after `Domain.join` — and every one is `SIG_DFL`. The reading has to be taken from inside +OCaml rather than from C after `caml_startup` returns, because OCaml 5 starts domains later; the first pass got this +wrong and read `SIG_DFL` for the wrong reason. A plain `ocamlopt` executable was built as a control and behaves +identically, so embedding changes nothing about signals. + +The reason is structural, not incidental: OCaml 5 detects stack overflow with an explicit stack-limit check and calls +`caml_raise_stack_overflow` directly, rather than with a guard page and a SIGSEGV handler. `nm` on `libasmrun.a` shows +`sigaltstack` referenced but no SIGSEGV handler defined, which matches. + +So **the break loop can take `SIGSEGV` outright, and it does not have to install first or last** — order does not +matter when there is nothing to displace. Measured directly: with the break loop's handler installed, deep OCaml +recursion still raises `Stack_overflow` normally, and a genuine fault at `0x10` reaches the break loop's handler with +`si_addr` correct. Chaining was implemented and compared; it is unnecessary here, but `harness5b.c` keeps it, because +it is what the merged build should do on any platform where the sweep comes back non-empty. + +**The caveat, and it is the one thing in this report to re-check rather than trust:** this is `x86_64-pc-linux-gnu` +only. macOS/arm64 OCaml 5 was not testable here, and item 11 raises signals in the same breath as macOS. Re-run +`sig.sh` there before relying on it. `flan_agent.c` needs nothing from this either way — it sends with `MSG_NOSIGNAL` +throughout rather than depending on a SIGPIPE disposition. + +### Question 4 — the GC and raw memory + +Confirmed, and the confirmation is narrow on purpose. An 8 MiB arena was filled with a checkable pattern and 64 raw +interior pointers taken into it; OCaml then allocated 26.4 million words, took 7 major collections and a full +`Gc.compact ()`. Afterwards: the arena base is unmoved, **0 of 8,388,608 bytes altered**, 0 of 64 interior pointers +invalidated. + +What that proves is that the collector traces its own roots and foreign memory is invisible to it. Arenas, `Vec`s and +`Map`s are safe because OCaml never learns they exist. + +**What would break the assumption**, stated so it is not rediscovered the hard way: + +1. Storing an OCaml `value` in Flan memory — an arena, a `Vec`, a global — across any allocation. The collector will + move the block and will not update that word, because it is not a root. `caml_register_global_root` (or the + generational one) is the only way that is legal, and it is a rule the merged design has to hold: **the boundary + passes pointers and scalars, never `value`s into Flan storage** — the same rule `dynload_stubs.c` already states + for its own reason. +2. A `value` held in a C local across a call that allocates, without `CAMLparam`/`CAMLlocal`. Ordinary OCaml-FFI + discipline; it applies to every stub the merge adds. +3. Long C work on the compiler thread without releasing the runtime system — which corrupts nothing but stalls + whichever domains want a stop-the-world. `llc` and `ld` are `exec`s and would want `caml_release_runtime_system` + around them. + +### Question 5 — what it costs + +| | bytes | +|---|---| +| `edn.flan`, release build | 71,824 | +| `edn.flan`, dev build, as built today | 114,296 | +| the same dev build with the whole compiler linked in | 4,257,624 | +| **what the compiler adds** | **~4.14 MB** | + +**Startup: `caml_startup` takes 0.58–0.72 ms** across six runs — the runtime coming up and every module initialiser in +the compiler running. Measured with `clock_gettime` around the call itself, not `time(1)` on the process, because exec +and dynamic linking are paid today anyway. + +The size reads as 37x, and that framing is misleading. **A dev session today runs two binaries, and the daemon alone is +4,815,368 bytes.** The merged dev build is *smaller than today's compiler process by itself*, and there is one of it +instead of two. Sub-millisecond startup and ~4 MB is not a cost worth designing around. + +One more number, recorded because item 13's step 3 will want it and for no other reason: **a full in-process compile of +`edn.flan` — read, parse, load, typecheck, emit — is 12.1–12.5 ms**, warm and cold alike, with `llc` and `ld` excluded +because they are separate processes. That is where the remaining cost sits once transport is gone. It argues for +nothing; item 13 says the backend is decided at step 4 on a measurement taken at step 3, and this is not that +measurement. + +### What this does not answer + +- **The agent's handlers are a port, not a recompile.** `harness4.c` proves the *pattern* — a C-created listener + thread can register with the runtime and call OCaml. It does not port `flan_agent.c`'s handlers, which today answer + requests out of the program's own memory and would instead be calling into the compiler. +- **Crash isolation is gone by construction.** Not a finding; item 11 already accepts it knowingly. A bad pointer + through the FFI takes the session, and conditions still catch everything the *language* signals. Noted only so this + entry stands alone. +- **macOS, for signals and for the main-thread inversion.** See above. +- **The backend.** Untouched deliberately. + +### The order the real work goes in + +1. **Make the `-output-complete-obj` step a dune rule**, producing the compiler-as-object that a dev build links. This + is the only build-system work, and `lib/dune`/`bin/dune` did not need changing to prove it. +2. **Land the `dynload_stubs.c` branch** (`9e0ae3a`, currently reverted). The merged build needs the same + pointer-and-scalar boundary, and it is already written. +3. **Invert the startup**: the Flan program keeps `main()`, and `flan_agent_start` also brings up the OCaml runtime on + its side thread. `merged_main.c` is the sketch. +4. **Port the agent's handlers** from answering out of the program's memory to calling the compiler directly — this is + where the socket, the wire protocol, the 4K result cap, the seqlock, the snapshot copying, the generation stamping + and the render-thunk-per-inspection all get deleted. It is the bulk of the work and the whole of the prize. +5. **Keep `llc` + `ld` + `dlopen` exactly as they are.** Item 13's third option. Nothing here argues against it. +6. **Measure what is left.** Then, and only then, item 13 step 4.