The compiler embeds, and the sharpest objection was not there at all

Item 12 asked five questions. All five come back clean, and the answer is
feasible with no obstacle that argues for porting the compiler.

The one that mattered most was signals, and it turned out to be a non-question:
OCaml 5.2 on Linux/amd64 installs no signal handlers at all -- fifteen swept at
four moments, every one SIG_DFL, and a plain ocamlopt executable behaves the
same, so embedding changes nothing. OCaml 5 checks the stack limit explicitly
instead of arming a guard page, so the SIGSEGV the break loop wants was never
taken. The break loop can have it outright and Stack_overflow still works. That
is measured on one platform only, and the entry says so: macOS/arm64 is the one
claim here that must be re-run rather than trusted.

The rest: -output-complete-obj links with dune uninvolved and the existing C
stubs intact, no symbol collides across the four .c files and libasmrun, the
game keeps the main thread while a C-created listener calls into OCaml after
caml_c_thread_register, and a compaction left an 8 MiB arena byte-for-byte
untouched. Startup is 0.6ms and the compiler adds 4.14MB -- which makes the
merged dev build smaller than today's daemon alone, and there is one of it
rather than two.

Written as item 14 with the numbers, the three ways the GC assumption would
break, what the spike does not cover, and the order the real work goes in. The
backend is left exactly where item 13 put it.

DISCUSS.md

@@ -559,3 +559,164 @@ way an SBCL executable can. The dev/release split is a build flag, not an archit
 **And one correction to keep:** OCaml's own wasm support is irrelevant to any of this. OCaml is the compiler's
 implementation language; Flan programs reach wasm through LLVM. The two only meet if the *compiler* should run in a
 browser, which is not a goal.
+
+## 14. The embedding spike, answered: OCaml 5.2 goes into a Flan dev build, and nothing objects
+
+Item 12's brief, run. **Feasible.** No obstacle was found that argues for porting the compiler, and the one expected to
+be sharpest — signals — turned out not to exist on the platform measured.
+
+The apparatus is `spike/embed/`: three shell scripts and nine small sources, deliberately not a dune target, driving
+`ocamlfind` and `clang` by hand against the `flan.cmxa` dune already builds. `bash spike/embed/run.sh` reproduces
+everything below; `sig.sh`, `symbols.sh` and `merged.sh` each answer a question on their own. Nothing under `spike/` is
+wired into the build, and `dune test --root .` is green either side of it.
+
+### The headline: one binary, and it compiles itself
+
+`spike/embed/merged.sh` builds a single executable out of, in one `clang` link:
+
+- the emitted Flan program (`test/programs/edn.flan`, `--dev`, `@main` renamed to `flan_program_main`),
+- `runtime/flan_rt.c`, `runtime/flan_dev.c`, `vendor/agent/flan_agent.c`,
+- and the entire OCaml compiler as one `-output-complete-obj` object.
+
+It runs. A C `main()` holds the main thread and runs the Flan program there; `caml_startup` happens on a `pthread`
+beside it, next to where `flan_agent_start` already puts its listener. The compiler inside the binary then compiles
+`edn.flan` — the source the program itself was built from — and emits 336,579 bytes of LLVM IR. That single result
+answers questions 1 and 2 together; the ladder of smaller probes underneath it is support, not evidence in its own
+right.
+
+Nothing is wired up. The two halves share an address space and do not speak to each other. That is the point: the
+question was whether they *can*, not what they would say.
+
+### Question 1 — does OCaml link into a native binary here?
+
+Yes, and with less friction than expected.
+
+- `ocamlopt -output-complete-obj` is the one to use, not `-output-obj`: it bundles the runtime, so there is no hunt for
+  `libasmrun`. The final link needs `-lm -lpthread -ldl` and, on 5.2, **`-lzstd`** — 5.x's marshaller is compressed,
+  and the missing `ZSTD_*` symbols are the first thing a naive link fails on. That is the whole of the surprise.
+- **dune is not in the way, because it does not have to be involved.** dune builds `lib/flan.cmxa` as it does today;
+  the `-output-complete-obj` step consumes that artifact afterwards. No dune rule had to change, and `lib/dune` and
+  `bin/dune` are untouched. A real merge would want a dune rule to drive that step, but the spike shows the artifact
+  boundary is clean, which is the part that could have failed.
+- **The existing C stubs come through.** `lib/dynload_stubs.c` was taken verbatim from the unmerged `9e0ae3a` dlopen
+  branch and compiled into the same object; from inside the embedded runtime, `flan_mem_alloc`/`poke`/`peek`
+  round-trip correctly and `dlopen`+`dlsym` work. `-output-complete-obj` carries a `foreign_stubs`-shaped C file
+  through without special handling.
+- **No symbol collides** (`symbols.sh`). Flan's own C — `flan_rt.c`, `flan_dev.c`, `flan_agent.c`, `dynload_stubs.c` —
+  defines 211 symbols; `libasmrun.a` defines 1,379; the intersection is empty, and the four Flan files do not collide
+  with each other either. Worth checking rather than assuming: those four are compiled into *two different processes*
+  today, and merging puts them in one link for the first time.
+
+### Question 2 — threads, and who owns the main loop
+
+**OCaml 5.2 is confirmed multicore**, and `Domain.recommended_domain_count ()` reports 16 here. A spawned domain does
+real work in parallel with the main thread. The objection that would have been fatal is gone.
+
+The macOS shape works, and it was tested as the thing that matters rather than as "can OCaml use threads":
+
+- `caml_startup` can be called from a **C-created, non-main pthread**, while `main()` goes on to a loop it does not
+  leave. `harness4.c` runs a 298-frame mock game loop on the main thread that never once enters OCaml.
+- **A second C thread — one the runtime never created, which is exactly the agent's listener — can call into OCaml**
+  after `caml_c_thread_register()`, bracketed by `caml_acquire_runtime_system`/`caml_release_runtime_system`. It
+  compiled the same program successfully from that thread. This is the specific capability the merged design needs
+  from `flan_agent.c`, and it exists.
+
+The spike could not test macOS. Nothing here is macOS-specific — the inversion is a portable pthread arrangement — but
+it is a Linux measurement.
+
+### Question 3 — signals
+
+**On Linux/amd64, OCaml 5.2 installs no signal handlers at all.** Not SIGSEGV, not SIGINT, not SIGFPE, not SIGPIPE,
+nothing. The expected conflict does not exist.
+
+`sig.sh` sweeps fifteen signals from inside the runtime at four moments — before `caml_startup`, at module init, from
+inside a spawned domain, and after `Domain.join` — and every one is `SIG_DFL`. The reading has to be taken from inside
+OCaml rather than from C after `caml_startup` returns, because OCaml 5 starts domains later; the first pass got this
+wrong and read `SIG_DFL` for the wrong reason. A plain `ocamlopt` executable was built as a control and behaves
+identically, so embedding changes nothing about signals.
+
+The reason is structural, not incidental: OCaml 5 detects stack overflow with an explicit stack-limit check and calls
+`caml_raise_stack_overflow` directly, rather than with a guard page and a SIGSEGV handler. `nm` on `libasmrun.a` shows
+`sigaltstack` referenced but no SIGSEGV handler defined, which matches.
+
+So **the break loop can take `SIGSEGV` outright, and it does not have to install first or last** — order does not
+matter when there is nothing to displace. Measured directly: with the break loop's handler installed, deep OCaml
+recursion still raises `Stack_overflow` normally, and a genuine fault at `0x10` reaches the break loop's handler with
+`si_addr` correct. Chaining was implemented and compared; it is unnecessary here, but `harness5b.c` keeps it, because
+it is what the merged build should do on any platform where the sweep comes back non-empty.
+
+**The caveat, and it is the one thing in this report to re-check rather than trust:** this is `x86_64-pc-linux-gnu`
+only. macOS/arm64 OCaml 5 was not testable here, and item 11 raises signals in the same breath as macOS. Re-run
+`sig.sh` there before relying on it. `flan_agent.c` needs nothing from this either way — it sends with `MSG_NOSIGNAL`
+throughout rather than depending on a SIGPIPE disposition.
+
+### Question 4 — the GC and raw memory
+
+Confirmed, and the confirmation is narrow on purpose. An 8 MiB arena was filled with a checkable pattern and 64 raw
+interior pointers taken into it; OCaml then allocated 26.4 million words, took 7 major collections and a full
+`Gc.compact ()`. Afterwards: the arena base is unmoved, **0 of 8,388,608 bytes altered**, 0 of 64 interior pointers
+invalidated.
+
+What that proves is that the collector traces its own roots and foreign memory is invisible to it. Arenas, `Vec`s and
+`Map`s are safe because OCaml never learns they exist.
+
+**What would break the assumption**, stated so it is not rediscovered the hard way:
+
+1. Storing an OCaml `value` in Flan memory — an arena, a `Vec`, a global — across any allocation. The collector will
+   move the block and will not update that word, because it is not a root. `caml_register_global_root` (or the
+   generational one) is the only way that is legal, and it is a rule the merged design has to hold: **the boundary
+   passes pointers and scalars, never `value`s into Flan storage** — the same rule `dynload_stubs.c` already states
+   for its own reason.
+2. A `value` held in a C local across a call that allocates, without `CAMLparam`/`CAMLlocal`. Ordinary OCaml-FFI
+   discipline; it applies to every stub the merge adds.
+3. Long C work on the compiler thread without releasing the runtime system — which corrupts nothing but stalls
+   whichever domains want a stop-the-world. `llc` and `ld` are `exec`s and would want `caml_release_runtime_system`
+   around them.
+
+### Question 5 — what it costs
+
+| | bytes |
+|---|---|
+| `edn.flan`, release build | 71,824 |
+| `edn.flan`, dev build, as built today | 114,296 |
+| the same dev build with the whole compiler linked in | 4,257,624 |
+| **what the compiler adds** | **~4.14 MB** |
+
+**Startup: `caml_startup` takes 0.58–0.72 ms** across six runs — the runtime coming up and every module initialiser in
+the compiler running. Measured with `clock_gettime` around the call itself, not `time(1)` on the process, because exec
+and dynamic linking are paid today anyway.
+
+The size reads as 37x, and that framing is misleading. **A dev session today runs two binaries, and the daemon alone is
+4,815,368 bytes.** The merged dev build is *smaller than today's compiler process by itself*, and there is one of it
+instead of two. Sub-millisecond startup and ~4 MB is not a cost worth designing around.
+
+One more number, recorded because item 13's step 3 will want it and for no other reason: **a full in-process compile of
+`edn.flan` — read, parse, load, typecheck, emit — is 12.1–12.5 ms**, warm and cold alike, with `llc` and `ld` excluded
+because they are separate processes. That is where the remaining cost sits once transport is gone. It argues for
+nothing; item 13 says the backend is decided at step 4 on a measurement taken at step 3, and this is not that
+measurement.
+
+### What this does not answer
+
+- **The agent's handlers are a port, not a recompile.** `harness4.c` proves the *pattern* — a C-created listener
+  thread can register with the runtime and call OCaml. It does not port `flan_agent.c`'s handlers, which today answer
+  requests out of the program's own memory and would instead be calling into the compiler.
+- **Crash isolation is gone by construction.** Not a finding; item 11 already accepts it knowingly. A bad pointer
+  through the FFI takes the session, and conditions still catch everything the *language* signals. Noted only so this
+  entry stands alone.
+- **macOS, for signals and for the main-thread inversion.** See above.
+- **The backend.** Untouched deliberately.
+
+### The order the real work goes in
+
+1. **Make the `-output-complete-obj` step a dune rule**, producing the compiler-as-object that a dev build links. This
+   is the only build-system work, and `lib/dune`/`bin/dune` did not need changing to prove it.
+2. **Land the `dynload_stubs.c` branch** (`9e0ae3a`, currently reverted). The merged build needs the same
+   pointer-and-scalar boundary, and it is already written.
+3. **Invert the startup**: the Flan program keeps `main()`, and `flan_agent_start` also brings up the OCaml runtime on
+   its side thread. `merged_main.c` is the sketch.
+4. **Port the agent's handlers** from answering out of the program's memory to calling the compiler directly — this is
+   where the socket, the wire protocol, the 4K result cap, the seqlock, the snapshot copying, the generation stamping
+   and the render-thunk-per-inspection all get deleted. It is the bulk of the work and the whole of the prize.
+5. **Keep `llc` + `ld` + `dlopen` exactly as they are.** Item 13's third option. Nothing here argues against it.
+6. **Measure what is left.** Then, and only then, item 13 step 4.