flan/plan.org

#+TITLE: Flan — Design Plan
#+DATE: 2026-09-10

* Specs
Two documents are normative and are settled ahead of implementation. Anything in
this plan that contradicts them is out of date.
- [[file:spec-memory.md][spec-memory.md]] — ownership, the four container types,
  copies and moves, assignable places, generics without type classes, function
  values.
- [[file:spec-conditions.md][spec-conditions.md]] — the six hard cases of
  conditions/restarts: what ~signal~ returns, no-handler behaviour, restart
  signatures, name shadowing, cleanup during a transfer, and crossing
  compiler-generated and foreign frames.

* What Flan is
A minimal Lisp for game development. Clojure's brackets and a small slice of its
API, C's memory and value model. No mandatory GC.

Not a Common Lisp, not a Clojure. In one line: *Odin with a Lisp frontend and a
live REPL.*

** References
- *Odin* — closest existing language. LLVM backend, manual memory, no GC, ships
  amd64 + arm64 + wasm32 (~js_wasm32~, ~wasi_wasm32~, ~freestanding_wasm32~).
  Proves this exact pipeline. Take directly:
  - ~context.allocator~ / ~context.temp_allocator~ (already this plan's design;
    ~free_all(context.temp_allocator)~ per frame /is/ the frame arena)
  - ~$T~ compile-time parametric polymorphism — monomorphisation without a heavy
    type system
  - ~#soa~ struct-of-arrays syntax
  - fixed arrays with component-wise ops and swizzles (~[4]f32~, ~v.xyzw~),
    built-in ~matrix~ type
  - ~defer~, tagged unions, ~distinct~ types, bit sets
  - ~vendor:raylib~
  Flan diverges by adding what Odin deliberately lacks: s-expressions, macros,
  conditions/restarts, hot reload, interactive development.
- *SBCL* — indirection cells for redefinition, conditions/restarts, the break loop.
- *Janet* — reference only; its immutable/mutable split is rejected (see Data model).
- *Carp* — statically typed Lisp with inference, no GC (ownership-based), compiles
  to C, aimed at games. Closest precedent for the /language/ shape, as Odin is for
  the /implementation/ shape.
- *jank* — cautionary; see Hot reload.

* Non-goals
- Numeric tower (no bignums, rationals, complex)
- Full CLOS/MOP (metaclasses, method combination, ~slot-missing~, arbitrary
  ~change-class~), ~format~, pathnames, streams, sequences-over-anything, CL reader
- Clojure's lazy seqs, JVM interop, ~core.async~
- Persistent collections, and immutable collection types generally (see Data model)
- Live-image development at SBCL's level
- Consoles

* Memory — no GC
Allocators all the way down; ~malloc~ hidden behind them.

| Tier      | Strategy                        | Cost           |
|-----------+---------------------------------+----------------|
| Frame     | arena, bulk reset each frame    | free           |
| Entities  | pool + generational handles     | free           |
| Subsystem | region, freed wholesale         | free           |
| Dev/REPL  | leaks by design, reset on reload| dev only       |

- Allocator is part of the calling convention, so a refcounted allocator can be
  added later without a language change.
- Generational handles instead of pointers for cross-references: a stale
  reference is detectable, not undefined behaviour.
- Symbols and code live in a permanent arena that only grows.

** Why no persistent collections
Structure sharing destroys clear ownership, which is the only thing that forces a
collector. Replaced by value structs that copy on assignment — see Data model.

** The consequence that drives everything
Plain ~struct~ means *no object headers*. Every struct is exactly its C layout,
arrays are C arrays, so there is no marshalling layer and no wrapper allocation.
The future managed ~class~ facility below is deliberately a separate kind of
value: it has identity, metadata, and an implementation-defined representation.

FFI is still a boundary — ownership, who allocates, string and slice
representation, struct padding, callbacks into Flan, error handling, and the
platform ABI all remain real work. What C layout buys is that the *data* crosses
for free. See spec-conditions.md §6 for the one hard rule: a restart transfer
cannot cross a foreign frame.

* Data model
No immutable collection types. Value semantics plus ~const~, as in C/Zig/Odin —
not Janet's tuple/struct vs array/table split, which was an answer to a GC'd
world.

- Four container types, distinct in both type and ownership — see spec-memory.md,
  which is normative:
  | Notation    | Layout          | Assignment       | Owns |
  |-------------+-----------------+------------------+------|
  | ~[n T]~     | inline, n items | copies           | no   |
  | ~[T]~       | ptr+len         | copies the view  | no   |
  | ~(Vec T)~   | ptr+len+cap     | *moves*          | yes  |
  | ~(Map K V)~ | open addressing | *moves*          | yes  |
  ~Vec~ and ~Map~ are monomorphic on element type and record their allocator. Not
  a Lua-style array/hash hybrid — that is what makes Lua's layout and performance
  unpredictable.
- Map keys initially use compiler-provided structural equality and hashing for
  integers, enums, strings, fixed arrays and value structs; pointers, slices and
  owning containers are excluded. A map is homogeneous, and empty construction
  names its types: ~(let [enemies (map-new string Enemy)] ...)~. ~get~
  returns ~(Option V)~; ~put~ is the ~()~-returning upsert. See
  spec-memory.md for the deferred move-aware operations.
- Operations: ~get~, ~put~, ~remove~, ~push~, ~pop~, ~at~, ~len~, ~update~.
  Copying is explicit: ~(clone m)~, and owning containers move rather than copy on
  assignment. No ~!~ convention — nothing is immutable, so it
  would carry no information. No ~assoc~; it only existed as the copy-returning form.
- ~const~ qualifier on references and slices: compile-time contract that a
  callee will not mutate. Zero runtime cost.
- Value structs copy on assignment — but only *value* structs. Ownership is
  structural: a struct is a value type iff every field is, so one ~Vec~ field
  makes it move-only. This is what keeps "copies on assignment" from meaning a
  shallow copy that aliases owned storage. Deep copies are always explicit:
  ~(clone x)~. Value structs are the snapshot / undo / replay story; they need no
  separate type.
- Literals live in read-only memory.
- Struct literals name fields: ~(Cursor {.src src .pos 0})~. *Omitted fields are
  zeroed*, as in Odin — the same rule as a declaration with no initialiser, so
  ~(Cursor {.src src})~ is complete and means ~pos~ is 0.
- *Zero is initialisation (ZII), with an opt-out.* No initialiser means
  all-bytes-zero. ~(defvar buf [65536 u8] uninit)~ skips it, exactly as Odin's
  ~---~ does, for a large buffer that is about to be overwritten. ~uninit~ is
  greppable and rare by design; reading an ~uninit~ value before writing it is
  undefined, and dev builds poison the memory so the bug is loud.
- Slices as ptr+len, non-owning; raw pointers ~(Ptr T)~ with explicit ~deref~;
  visible casts. ~resolve~ yields ~(Ptr a)~ where ~deref~ yields a value — that is
  how a matched struct is mutated in place rather than as a copy.
- Flat and SoA arrays.
- No implicit allocation anywhere in the core.

** Managed classes — planned, deliberately separate from structs
~struct~ remains Flan's default: fixed-layout values suitable for stack storage,
arrays, SIMD/SoA, FFI, and hot paths. It cannot acquire fields while live values
exist. A future ~class~ is for long-lived gameplay/editor objects that need
identity, extensibility, and live schema changes:

#+begin_src lisp
(defclass Enemy
  (x f32) (y f32) (health i32))

(defmethod update ((e Enemy) dt) ...)
#+end_src

Classes require a managed allocation strategy and runtime class/shape metadata,
but *not necessarily a tracing GC*. The initial likely choices are a world or
session arena, pool allocation behind generational ~(Handle T)~ values, or an
explicitly owned region. A small tracing GC confined to class instances remains
an option if cyclic graphs prove burdensome; it never changes ~struct~ layout or
the C ABI.

Generic functions attach operations independently of class definitions. This
addresses the expression problem for dynamic game objects: a later module can
add either ~(defmethod draw ((e Enemy)) ...)~ or a new ~Capsule~ class plus its
~draw~ method without editing the original class or a central closed ~match~.
Start with exact-class, single-argument dispatch. Multi-argument dispatch is
the later extension for interactions such as ~(collide Player Enemy)~. Method
specificity/ambiguity rules are required before inheritance or multiple dispatch
is enabled.

Class redefinition has a stable class identity and numbered layouts. Changing
slots leaves existing instances at their old layout until an explicit migration
at a reload/frame boundary. Added fields use declared defaults; removed fields
are discarded; renamed or type-changed fields require user migration code. This
is the small, eager and debuggable analogue of CLOS
~make-instances-obsolete~ / ~update-instance-for-redefined-class~, rather than
surprising lazy mutation on field access:

#+begin_src lisp
(redefine-class Enemy
  (x f32) (y f32) (health i32) (shield i32))
(defmethod migrate ((old Enemy@1) (new Enemy@2))
  (set (.shield new) (.health old)))     ; a default would not have been right
(migrate-instances Enemy)
#+end_src

The precise class syntax, inheritance, storage strategy, and migration API are
not frozen. Do not add classes until ordinary ~struct~, ~Handle~, and reload
semantics are working.

Immutability also serves the optimiser: a value known never to be mutated can be
copied into registers and stack-allocated freely. Mutability is what forces heap
identity.

* Types
*Statically typed.* Not forced — tagging and boxing give dynamic typing without a
collector, as Forth-lineage and refcounted dynamic languages show. It is chosen,
and the reason is that dynamic typing would require paying a tag word on every
value, which is exactly the header cost dropping the GC was meant to avoid. Under
static typing the tag is paid only where it is asked for: in ~any~, in ~Error~,
and on a managed ~class~ instance. An ordinary ~struct~ never carries one.

- Types are mandatory; *inference* makes them feel optional. Annotate function
  signatures, infer locals — Odin/Zig/Rust ergonomics.
- Signatures are annotated as inline name/type pairs, as in ~defstruct~ and a
  ~restart-case~ clause: ~(defn area [s Shape] f32 ...)~. A ~let~ is not one of
  them — a local is inferred from its initialiser and takes no annotation at
  all. No separate ~declare~ form —
  ~declare~ is kept only where there is no body (forward declarations, FFI).
- Annotations at function boundaries are unavoidable, because compile-time
  overloading is incompatible with full inference. Locals are inferred.
- The return type is always written, and a function that returns nothing writes
  ~()~ — a real zero-sized type with one value, not C's ~void~. Generic code over
  it works, so there is no ~Action~/~Func~ split. The slot was optional once and
  the parser guessed between a return type and a body form from a table of type
  names; the guess was silently wrong twice, so the slot is mandatory.
- Every type notation reads as exactly one data item: ~[f32]~, ~[4 f32]~,
  ~(Vec f32)~, ~(Map string i32)~, ~(Ptr World)~, ~(Fn [f32] bool)~,
  ~(Option $t)~, ~(Handle $t)~. The map spelling was ~{string i32}~ once and is
  not any more: braces in type position are refused by name. The ~where~ clause
  below is a brace form at the head of a body, so keeping the brace type would
  have put ~(defn f [...] {string i32} {:where ...} body)~ in the language — two
  braces in a row meaning different things.
- ~i8..i64~, ~u8..u64~, ~f32~, ~f64~ as real machine types; wrapping arithmetic.
- Vector width 128-bit. Fixed arrays with component-wise ops and swizzles.
- Parametric polymorphism by monomorphisation (Odin's model, no type classes, no
  HKTs). A type variable is written ~$t~ wherever a *type* goes — parameter,
  return type, or nested as ~[$t]~ or ~(Vec $t)~ — and bare ~t~ wherever a type's
  *name* is an argument in expression position: ~(vec-new t)~, ~(map-new t i32)~,
  ~(pool-new t)~, and the cast ~(t x)~. This is what makes
  ~map!~/~filter~/~reduce~ and the monomorphic containers work; it collapsed the
  prelude's per-type families into one function each.
  A generic body is checked *abstractly*, with nothing substituted, so ~=~, ~<~,
  ~+~ and ~hash~ over an unconstrained variable are rejected at the definition
  rather than at whichever call site first instantiates it. **This is not Odin's
  rule** — Odin checks a polymorphic body only per instantiation — and the two
  decisions are independent even though this line once bundled them.
  What makes the rejection liveable is a ~where~ clause over compile-time type
  predicates, which is Odin's (~core/slice/slice.odin:289~,
  ~where intrinsics.type_is_ordered(T)~). It is written as a Clojure-style map at
  the head of the body — ~{:where (ordered? $t)}~, or a vector for more than one,
  ~{:where [(copyable? $t) (copyable? $u)]}~ — on the precedent of Clojure's
  ~{:pre ... :post ...}~, and because a bare ~{}~ in expression position is
  already refused so nothing else it could be. ~sort!~ declares ~ordered?~ of its
  variable, the abstract pass then allows ~<~ in the body, and each instantiation
  checks the concrete type satisfies the predicate and refuses the call site if it
  does not. There are *five* predicates — ~ordered?~, ~equal?~, ~hashable?~,
  ~numeric?~, ~copyable?~ — against Odin's forty-one, and they entail one another
  in one direction, so one clause usually does: ~numeric?~ gives ~ordered?~,
  ~ordered?~ gives ~equal?~, and any of the four gives ~copyable?~.
  ~copyable?~ has no Odin counterpart, because Odin has no move semantics and a
  ~$T~ there never has to answer the question. *A type variable is move-only by
  default* and ~copyable?~ is the opt-out: whether a variable is move-only is not
  decidable abstractly — ~i32~ at one instantiation, ~(Vec i32)~ at the next — so
  the checker takes the stricter rule, which can only refuse a program that would
  have been fine and never admit one that double-frees. The prior art is Rust's
  ~T: Copy~, differing in that the compiler answers the question rather than a
  user implementing a trait.
  ~hashable?~ is what lets a variable *key a map*: without it the type
  ~(Map $t i32)~ is refused where it is written, and with it the refusal moves to
  the call site that names an unhashable key.
  This is not a type class and the difference is worth keeping straight: a type
  class carries *implementations* selected per instance and extensible by anyone,
  and needs dictionaries and coherence rules. A predicate carries nothing — it
  gates a builtin the compiler already has. The ceiling is that no one can supply
  a user-defined ~<~; every operation the prelude and the containers need is a
  primitive, so it does not bind. Compile-time interfaces, if they are ever
  wanted, come after the base checker is stable.
  ~println~ is a compiler-provided exception and is on an explicit allow-list of
  forms the abstract pass defers to instantiation: whether a printer exists for a
  variable is only decidable once it is substituted. The ~Map~ operations over a
  variable key are the other member, for the same reason — the hash and the
  equality are concrete symbols chosen from the concrete key type — and that one
  is paid for by ~hashable?~ being written in the signature, so the refusal it
  moves still lands against a requirement somebody wrote down. Keep the list
  short: every member moves a refusal from the definition to a call site, which
  is what the abstract rule exists to avoid.
- Function values split three ways (spec-memory.md): ~(Fn [T1 T2] R)~ is a plain
  pointer with no environment — the only kind that crosses FFI or sits in a reload
  cell; a *non-escaping* ~fn~ captures enclosing locals by value into a stack
  environment, which is what ~reduce~ callbacks and ~handler-bind~ handlers use;
  an *escaping* closure needs a heap environment and is still an open decision.
- No monads, no HKTs, no type classes. Effects are direct; error handling is
  conditions plus ~Option~ and ~or-else~. Monadic sequencing, if ever wanted, is a
  macro.
- ~any~ is an explicit opt-in tagged union, for heterogeneous containers and debug
  printing. ~Error~ is a second, specialised erased carrier for fallible results;
  neither adds a header to ordinary values.
- Tagged unions, ~distinct~ types, bit sets.

** Consequences
- For ~struct~s, multimethods are *compile-time overload resolution*, not runtime
  dispatch. Genuine runtime dispatch is on an explicit enum or tagged union.
  Planned managed classes instead use generic-function dispatch as described in
  "Managed classes"; this is an intentionally separate facility.
- Conditions still work: a condition is a struct, the signal channel is a tagged
  union, ~handler-case~ type matching resolves at compile time.
- The REPL works — the compiler knows each site's type and emits the right printer,
  as in the OCaml and Haskell REPLs.
- Macros are unaffected; they run on syntax before typing.

* Semantics kept
** Conditions and restarts
The headline feature. Four operators: ~handler-bind~, ~handler-case~,
~restart-case~, ~invoke-restart~. No condition class hierarchy — struct types plus
predicate matching. *Operational semantics: spec-conditions.md, which is
normative.* The ~500-line estimate below was for the operators alone and does not
include the explicit transfer lowering, which is compiler work.

~signal~ does not mean "fail". It means: here is something notable, here is the
data, and here are the ways I know how to continue. Restarts are a menu; a handler
installed by an outer caller reads the data and picks one — or picks nothing and
returns, in which case the signaller simply carries on.

| Handler does        | Behaviour you get       |
|---------------------+-------------------------|
| returns normally    | accumulate and continue |
| invokes a restart   | recover / retry / substitute |
| unwinds (~handler-case~) | try/catch          |

*Why this replaces most error types.* A condition signalled deep in a call stack
never appears in intermediate signatures. Nothing to thread, no ~From~ conversions,
no ~anyhow~ equivalent. Compare Rust, where every layer must name every error type
it passes through.

*Accumulation.* A handler that records and invokes a ~continue~-style restart gives
error collection with no applicative or monad. This is how CL compilers report every
error in one pass.

*Costs nothing when unused.* The handler stack is a linked list of stack-allocated
frames: ~handler-bind~ is a couple of stores, ~signal~ with no handler is a null
check. No allocation, safe inside a frame loop.

*Condition objects live on the signalling frame's stack*, since nothing unwinds
before the handler runs. Because conditions are value structs, accumulating one into
an outliving array copies it. A pointer-to-condition would dangle.

*Under static typing.* Restarts are dynamically scoped and named, so
~(invoke-restart 'skip-form)~ cannot be fully checked at compile time. Accept a
runtime error initially; a statically tracked restart set (as Zig tracks error sets)
is a nice-to-have, not a blocker. ~signal~ has type ~()~, ~invoke-restart~ and
~error~ have type ~Never~, and every restart clause shares one type with the
~restart-case~ body — so a ~restart-case~ in value position needs a fall-through
that produces the type or diverges.

*Unwinding* is only the /transfer/ — invoking an outer restart. Lowered explicitly,
see Compilation.

** Error handling, layered
1. ~Option~ for expected absence: lookup miss, empty collection, end of stream.
2. Conditions for exceptional failure where a caller may have a recovery policy.
3. ~Result~ where failure should be visible in the signature (parsers, fallible
   pure functions). The ordinary form is ~(Result T Error)~: an erased built-in
   error carrier, not an external ~anyhow~-style library. A precise ~(Result T E)~
   remains available where it materially helps.

~Error~ is a small value carrying a type id, pointer to an allocator-owned typed
payload, source location, context and a rendered summary. It preserves structured
fields for debugger/editor inspection while the summary is stable to print and
send over the wire. This makes ~(Vec Error)~ the standard heterogeneous diagnostic
collection. Wrapping an error uses the current implicit allocator; metadata is
out-of-band, so ordinary values keep their header-free C layout.

Rule of thumb: if you can name the one correct recovery at the point of failure,
return a ~Result~. If the answer is "depends who is calling", signal a condition.

Mechanisms — two unwrap operators, because they are two different things (Zig's
split):
- ~try~ — unwrap ~Ok~, else early-return ~Err~
- ~some~ — unwrap ~Some~, else early-return ~None~
- ~(ok-or opt err)~ / ~(ok res)~ — conversions, always *explicit*. ~try~ does not
  accept an ~Option~ in a ~Result~-returning function. Converting a precise error
  to ~Error~ is explicit too; there is no trait-based conversion search.
- ~some->~ (short-circuiting thread), ~or-else~, ~if-let~
- ~errdefer~ — cleanup on the failure path only, pairs with arenas

Async composes by nesting, no new mechanism: ~(try (await (http-get url)))~ —
~await~ unwraps the task, ~try~ unwraps the ~Result~ inside it.

~try~ and ~some~ are macros expanding to early returns, and early return is the
explicit non-local-exit lowering, so all of this compiles identically on native and
wasm32.

*Restarts go at the resync point, once* — the loop over top-level forms in a
parser, not inside every function below it. Intermediate frames stay silent about
restarts for the same reason they stay silent about conditions.

No monads, no HKTs. Chaining that would want do-notation is a macro.

*Async coupling:* with a state-machine transform the handler stack must live in the
/task/ state, not thread-local, or a handler established before an ~await~ is out of
scope after resumption. Cheap if designed in, painful later.

** Multimethods
Compile-time overload resolution on argument types. No precedence lists, no method
combination, no MOP, no runtime dispatch table — see Types.

** Macros
- ~&env~ equivalent: macros seeing the lexical environment (names of locals in
  scope). No longer urgent — its "decide now" status came from the
  instrumentation-based step debugger, which is cut (see Tooling). Milestone 5.
- Macros are initially deliberately non-hygienic, in the Common Lisp/Clojure
  style. A macro uses explicit ~gensym~ for introduced bindings; there is no
  syntax-object hygiene system. Lexically local macros (~macrolet~-style) wait
  for a concrete use case.

* Host language
*OCaml.* The compiler only; the runtime and stdlib are Flan with a few C
primitives, and are never bootstrapped away.

The LLVM question does not bear on this: the release backend emits LLVM IR *as
text* and shells out to ~clang~, so no language needs LLVM bindings, and C++ or
Rust buy nothing here. What the choice actually turns on is that milestones 2–5
are a reader, a typed IR, a checker and the code generators behind it — variants
and exhaustive pattern matching, which is the one domain where OCaml is not a
preference but a clear win. There is also a menhir lexer/parser already started
in ~old-ocaml/~.

The honest alternative is Rust, and it wins on exactly one axis: if the compiler
is a language you will not enjoy maintaining in three months, that outweighs
being 30% shorter. Nothing technical breaks either way.

*Self-hosting is not a goal* and must not drive this. It appears nowhere in the
build sequence. For a game language it buys dogfooding at the price of a second
compiler to maintain forever. Choose as if the host language is permanent.

* Milestone-2 primitives
The runtime provides these; everything else is written in Flan. Keeping the
list short is the whole strategy — it is what makes the LLVM backend and the
wasm32 target cheap, because a primitive is the only thing implemented twice.

| Primitive              | Notes |
|------------------------+-------|
| ~argv~                 | ~[string]~, borrowed, never freed |
| ~write-stdout~         | takes ~[u8]~; the ONE output primitive |
| ~exit~                 | ~i32~ status |
| ~len~ ~at~ ~slice~     | on fixed arrays and slices |
| ~bytes~                | ~string~ → ~[u8]~, a view, no copy |
| ~bytes->f64~ ~bytes->i64~ | and the inverses, for printing |
| ~addr~                 | address of a place |
| arithmetic, comparison, casts | per machine type |

Printing is *not* a primitive: ~write-stdout~ is the one output primitive and
~println~ and ~print~ are written over it. Both are compiler-provided — the
checker walks the concrete type at the call site and emits the printer for it
(lib/render.ml, shared with the REPL's ~C-x C-e~). This is intentionally not
user-defined overload resolution — ordinary values remain untagged, and there
is nothing to dispatch on at run time — which is why it did not have to wait
for generics as this line once said it would. Generic instantiations are
concrete types by the time the checker sees them, so they need nothing further;
~any~ and ~Error~ are the remaining case, and they carry the metadata their
dynamic printers need.

*Entry point.* ~(defn main [args [string]] i32)~. Both the parameter and the
return type are optional: omitting ~args~ means the program ignores argv,
a return type of ~()~ means an exit status of 0. sand.flan uses
the short form, calc-me the long one.

*RNG is ours, not libc's.* ~rand-f32~ is a seeded PRNG implemented in Flan
(xoshiro or PCG), because a grid hash is only a regression test if the sequence
is byte-identical on native and wasm32. Decided here rather than at milestone 4,
since a headless deterministic sand run is the cross-target test.

* Modules
There *are* modules — Odin calls them packages, and so does Flan. What is removed
is Clojure's ~ns~ form: no path that must mirror the directory, no
~:require~/~:refer~/~:as~/~:import~ vocabulary, no per-file namespace object.

- *The directory is the package.* Every file in a directory shares one top-level
  scope. Files in a package do not import each other, and top-level names are
  order-independent, so mutually recursive functions need no forward declaration.
- *The package declaration is optional*, which is the one place Flan diverges
  from Odin — Odin requires ~package foo~ as the first line of every file and
  requires it to agree across the directory. Flan infers the package name from
  the directory name, and ~(package parser)~ is written only when the name must
  differ from the directory (a directory named ~flan-parser~, a scratch directory
  with a name that is not an identifier). When present it must agree across the
  directory, as in Odin.
- *A loose file in ~~/scratch/~ is a package of one.* No project file, no
  manifest, no declaration. Open it, connect the REPL, start working. The
  ceremony budget for "new file, running REPL" is zero — this is the requirement
  the whole scheme is designed around, and it is why the declaration is optional
  rather than mandatory.
- *Cross-package:* ~(import rl "vendor:raylib")~, and everything from it is
  qualified ~rl/foo~. One form, one meaning, no unqualified-import mode.
- Collections in the path (~vendor:~, ~core:~) are Odin's, and are just
  root-directory aliases.

* Compilation
*One evaluator and three paths.* The split is not dev-vs-release; it is
/does this code have a frame budget/. There is no interpreter: open decision #7
is settled the other way from how this section was first written, and BUILT.md's
"Why there is no interpreter" carries the reasoning. Compiling is the only way a
form is ever run, so there is no second evaluator that could disagree with the
first about what a program means.

#+begin_src
expression eval:    flan → typed IR → .ll → llc → ld -shared → dlopen → call
                                                                       ~19ms  (MEASURED)
dev redefinition:   the same path, ending in a cell store rather than a call
release build:      flan → typed IR → .ll → clang --target={native,wasm32}
#+end_src

*Hard requirement: eval is immediate.* Not "fast enough for a build" — immediate,
because the whole point of the live loop is that you see the result. 19ms is
around one frame at 60fps and under the ~50ms threshold where a response stops
feeling instantaneous. The rule that buys it: *never invoke the ~clang~ driver on
the dev path.*

*Expression eval* — ~C-x C-e~, calling a function, inspecting a var, running a
test — is compiled like everything else, into its own shared object, which is
then loaded and called. What made an interpreter look necessary was the
assumption that this had to be sub-millisecond; the measurement below is that
the compiled route is already inside the threshold, and the one thing an
interpreter would have bought is an oracle the hand-written acceptance table
supplies instead.

*Dev redefinition* — ~C-c C-c~ on a function inside a running game — has an 8ms
frame budget to respect. It recompiles the one function, links it, and does the
atomic indirection-cell store. This is what the Hot reload section has always
described, and it is the same machinery expression eval uses, one step further
on.

*Release* is whole-program AOT with direct calls and no cells.

*A second code generator, not a second evaluator.* ~lib/x86.ml~ emits x86-64
machine code directly and is selected with ~--x86~; it exists because ~llc~ is
most of the 19ms above. It is a different route from the same typed IR to the
same observable behaviour, not a different semantics, and what holds it to that
is ~spike/x86/survey.sh~: every program in the corpus is built both ways and
byte-compared on stdout, stderr and exit status. At the time of writing that is
103 MATCH, 0 DIFFER, 0 refused by name. It handles conditions, bounds checks,
indirection cells, redefinition modules and DWARF line tables; what it does not
have, and must not grow, is an aggregate classifier — an ~--x86~ host therefore
takes ~--x86~ modules and an LLVM host takes LLVM ones, and ~lib/build.ml~
refuses the crossed pair by name.

** Measured redefinition latency
Single function, x86-64, clang 20.1.8, 20 iterations each:

| Step                                     | Per call | Dev path? |
|------------------------------------------+----------+-----------|
| ~clang -shared~ (driver: compile + link)  | 52.0ms   | *no*      |
| ~clang -c~ (driver: compile only)         | 22.5ms   | no        |
| ~llc -filetype=obj~                       | 13.9ms   | yes       |
| ~ld -shared~ from the ~.o~                | 2.4ms    | yes       |

~llc~ + ~ld~ + ~dlopen~ ≈ *16ms*. The clang driver is the cost, not codegen —
it forks a second process and re-does argument and target resolution. Codegen
itself barely scales with function size: 721 lines of IR took 16.7ms against
13.9ms for 8 lines, because ~10ms is ~llc~ startup loading libLLVM. A realistic
redefined function lands in the same 15–17ms.

This is why an in-process ORC JIT is not needed. It would take 16ms to ~3ms; the
difference is below perception, and the price is a version-pinned libLLVM and C++
linkage from the host language, forever.

** Redefinition must not stutter the running game
The 16ms is /not/ paid by the game thread. ~llc~ and ~ld~ are already separate
processes running on other cores. What the game process does is smaller:

| Step                             | Cost      | Game thread? |
|----------------------------------+-----------+--------------|
| ~llc~, ~ld~                       | 16.3ms    | no, separate processes |
| ~dlopen~ the new ~.so~            | ~0.1–1ms  | *must not be* |
| atomic store into the cell        | ns        | yes, and free |

Two design choices are load-bearing, and neither is automatic:

1. *~dlopen~ happens on the reload thread.* It mmaps, relocates and takes the
   loader lock; off-thread it blocks nobody, because the game thread is not doing
   dynamic linking. Load with ~RTLD_NOW~ so lazy PLT resolution cannot ambush the
   game thread on a later first call.
2. *Publish at a frame boundary, in a batch.* This matters more than the
   threading. Storing each cell the moment it is ready lets the game observe a
   half-applied redefinition — two functions that changed together applied one
   frame apart, or a function swapped mid-frame with half the entities already
   updated by the old code. Instead the reload thread stages the complete set of
   new pointers and sets a flag; the game loop tests the flag once at the top of
   the frame and does N stores. One relaxed atomic load per frame when nothing
   changed.

Residual cost: the first call into new code page-faults and misses i-cache. Tens
of microseconds, not visible.

*** Dev architecture: daemon plus agent
- *Compiler daemon*, a separate process: the OCaml frontend, the nREPL server,
  and the ~llc~/~ld~ invocations. Editors talk to this.
- *Reload agent*, linked into the game binary: a socket listener, ~dlopen~, and
  the frame-boundary cell publisher. A few hundred lines, and no OCaml runtime
  in the game.

This is why *the dev runtime is multithreaded* — it needs the reload thread. That
is settled, and is independent of whether the /language/ exposes threads, which
is still open decision #4.

This is also what settled the interpreter question: if the agent can ~dlopen~ and
call anything in under 20ms, then even "eval this expression against live game
state" is a compiled ~.so~, and no interpreter is needed inside the game process.
That is the route ~C-x C-e~ actually takes.

** Why LLVM IR as text
| | text ~.ll~ → ~clang~ | libLLVM bindings | emit C |
|---+---+---+---|
| Build dependency | a ~clang~ on PATH | matching libLLVM, version-pinned, C++ linkage | any C compiler |
| Breaks on LLVM upgrade | no | routinely | no |
| Debuggable | ~.ll~ is readable | print-from-memory | readable, but lies about origin |
| In-process JIT | *no* | yes (ORC) | no |
| Control of layout / ABI / tail calls | full | full | poor |

The only column text loses is the JIT one, and the measurement above shows the
loss is ~13ms — below perception. ORC remains addable later behind the same typed
IR without touching the language, but nothing currently argues for it.

** The interpreter, and why there is not one
An interpreter could never have run sand, and that was the first half of the
argument. 200 × 280 = 56,000 cells, scanned by ~game-update~ and again by
~game-draw~ — ~112,000 interpreted cell-visits per frame against an 8.3ms budget
at 120fps. At an optimistic 100ns per visit (environment allocation, argument
binding, two index computations, a compare) that is 11ms before ~settle~,
~paint~, or a single raylib call. Expect 20–30fps. Milestone 4's interactive
acceptance test was always going to run on the compiled dev build.

*** Settled: the compiled path is the only backend
This was open decision #7 and it is closed. Compiled redefinition measured at
~19ms is perceptually instant for expression eval too, so the one thing an
interpreter was still wanted for went away; the instrumentation-based step
debugger that wanted it is cut (see Tooling); and milestone 3 did not need it as
an oracle either, because the acceptance table is hand-written and the table /is/
the oracle. What is bought by dropping it is the standing obligation: two
evaluators must agree on observable behaviour forever, and every divergence is a
bug that reproduces in only one of them. BUILT.md's "Why there is no interpreter"
records the decision; ~lib/expand.ml~ states it at the top of the file, because
macros are where the absence stopped being free — a macro has to run at compile
time and there is nothing to interpret it with, so the compiler compiles it into
a shared object and loads it with ~dlopen~ into its own process.

Consequences applied elsewhere in this document: milestone 2's "interpreted calls
per second" exit criterion is dropped, and the host ABI moved onto the critical
path in its place.

The paths that remain share the frontend and the typed IR and must agree on
observable behaviour. That agreement is what the acceptance programs test, and
for the two code generators it is tested byte for byte.

- Non-local exit lowered *explicitly* (result propagation + branch targets), not
  via platform unwinding. Same on both targets, no dependency on the WASM
  exception-handling proposal. Every Flan function carries the transfer channel;
  uniformity keeps indirect calls and hot reload ABI-safe. A later optimisation may
  narrow that cost, but it cannot change the ABI.
- Stdlib written *in Flan*, not the host language. A few hundred primitives per
  backend, everything else on top. This is what keeps a second backend cheap.

* Targets
- *Desktop*: AOT to native, x86-64 and arm64. Primary development target.
- *WASM*: AOT build artifact only, *not interactive-first* — no REPL, no hot
  reload, no debugger there. That is not the same as untested: every runtime or
  ABI feature ships with automated wasm32 tests in CI from the first one, because
  a divergence found at ship time is a rewrite.
- Host binary links raylib natively; web links raylib via emscripten.

** One narrow host ABI, implemented twice
The portability risk is the host interface, not the language. Divergence points:
- Filesystem — pack assets, one abstraction, never touch paths
- Threads — decide now; retrofitting is worse than the reverse
- Blocking — browser main thread cannot block
- Audio/input/window — constrain to the raylib subset identical on both

* Dev vs release builds
Deliberately different.

|         | Dev                       | Release    |
|---------+---------------------------+------------|
| Backend | LLVM, or ~--x86~          | LLVM/clang |
| Calls   | indirection cells | direct  |
| Code    | never freed       | static  |
| Frames  | shadow stack      | none    |
| Structs | version word      | none    |
| Reload  | yes               | no      |

Build and run the release config regularly, not just at ship time.

* Hot reload
Every cross-function call goes through an *indirection cell*; body redefinition
is one atomic pointer store. A top-level function value is a stable trampoline
over the same cell, never a pointer to a specific body, so callers holding it
observe a body redefinition too. *Old code is never unloaded*, so a thread
mid-execution finishes safely in the old version.

A signature-changing redefinition creates a new internal function version and
trampoline. Newly compiled callers use it; existing callers and stored function
values safely retain the old version. The session immediately warns at each
tracked old caller site, and recompiling that caller either updates it or gives a
normal type error. This preserves live running code without hiding stale calls.

This is the fix for jank issue #947 (segfault redefining a running loop's
function plus its callee — their JIT relinks and unloads under a running thread).

** What redefinition cannot do
Patch a mid-execution frame and continue at the same PC — its register
allocation belongs to the old compilation. No implementation does this. "Resume"
means re-entering from an established restart point.

* Tooling
Server speaks *nREPL* (bencode over socket) — the transport and the core ops
(~eval~, ~load-file~, ~describe~, ~interrupt~) are genuinely reusable, and that is
what the ~500–1000 lines buys. It does *not* buy CIDER/Conjure/Calva
compatibility: their useful operations assume Clojure-shaped vars, namespaces,
nses-of-symbols and middleware. Treat "speaks nREPL" as milestone 7a and "an
editor client that is pleasant" as a separate milestone 7b. In the dev runtime:
- eval string in package; compile form/file with source locations
- completion, arglist, describe, find-definition
- backtrace + restarts; interrupt

** Emacs client
Focused client, ~3–5k lines. Do *not* fork CIDER (~30k lines elisp, deeply
Clojure-coupled) — reference it. ~clojure-mode~-derived major mode, overlay
rendering, hydra for stepping bindings (~transient~ is the maintained
alternative).

** Debugging: two layers, two protocols
Do not extend nREPL with registers or memory. The protocol is the easy part —
adding an op is trivial; /implementing/ it means becoming a debugger (ptrace,
trap handling, frame unwinding), duplicating an enormous existing project.

| Capability                               | DAP (lldb-dap + dape) | nREPL |
|------------------------------------------+-----------------------+-------|
| Registers, memory, native stack           | yes                   | never |
| Breakpoints, watchpoints, stepping        | yes                   | —     |
| Eval Flan expressions with real semantics | no                    | yes   |
| Redefine a function and continue          | no                    | yes   |
| Invoke a restart                          | no                    | yes   |
| Pause /before/ unwinding                  | no                    | yes   |

Neither covers the other's column, so this is not a choice between them.

*DAP is nearly free.* No debug adapter is written: emit DWARF from the backend
and point ~lldb-dap~ at the binary; dape speaks to that. Both code generators do
— the hand-written one writes its compile unit, subprograms and line table out as
bytes, since ~.loc~ cannot work against a file whose instructions are ~.byte~
blobs, and what it does not describe is locals and types. lldb and gdb
both ship DAP interfaces already.

This is where /no object headers/ pays off a second time. Flan structs *are* C
structs — real machine types, no tag words, no boxing — so DWARF describes them
with no impedance mismatch and lldb shows a ~Cursor~ correctly with no plugin
and no formatters.

*** Reload and DAP
Mostly automatic. lldb watches the loader rendezvous structure, so a ~dlopen~'d
generation is noticed and its DWARF read; and lldb keeps breakpoints as
unresolved /specs/, re-resolving them against newly loaded modules, so a
breakpoint on ~sand.flan:85~ binds to the new function by itself.

*The gotcha is our own never-unload rule.* After N redefinitions there are N
copies of a function, each with DWARF claiming ~sand.flan:85~, so one breakpoint
resolves to N locations and stops in stale generations. Needs glue, and it is
small: give each generation a distinct source identity in its DWARF
(~sand.flan#7~), and have the reload agent disable breakpoint locations in
superseded modules on each reload. This is the one piece that cannot just be
wired up and left alone.

Not available at any price: if execution is paused /inside/ the old function,
redefinition does not move it — register allocation belongs to the old
compilation (see "What redefinition cannot do"). The current frame finishes in
the old code; the next call enters the new one. Also, dape reads source from
disk, so editing further after a generation was compiled drifts the highlighted
line from what is executing.

*** Cut: the instrumentation-based step debugger
Previously planned as CIDER's model — macroexpansion wrapping subforms with
breakpoints. Dropped. It is the most expensive piece of milestone 8
(instrumentation, an overlay protocol, ~&env~, a bespoke Emacs client) and
DAP+DWARF covers most of its use. What DAP cannot cover — pausing before the
stack unwinds with restarts available — comes from ~handler-bind~ over nREPL.

Consequences: *~&env~ loses its urgency* (its "decide now" status came from the
step debugger requiring it — now an ordinary milestone-5 decision), and *the
shadow stack shrinks* to serving only nREPL backtraces and restart enumeration.

Forcing the live-programming workflow into DAP would repeat the CIDER mistake in
the other direction: DAP's model is stop/step/inspect, with no vocabulary for
"invoke ~use-placeholder~ and resume". That stays on nREPL.

** Pause on exception
Better than CIDER's, because ~handler-bind~ has not unwound the stack. Dev-mode
global handler messages the editor and blocks with the full live stack and all
restarts available. Fix the function, resume via restart.

* Build sequence
Deliberately ordered so each step is runnable and the next one cannot start until
the previous checker is stable. The failure mode this exists to prevent is
building the whole live environment at once.

1. *Freeze the model.* spec-memory.md and spec-conditions.md — done before any
   code. Fixed arrays, non-owning slices, move-only ~Vec~/~Map~, allocators,
   ~Ptr~, explicit ~clone~; the six restart cases. /Done./
2. *Run calc-me.flan.* Reader, typed IR, checker, and a backend that can carry
   the program end to end. The exit criterion was once a measured interpreter
   throughput number; with no interpreter that criterion is gone and the narrow
   host ABI took its place on the critical path (see Compilation). Packages,
   structs, ~(Ptr T)~ + ~addr~, byte slices,
   ~at~/~len~, ~while~, ~set~ on the fixed place list, ~cond~, ~match~, ~Option~
   + ~some~, ~i32~/~u8~/~f64~, recursion, argv, stdout. No allocator, no ~Vec~,
   no generics, no user macros, no FFI, no window. Headless, so the acceptance
   test is a table of expression/result pairs.
3. *Emit LLVM IR and pass the same calc-me test AOT*, on native and wasm32 in CI.
   Both targets, one test table, one narrow host ABI (argv, stdout, exit). This
   is where the second target gets proven — while there is almost nothing to port.
   The hand-written x86-64 code generator is not on this path: it arrived later,
   as a second route to the same behaviour rather than a milestone of its own,
   and is held to the LLVM backend's output byte for byte (see Compilation).
4. *Run sand.flan.* Fixed 2-D arrays, ~dotimes~, ~defer~, and typed FFI to
   raylib including keyword→enum coercion. Acceptance test twice: headless (N
   frames, hash the grid — runnable in CI on both targets) and interactive at
   120 fps.
5. *Generics and macro expansion*, once the base checker is stable. ~defmacro~,
   ~&env~, hygiene. Until here, ~when~/~unless~/~until~/~cond~/~dotimes~ are
   special forms in the compiler.
6. *Allocators, ~Vec~/~Map~, ~Result~/~try~/~errdefer~, then conditions and
   restarts* against spec-conditions.md, with dedicated tests per numbered case.
7. *Hot reload* — indirection cells in dev builds, with signature generations
   and stale-caller warnings, plus the
   remaining compatibility limits written down and enforced: struct layout
   changes, live callbacks held by C, captured environments.
8. *Debugger, nREPL, async* — last, and 8 splits into transport (8a) and editor
   client (8b).

Milestones 1–4 are the project. Everything from 5 on is optional in the sense
that a language that stops there is still usable; nothing before 5 is.

Ordering note: sand cannot be first even though it is the better demo, because
it needs raylib FFI, keyword→enum coercion and a window before a single line of
it runs. calc-me needs argv and stdout.

* Runtime budget
Scoped to *milestones 1–3 only*. The earlier version of this table put the whole
plan — conditions, explicit transfer lowering, macros, hot reload, debugger
support — at 15–25k, which is not credible: each of those is architecture work
that touches the frontend, the IR and the backend at once.

| Piece                             | Lines  | Milestone |
|-----------------------------------+--------+-----------|
| Allocators                        | 2–3k   | 2         |
| Core data (fixed/slice/Vec/Map)   | 2–3k   | 2         |
| Reader + frontend + checker       | 5–8k   | 2         |
| LLVM IR lowering                  | 3–5k   | 2         |
| raylib FFI + host ABI ×2          | 1–2k   | 3         |
| *Subtotal, a language that runs sand* | 13–21k | |

Beyond that, estimated but not budgeted, because these are the parts that are
architecture rather than volume:

| Piece                                  | Note |
|----------------------------------------+------|
| Macroexpander + ~&env~                  | touches the reader and the checker |
| Generics / monomorphisation            | touches the whole checker |
| Conditions + explicit transfer lowering | frontend *and* IR *and* backend |
| Hot reload cells + compatibility rules  | changes the calling convention |
| Debugger instrumentation + nREPL        | needs ~&env~ and source locations |

Janet is 36k including a bytecode VM, and Janet has no static types, no
monomorphisation, no restarts and no reload.

* Open decisions
None of these block milestone 2. The milestone each one must be answered by is
marked.

1. *Host language: OCaml or Rust* — /settled: OCaml,/ and the compiler has been
   written in it since. See Host language for what the choice turned on.
2. Macro hygiene is settled for milestone 5: explicit ~gensym~, deliberately
   non-hygienic expansion, no local macros until a concrete use case appears.
3. Borrow checking and escaping frame-arena values. /Deferred; revisit after
   Vec/Map are useful in real programs./ The first implementation deliberately
   follows Zig/Odin: slices and pointers have explicit lifetime and
   reallocation contracts, with dev generation checks. A later lightweight
   provenance pass may catch obvious escaping/stale-borrow mistakes, but must
   not impose Rust-style lifetime annotations, automatic promotion, or an
   ECS-shaped object model.
4. Threads *in the language*: yes or no, decided before the host ABI. (The dev
   /runtime/ is multithreaded regardless — it needs a reload thread. Settled; see
   Compilation.) /Milestone 3, before the host ABI./
5. /Escaping/ closures. /Deferred until a concrete use case./ Settled in
   spec-memory.md:
   ~(Fn ...)~ is a bare pointer with no environment (callbacks, reload cells, FFI),
   and a *non-escaping* ~fn~ captures by value into a stack environment — which is
   what makes ~handler-bind~ handlers able to see enclosing locals, without which
   conditions are not worth building. Still open: a closure that is stored,
   returned, or pushed into a container. Which allocator owns its environment, and
   what happens when the frame arena resets? Do not design this speculatively;
   revisit it alongside the deferred provenance work only when it is needed.
6. Hot-reload compatibility rules. /Milestone 7./ A changed function signature
   creates a new version: new callers resolve it, old callers keep the old one,
   and the session warns at tracked stale caller sites. Still open: a changed
   *struct* layout with live values is rejected; a managed class layout has an
   explicit frame-boundary migration path, as described above. Still open: a
   function pointer already handed to C, a captured environment, and redefining
   a ~defvar~. Each needs an answer of the form "rejected", "accepted with a
   migration", or "accepted and the old code keeps running".
7. Does the interpreter survive milestone 3, or is the compiled path the only
   backend? /Settled: the compiled path is the only one, and there is no
   interpreter./ See Compilation, and BUILT.md's "Why there is no interpreter".
8. ~(Option a)~ /settled:/ an ordinary stdlib union with ~Some~/~None~; the
   compiler niche-optimises ~(Option (Ptr T))~ to a nullable pointer. The
   CL-vs-Clojure truthiness question is moot under static typing.

*Settled since the first draft* (see the specs):
- Module system → the directory is the package, Odin's model. No ~ns~ form. See
  Modules.
- Zero values → ZII by default, with an opt-out. A declaration with no
  initialiser is all-bytes-zero; ~(zeroed)~ re-zeroes something later; ~uninit~
  opts out for large buffers about to be overwritten. It is a ~memset~, not a
  ~memcpy~; zeroed globals live in BSS and cost nothing.
- Package declaration → optional, inferred from the directory name. See Modules.
- Keywords at typed call sites → yes. ~:space~ resolves at compile time against
  the parameter's enum type, ~rl/key-space~ names the same value, and a typo is
  a compile error checked against the enum's members. No runtime cost. Needs the
  FFI enum declared, so it lands with milestone 4.
- Dev redefinition latency → ~16ms, measured: ~llc~ + ~ld -shared~ + ~dlopen~,
  never the ~clang~ driver, ~dlopen~ off the game thread, cells published in a
  batch at a frame boundary. See Compilation.
- Dev backend → compiled, and only compiled. The interpreter that milestone 2
  was going to be written against was never needed and does not exist: expression
  eval is a compiled shared object like everything else, and a macro is the case
  that made the absence load-bearing rather than merely tidy. See Compilation.
- ~set~ on places → a fixed list of assignable forms, not ~setf~.
- Loop story → imperative ~while~/~until~/~dotimes~ with ~break~/~continue~ and
  ~return~; ~loop~/~recur~ only if it later earns its place. It did: both are
  built. ~break~ and ~continue~ take a label for the loop that is not the
  innermost, and ~loop~ is an *expression* whose value is its body's, with
  ~recur~ rebinding every name at once and jumping rather than calling.
  sand.flan is ported.
- Generic parameters → inferred at call sites, no explicit instantiation. No type
  classes. An operator over a variable with no ~where~ clause asserting it is
  rejected at the definition; a ~where~ predicate is what admits it.

* Unverified claims in this plan
- *Scope is the biggest risk, not any single feature.* A language, inference,
  monomorphisation, macros, an explicit-memory runtime, conditions/restarts, hot
  reload, a debugger, nREPL tooling, native and WASM — each is reasonable, the
  set is not one project. The build sequence above exists because of this; if
  something has to give, it gives from milestone 4 upward.
- Line-count estimates are extrapolations.
- LLVM → wasm32 with manual memory is validated by Odin shipping it; not yet
  validated for s-expression macros + conditions/restarts on top.