flan/plan.org
Joseph Ferano 93231e8c9e println, the structural printer, shared with the REPL
session.ml already had this: a compile-time walk over a Tast type that
emits the calls to print a value of it, handling every concrete type the
language has. It was dev-build-only and went to flan_dev_emit, and
prelude.ml justified the per-type print-* functions by saying a real
println had to wait for milestone 5 and generics. It did not. plan.org
specifies println as compiler-provided and per concrete type, which is
not overloading: there is nothing to dispatch on at run time and no
user-supplied printer to choose between, so no type variables appear.

The walk moves to render.ml, parameterised on an emitter and a slot
allocator. The emitter is five functions rather than five extern names
because the two sides are not both extern calls -- the REPL's are, and
stdout's compose a conversion with a write. The slot allocator differs
too: the REPL builds a thunk's frame, println takes slots from the
enclosing function being checked, once per call site.

Two runtime shims, both only reachable from the walk. flan_u64_to_bytes,
because routing u64 through the signed printer makes 0xFFFF...F read as
-1, which is the one way println could disagree with the REPL about a
value both can hold. flan_escape_bytes, so a string nested in a printed
structure is quoted and escaped -- same table as flan_dev_emit_str, noted
in both, because the REPL and println must not disagree about what a
struct looks like.

A string at top level prints raw and nested prints quoted. Not a conflict:
(println "hello") has to print hello, and a struct's string field has to
be distinguishable from the punctuation around it. The split is top-level
vs nested, so it lives in check.ml and not in the walk.

Found on the way: a field of an Option had no gep in emit.ml, so the
walk's Option arm had never run -- the REPL would have failed on one too.
Option is { i8, T } with no declared name, so its layout is now spelled
out. Nothing in the surface language reaches a field of an Option; the
printer does, to read the tag without unwrapping a None.

The print-* functions stay. They print without a newline, which println
cannot express -- slices.flan's show prints elements separated by spaces
-- and they are raw where print is structural.

println.flan covers every arm at -O0 and -O2: the u64, the raw/quoted
split, both Option arms, the depth and span caps, and the slice arm's
loop twice over plus once inside a dotimes, which is where per-call-site
slot allocation would show if it were per-iteration.
2026-09-12 04:55:42 +07:00

46 KiB
Raw Blame History

Flan — Design Plan

Specs

Two documents are normative and are settled ahead of implementation. Anything in this plan that contradicts them is out of date.

  • spec-memory.md — ownership, the four container types, copies and moves, assignable places, generics without type classes, function values.
  • spec-conditions.md — the six hard cases of conditions/restarts: what signal returns, no-handler behaviour, restart signatures, name shadowing, cleanup during a transfer, and crossing compiler-generated and foreign frames.

What Flan is

A minimal Lisp for game development. Clojure's brackets and a small slice of its API, C's memory and value model. No mandatory GC.

Not a Common Lisp, not a Clojure. In one line: Odin with a Lisp frontend and a live REPL.

References

  • Odin — closest existing language. LLVM backend, manual memory, no GC, ships amd64 + arm64 + wasm32 (js_wasm32, wasi_wasm32, freestanding_wasm32). Proves this exact pipeline. Take directly:

    • context.allocator / context.temp_allocator (already this plan's design; free_all(context.temp_allocator) per frame is the frame arena)
    • $T compile-time parametric polymorphism — monomorphisation without a heavy type system
    • #soa struct-of-arrays syntax
    • fixed arrays with component-wise ops and swizzles ([4]f32, v.xyzw), built-in matrix type
    • defer, tagged unions, distinct types, bit sets
    • vendor:raylib

    Flan diverges by adding what Odin deliberately lacks: s-expressions, macros, conditions/restarts, hot reload, interactive development.

  • SBCL — indirection cells for redefinition, conditions/restarts, the break loop.
  • Janet — reference only; its immutable/mutable split is rejected (see Data model).
  • Carp — statically typed Lisp with inference, no GC (ownership-based), compiles to C, aimed at games. Closest precedent for the language shape, as Odin is for the implementation shape.
  • jank — cautionary; see Hot reload.

Non-goals

  • Numeric tower (no bignums, rationals, complex)
  • Full CLOS/MOP (metaclasses, method combination, slot-missing, arbitrary change-class), format, pathnames, streams, sequences-over-anything, CL reader
  • Clojure's lazy seqs, JVM interop, core.async
  • Persistent collections, and immutable collection types generally (see Data model)
  • Live-image development at SBCL's level
  • Consoles

Memory — no GC

Allocators all the way down; malloc hidden behind them.

Tier Strategy Cost
Frame arena, bulk reset each frame free
Entities pool + generational handles free
Subsystem region, freed wholesale free
Dev/REPL leaks by design, reset on reload dev only
  • Allocator is part of the calling convention, so a refcounted allocator can be added later without a language change.
  • Generational handles instead of pointers for cross-references: a stale reference is detectable, not undefined behaviour.
  • Symbols and code live in a permanent arena that only grows.

Why no persistent collections

Structure sharing destroys clear ownership, which is the only thing that forces a collector. Replaced by value structs that copy on assignment — see Data model.

The consequence that drives everything

Plain struct means no object headers. Every struct is exactly its C layout, arrays are C arrays, so there is no marshalling layer and no wrapper allocation. The future managed class facility below is deliberately a separate kind of value: it has identity, metadata, and an implementation-defined representation.

FFI is still a boundary — ownership, who allocates, string and slice representation, struct padding, callbacks into Flan, error handling, and the platform ABI all remain real work. What C layout buys is that the data crosses for free. See spec-conditions.md §6 for the one hard rule: a restart transfer cannot cross a foreign frame.

Data model

No immutable collection types. Value semantics plus const, as in C/Zig/Odin — not Janet's tuple/struct vs array/table split, which was an answer to a GC'd world.

  • Four container types, distinct in both type and ownership — see spec-memory.md, which is normative:

    Notation Layout Assignment Owns
    [n T] inline, n items copies no
    [T] ptr+len copies the view no
    (Vec T) ptr+len+cap moves yes
    (Map K V) open addressing moves yes

    Vec and Map are monomorphic on element type and record their allocator. Not a Lua-style array/hash hybrid — that is what makes Lua's layout and performance unpredictable.

  • Map keys initially use compiler-provided structural equality and hashing for integers, enums, strings, fixed arrays and value structs; pointers, slices and owning containers are excluded. A map is homogeneous, and empty construction is type-directed: (defvar enemies (Map string Enemy) (map-new)). get returns (Option V); put is the `Unit`-returning upsert. See spec-memory.md for the deferred move-aware operations.
  • Operations: get, put, remove, push, pop, at, len, update. Copying is explicit: (clone m), and owning containers move rather than copy on assignment. No ! convention — nothing is immutable, so it would carry no information. No assoc; it only existed as the copy-returning form.
  • const qualifier on references and slices: compile-time contract that a callee will not mutate. Zero runtime cost.
  • Value structs copy on assignment — but only value structs. Ownership is structural: a struct is a value type iff every field is, so one Vec field makes it move-only. This is what keeps "copies on assignment" from meaning a shallow copy that aliases owned storage. Deep copies are always explicit: (clone x). Value structs are the snapshot / undo / replay story; they need no separate type.
  • Literals live in read-only memory.
  • Struct literals name fields: (Cursor {:src src :pos 0}). Omitted fields are zeroed, as in Odin — the same rule as a declaration with no initialiser, so (Cursor {:src src}) is complete and means pos is 0.
  • Zero is initialisation (ZII), with an opt-out. No initialiser means all-bytes-zero. (defvar buf [65536 u8] uninit) skips it, exactly as Odin's --- does, for a large buffer that is about to be overwritten. uninit is greppable and rare by design; reading an uninit value before writing it is undefined, and dev builds poison the memory so the bug is loud.
  • Slices as ptr+len, non-owning; raw pointers (Ptr T) with explicit deref; visible casts. resolve yields (Ptr a) where deref yields a value — that is how a matched struct is mutated in place rather than as a copy.
  • Flat and SoA arrays.
  • No implicit allocation anywhere in the core.

Managed classes — planned, deliberately separate from structs

struct remains Flan's default: fixed-layout values suitable for stack storage, arrays, SIMD/SoA, FFI, and hot paths. It cannot acquire fields while live values exist. A future class is for long-lived gameplay/editor objects that need identity, extensibility, and live schema changes:

(defclass Enemy
  (x f32) (y f32) (health i32))

(defmethod update ((e Enemy) dt) ...)

Classes require a managed allocation strategy and runtime class/shape metadata, but not necessarily a tracing GC. The initial likely choices are a world or session arena, pool allocation behind generational (Handle T) values, or an explicitly owned region. A small tracing GC confined to class instances remains an option if cyclic graphs prove burdensome; it never changes struct layout or the C ABI.

Generic functions attach operations independently of class definitions. This addresses the expression problem for dynamic game objects: a later module can add either (defmethod draw ((e Enemy)) ...) or a new Capsule class plus its draw method without editing the original class or a central closed match. Start with exact-class, single-argument dispatch. Multi-argument dispatch is the later extension for interactions such as (collide Player Enemy). Method specificity/ambiguity rules are required before inheritance or multiple dispatch is enabled.

Class redefinition has a stable class identity and numbered layouts. Changing slots leaves existing instances at their old layout until an explicit migration at a reload/frame boundary. Added fields use declared defaults; removed fields are discarded; renamed or type-changed fields require user migration code. This is the small, eager and debuggable analogue of CLOS make-instances-obsolete / update-instance-for-redefined-class, rather than surprising lazy mutation on field access:

(redefine-class Enemy
  (x f32) (y f32) (health i32) (shield i32))
(defmethod migrate ((old Enemy@1) (new Enemy@2))
  (set (.shield new) (.health old)))     ; a default would not have been right
(migrate-instances Enemy)

The precise class syntax, inheritance, storage strategy, and migration API are not frozen. Do not add classes until ordinary struct, Handle, and reload semantics are working.

Immutability also serves the optimiser: a value known never to be mutated can be copied into registers and stack-allocated freely. Mutability is what forces heap identity.

Types

Statically typed. Not forced — tagging and boxing give dynamic typing without a collector, as Forth-lineage and refcounted dynamic languages show. It is chosen, and the reason is that dynamic typing would require paying a tag word on every value, which is exactly the header cost dropping the GC was meant to avoid. Under static typing the tag is paid only where it is asked for: in any, in Error, and on a managed class instance. An ordinary struct never carries one.

  • Types are mandatory; inference makes them feel optional. Annotate function signatures, infer locals — Odin/Zig/Rust ergonomics.
  • Signatures are annotated as inline name/type pairs, as in let and defstruct: (defn area [s Shape] f32 ...). No separate declare form — declare is kept only where there is no body (forward declarations, FFI).
  • Annotations at function boundaries are unavoidable, because compile-time overloading is incompatible with full inference. Locals are inferred.
  • An omitted return type means Unit — a real zero-sized type with one value, not C's void. Generic code over it works, so there is no Action~/~Func split.
  • Every type notation reads as exactly one data item: [f32], [4 f32], (Vec f32), {string i32}, (Ptr World), (Fn [f32] bool), (Option a), (Handle a).
  • i8..i64, u8..u64, f32, f64 as real machine types; wrapping arithmetic.
  • Vector width 128-bit. Fixed arrays with component-wise ops and swizzles.
  • Parametric polymorphism by monomorphisation (Odin's model, no type classes, no HKTs). Lowercase type names are variables, Capitalized are concrete — no sigil. This is what makes map~/~filter~/~reduce and the monomorphic containers work. The price, made explicit in spec-memory.md: with no constraints, a type variable supports only what every type supports. =, <, + and hash over an unconstrained a are rejected, not silently instantiated — they are passed in as function values. println is the one compiler-provided exception: it selects a structural printer at each concrete instantiation. Compile-time interfaces, if they are ever wanted, come after the base checker is stable.
  • Function values split three ways (spec-memory.md): (Fn [T1 T2] R) is a plain pointer with no environment — the only kind that crosses FFI or sits in a reload cell; a non-escaping fn captures enclosing locals by value into a stack environment, which is what reduce callbacks and handler-bind handlers use; an escaping closure needs a heap environment and is still an open decision.
  • No monads, no HKTs, no type classes. Effects are direct; error handling is conditions plus Option and or-else. Monadic sequencing, if ever wanted, is a macro.
  • any is an explicit opt-in tagged union, for heterogeneous containers and debug printing. Error is a second, specialised erased carrier for fallible results; neither adds a header to ordinary values.
  • Tagged unions, distinct types, bit sets.

Consequences

  • For ~struct~s, multimethods are compile-time overload resolution, not runtime dispatch. Genuine runtime dispatch is on an explicit enum or tagged union. Planned managed classes instead use generic-function dispatch as described in "Managed classes"; this is an intentionally separate facility.
  • Conditions still work: a condition is a struct, the signal channel is a tagged union, handler-case type matching resolves at compile time.
  • The REPL works — the compiler knows each site's type and emits the right printer, as in the OCaml and Haskell REPLs.
  • Macros are unaffected; they run on syntax before typing.

Semantics kept

Conditions and restarts

The headline feature. Four operators: handler-bind, handler-case, restart-case, invoke-restart. No condition class hierarchy — struct types plus predicate matching. Operational semantics: spec-conditions.md, which is normative. The ~500-line estimate below was for the operators alone and does not include the explicit transfer lowering, which is compiler work.

signal does not mean "fail". It means: here is something notable, here is the data, and here are the ways I know how to continue. Restarts are a menu; a handler installed by an outer caller reads the data and picks one — or picks nothing and returns, in which case the signaller simply carries on.

Handler does Behaviour you get
returns normally accumulate and continue
invokes a restart recover / retry / substitute
unwinds (handler-case) try/catch

Why this replaces most error types. A condition signalled deep in a call stack never appears in intermediate signatures. Nothing to thread, no From conversions, no anyhow equivalent. Compare Rust, where every layer must name every error type it passes through.

Accumulation. A handler that records and invokes a continue-style restart gives error collection with no applicative or monad. This is how CL compilers report every error in one pass.

Costs nothing when unused. The handler stack is a linked list of stack-allocated frames: handler-bind is a couple of stores, signal with no handler is a null check. No allocation, safe inside a frame loop.

Condition objects live on the signalling frame's stack, since nothing unwinds before the handler runs. Because conditions are value structs, accumulating one into an outliving array copies it. A pointer-to-condition would dangle.

Under static typing. Restarts are dynamically scoped and named, so (invoke-restart 'skip-form) cannot be fully checked at compile time. Accept a runtime error initially; a statically tracked restart set (as Zig tracks error sets) is a nice-to-have, not a blocker. signal has type Unit, invoke-restart and error have type Never, and every restart clause shares one type with the restart-case body — so a restart-case in value position needs a fall-through that produces the type or diverges.

Unwinding is only the transfer — invoking an outer restart. Lowered explicitly, see Compilation.

Error handling, layered

  1. Option for expected absence: lookup miss, empty collection, end of stream.
  2. Conditions for exceptional failure where a caller may have a recovery policy.
  3. Result where failure should be visible in the signature (parsers, fallible pure functions). The ordinary form is (Result T Error): an erased built-in error carrier, not an external anyhow-style library. A precise (Result T E) remains available where it materially helps.

Error is a small value carrying a type id, pointer to an allocator-owned typed payload, source location, context and a rendered summary. It preserves structured fields for debugger/editor inspection while the summary is stable to print and send over the wire. This makes (Vec Error) the standard heterogeneous diagnostic collection. Wrapping an error uses the current implicit allocator; metadata is out-of-band, so ordinary values keep their header-free C layout.

Rule of thumb: if you can name the one correct recovery at the point of failure, return a Result. If the answer is "depends who is calling", signal a condition.

Mechanisms — two unwrap operators, because they are two different things (Zig's split):

  • try — unwrap Ok, else early-return Err
  • some — unwrap Some, else early-return None
  • (ok-or opt err) / (ok res) — conversions, always explicit. try does not accept an Option in a Result-returning function. Converting a precise error to Error is explicit too; there is no trait-based conversion search.
  • some-> (short-circuiting thread), or-else, if-let
  • errdefer — cleanup on the failure path only, pairs with arenas

Async composes by nesting, no new mechanism: (try (await (http-get url)))await unwraps the task, try unwraps the Result inside it.

try and some are macros expanding to early returns, and early return is the explicit non-local-exit lowering, so all of this compiles identically on native and wasm32.

Restarts go at the resync point, once — the loop over top-level forms in a parser, not inside every function below it. Intermediate frames stay silent about restarts for the same reason they stay silent about conditions.

No monads, no HKTs. Chaining that would want do-notation is a macro.

Async coupling: with a state-machine transform the handler stack must live in the task state, not thread-local, or a handler established before an await is out of scope after resumption. Cheap if designed in, painful later.

Multimethods

Compile-time overload resolution on argument types. No precedence lists, no method combination, no MOP, no runtime dispatch table — see Types.

Macros

  • &env equivalent: macros seeing the lexical environment (names of locals in scope). No longer urgent — its "decide now" status came from the instrumentation-based step debugger, which is cut (see Tooling). Milestone 5.
  • Macros are initially deliberately non-hygienic, in the Common Lisp/Clojure style. A macro uses explicit gensym for introduced bindings; there is no syntax-object hygiene system. Lexically local macros (macrolet-style) wait for a concrete use case.

Host language

OCaml. The compiler only; the runtime and stdlib are Flan with a few C primitives, and are never bootstrapped away.

The LLVM question does not bear on this: the release backend emits LLVM IR as text and shells out to clang, so no language needs LLVM bindings, and C++ or Rust buy nothing here. What the choice actually turns on is that milestones 25 are a reader, a typed IR, a checker and a tree-walking interpreter — variants and exhaustive pattern matching, which is the one domain where OCaml is not a preference but a clear win. There is also a menhir lexer/parser already started in old-ocaml/.

The honest alternative is Rust, and it wins on exactly one axis: if the compiler is a language you will not enjoy maintaining in three months, that outweighs being 30% shorter. Nothing technical breaks either way.

Self-hosting is not a goal and must not drive this. It appears nowhere in the build sequence. For a game language it buys dogfooding at the price of a second compiler to maintain forever. Choose as if the host language is permanent.

Milestone-2 primitives

The interpreter provides these; everything else is written in Flan. Keeping the list short is the whole strategy — it is what makes the LLVM backend and the wasm32 target cheap, because a primitive is the only thing implemented twice.

Primitive Notes
argv [string], borrowed, never freed
write-stdout takes [u8]; the ONE output primitive
exit i32 status
len at slice on fixed arrays and slices
bytes string[u8], a view, no copy
bytes->f64 bytes->i64 and the inverses, for printing
addr address of a place
arithmetic, comparison, casts per machine type

Printing is not a primitive. print-str, print-f64 and friends are Flan functions over write-stdout. println and print are compiler-provided: the checker walks the concrete type at the call site and emits the printer for it (lib/render.ml, shared with the REPL's C-x C-e). This is intentionally not user-defined overload resolution — ordinary values remain untagged, and there is nothing to dispatch on at run time — which is why it did not have to wait for generics as this line once said it would. Generic instantiations are concrete types by the time the checker sees them, so they need nothing further; any and Error are the remaining case, and they carry the metadata their dynamic printers need.

Entry point. (defn main [args [string]] i32). Both the parameter and the return type are optional: omitting args means the program ignores argv, omitting the return type means Unit and an exit status of 0. sand.flan uses the short form, calc-me the long one.

RNG is ours, not libc's. rand-f32 is a seeded PRNG implemented in Flan (xoshiro or PCG), because a grid hash is only a regression test if the sequence is byte-identical on native and wasm32. Decided here rather than at milestone 4, since a headless deterministic sand run is the cross-target test.

Modules

There are modules — Odin calls them packages, and so does Flan. What is removed is Clojure's ns form: no path that must mirror the directory, no :require~/:refer~/~:as~/~:import~ vocabulary, no per-file namespace object.

  • The directory is the package. Every file in a directory shares one top-level scope. Files in a package do not import each other, and top-level names are order-independent, so mutually recursive functions need no forward declaration.
  • The package declaration is optional, which is the one place Flan diverges from Odin — Odin requires package foo as the first line of every file and requires it to agree across the directory. Flan infers the package name from the directory name, and (package parser) is written only when the name must differ from the directory (a directory named flan-parser, a scratch directory with a name that is not an identifier). When present it must agree across the directory, as in Odin.
  • A loose file in ~/scratch/ is a package of one. No project file, no manifest, no declaration. Open it, connect the REPL, start working. The ceremony budget for "new file, running REPL" is zero — this is the requirement the whole scheme is designed around, and it is why the declaration is optional rather than mandatory.
  • Cross-package: (import rl "vendor:raylib"), and everything from it is qualified rl/foo. One form, one meaning, no unqualified-import mode.
  • Collections in the path (vendor:, core:) are Odin's, and are just root-directory aliases.

Compilation

Two backends and three paths. The split is not dev-vs-release; it is does this code have a frame budget.

expression eval:    flan → typed IR → interpreter                      ~1ms
dev redefinition:   flan → typed IR → .ll → llc → ld -shared → dlopen → cell store
                                                                       ~16ms  (MEASURED)
release build:      flan → typed IR → .ll → clang --target={native,wasm32}

Hard requirement: eval is immediate. Not "fast enough for a build" — immediate, because the whole point of the live loop is that you see the result. 16ms is one frame at 60fps and under the 50ms threshold where a response stops feeling instantaneous. The rule that buys it: *never invoke the ~clang driver on the dev path.*

Expression evalC-c C-e, calling a function, inspecting a var, running a test — goes to the tree-walking interpreter. Sub-millisecond, no subprocess. This is the permanent REPL backend, not a milestone-2 scaffold.

Dev redefinitionC-c C-c on a function inside a running game — cannot use the interpreter, because that code has an 8ms frame budget. It recompiles the one function, links it, and does the atomic indirection-cell store. This is what the Hot reload section has always described; the interpreter does not replace it.

Release is whole-program AOT with direct calls and no cells.

Measured redefinition latency

Single function, x86-64, clang 20.1.8, 20 iterations each:

Step Per call Dev path?
clang -shared (driver: compile + link) 52.0ms no
clang -c (driver: compile only) 22.5ms no
llc -filetype=obj 13.9ms yes
ld -shared from the .o 2.4ms yes

llc + ld + dlopen16ms. The clang driver is the cost, not codegen — it forks a second process and re-does argument and target resolution. Codegen itself barely scales with function size: 721 lines of IR took 16.7ms against 13.9ms for 8 lines, because 10ms is ~llc startup loading libLLVM. A realistic redefined function lands in the same 1517ms.

This is why an in-process ORC JIT is not needed. It would take 16ms to ~3ms; the difference is below perception, and the price is a version-pinned libLLVM and C++ linkage from the host language, forever.

Redefinition must not stutter the running game

The 16ms is not paid by the game thread. llc and ld are already separate processes running on other cores. What the game process does is smaller:

Step Cost Game thread?
llc, ld 16.3ms no, separate processes
dlopen the new .so ~0.11ms must not be
atomic store into the cell ns yes, and free

Two design choices are load-bearing, and neither is automatic:

  1. dlopen happens on the reload thread. It mmaps, relocates and takes the loader lock; off-thread it blocks nobody, because the game thread is not doing dynamic linking. Load with RTLD_NOW so lazy PLT resolution cannot ambush the game thread on a later first call.
  2. Publish at a frame boundary, in a batch. This matters more than the threading. Storing each cell the moment it is ready lets the game observe a half-applied redefinition — two functions that changed together applied one frame apart, or a function swapped mid-frame with half the entities already updated by the old code. Instead the reload thread stages the complete set of new pointers and sets a flag; the game loop tests the flag once at the top of the frame and does N stores. One relaxed atomic load per frame when nothing changed.

Residual cost: the first call into new code page-faults and misses i-cache. Tens of microseconds, not visible.

Dev architecture: daemon plus agent

  • Compiler daemon, a separate process: the OCaml frontend, the nREPL server, and the llc~/~ld invocations. Editors talk to this.
  • Reload agent, linked into the game binary: a socket listener, dlopen, and the frame-boundary cell publisher. A few hundred lines, and no OCaml runtime in the game.

This is why the dev runtime is multithreaded — it needs the reload thread. That is settled, and is independent of whether the language exposes threads, which is still open decision #4.

It also bears on whether the interpreter survives: if the agent can dlopen and call anything in 16ms, then even "eval this expression against live game state" can be a compiled .so, and no interpreter is needed inside the game process.

Why LLVM IR as text

text .llclang libLLVM bindings emit C
Build dependency a clang on PATH matching libLLVM, version-pinned, C++ linkage any C compiler
Breaks on LLVM upgrade no routinely no
Debuggable .ll is readable print-from-memory readable, but lies about origin
In-process JIT no yes (ORC) no
Control of layout / ABI / tail calls full full poor

The only column text loses is the JIT one, and the measurement above shows the loss is ~13ms — below perception. ORC remains addable later behind the same typed IR without touching the language, but nothing currently argues for it.

The interpreter cannot run sand

Do not plan around it. 200 × 280 = 56,000 cells, scanned by game-update and again by game-draw — ~112,000 interpreted cell-visits per frame against an 8.3ms budget at 120fps. At an optimistic 100ns per visit (environment allocation, argument binding, two index computations, a compare) that is 11ms before settle, paint, or a single raylib call. Expect 2030fps.

This is an estimate, not a measurement, which is why milestone 2 exits with a measured interpreter throughput number — before milestone 4 depends on it. Milestone 4's interactive acceptance test runs on the compiled dev build; the interpreter is not in that loop.

Open: does the interpreter survive milestone 3?

Now that compiled redefinition is measured at 16ms, the case for a permanent interpreter is weaker than it looked. 16ms is perceptually instant for expression eval too, and one backend removes a standing obligation — two backends must agree on observable behaviour forever, and every divergence is a bug that reproduces in only one of them.

Against dropping it: the interpreter is clearly right for milestone 2 (far less work than an LLVM backend, better error messages, no linking), and the instrumentation-based step debugger wants it. Decide at milestone 3 exit on measured numbers, not now.

All three paths share the frontend and the typed IR and must agree on observable behaviour. That agreement is what the acceptance programs test.

  • Non-local exit lowered explicitly (result propagation + branch targets), not via platform unwinding. Same on both targets, no dependency on the WASM exception-handling proposal. Every Flan function carries the transfer channel; uniformity keeps indirect calls and hot reload ABI-safe. A later optimisation may narrow that cost, but it cannot change the ABI.
  • Stdlib written in Flan, not the host language. A few hundred primitives per backend, everything else on top. This is what keeps a second backend cheap.

Targets

  • Desktop: AOT to native, x86-64 and arm64. Primary development target.
  • WASM: AOT build artifact only, not interactive-first — no REPL, no hot reload, no debugger there. That is not the same as untested: every runtime or ABI feature ships with automated wasm32 tests in CI from the first one, because a divergence found at ship time is a rewrite.
  • Host binary links raylib natively; web links raylib via emscripten.

One narrow host ABI, implemented twice

The portability risk is the host interface, not the language. Divergence points:

  • Filesystem — pack assets, one abstraction, never touch paths
  • Threads — decide now; retrofitting is worse than the reverse
  • Blocking — browser main thread cannot block
  • Audio/input/window — constrain to the raylib subset identical on both

Dev vs release builds

Deliberately different.

Dev Release
Backend interpreter and LLVM LLVM/clang
Calls indirection cells direct
Code never freed static
Frames shadow stack none
Structs version word none
Reload yes no

Build and run the release config regularly, not just at ship time.

Hot reload

Every cross-function call goes through an indirection cell; body redefinition is one atomic pointer store. A top-level function value is a stable trampoline over the same cell, never a pointer to a specific body, so callers holding it observe a body redefinition too. Old code is never unloaded, so a thread mid-execution finishes safely in the old version.

A signature-changing redefinition creates a new internal function version and trampoline. Newly compiled callers use it; existing callers and stored function values safely retain the old version. The session immediately warns at each tracked old caller site, and recompiling that caller either updates it or gives a normal type error. This preserves live running code without hiding stale calls.

This is the fix for jank issue #947 (segfault redefining a running loop's function plus its callee — their JIT relinks and unloads under a running thread).

What redefinition cannot do

Patch a mid-execution frame and continue at the same PC — its register allocation belongs to the old compilation. No implementation does this. "Resume" means re-entering from an established restart point.

Tooling

Server speaks nREPL (bencode over socket) — the transport and the core ops (eval, load-file, describe, interrupt) are genuinely reusable, and that is what the ~5001000 lines buys. It does not buy CIDER/Conjure/Calva compatibility: their useful operations assume Clojure-shaped vars, namespaces, nses-of-symbols and middleware. Treat "speaks nREPL" as milestone 7a and "an editor client that is pleasant" as a separate milestone 7b. In the dev runtime:

  • eval string in package; compile form/file with source locations
  • completion, arglist, describe, find-definition
  • backtrace + restarts; interrupt

Emacs client

Focused client, 35k lines. Do *not* fork CIDER (~30k lines elisp, deeply Clojure-coupled) — reference it. ~clojure-mode-derived major mode, overlay rendering, hydra for stepping bindings (transient is the maintained alternative).

Debugging: two layers, two protocols

Do not extend nREPL with registers or memory. The protocol is the easy part — adding an op is trivial; implementing it means becoming a debugger (ptrace, trap handling, frame unwinding), duplicating an enormous existing project.

Capability DAP (lldb-dap + dape) nREPL
Registers, memory, native stack yes never
Breakpoints, watchpoints, stepping yes
Eval Flan expressions with real semantics no yes
Redefine a function and continue no yes
Invoke a restart no yes
Pause before unwinding no yes

Neither covers the other's column, so this is not a choice between them.

DAP is nearly free. No debug adapter is written: emit DWARF from the LLVM backend and point lldb-dap at the binary; dape speaks to that. lldb and gdb both ship DAP interfaces already.

This is where no object headers pays off a second time. Flan structs are C structs — real machine types, no tag words, no boxing — so DWARF describes them with no impedance mismatch and lldb shows a Cursor correctly with no plugin and no formatters.

Reload and DAP

Mostly automatic. lldb watches the loader rendezvous structure, so a dlopen'd generation is noticed and its DWARF read; and lldb keeps breakpoints as unresolved specs, re-resolving them against newly loaded modules, so a breakpoint on sand.flan:85 binds to the new function by itself.

The gotcha is our own never-unload rule. After N redefinitions there are N copies of a function, each with DWARF claiming sand.flan:85, so one breakpoint resolves to N locations and stops in stale generations. Needs glue, and it is small: give each generation a distinct source identity in its DWARF (sand.flan#7), and have the reload agent disable breakpoint locations in superseded modules on each reload. This is the one piece that cannot just be wired up and left alone.

Not available at any price: if execution is paused inside the old function, redefinition does not move it — register allocation belongs to the old compilation (see "What redefinition cannot do"). The current frame finishes in the old code; the next call enters the new one. Also, dape reads source from disk, so editing further after a generation was compiled drifts the highlighted line from what is executing.

Cut: the instrumentation-based step debugger

Previously planned as CIDER's model — macroexpansion wrapping subforms with breakpoints. Dropped. It is the most expensive piece of milestone 8 (instrumentation, an overlay protocol, &env, a bespoke Emacs client) and DAP+DWARF covers most of its use. What DAP cannot cover — pausing before the stack unwinds with restarts available — comes from handler-bind over nREPL.

Consequences: &env loses its urgency (its "decide now" status came from the step debugger requiring it — now an ordinary milestone-5 decision), and the shadow stack shrinks to serving only nREPL backtraces and restart enumeration.

Forcing the live-programming workflow into DAP would repeat the CIDER mistake in the other direction: DAP's model is stop/step/inspect, with no vocabulary for "invoke use-placeholder and resume". That stays on nREPL.

Pause on exception

Better than CIDER's, because handler-bind has not unwound the stack. Dev-mode global handler messages the editor and blocks with the full live stack and all restarts available. Fix the function, resume via restart.

Build sequence

Deliberately ordered so each step is runnable and the next one cannot start until the previous checker is stable. The failure mode this exists to prevent is building the whole live environment at once.

  1. Freeze the model. spec-memory.md and spec-conditions.md — done before any code. Fixed arrays, non-owning slices, move-only Vec~/~Map, allocators, Ptr, explicit clone; the six restart cases. Done.
  2. Run calc-me.flan on the interpreter. Reader, typed IR, checker, tree-walking backend. Exit criterion includes a measured throughput number — interpreted calls per second on a tight loop — because milestone 4's frame budget depends on it (see Compilation). Packages, structs, (Ptr T) + addr, byte slices, at~/~len, while, set on the fixed place list, cond, match, Option

    • some, i32~/~u8~/~f64, recursion, argv, stdout. No allocator, no Vec,

    no generics, no user macros, no FFI, no window. Headless, so the acceptance test is a table of expression/result pairs.

  3. Emit LLVM IR and pass the same calc-me test AOT, on native and wasm32 in CI. Both backends, one test table, one narrow host ABI (argv, stdout, exit). This is where the second target gets proven — while there is almost nothing to port.
  4. Run sand.flan. Fixed 2-D arrays, dotimes, defer, and typed FFI to raylib including keyword→enum coercion. Acceptance test twice: headless (N frames, hash the grid — runnable in CI on both targets) and interactive at 120 fps.
  5. Generics and macro expansion, once the base checker is stable. defmacro, &env, hygiene. Until here, when~/~unless~/~until~/~cond~/~dotimes are special forms in the compiler.
  6. Allocators, Vec~/~Map, Result~/~try~/~errdefer, then conditions and restarts against spec-conditions.md, with dedicated tests per numbered case.
  7. Hot reload — free in the interpreter, indirection cells for compiled dev builds, with signature generations and stale-caller warnings, plus the remaining compatibility limits written down and enforced: struct layout changes, live callbacks held by C, captured environments.
  8. Debugger, nREPL, async — last, and 8 splits into transport (8a) and editor client (8b).

Milestones 14 are the project. Everything from 5 on is optional in the sense that a language that stops there is still usable; nothing before 5 is.

Ordering note: sand cannot be first even though it is the better demo, because it needs raylib FFI, keyword→enum coercion and a window before a single line of it runs. calc-me needs argv and stdout.

Runtime budget

Scoped to milestones 13 only. The earlier version of this table put the whole plan — conditions, explicit transfer lowering, macros, hot reload, debugger support — at 1525k, which is not credible: each of those is architecture work that touches the frontend, the IR and the backend at once.

Piece Lines Milestone
Allocators 23k 2
Core data (fixed/slice/Vec/Map) 23k 2
Reader + frontend + checker 58k 2
LLVM IR lowering 35k 2
raylib FFI + host ABI ×2 12k 3
Subtotal, a language that runs sand 1321k

Beyond that, estimated but not budgeted, because these are the parts that are architecture rather than volume:

Piece Note
Macroexpander + &env touches the reader and the checker
Generics / monomorphisation touches the whole checker
Conditions + explicit transfer lowering frontend and IR and backend
Hot reload cells + compatibility rules changes the calling convention
Debugger instrumentation + nREPL needs &env and source locations

Janet is 36k including a bytecode VM, and Janet has no static types, no monomorphisation, no restarts and no reload.

Open decisions

None of these block milestone 2. The milestone each one must be answered by is marked.

  1. Host language: OCaml or Rust — the only thing blocking the scaffold. See Host language. Milestone 2.
  2. Macro hygiene is settled for milestone 5: explicit gensym, deliberately non-hygienic expansion, no local macros until a concrete use case appears.
  3. Borrow checking and escaping frame-arena values. Deferred; revisit after Vec/Map are useful in real programs. The first implementation deliberately follows Zig/Odin: slices and pointers have explicit lifetime and reallocation contracts, with dev generation checks. A later lightweight provenance pass may catch obvious escaping/stale-borrow mistakes, but must not impose Rust-style lifetime annotations, automatic promotion, or an ECS-shaped object model.
  4. Threads in the language: yes or no, decided before the host ABI. (The dev runtime is multithreaded regardless — it needs a reload thread. Settled; see Compilation.) Milestone 3, before the host ABI.
  5. Escaping closures. Deferred until a concrete use case. Settled in spec-memory.md: (Fn ...) is a bare pointer with no environment (callbacks, reload cells, FFI), and a non-escaping fn captures by value into a stack environment — which is what makes handler-bind handlers able to see enclosing locals, without which conditions are not worth building. Still open: a closure that is stored, returned, or pushed into a container. Which allocator owns its environment, and what happens when the frame arena resets? Do not design this speculatively; revisit it alongside the deferred provenance work only when it is needed.
  6. Hot-reload compatibility rules. Milestone 7. A changed function signature creates a new version: new callers resolve it, old callers keep the old one, and the session warns at tracked stale caller sites. Still open: a changed struct layout with live values is rejected; a managed class layout has an explicit frame-boundary migration path, as described above. Still open: a function pointer already handed to C, a captured environment, and redefining a defvar. Each needs an answer of the form "rejected", "accepted with a migration", or "accepted and the old code keeps running".
  7. Does the interpreter survive milestone 3, or is the compiled path the only backend? Milestone 3, on measured numbers. See Compilation.
  8. (Option a) settled: an ordinary stdlib union with Some~/~None; the compiler niche-optimises (Option (Ptr T)) to a nullable pointer. The CL-vs-Clojure truthiness question is moot under static typing.

Settled since the first draft (see the specs):

  • Module system → the directory is the package, Odin's model. No ns form. See Modules.
  • Zero values → ZII by default, with an opt-out. A declaration with no initialiser is all-bytes-zero; (zeroed) re-zeroes something later; uninit opts out for large buffers about to be overwritten. It is a memset, not a memcpy; zeroed globals live in BSS and cost nothing.
  • Package declaration → optional, inferred from the directory name. See Modules.
  • Keywords at typed call sites → yes. :space resolves at compile time against the parameter's enum type, rl/key-space names the same value, and a typo is a compile error checked against the enum's members. No runtime cost. Needs the FFI enum declared, so it lands with milestone 4.
  • Dev redefinition latency → 16ms, measured: ~llc + ld -shared + dlopen, never the clang driver, dlopen off the game thread, cells published in a batch at a frame boundary. See Compilation.
  • Dev backend → interpreter for milestone 2 certainly. Whether it survives milestone 3 is open, not settled — see Compilation.
  • set on places → a fixed list of assignable forms, not setf.
  • Loop story → imperative while~/~for with break~/~continue and return; loop~/~recur only if it later earns its place. sand.flan is ported.
  • Generic parameters → inferred at call sites, no explicit instantiation; and no type classes, so unconstrained operators over a type variable are rejected.

Unverified claims in this plan

  • Scope is the biggest risk, not any single feature. A language, inference, monomorphisation, macros, an explicit-memory runtime, conditions/restarts, hot reload, a debugger, nREPL tooling, native and WASM — each is reasonable, the set is not one project. The build sequence above exists because of this; if something has to give, it gives from milestone 4 upward.
  • Line-count estimates are extrapolations.
  • LLVM → wasm32 with manual memory is validated by Odin shipping it; not yet validated for s-expression macros + conditions/restarts on top.