flan/docs/SBCL-REDEFINITION-NOTES.md

SBCL's redefinition model, as a reference point for Flan

Research notes, 2026-09-20. Untracked. Written for DISCUSS.org's item "investigate SBCL's redefinition model — warn + keep old value until callers update". Nothing here is a decision; the last section lists candidates and stops.

The question behind it: Flan's dev session hard-refuses a struct layout change and a global type change (lib/session.ml:304-338, docs/BUILT.md:1213-1219), and plan.org's dev/release table (~671-682) lists "Structs: version word" for dev builds with nothing behind it. SBCL is the system the author half-remembered as "warn, keep the old layout, complain at the stale reader". It does do something close to that — but the shape of it is not quite what the note says, and the difference matters for what Flan could copy.

---

1. defstruct redefinition

It is an error, not a warning

This is the one place DISCUSS.org's summary is wrong, and it is worth stating first because it inverts the precedent. The note says SBCL "warns rather than refusing". SBCL signals a continuable error — the default behaviour of an incompatible defstruct redefinition at a REPL with no handler is to drop you into the debugger, and in a non-interactive build it aborts. It proceeds only because a human (or a handler) picks a restart.

From src/code/defstruct.lisp, %redefine-defstruct:

"attempt to redefine the ~S class ~S incompatibly with the current definition"

with two restarts:

• continue — "Use the new definition of ~S, invalidating already-loaded code and instances."
• recklessly-continue — "Use the new definition of ~S as if it were compatible, allowing old accessors to use new instances and allowing new accessors to use old instances." Offered only when mutable-layout-p holds, i.e. when the new layout can be overwritten in place onto the old one (compatible sizes/rawness), so it is not offered for arbitrary changes.

There is a separate genuine warning for the compatible-ish case, redefine-structure-warning:

"incompatibly redefining slots of structure class S@Make sure any uses of affected accessors are recompiled"

The comparison that drives all of this is compare-slots, which returns three lists: slots that moved, slots that were retyped, slots that were deleted. So SBCL's notion of "incompatible" is finer than Flan's compatible_structs all-or-nothing name+type list equality — a pure append of trailing slots is a different case from a reorder, and SBCL distinguishes them.

Note also that ANSI leaves structure redefinition undefined (CLHS defstruct); everything above is an SBCL extension, not a standard protocol. SBCL is not implementing a spec here, it invented a policy.

What happens to the live instances

Picking continue calls register-layout with :invalidate t, which reaches %invalidate-layout in src/code/class.lisp:

"Mark LAYOUT as invalid. This is called only on CONDITION and STRUCTURE subtypes when redefining incompatibly."

It sets the layout's invalid slot, zeroes layout-clos-hash, and unhooks the classoid from every superclass's subclass table. The instances themselves are not touched, not walked, not found. There is no heap scan. Each instance keeps pointing at the same layout object it always pointed at; that object is now flagged invalid.

This is the crucial mechanism, and it is the one Flan cannot have: an SBCL structure instance carries a pointer to its layout in its header word. The invalidation is one store into a shared object, and every instance in the heap learns about it for free, because they all reach it through that pointer.

When the error surfaces

At the next access, not at redefinition time. Two paths, and they give different conditions, which is worth being precise about:

• The typed-accessor path. A defstruct accessor in safe code checks the instance's layout against the expected one. With the layout invalidated the check fails and you get an ordinary type-error — the value is no longer typep the struct type it used to be. (I did not read the exact accessor emission path in src/compiler/; treat "it is specifically a type-error rather than some dedicated condition" as unverified.)

• The PCL path. When a structure-object with an invalid wrapper goes through PCL (generic function dispatch, slot-value), %obsolete-instance-trap fires. For a structure it signals sb-pcl::obsolete-structure, defined in src/pcl/std-class.lisp, reported as:

  "obsolete structure error for a structure of type ~S"

  This is the condition people quote when they say "SBCL tells you your instance is stale". There is no restart on it that repairs the instance. There is no update protocol for structures.

So the honest one-line summary of the struct story: SBCL refuses by default, proceeds only on an explicit human restart, and then converts your old instances into landmines that raise at first touch. It never silently reads at the wrong offsets — except under recklessly-continue, which is exactly the "silent argument mismatch" outcome docs/BUILT.md:1223-1230 says Flan refuses to ship, and SBCL offers it only behind a name chosen to shame you.

2. defclass redefinition — the part with a real protocol

This is where CL earns the reputation, and it is a standard protocol (CLHS 4.3.6, update-instance-for-redefined-class), not an SBCL extension.

Redefining a class does not error. The sequence:

1. The new class definition is installed. make-instances-obsolete runs — in SBCL, the std-class method calls %update-lisp-class-layout and %invalidate-wrapper with the :obsolete flag. Again: a flag on the shared wrapper, no heap walk. layout-invalid can hold nil, t, (:flush <wrapper>) or (:obsolete <wrapper>); invalid-wrapper-p is the predicate on the access path.

2. Nothing else happens until someone touches an instance. CLHS 4.3.6: the update occurs "at an implementation-dependent time, but no later than the next time a slot of that instance is read or written", and the instance's eq identity is preserved across it.

3. On that first touch, %obsolete-instance-trap runs. It computes the added slots and the discarded slots by name, builds a property list of the discarded slots' values, swaps the instance's storage for the new shape, and calls:

   (sb-sys:nlx-protect (update-instance-for-redefined-class
                        instance added discarded plist)
     (replace-wrapper-and-slots instance owrapper oslots))
   

   The nlx-protect is a detail worth noting: if the user's update-instance-for-redefined-class method signals and the stack unwinds, the instance is rolled back to its old wrapper and slots rather than left half-migrated.

What the protocol guarantees:

• Slots present in both definitions keep their values, matched by name. This is the whole reason it works, and it is only possible because a CLOS instance's slots are addressed through a name→index map held in the class, not baked into compiled call sites.
• Added slots are initialized from their :initforms — the system-supplied primary method calls shared-initialize on exactly the added-slot names, passing along the initargs it received.
• Discarded slots' values are not lost, they are handed to you in the property list, so a user method can do (getf plist 'old-name) and derive the new slots from the old ones (the canonical CLHS example is a cartesian→polar coordinate change).
• Initargs are validated; the default method signals on an initarg not declared valid for the class.
• The return value is ignored.

The pattern to take away: the automatic part is name-matching, and the interesting part is a user hook that gets the old values. CLOS does not try to guess what a renamed slot meant; it hands you the corpse and lets you decide.

3. Global variables

Trivial, as expected, and the interesting bit is that CL already draws Flan's distinction.

• defparameter re-evaluated "unconditionally assigns the initial-value to the dynamic variable named name".
• defvar "assigns initial-value (if supplied) to the dynamic variable named name only if name is not already bound".

That is precisely docs/BUILT.md's rule that a defvar's initial value is deliberately not in the refusal table — "edit the code, keep the sand". CL's defvar is the same policy, reached by the same reasoning, thirty years earlier. Flan's defvar and CL's behave the same way here: both accept a changed initialiser and ignore it while the variable is bound. The divergence is only in the const twin — Flan's defconst refuses a changed value (when the checker consumed it), whereas CL's defparameter simply overwrites.

There is no type-change problem to speak of, because a CL special variable has no compile-time type and no fixed-shape storage — it is one boxed word. The entire class of failure that lib/session.ml:304-311 is protecting against (storage laid out to a type, reused for another) does not exist in CL. So CL offers no precedent for Flan's typed-global refusal; it sidesteps it by being dynamically typed. Worth saying plainly rather than pretending there is guidance here.

defconstant is the exception and is famously annoying: redefining one to a non-eql value is an error in SBCL, which is the same shape as Flan's compile-time-consumed defconst refusal, for a related reason (the old value may already have been folded into compiled code).

4. Function redefinition

Trivially supported, and the reason is the reason Flan's cell design already works: a global function call goes through the fdefinition, a name→function indirection, not to a body address. Redefining foo stores a new function object into the symbol's function cell, and every call site compiled against foo picks it up on its next call with no recompilation and no stale caller. This is exactly plan.org's "every cross-function call goes through an indirection cell; body redefinition is one atomic pointer store". Flan already matches SBCL here for the body-only case.

Two nuances worth carrying over:

• Stale callers do exist in SBCL, in one place: block compilation. Under (declaim (start-block ...)) / :block-compile, calls within the block become local calls resolved at compile time. Redefining a function in a block-compiled unit leaves its in-block callers calling the old body — they were never going through the fdefinition. SBCL's answer is not a warning or a trampoline; it is "block compilation prevents redefinition, that is the trade you made", and the feature is off by default. Self-recursive calls are the same story in miniature: SBCL deliberately compiles a self-call as a full call precisely so that a function can be redefined while running.
• SBCL does not version signatures. A function whose lambda list changes is just a redefinition; there is no new version, no trampoline, no per-caller warning. Callers compiled against the old arglist get a normal wrong-number-of-arguments error at runtime (or a compile-time style-warning if SBCL still has the old ftype recorded and sees the call again). plan.org's versioned-function design with tracked caller sites is more than SBCL does, not a port of it. If the author was remembering SBCL as the source of that design, the memory is of something else — the nearest real relatives are Erlang's two-version code loading (old code + current code, a process running old code is killed on the third load) and Smalltalk's become:.

5. What maps to Flan and what cannot

The structural obstacle, stated once

Every SBCL mechanism above — struct layout invalidation, CLOS wrapper obsolescence, the lazy trap — rests on a single fact: an instance carries a pointer to its shape descriptor in its header. That is what makes "invalidate the shape" an O(1) operation that reaches every live instance, and what makes "detect staleness at next access" a load-and-compare on a word the instance already has.

Flan's typed side has none of that. A defstruct value is flat and unboxed; it lives inline in a global, inline in an array element, in registers, spilled on a stack frame. There is no header word, no per-instance shape pointer, and no way to enumerate the live instances of a type. So:

• Layout invalidation as SBCL does it: structurally unavailable. There is nothing to invalidate that instances reach.
• Detecting an obsolete instance at access time: unavailable in release shape, because detection requires a per-instance word. It is available only if you add that word, which is exactly what plan.org's "Structs: version word, dev only" line means.
• update-instance-for-redefined-class-style migration: doubly unavailable, because it needs both the per-instance shape tag and the ability to change an instance's size in place. A flat struct embedded in an array cannot grow.

The dev-build version word is SBCL's generation counter

The connection plan.org's table implies but does not spell out: a version word in the dev build is the degenerate form of SBCL's layout. SBCL's instance points at a layout object whose invalid flag can be flipped; a Flan dev-build struct would carry an integer stamped at construction, compared against the type's current generation on access. Flip the generation on redefinition, and the next field access on an old-generation value traps. That is layout-invalid with the indirection collapsed into an immediate, which is the only form that survives "instances have no headers, but dev builds may pay for one extra word".

What it buys, honestly: not the ability to keep using old values. It buys a diagnosis instead of a refusal — the reload is accepted, the program keeps running, and the first read of a stale value says "this Cursor was built before you changed Cursor" with a location, instead of the session saying "restart" before anything runs. That is genuinely SBCL's bargain (continue then trap), and it is the honest framing: the version word converts a compile-time refusal into a runtime trap. Whether that is an improvement is a judgement about which failure the author would rather debug.

What it costs, all of which is real:

• A word per struct instance in dev builds, so dev and release layouts differ — which means array strides, FFI structs, and anything crossing to C differ too, or need to be excluded. The dev/release divergence plan.org already accepts for frames and cells gets bigger and more observable.
• A check on every field access in dev builds.
• Structs embedded in other structs, in arrays, and in Vecs each need a story; an array of 4096 cells would need every element stamped and every element checked.
• Globals are not covered by it at all. A global's storage is a fixed slot with a shape; a version word on the global could detect a stale read, but there is nothing to read — the point of the refusal is that the old bytes mean something else now. SBCL has no precedent to offer, per section 3.

The dyn side is where CLOS-style update is actually available

Grounding the claim in the code as landed (FIX.org item 6, commits 8d2bf2a, 5af990e, 6c6024e; runtime/flan_dyn.c):

• A defclass instance is an ordinary dyn map with a klass field in the flan_obj header (flan_dyn.c:255), set by flan_dyn_map_new_class (:1088), read by flan_dyn_class_of (:1100). It is deliberately in the header and not an entry, so len, render and dyn_equal do not see it.
• Slots are keys. lib/classes.ml:150-175: (defclass point [x y]) desugars to (defn point [x dyn y dyn] dyn #point{:x x :y y}) and nothing else. Access is get/put by keyword.

Two consequences, and the second is the one that surprised me:

1. Name-based slot carrying — the hard half of update-instance-for-redefined-class — is free here. A dyn instance already stores its slots by name. Any migration is a map operation.
2. There is no class object at runtime to invalidate, because there is no class registry at all. The klass field holds an interned keyword, not a pointer to a class descriptor. Redefining a defclass today is just redefining its constructor defn — which the session already permits as an ordinary body change. Old instances keep their old keys and keep working; new ones get the new keys; class-of answers the same keyword for both. So the dyn side currently has neither the refusal nor the protocol: it has silence.

Whether that silence is a bug depends on what the author wants. It is the Clojure answer (a map is a map; a "class" is a tag) rather than the CLOS answer. The CLOS answer would need, concretely:

• A runtime class registry: keyword → (slot-name list, generation counter). Nothing like this exists today; lib/classes.ml is compile-time only and emits a plain defn.
• A generation stamp per instance, or a cheaper trick: compare the instance's key set against the registry's current slot list lazily on get/put. With a generation word in the flan_obj header — there is room, and unlike the typed side these objects have headers — the check is a compare.
• A migration step at first access after a bump: add missing slots as nil, collect removed slots into a property list, and call a user hook. The obvious Flan spelling of that hook is a generic function, e.g. (defmethod update-for-redefined point [p added discarded] ...), which fits the dispatch mechanism that already exists.
• A decision on whether put of an unknown slot stays legal. Today it is — a class instance is an open map, and FIX.org already defers "refusing an unknown slot at (get p :z)". If unknown slots stay legal, the registry's slot list is advisory and the whole update protocol is advisory with it.

This is a real, SBCL/CLOS-precedented design that Flan's runtime can actually support. It is also a feature with no user yet, since redefinition on the dyn side currently fails silently rather than loudly.

---

6. Candidate designs

Three, none recommended over the others. Cost estimates are rough.

A. Keep the refusal for typed structs and globals; fix the message

Leave lib/session.ml refusing, but rewrite both messages to say what SBCL's restarts say: name which fields moved, were retyped, or were deleted (SBCL's compare-slots split — Flan compares whole field lists and can cheaply do the same split), and state the two things the author can do instead (restart, or rename the type and migrate by hand). Optionally accept the case SBCL's mutable-layout-p accepts and Flan currently refuses: appending fields to a struct no array/FFI type depends on is layout-compatible for every existing instance, so it need not be refused at all.

• Cost: small, days. Frontend only, no runtime change, no dev/release divergence.
• Behaviour: unchanged except the append case starts working and the diagnostic names the offending field.
• Precedent: SBCL refuses by default too — this is the %redefine-defstruct error with no restart taken, plus compare-slots' finer classification. The "SBCL warns instead of refusing" premise in DISCUSS.org does not survive contact with the source, so "keep refusing" is the SBCL-consistent option, not the timid one.

B. Dev-build struct version word — SBCL's layout generation, collapsed

Stamp every dev-build struct instance with the type's generation at construction; bump the generation on an incompatible redefinition; check the stamp on field access in dev builds and trap with "this value was built before Cursor changed shape" instead of refusing the reload. Release builds carry neither word nor check, which is plan.org's dev/release table as written.

• Cost: large. Touches layout, both backends, arrays and Vecs of structs, FFI boundary exclusions, and every field-access site; the dev/release layout divergence becomes observable wherever a struct crosses to C. This is the one that wants a written design before any code.
• Behaviour: the reload is accepted and the program keeps running; stale values trap loudly at first touch rather than reading wrong offsets. It does not let old values keep working — no migration is possible for a flat unboxed struct — so it converts a pre-run refusal into a mid-run trap. Globals are not covered and still need option A's refusal.
• Precedent: SBCL's %invalidate-layout plus the obsolete-instance trap, with the layout pointer replaced by an immediate because Flan instances have no headers. Note that SBCL, having the same choice, still puts the error in the redefinition path as well and requires a restart to reach the trap state.

C. CLOS-style update for dyn defclass instances only

Give defclass a runtime registry (keyword → slots + generation), stamp a generation into the flan_obj header beside klass, and on the first get/put after a bump, migrate the instance: add new slots as nil, gather removed slots into a list, and dispatch a user-overridable generic update-for-redefined before returning. Typed structs and globals are untouched and keep option A's refusal.

• Cost: medium and well-contained — runtime/flan_dyn.c header field plus a registry, lib/classes.ml emitting the registration, one lazy check on the map access path. It rides on machinery that already exists: headers, interned keyword tags, generic dispatch, and slots that are already addressed by name.
• Behaviour: redefining a class stops being silent. Instances survive redefinition with their shared slots intact, new slots appear as nil, and the author can write the coordinate-change-style migration by hand. The risk is that it is a ceremony on top of maps that are already open — if put of an arbitrary key stays legal, the registry describes an intention rather than a constraint.
• Precedent: this is CLHS 4.3.6 and update-instance-for-redefined-class, the only part of CL's story that is a standard protocol rather than an implementation's policy, and the only part whose prerequisites Flan already satisfies.

B and C are independent and could both happen, or neither. A is close to free and is compatible with both.

---

Sources

• SBCL source, src/code/defstruct.lisp (%redefine-defstruct, compare-slots, redefine-structure-warning, mutable-layout-p, restart texts) — https://github.com/sbcl/sbcl/blob/master/src/code/defstruct.lisp
• SBCL source, src/code/class.lisp (%invalidate-layout, register-layout, layout-invalid) — https://github.com/sbcl/sbcl/blob/master/src/code/class.lisp
• SBCL source, src/pcl/std-class.lisp (obsolete-structure condition, %obsolete-instance-trap, make-instances-obsolete, invalid-wrapper-p, replace-wrapper-and-slots) — https://github.com/sbcl/sbcl/blob/master/src/pcl/std-class.lisp
• CLHS 4.3.6 Redefining Classes — https://www.lispworks.com/documentation/HyperSpec/Body/04_cf.htm
• CLHS update-instance-for-redefined-class — https://www.lispworks.com/documentation/HyperSpec/Body/f_upda_1.htm
• CLHS defparameter/defvar — https://www.lispworks.com/documentation/HyperSpec/Body/m_defpar.htm
• SBCL Internals, Local Calls — http://www.sbcl.org/sbcl-internals/Local-Calls.html
• Block compilation in SBCL 2.0.2 — https://mstmetent.blogspot.com/2020/02/block-compilation-fresh-in-sbcl-202.html
• Lisp journey, "Structures: lightweight records" (the user-visible restart menu) — https://lisp-journey.gitlab.io/blog/data-structures-structs/

The SBCL manual (https://www.sbcl.org/manual/) does not document structure redefinition, obsolete instances or layout invalidation at all — checked, and it is silent. Everything in sections 1 and 4 above comes from source and mailing list/blog material, not from the manual.

Marked unverified

• The exact condition signalled when a compiled typed accessor (as opposed to the PCL path) is applied to an instance with an invalidated layout. Stated above as "an ordinary type-error"; I did not read the accessor emission in src/compiler/, so treat the specific condition name as unverified. The sb-pcl::obsolete-structure text is verified from src/pcl/std-class.lisp.
• Whether SBCL's compare-slots classification ever avoids the error for a pure append (i.e. whether appending slots is accepted silently). The mutable-layout-p gate on recklessly-continue strongly implies a compatible-layout notion exists, but I did not confirm the append case is error-free.