flan/docs/SBCL-REDEFINITION-NOTES.md

# SBCL's redefinition model, as a reference point for Flan

Research notes, 2026-09-20. Untracked. Written for DISCUSS.org's item
"investigate SBCL's redefinition model — warn + keep old value until callers
update". Nothing here is a decision; the last section lists candidates and stops.

The question behind it: Flan's dev session hard-refuses a struct layout change
and a global type change (`lib/session.ml:304-338`, `docs/BUILT.md:1213-1219`),
and plan.org's dev/release table (~671-682) lists "Structs: version word" for dev
builds with nothing behind it. SBCL is the system the author half-remembered as
"warn, keep the old layout, complain at the stale reader". It does do something
close to that — but the shape of it is not quite what the note says, and the
difference matters for what Flan could copy.

---

## 1. `defstruct` redefinition

### It is an error, not a warning

This is the one place DISCUSS.org's summary is wrong, and it is worth stating
first because it inverts the precedent. The note says SBCL "warns rather than
refusing". SBCL signals a **continuable error** — the default behaviour of an
incompatible `defstruct` redefinition at a REPL with no handler is to drop you
into the debugger, and in a non-interactive build it aborts. It proceeds only
because a human (or a handler) picks a restart.

From `src/code/defstruct.lisp`, `%redefine-defstruct`:

> "attempt to redefine the ~S class ~S incompatibly with the current definition"

with two restarts:

- `continue` — "Use the new definition of ~S, invalidating already-loaded code
  and instances."
- `recklessly-continue` — "Use the new definition of ~S as if it were
  compatible, allowing old accessors to use new instances and allowing new
  accessors to use old instances." Offered only when `mutable-layout-p` holds,
  i.e. when the new layout can be overwritten in place onto the old one
  (compatible sizes/rawness), so it is not offered for arbitrary changes.

There is a separate genuine warning for the compatible-ish case,
`redefine-structure-warning`:

> "incompatibly redefining slots of structure class ~S~@Make sure any uses of
> affected accessors are recompiled"

The comparison that drives all of this is `compare-slots`, which returns three
lists: slots that **moved**, slots that were **retyped**, slots that were
**deleted**. So SBCL's notion of "incompatible" is finer than Flan's
`compatible_structs` all-or-nothing name+type list equality — a pure append of
trailing slots is a different case from a reorder, and SBCL distinguishes them.

Note also that ANSI leaves structure redefinition **undefined**
(CLHS `defstruct`); everything above is an SBCL extension, not a standard
protocol. SBCL is not implementing a spec here, it invented a policy.

### What happens to the live instances

Picking `continue` calls `register-layout` with `:invalidate t`, which reaches
`%invalidate-layout` in `src/code/class.lisp`:

> "Mark LAYOUT as invalid. This is called only on CONDITION and STRUCTURE
> subtypes when redefining incompatibly."

It sets the layout's `invalid` slot, zeroes `layout-clos-hash`, and unhooks the
classoid from every superclass's subclass table. **The instances themselves are
not touched, not walked, not found.** There is no heap scan. Each instance keeps
pointing at the same layout object it always pointed at; that object is now
flagged invalid.

This is the crucial mechanism, and it is the one Flan cannot have: **an SBCL
structure instance carries a pointer to its layout in its header word.** The
invalidation is one store into a shared object, and every instance in the heap
learns about it for free, because they all reach it through that pointer.

### When the error surfaces

At the next access, not at redefinition time. Two paths, and they give different
conditions, which is worth being precise about:

- **The typed-accessor path.** A `defstruct` accessor in safe code checks the
  instance's layout against the expected one. With the layout invalidated the
  check fails and you get an ordinary `type-error` — the value is no longer
  `typep` the struct type it used to be. (I did not read the exact accessor
  emission path in `src/compiler/`; treat "it is specifically a `type-error`
  rather than some dedicated condition" as **unverified**.)
- **The PCL path.** When a structure-object with an invalid wrapper goes through
  PCL (generic function dispatch, `slot-value`), `%obsolete-instance-trap`
  fires. For a structure it signals `sb-pcl::obsolete-structure`, defined in
  `src/pcl/std-class.lisp`, reported as:

  > "obsolete structure error for a structure of type ~S"

  This is the condition people quote when they say "SBCL tells you your instance
  is stale". There is no restart on it that repairs the instance. There is no
  update protocol for structures.

So the honest one-line summary of the struct story: **SBCL refuses by default,
proceeds only on an explicit human restart, and then converts your old instances
into landmines that raise at first touch.** It never silently reads at the wrong
offsets — except under `recklessly-continue`, which is exactly the "silent
argument mismatch" outcome `docs/BUILT.md:1223-1230` says Flan refuses to ship,
and SBCL offers it only behind a name chosen to shame you.

## 2. `defclass` redefinition — the part with a real protocol

This is where CL earns the reputation, and it is a standard protocol (CLHS
4.3.6, `update-instance-for-redefined-class`), not an SBCL extension.

Redefining a class does **not** error. The sequence:

1. The new class definition is installed. `make-instances-obsolete` runs —
   in SBCL, the `std-class` method calls `%update-lisp-class-layout` and
   `%invalidate-wrapper` with the `:obsolete` flag. Again: a flag on the shared
   wrapper, no heap walk. `layout-invalid` can hold `nil`, `t`,
   `(:flush <wrapper>)` or `(:obsolete <wrapper>)`; `invalid-wrapper-p` is the
   predicate on the access path.
2. Nothing else happens until someone touches an instance. CLHS 4.3.6: the
   update occurs "at an implementation-dependent time, but no later than the
   next time a slot of that instance is read or written", and the instance's
   `eq` identity is preserved across it.
3. On that first touch, `%obsolete-instance-trap` runs. It computes the
   **added** slots and the **discarded** slots by name, builds a property list
   of the discarded slots' values, swaps the instance's storage for the new
   shape, and calls:

   ```lisp
   (sb-sys:nlx-protect (update-instance-for-redefined-class
                        instance added discarded plist)
     (replace-wrapper-and-slots instance owrapper oslots))
   ```

   The `nlx-protect` is a detail worth noting: if the user's
   `update-instance-for-redefined-class` method signals and the stack unwinds,
   the instance is rolled back to its old wrapper and slots rather than left
   half-migrated.

What the protocol guarantees:

- **Slots present in both definitions keep their values, matched by name.**
  This is the whole reason it works, and it is only possible because a CLOS
  instance's slots are addressed through a name→index map held in the class, not
  baked into compiled call sites.
- **Added slots are initialized from their `:initform`s** — the system-supplied
  primary method calls `shared-initialize` on exactly the added-slot names,
  passing along the initargs it received.
- **Discarded slots' values are not lost, they are handed to you** in the
  property list, so a user method can do `(getf plist 'old-name)` and derive the
  new slots from the old ones (the canonical CLHS example is a cartesian→polar
  coordinate change).
- **Initargs are validated**; the default method signals on an initarg not
  declared valid for the class.
- The return value is ignored.

The pattern to take away: **the automatic part is name-matching, and the
interesting part is a user hook that gets the old values.** CLOS does not try to
guess what a renamed slot meant; it hands you the corpse and lets you decide.

## 3. Global variables

Trivial, as expected, and the interesting bit is that CL already draws Flan's
distinction.

- `defparameter` re-evaluated "unconditionally assigns the initial-value to the
  dynamic variable named name".
- `defvar` "assigns initial-value (if supplied) to the dynamic variable named
  name **only if name is not already bound**".

That is precisely `docs/BUILT.md`'s rule that a `defvar`'s *initial value* is
deliberately not in the refusal table — "edit the code, keep the sand". CL's
`defvar` is the same policy, reached by the same reasoning, thirty years earlier.
Flan's `defvar` and CL's behave the same way here: both accept a changed
initialiser and ignore it while the variable is bound. The divergence is only in
the const twin — Flan's `defconst` refuses a changed value (when the checker
consumed it), whereas CL's `defparameter` simply overwrites.

There is **no** type-change problem to speak of, because a CL special variable
has no compile-time type and no fixed-shape storage — it is one boxed word. The
entire class of failure that `lib/session.ml:304-311` is protecting against
(storage laid out to a type, reused for another) does not exist in CL. So CL
offers no precedent for Flan's typed-global refusal; it sidesteps it by being
dynamically typed. Worth saying plainly rather than pretending there is guidance
here.

`defconstant` is the exception and is famously annoying: redefining one to a
non-`eql` value is an error in SBCL, which is the same shape as Flan's
compile-time-consumed `defconst` refusal, for a related reason (the old value
may already have been folded into compiled code).

## 4. Function redefinition

Trivially supported, and the reason is the reason Flan's cell design already
works: **a global function call goes through the `fdefinition`**, a name→function
indirection, not to a body address. Redefining `foo` stores a new function
object into the symbol's function cell, and every call site compiled against
`foo` picks it up on its next call with no recompilation and no stale caller.
This is exactly plan.org's "every cross-function call goes through an
indirection cell; body redefinition is one atomic pointer store". Flan already
matches SBCL here for the body-only case.

Two nuances worth carrying over:

- **Stale callers do exist in SBCL, in one place: block compilation.** Under
  `(declaim (start-block ...))` / `:block-compile`, calls within the block become
  local calls resolved at compile time. Redefining a function in a block-compiled
  unit leaves its in-block callers calling the old body — they were never going
  through the fdefinition. SBCL's answer is not a warning or a trampoline; it is
  "block compilation prevents redefinition, that is the trade you made", and the
  feature is off by default. Self-recursive calls are the same story in
  miniature: SBCL deliberately compiles a self-call as a full call precisely so
  that a function can be redefined while running.
- **SBCL does not version signatures.** A function whose lambda list changes is
  just a redefinition; there is no new version, no trampoline, no per-caller
  warning. Callers compiled against the old arglist get a normal
  wrong-number-of-arguments error at runtime (or a compile-time style-warning if
  SBCL still has the old ftype recorded and sees the call again). plan.org's
  versioned-function design with tracked caller sites is **more** than SBCL does,
  not a port of it. If the author was remembering SBCL as the source of that
  design, the memory is of something else — the nearest real relatives are
  Erlang's two-version code loading (old code + current code, a process running
  old code is killed on the third load) and Smalltalk's `become:`.

## 5. What maps to Flan and what cannot

### The structural obstacle, stated once

Every SBCL mechanism above — struct layout invalidation, CLOS wrapper
obsolescence, the lazy trap — rests on a single fact: **an instance carries a
pointer to its shape descriptor in its header.** That is what makes "invalidate
the shape" an O(1) operation that reaches every live instance, and what makes
"detect staleness at next access" a load-and-compare on a word the instance
already has.

Flan's typed side has none of that. A `defstruct` value is flat and unboxed; it
lives inline in a global, inline in an array element, in registers, spilled on a
stack frame. There is no header word, no per-instance shape pointer, and no way
to enumerate the live instances of a type. So:

- Layout invalidation as SBCL does it: **structurally unavailable.** There is
  nothing to invalidate that instances reach.
- Detecting an obsolete instance at access time: **unavailable in release
  shape,** because detection requires a per-instance word. It is available only
  if you *add* that word, which is exactly what plan.org's "Structs: version
  word, dev only" line means.
- `update-instance-for-redefined-class`-style migration: **doubly unavailable**,
  because it needs both the per-instance shape tag *and* the ability to change
  an instance's size in place. A flat struct embedded in an array cannot grow.

### The dev-build version word is SBCL's generation counter

The connection plan.org's table implies but does not spell out: a version word
in the dev build is the degenerate form of SBCL's layout. SBCL's instance points
at a layout object whose `invalid` flag can be flipped; a Flan dev-build struct
would carry an integer stamped at construction, compared against the type's
current generation on access. Flip the generation on redefinition, and the next
field access on an old-generation value traps. That is `layout-invalid` with the
indirection collapsed into an immediate, which is the only form that survives
"instances have no headers, but dev builds may pay for one extra word".

What it buys, honestly: **not** the ability to keep using old values. It buys a
*diagnosis* instead of a refusal — the reload is accepted, the program keeps
running, and the first read of a stale value says "this `Cursor` was built
before you changed `Cursor`" with a location, instead of the session saying
"restart" before anything runs. That is genuinely SBCL's bargain (`continue`
then trap), and it is the honest framing: the version word converts a
compile-time refusal into a runtime trap. Whether that is an improvement is a
judgement about which failure the author would rather debug.

What it costs, all of which is real:

- A word per struct instance in dev builds, so dev and release layouts differ —
  which means array strides, FFI structs, and anything crossing to C differ too,
  or need to be excluded. The dev/release divergence plan.org already accepts
  for frames and cells gets bigger and more observable.
- A check on every field access in dev builds.
- Structs embedded in other structs, in arrays, and in `Vec`s each need a story;
  an array of 4096 cells would need every element stamped and every element
  checked.
- Globals are not covered by it at all. A global's storage is a fixed slot with
  a shape; a version word on the *global* could detect a stale read, but there
  is nothing to read — the point of the refusal is that the old bytes mean
  something else now. SBCL has no precedent to offer, per section 3.

### The dyn side is where CLOS-style update is actually available

Grounding the claim in the code as landed (FIX.org item 6, commits `8d2bf2a`,
`5af990e`, `6c6024e`; `runtime/flan_dyn.c`):

- A `defclass` instance is an ordinary dyn map with a `klass` field in the
  `flan_obj` header (`flan_dyn.c:255`), set by `flan_dyn_map_new_class`
  (`:1088`), read by `flan_dyn_class_of` (`:1100`). It is deliberately in the
  header and not an entry, so `len`, `render` and `dyn_equal` do not see it.
- **Slots are keys.** `lib/classes.ml:150-175`: `(defclass point [x y])`
  desugars to `(defn point [x dyn y dyn] dyn #point{:x x :y y})` and nothing
  else. Access is `get`/`put` by keyword.

Two consequences, and the second is the one that surprised me:

1. **Name-based slot carrying — the hard half of
   `update-instance-for-redefined-class` — is free here.** A dyn instance already
   stores its slots by name. Any migration is a map operation.
2. **There is no class object at runtime to invalidate, because there is no class
   registry at all.** The `klass` field holds an interned keyword, not a pointer
   to a class descriptor. Redefining a `defclass` today is just redefining its
   constructor `defn` — which the session already permits as an ordinary body
   change. Old instances keep their old keys and keep working; new ones get the
   new keys; `class-of` answers the same keyword for both. So the dyn side
   currently has neither the refusal *nor* the protocol: it has silence.

Whether that silence is a bug depends on what the author wants. It is the
Clojure answer (a map is a map; a "class" is a tag) rather than the CLOS answer.
The CLOS answer would need, concretely:

- A runtime class registry: keyword → (slot-name list, generation counter).
  Nothing like this exists today; `lib/classes.ml` is compile-time only and
  emits a plain `defn`.
- A generation stamp per instance, or a cheaper trick: compare the instance's
  key set against the registry's current slot list lazily on `get`/`put`. With
  a generation word in the `flan_obj` header — there is room, and unlike the
  typed side these objects *have* headers — the check is a compare.
- A migration step at first access after a bump: add missing slots as nil,
  collect removed slots into a property list, and call a user hook. The obvious
  Flan spelling of that hook is a generic function, e.g.
  `(defmethod update-for-redefined point [p added discarded] ...)`, which fits
  the dispatch mechanism that already exists.
- A decision on whether `put` of an unknown slot stays legal. Today it is — a
  class instance is an open map, and FIX.org already defers "refusing an unknown
  slot at `(get p :z)`". If unknown slots stay legal, the registry's slot list is
  advisory and the whole update protocol is advisory with it.

This is a real, SBCL/CLOS-precedented design that Flan's runtime can actually
support. It is also a feature with no user yet, since redefinition on the dyn
side currently fails silently rather than loudly.

---

## 6. Candidate designs

Three, none recommended over the others. Cost estimates are rough.

### A. Keep the refusal for typed structs and globals; fix the message

Leave `lib/session.ml` refusing, but rewrite both messages to say what SBCL's
restarts say: name which fields moved, were retyped, or were deleted (SBCL's
`compare-slots` split — Flan compares whole field lists and can cheaply do the
same split), and state the two things the author can do instead (restart, or
rename the type and migrate by hand). Optionally accept the case SBCL's
`mutable-layout-p` accepts and Flan currently refuses: **appending** fields to a
struct no array/FFI type depends on is layout-compatible for every existing
instance, so it need not be refused at all.

- Cost: small, days. Frontend only, no runtime change, no dev/release divergence.
- Behaviour: unchanged except the append case starts working and the diagnostic
  names the offending field.
- Precedent: SBCL refuses by default too — this is the `%redefine-defstruct`
  error with no restart taken, plus `compare-slots`' finer classification. The
  "SBCL warns instead of refusing" premise in DISCUSS.org does not survive
  contact with the source, so "keep refusing" is the SBCL-consistent option, not
  the timid one.

### B. Dev-build struct version word — SBCL's layout generation, collapsed

Stamp every dev-build struct instance with the type's generation at
construction; bump the generation on an incompatible redefinition; check the
stamp on field access in dev builds and trap with "this value was built before
`Cursor` changed shape" instead of refusing the reload. Release builds carry
neither word nor check, which is plan.org's dev/release table as written.

- Cost: large. Touches layout, both backends, arrays and `Vec`s of structs, FFI
  boundary exclusions, and every field-access site; the dev/release layout
  divergence becomes observable wherever a struct crosses to C. This is the one
  that wants a written design before any code.
- Behaviour: the reload is accepted and the program keeps running; stale values
  trap loudly at first touch rather than reading wrong offsets. It does **not**
  let old values keep working — no migration is possible for a flat unboxed
  struct — so it converts a pre-run refusal into a mid-run trap. Globals are not
  covered and still need option A's refusal.
- Precedent: SBCL's `%invalidate-layout` plus the obsolete-instance trap, with
  the layout pointer replaced by an immediate because Flan instances have no
  headers. Note that SBCL, having the same choice, still puts the error in the
  redefinition path *as well* and requires a restart to reach the trap state.

### C. CLOS-style update for dyn `defclass` instances only

Give `defclass` a runtime registry (keyword → slots + generation), stamp a
generation into the `flan_obj` header beside `klass`, and on the first `get`/`put`
after a bump, migrate the instance: add new slots as nil, gather removed slots
into a list, and dispatch a user-overridable generic `update-for-redefined`
before returning. Typed structs and globals are untouched and keep option A's
refusal.

- Cost: medium and well-contained — `runtime/flan_dyn.c` header field plus a
  registry, `lib/classes.ml` emitting the registration, one lazy check on the map
  access path. It rides on machinery that already exists: headers, interned
  keyword tags, generic dispatch, and slots that are already addressed by name.
- Behaviour: redefining a class stops being silent. Instances survive
  redefinition with their shared slots intact, new slots appear as nil, and the
  author can write the coordinate-change-style migration by hand. The risk is
  that it is a ceremony on top of maps that are already open — if `put` of an
  arbitrary key stays legal, the registry describes an intention rather than a
  constraint.
- Precedent: this *is* CLHS 4.3.6 and `update-instance-for-redefined-class`, the
  only part of CL's story that is a standard protocol rather than an
  implementation's policy, and the only part whose prerequisites Flan already
  satisfies.

B and C are independent and could both happen, or neither. A is close to free
and is compatible with both.

---

## Sources

- SBCL source, `src/code/defstruct.lisp` (`%redefine-defstruct`, `compare-slots`,
  `redefine-structure-warning`, `mutable-layout-p`, restart texts) —
  <https://github.com/sbcl/sbcl/blob/master/src/code/defstruct.lisp>
- SBCL source, `src/code/class.lisp` (`%invalidate-layout`, `register-layout`,
  `layout-invalid`) — <https://github.com/sbcl/sbcl/blob/master/src/code/class.lisp>
- SBCL source, `src/pcl/std-class.lisp` (`obsolete-structure` condition,
  `%obsolete-instance-trap`, `make-instances-obsolete`, `invalid-wrapper-p`,
  `replace-wrapper-and-slots`) —
  <https://github.com/sbcl/sbcl/blob/master/src/pcl/std-class.lisp>
- CLHS 4.3.6 Redefining Classes —
  <https://www.lispworks.com/documentation/HyperSpec/Body/04_cf.htm>
- CLHS `update-instance-for-redefined-class` —
  <https://www.lispworks.com/documentation/HyperSpec/Body/f_upda_1.htm>
- CLHS `defparameter`/`defvar` —
  <https://www.lispworks.com/documentation/HyperSpec/Body/m_defpar.htm>
- SBCL Internals, Local Calls — <http://www.sbcl.org/sbcl-internals/Local-Calls.html>
- Block compilation in SBCL 2.0.2 —
  <https://mstmetent.blogspot.com/2020/02/block-compilation-fresh-in-sbcl-202.html>
- Lisp journey, "Structures: lightweight records" (the user-visible restart menu)
  — <https://lisp-journey.gitlab.io/blog/data-structures-structs/>

The SBCL *manual* (<https://www.sbcl.org/manual/>) does not document structure
redefinition, obsolete instances or layout invalidation at all — checked, and it
is silent. Everything in sections 1 and 4 above comes from source and mailing
list/blog material, not from the manual.

### Marked unverified

- The exact condition signalled when a *compiled typed accessor* (as opposed to
  the PCL path) is applied to an instance with an invalidated layout. Stated
  above as "an ordinary `type-error`"; I did not read the accessor emission in
  `src/compiler/`, so treat the specific condition name as unverified. The
  `sb-pcl::obsolete-structure` text is verified from `src/pcl/std-class.lisp`.
- Whether SBCL's `compare-slots` classification ever *avoids* the error for a
  pure append (i.e. whether appending slots is accepted silently). The
  `mutable-layout-p` gate on `recklessly-continue` strongly implies a
  compatible-layout notion exists, but I did not confirm the append case is
  error-free.