Each top-level node is wrapped in a marker carrying its class (infrastructure vs
visual) and byte span, so the reader can serve or skip a whole node per load pass
without knowing its terminator token. The bake recognizes nodes by the "node"
keyword + type token (map type->terminator confirmed against the deserializers)
and classifies them; the writer buffers a node's entries and emits the marker.
The reader gains a pass selector (all / infrastructure / visual) and skips nodes
not in the pass by advancing past their byte span.
This is the foundation only: the default pass is "all", so loading is identical
to v5 (markers are transparent) -- verified by loading td.scn to the same point.
The eager/visual split is wired up by the upcoming loader/driver integration.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Reader is buffer-based and streaming: the twin is read once and parsed by
pointer, entries decoded on demand (no per-byte stream, no vector-of-entry
materialisation), string tokens served as views into the buffer.
- Format v5 is markedly smaller without compression: string interning with
varint indices, entry type packed into the head varint, integers as zig-zag
varint and fractional numbers as f32 (f64 only when needed). Tokens are stored
in original case with a quoted flag and lower-cased per consumer at replay, so
capture is grammar-independent.
- Writer encodes entries incrementally into a compact buffer instead of holding
a struct per token, keeping memory bounded when baking huge files in parallel.
- New headless bake mode (-bake) precompiles a scenario and all its includes into
twins on a thread pool, with no window/renderer/scene; each file is tokenised in
isolation and baked exactly once.
- Fix stack overflow on files with long runs of consecutive includes by handling
include directives iteratively instead of recursively (also hardens the normal
text load).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Text scenery component files (.scn/.inc/.scm/.ctr) are compiled into
per-file binary twins (.scnb/.incb/.scmb/.ctrb), handled transparently
at the cParser layer: a fresh twin (mtime-checked, version-matched) is
replayed instead of re-tokenizing text; otherwise the text is parsed and
a twin is compiled alongside it for next time.
Format details:
- per-file string interning: keywords/paths stored once, referenced by
varint index (so node/endmodel/... are not repeated as text)
- numeric tokens stored as 8-byte IEEE doubles, not ASCII
- includes kept as references with parameters; random sets stored verbatim
and re-evaluated on every load (choice not frozen at compile time)
- twin writing offloaded to a bounded thread pool so baking overlaps
scene construction instead of blocking the load
The legacy terrain-only .sbt path is removed; terrain now loads as
ordinary scenery content.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
