On a million-instance scenery (tomaszewo) the infra pass was spending ~24s
reopening pure-visual leaf twins (grass.incb etc.) just to skip their content --
~200k cParser constructions, because every flora include reopens the same leaf.
Twin header now flags whether a file has any infrastructure node or include
(format bumped to v10). A pure-visual leaf (flora .incb: triangles + transform
directives only) has it clear, so the infra pass skips opening it: the first open
of each file caches the verdict, later opens are dropped before construction.
Result on tomaszewo: infra 55s -> 31s, getToken 1.06M -> 89k. Also adds a load
profiler (per-type build time, dispatch time, getToken count) behind WriteLog so
the next bottleneck is measured, not guessed (it's now the 25s of decorative
vehicle media loading).
Replaces the camera-distance ring passes with section-following streaming and a
persistent section index, so a million-node scenery no longer re-scans the whole
twin every time the camera moves into new ground.
- v9 node marker also stores range_max; a model visible from beyond the stream
radius (or unlimited) is built once up front, the rest stream by section so
distant landmarks/traction/buildings don't pop out while flora stays local.
- Reader gains node_offset()/seek_node(); cParser exposes the deepest twin's
file/path/offset/params and seekReplayNode/setReplayParams to rebuild one node.
- First visual pass indexes every deferred node (models via the dispatch fast
path, origin-placed flora/shapes in deserialize_node) under its region section
while building the spawn area; later cycles rebuild only the newly-wanted
sections by seeking straight to their nodes -- O(visible), not O(whole twin).
- Build-all fallback retained for ghostview with no camera centre.
Known: absolute terrain triangles (no origin) still build in the first pass; only
origin-placed content section-streams. Untested in-game (format bump needs a
rebake).
Pass filtering (infrastructure eager / visual deferred) relies on the per-node
markers that only a binary twin carries. An un-baked text include has no markers,
so it can't be split: it loads in full during the infrastructure pass. Reopening
it on the visual pass reprocessed every node a second time and, for parameterized
text includes, re-ran their models against a parser whose parameters no longer
applied, leaking a literal "(pN)" into TAnimModel::Load -> std::stoi throw ->
crash while streaming visuals. Drop the text include on the visual pass; its
content already loaded on the first pass. Fully-baked include trees are unaffected
(their children replay as twins and defer normally).
The camera-ring visual load replays the twin once per distance ring, and each pass
had to read every node's header + X Y Z tokens (~8 tokens through the cParser pipeline)
just to decide whether the node is in the current ring -- expensive when a scenery has
a million flora instances.
Bump the twin format to v7: a visual model node's marker now carries its local position
(3 f32), captured at bake time from the node's first three numbers. The reader hands the
position out (scenery_binary_reader::node_position), and the deserializer ring-tests a
model and skips it in O(1) over the marker span, straight from the dispatch loop, without
deserialize_node decoding any of its tokens. deserialize_model uses the same marker
position for its own ring test so the two decisions agree exactly (a node near a ring
boundary can't be dropped by both adjacent rings). Shapes/older twins have no marker
position and fall back to the token-based test. The per-frame visual budget is factored
into VISUAL_BUDGET_MS.
Verified: td.scn rebakes to v7, no duplicates, no unexpected tokens, completes ~1s.
tomaszewo (1M+ flora) stays memory-bounded with no duplicates; full streaming is still
paced by walking every node per ring (the remaining cost is the replay pipeline itself,
which a spatial section index would address).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous nearest-first build captured every deferred visual node as text into a
sorted vector, which does not scale: one tomaszewo flora file alone holds 440k model
nodes (the scenery has 1M+), so the capture ran the process to ~7 GB and its
enumeration never finished.
Stream the visual nodes in camera-distance rings instead, with no per-node capture
(O(1) memory). The visual pass replays the twin once per ring (nearest first); a node
is built only when its squared distance to the camera -- sampled once when the visual
phase starts, so the partition is stable across passes -- falls in the current ring,
otherwise the rest of its body is skipped in O(1) by jumping over the v6 marker span.
Each node is therefore built exactly once, in roughly nearest-first order, through the
normal node path (instancing buckets unchanged). Explicit triangles/lines shapes have
no single position to ring-test by, so they build in the nearest ring pass only.
Reader gains skip_to_node_end() (remembers the served node's end and jumps the cursor
there); cParser::skipReplayNode() delegates it down the active include child.
Verified: td.scn builds 4 rings, no duplicates, no unexpected tokens, complete ~1s
after the infrastructure pass. tomaszewo stays memory-bounded (no OOM, no duplicates)
where the capture approach previously hung.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When replaying a binary twin, load in two passes over the same data: the first
pass loads infrastructure (tracks/traction/events/memcells/sounds + directives),
the second pass loads the visual nodes (3d models, terrain shapes/lines) that the
reader skipped via the v6 node-class markers.
- cParser: setReplayPass() selects the served node class (propagated to include
children); restartReplay() rewinds the twin for the second pass.
- state_serializer: first pass uses the infrastructure pass; on completion it
restarts the twin for the visual pass. Stateful directives (trainset, event,
camera, light, sky, time, ...) are skipped on the visual pass so their side
effects do not duplicate; transform/group directives re-run so deferred visual
nodes get correct placement. A text/compile load (no twin) stays single-pass.
Verified: td.scn replays through both passes with no duplicate vehicles/events
and reaches the normal load endpoint. This is the sequential foundation; moving
the visual pass into the driver (so play starts after the infrastructure pass) is
the next step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each top-level node is wrapped in a marker carrying its class (infrastructure vs
visual) and byte span, so the reader can serve or skip a whole node per load pass
without knowing its terminator token. The bake recognizes nodes by the "node"
keyword + type token (map type->terminator confirmed against the deserializers)
and classifies them; the writer buffers a node's entries and emits the marker.
The reader gains a pass selector (all / infrastructure / visual) and skips nodes
not in the pass by advancing past their byte span.
This is the foundation only: the default pass is "all", so loading is identical
to v5 (markers are transparent) -- verified by loading td.scn to the same point.
The eager/visual split is wired up by the upcoming loader/driver integration.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Reader is buffer-based and streaming: the twin is read once and parsed by
pointer, entries decoded on demand (no per-byte stream, no vector-of-entry
materialisation), string tokens served as views into the buffer.
- Format v5 is markedly smaller without compression: string interning with
varint indices, entry type packed into the head varint, integers as zig-zag
varint and fractional numbers as f32 (f64 only when needed). Tokens are stored
in original case with a quoted flag and lower-cased per consumer at replay, so
capture is grammar-independent.
- Writer encodes entries incrementally into a compact buffer instead of holding
a struct per token, keeping memory bounded when baking huge files in parallel.
- New headless bake mode (-bake) precompiles a scenario and all its includes into
twins on a thread pool, with no window/renderer/scene; each file is tokenised in
isolation and baked exactly once.
- Fix stack overflow on files with long runs of consecutive includes by handling
include directives iteratively instead of recursively (also hardens the normal
text load).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Text scenery component files (.scn/.inc/.scm/.ctr) are compiled into
per-file binary twins (.scnb/.incb/.scmb/.ctrb), handled transparently
at the cParser layer: a fresh twin (mtime-checked, version-matched) is
replayed instead of re-tokenizing text; otherwise the text is parsed and
a twin is compiled alongside it for next time.
Format details:
- per-file string interning: keywords/paths stored once, referenced by
varint index (so node/endmodel/... are not repeated as text)
- numeric tokens stored as 8-byte IEEE doubles, not ASCII
- includes kept as references with parameters; random sets stored verbatim
and re-evaluated on every load (choice not frozen at compile time)
- twin writing offloaded to a bounded thread pool so baking overlaps
scene construction instead of blocking the load
The legacy terrain-only .sbt path is removed; terrain now loads as
ordinary scenery content.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
