16
0
mirror of https://github.com/MaSzyna-EU07/maszyna.git synced 2026-07-19 06:29:18 +02:00

Improve binary scenery format (v5): streaming reader, smaller files, headless parallel bake

- Reader is buffer-based and streaming: the twin is read once and parsed by
  pointer, entries decoded on demand (no per-byte stream, no vector-of-entry
  materialisation), string tokens served as views into the buffer.
- Format v5 is markedly smaller without compression: string interning with
  varint indices, entry type packed into the head varint, integers as zig-zag
  varint and fractional numbers as f32 (f64 only when needed). Tokens are stored
  in original case with a quoted flag and lower-cased per consumer at replay, so
  capture is grammar-independent.
- Writer encodes entries incrementally into a compact buffer instead of holding
  a struct per token, keeping memory bounded when baking huge files in parallel.
- New headless bake mode (-bake) precompiles a scenario and all its includes into
  twins on a thread pool, with no window/renderer/scene; each file is tokenised in
  isolation and baked exactly once.
- Fix stack overflow on files with long runs of consecutive includes by handling
  include directives iteratively instead of recursively (also hardens the normal
  text load).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
maj00r
2026-06-22 18:49:09 +02:00
parent 65fd1c10f3
commit eec5986056
5 changed files with 573 additions and 266 deletions

View File

@@ -20,6 +20,7 @@ Stele, firleju, szociu, hunter, ZiomalCl, OLI_EU and others
#include "application/application.h"
#include "utilities/Logs.h"
#include "scene/scenerybinary.h"
#include <cstdlib>
#ifdef WITHDUMPGEN
#ifdef _WIN32
@@ -98,6 +99,12 @@ int main(int argc, char *argv[])
int dynamic = std::stoi(std::string(argv[5]));
export_e3d_standalone(in, out, flags, dynamic);
}
// headless bake: precompile a scenery (and all its includes) into binary twins with
// no window, renderer or simulation. usage: eu07 -bake <sceneryfile relative to scenery/>
else if (argc >= 3 && std::string(argv[1]) == "-bake")
{
scene::scenerybinary_bake_headless(argv[2], Global.asCurrentSceneryPath, Global.bLoadTraction);
}
else
{
try