Cook the ground into a quadtree of mesh tiles, and keep the mesh the scenery drew

The ground in a scenery is already an irregular mesh, so the cook keeps it: it cuts the
triangles along the tile grid, welds what coincides, and builds each coarser level by
collapsing edges. Nothing is resampled, nothing is interpolated, and a hole the scenery left
stays a hole. What was here before threw the triangles away, kept their vertices and built a
fresh Delaunay triangulation over them, which filled every hole, merged the two heights of
every retaining wall into one, and - because it never erased the super-triangle - emitted
indices three too large together with triangles reaching out to the artificial corners.

Tiles form a quadtree, as Cesium Quantized Mesh intends. A distant tile is a large one, so the
horizon costs tens of tiles rather than the thousands a fixed grid needed, and the level to
draw follows from the tile's own geometric error against what the screen can resolve - which is
what the format carries that field for. Every vertex on a tile's border survives into all of
its coarser levels, and so does every edge between two of them, so two neighbours drawn at
different levels share their border exactly: there is nothing to stitch and nothing to morph.

The file layout is the format's, with two deliberate departures, both written down in
quantizedmeshformat.h: the geodetic header fields carry scenery metres, there being no
geographic reference to convert from, and the material a vertex belongs to rides in an
extension of ours. Texture coordinates are not stored at all - ground tiles in world space, so
the coordinate follows from the position and from how many metres one repeat of the material
covers, measured off the scenery once and kept in the terrain's table.

Tiles live in one archive beside the scenery, an entry each, compressed on its own. Baltyk
cooks to 546 kB against 2.2 MB of heightfields, quark to 361 kB against 9.6 MB, tomaszewo to
375 kB against 9.9 MB.

Drawing is one call a frame: every resident tile lives in the same vertex and index buffer,
with its vertices measured from the middle of the terrain so that the whole map fits in floats,
and a frame is one glMultiDrawElementsBaseVertex over the tiles the walk chose and the frustum
kept. On baltyk that settles at 291 tiles resident, 16 MB of mesh, 85 tiles and 6k triangles
drawn.

What counts as ground is what the scenery says, not what its triangles look like. A node under
origin, rotate or scale is an object standing on the ground; a material with an alpha channel is
a tree, a bush, a fence or a catenary mast. Everything else is laid whichever way it faces, so
the side of an embankment or a cutting goes in with the rest instead of leaving a hole along
every bank. Whatever is left out is recorded per node, and the scenery goes on drawing it.

A cook is a cache: it is used again only when the archive, the note of which nodes went into it
and the manifest of what the scenery was made from all agree. A manifest listing a file the
scenery names but does not have used to fail that test for ever, since an absent file never
matches - quark has such an include, and nothing beside it could ever be reused.

tools/terraincook cooks the same way outside the engine, and meshtest beside it checks what the
cook promises: a tile reads back as written, neighbouring tiles agree on their border to the
millimetre, a hole is not filled in, a wall is laid exactly once, and a material's tiling comes
out of the source. seamcheck.py goes: it tested the morphing constants of a heightfield nothing
reads any more.

rendering/terrainclipmap.cppm

@@ -11,37 +11,35 @@ module;
 #include <array>
 #include <cstdint>
 #include <functional>
-#include <memory>
 #include <optional>
 #include <string>
 #include <unordered_map>
+#include <unordered_set>
 #include <vector>
 
 #include "scene/quantizedmeshreader.h"
 
 export module eu07.rendering.terrainclipmap;
 import eu07.gl.shader;
-import eu07.gl.buffer;
-import eu07.gl.vao;
 import eu07.glm;
 import eu07.rendering.terraintileloader;
 import eu07.rendering.terrainstatus;
-import eu07.rendering.terraintilepool_tin;
+import eu07.rendering.terrainmesharena;
 import eu07.rendering.terrainground;
 
 export {
 
-// Draws TIN (Triangulated Irregular Network) terrain from Quantized Mesh tiles.
+// Draws a scenery's cooked terrain: a quadtree of mesh tiles, each a piece of the ground as the
+// scenery drew it, thinned out the further from the camera it is drawn.
 //
-// Replaces the old heightfield texture array approach with vertex/index buffers per tile.
-// Each tile is an independent triangulated mesh with multiple LOD levels. The terrain adapts
-// density to detail: fine triangulation along track corridors, coarse in open fields.
+// Each frame a walk of the quadtree settles on the tiles to draw: a tile is split when the ground
+// it stands for would be off by more pixels than the detail setting allows, and only when all four
+// of its children are on the card - otherwise it is drawn itself and the children are asked for, so
+// that streaming shows coarse ground rather than a hole. Because every vertex on a tile's border
+// survives into all of its coarser levels, two neighbours drawn at different levels share their
+// border exactly, and nothing has to be stitched or morphed.
 //
-// CDLOD (Continuous Distance LOD): tiles select their LOD based on distance from camera,
-// with edge stitching to prevent cracks between LOD transitions.
-//
-// The class owns GL resources and file handles. Given a viewpoint and visibility test,
-// it loads, uploads, and renders appropriate tiles each frame.
+// Every resident tile lives in the same pair of buffers, so the whole terrain is drawn in one call.
 class terrain_clipmap {
 
 public:
@@ -53,116 +51,121 @@ public:
     terrain_clipmap( terrain_clipmap const & ) = delete;
     terrain_clipmap &operator=( terrain_clipmap const & ) = delete;
 
-    // opens a cooked TIN directory and builds the gl resources. returns false when the
-    // directory cannot be read, leaving the object unusable but safe to destroy
+    // opens a cooked terrain - an archive or a directory of tiles - and builds the gl resources.
+    // returns false when it cannot be read, leaving the object unusable but safe to destroy
     bool open( std::string const &Path );
     bool ready() const { return m_ready; }
     void close();
 
-    // asks for the tiles around the viewpoint, uploads a bounded number of those that have
-    // arrived, and retires the ones that left range. safe to call every frame: the reading
-    // and decoding happen on the loader's thread
+    // walks the quadtree, asks for what is missing, puts in what has arrived and retires what has
+    // not been wanted for a while. safe to call every frame: the reading happens on the loader's
+    // thread
     void update( glm::dvec3 const &Viewpoint );
-    // draws the resident tiles in range that the test says can be seen. the caller is
-    // expected to have the scene and model uniform blocks already set for a camera-relative
-    // pass. tile positions are passed as uniforms per draw
+    // draws the tiles the walk settled on that the test says can be seen
     void render( glm::dvec3 const &Viewpoint, visibility const &Visible );
 
     // how far from the viewpoint terrain is drawn, in metres
     void range( float const Range ) { m_range = Range; }
     float range() const { return m_range; }
-    // what the screen can resolve: the viewport's pixels per unit of size over distance, and
-    // how many pixels a sample spacing may cover. decides how far each level reaches
+    // what the screen can resolve: the viewport's pixels per unit of size over distance, and how
+    // many pixels a tile's error may cover before a finer level is used
     void detail( double const Pixelsperunit, double const Pixels );
-    // the finest level's range the tiles in range were last worked out with
-    float finest() const { return static_cast<float>( m_finest ); }
-
-    // metres between samples at the finest level; a scenery's fields are drawn from the
-    // finest to the coarsest so that the depth test keeps the better surface
-    float gridstep() const { return m_reader.gridstep(); }
-    // pushes this field back in depth by the given rank, so that geometry lying on it wins the
-    // depth test, and a coarser field loses to a finer one describing the same ground. the
-    // offset is in depth, scaled by slope, so it holds at any distance and on any incline
+    // side of a tile at the finest level, in metres. terrains are drawn finest first, so that the
+    // depth test keeps the better surface where two of them describe the same ground
+    float gridstep() const { return m_finesttile; }
+    // pushes this terrain back in depth by the given rank, so that geometry lying on it wins the
+    // depth test, and a coarser terrain loses to a finer one describing the same ground
     void depthrank( unsigned const Rank ) { m_depthrank = Rank; }
 
     terrain_statistics const &stats() const { return m_stats; }
     std::string const &path() const { return m_path; }
-    // metres per tile side
-    float tilesize() const { return m_tilesize; }
-    // number of LOD levels available
-    std::uint32_t levels() const { return m_lodlevels; }
+    float tilesize() const { return m_finesttile; }
+    std::uint32_t levels() const { return m_levels; }
     std::size_t backlog() const { return m_loader.backlog(); }
+    float detailpixels() const { return static_cast<float>( m_detail ); }
+    // side of a tile at each level, and how far that level's ground is off, for the debug panel
+    float leveltile( std::uint32_t const Level ) const;
+    float levelerror( std::uint32_t const Level ) const;
 
 private:
-    // one tile resident on the gpu: VBO/IBO handle
+    // one tile held in the buffers
     struct resident_tile {
-        std::int32_t x { 0 }, z { 0 };
+        terrain_mesh_arena::piece piece;
+        // for culling, in world space
+        glm::dvec3 centre { 0.0 };
+        float radius { 0.f };
         std::uint32_t level { 0 };
-        terrain_tile_pool_tin::tile_handle gpu_data;
-        float minheight { 0.f }, maxheight { 0.f };
         std::uint64_t lastused { 0 };
     };
 
-    // one tile within range, and the level its distance earns
-    struct in_range {
-        std::int64_t key;
-        std::uint32_t level;
-    };
-
-    // mip level a tile at the given distance is drawn at
-    std::uint32_t level_for( double const Distance ) const;
-    // works out which tiles are in range and at what level, and tells the loader what is
-    // missing. the answer only changes once the camera has moved a fair part of a tile
-    void scan( glm::dvec3 const &Viewpoint );
-    // puts a tile the loader has read on the gpu, replacing whatever level it had before
-    void upload( terrain_tile_loader::payload const &Tile );
-    void retire( resident_tile const &Tile );
+    // the tiles the walk settled on, in the order they will be drawn
+    void walk( glm::dvec3 const &Viewpoint );
+    // asks the loader for every tile the wanted cut of the quadtree is missing, and says whether this
+    // tile and everything under it is already there
+    bool need( quantizedmesh::tile_address const &Tile, glm::dvec3 const &Viewpoint );
+    // whether the level this tile stands for is close enough to the ground below it
+    bool good_enough( quantizedmesh::tile_address const &Tile, double Distance ) const;
+    void choose( quantizedmesh::tile_address const &Tile, glm::dvec3 const &Viewpoint );
+    // where a tile's square is, and how far the viewpoint is from it
+    void square( quantizedmesh::tile_address const &Tile, double &West, double &North, double &Side ) const;
+    double distance_to( quantizedmesh::tile_address const &Tile, glm::dvec3 const &Viewpoint ) const;
+    bool present( quantizedmesh::tile_address const &Tile ) const;
+    void ask_for( quantizedmesh::tile_address const &Tile, double Distance );
+    void put_in( terrain_tile_loader::payload const &Tile );
+    void retire_unused();
     void resolve_uniforms();
     void report_residency();
 
-    static std::int64_t key( std::int32_t const X, std::int32_t const Z ) {
-        return ( static_cast<std::int64_t>( X ) << 32 ) ^ static_cast<std::uint32_t>( Z );
-    }
-
-    // Tile management
-    quantizedmesh::reader m_reader;
+    quantizedmesh::reader m_reader;      // the table and which tiles there are; reading is the loader's
     terrain_tile_loader m_loader;
-    terrain_tile_pool_tin m_pool;
-    
-    std::vector<terrain_tile_loader::request> m_wanted;
-    std::vector<terrain_tile_loader::payload> m_arrived;
-    // what the last scan found in range, and the level wanted for each tile
-    std::vector<in_range> m_inrange;
-    std::unordered_map<std::int64_t, std::uint32_t> m_wantedlevel;
-    glm::dvec3 m_scanpoint { 0.0 };
-    float m_scanrange { -1.f };
-    double m_finestwanted { 0.0 };
-    double m_finest { 0.0 };
-    
-    // Resident tiles (key = tile_key(x,z))
-    std::unordered_map<std::int64_t, resident_tile> m_tiles;
-    
-    // Ground texture array (materials)
+    terrain_mesh_arena m_arena;
     terrain_ground m_ground;
-    
-    // Shader and uniforms
+
+    quantizedmesh::terrain_table m_table;
+    glm::dvec3 m_origin { 0.0 };         // what the vertices in the buffers are measured from
+    float m_finesttile { 256.f };
+    std::uint32_t m_levels { 1 };
+
+    std::unordered_map<std::int64_t, resident_tile> m_tiles;
+    std::vector<std::int64_t> m_chosen;
+    std::vector<std::pair<double, quantizedmesh::tile_address>> m_wishes;
+    std::vector<quantizedmesh::tile_address> m_wanted;
+    std::vector<terrain_tile_loader::payload> m_arrived;
+    std::unordered_set<std::int64_t> m_asked;    // what the last walk asked the loader for
+    // whether a subtree could be drawn, worked out once per walk
+    std::unordered_map<std::int64_t, bool> m_readiness;
+
+    // what the draw hands to gl, kept between frames so that a frame allocates nothing
+    std::vector<std::int32_t> m_counts;
+    std::vector<void const *> m_offsets;
+    std::vector<std::int32_t> m_bases;
+
+    glm::dvec3 m_walkpoint { 0.0 };
+    bool m_walked { false };
+    double m_pixelsperunit { 0.0 };
+    double m_detail { 3.0 };
+    double m_walkdetail { 0.0 };
+    float m_range { 6000.f };
+    float m_walkrange { -1.f };
+
     std::optional<gl::program> m_shader;
     struct uniform_locations {
-        int tile_offset { -1 };      // vec3: tile origin relative to camera
-        int ground { -1 };            // sampler2DArray: material textures
+        int origin { -1 };       // where the terrain's origin is, relative to the camera
+        int world { -1 };        // and where it is in the world
+        int ground { -1 };
+        int materials { -1 };
     } m_uniforms;
 
-    // Configuration
-    float m_tilesize { 256.0f };
-    std::uint32_t m_lodlevels { 4 };
     terrain_statistics m_stats;
     std::uint64_t m_frame { 0 };
     std::size_t m_reportedresidency { 0 };
+    std::size_t m_reporteddrawn { 0 };
+    std::uint64_t m_reportedframe { 0 };
     bool m_reported { false };
-    float m_range { 6000.f };
     unsigned m_depthrank { 0 };
-    std::size_t m_budget { 2048 };         // tiles allowed resident
-    std::size_t m_uploadsperframe { 32 };
+    // what the buffers may hold before tiles nobody asked for lately are let go
+    std::size_t m_budget { 192u << 20 };
+    std::size_t m_putsperframe { 16 };
     bool m_ready { false };
     std::string m_path;
 };