Give the terrain its own archive, its normals, and a level per tile

The tiles moved out of the scenery's pak into an archive of their own: a
sorted index of fixed-size entries found by where a tile sits in the
quadtree rather than by a name, each entry compressed on its own so a
tile can be read without touching the rest. A name apiece would have been
seventy megabytes of index to read before the first tile could be drawn.

An entry carries what its tile's ground asks for - how far it departs
from the finest mesh, and how long its triangles are - because which
level to draw has to be settled for tiles that are not on the card yet,
and the index is the only thing about them that is read up front. Until
now the figure came from the tile's level, which is the worst tile in it,
so one steep corner spoke for every tile that shared its level.

The mesh carries a normal per vertex, octahedron-encoded into two bytes,
in place of the face normal the fragment stage worked out from how the
view position changed across a triangle. A face normal makes the ground
read as facets, and a coarser level's facets face differently from the
ones they stand for, so the light over a hillside changed as the level
did. A geometry stage hands the fragment stage the three materials of the
triangle it is in and the barycentric weights, so ground changes over a
triangle rather than along its edge; 3.3 has no gl_BaryCoordEXT.

And the level a tile is drawn at now counts the camera's height. The
distance was measured on the ground plane alone, which puts the tile
underfoot nought metres away from a camera a kilometre above it - and
nought metres asks for the finest level there is, so a view from the air
wanted the whole ground at full detail, more tiles than the buffers hold,
and what would not fit came out as holes.

Archives written under the old rules are refused at the door and baked
again.

application/driveruipanels.cpp

@@ -770,11 +770,13 @@ debug_panel::render_section_scenario() {
 		{
 			auto terrainrange = std::log(Global.TerrainRange);
 			if (ImGui::SliderFloat(
-				(to_string(std::exp(terrainrange), 0, 5) + " m###terrainrange").c_str(), &terrainrange, std::log(500.0f), std::log(100000.0f), "Terrain range")) {
+				(to_string(std::exp(terrainrange), 0, 5) + " m###terrainrange").c_str(), &terrainrange, std::log(500.0f), std::log(250000.0f), "Terrain loaded out to")) {
-				Global.TerrainRange = std::clamp(std::exp(terrainrange), 500.0f, 100000.0f);
+				Global.TerrainRange = std::clamp(std::exp(terrainrange), 500.0f, 250000.0f);
 			}
 			// pixels a terrain sample may cover before a finer level takes over
-			ImGui::SliderFloat("###terraindetail", &Global.TerrainDetail, 1.0f, 8.0f, "Terrain detail: %.1f px per sample");
+			// how many pixels a level's departure from the finest mesh may cover before the finer one
+			// is used. lower is more detail, and the panel above says what it comes to in metres
+			ImGui::SliderFloat("###terraindetail", &Global.TerrainDetail, 0.5f, 16.0f, "Terrain detail: %.1f px of error allowed");
 		}
     }
 
@@ -1505,33 +1507,52 @@ debug_panel::update_section_renderer( std::vector<text_line> &Output ) {
 
             Output.emplace_back( textline, Global.UITextColor );
 
-            // terrain streaming: how far tiles are loaded, how many are held, and how far
+            // terrain streaming, as a table: a header, a line per level, and nothing said twice
-            // behind the camera the loader is
             for( auto const &field : TerrainStatus ) {
                 auto const &stats { field.stats };
                 auto const slash { field.path.find_last_of( "/\\" ) };
+                auto const metres = []( double const Value ) {
+                    return ( Value >= 10000.0 )
+                        ? to_string( Value / 1000.0, 0 ) + " km"
+                        : to_string( Value, 0 ) + " m"; };
+
                 Output.emplace_back(
-                    "Terrain: " + ( slash == std::string::npos ? field.path : field.path.substr( slash + 1 ) )
+                    "Terrain " + ( slash == std::string::npos ? field.path : field.path.substr( slash + 1 ) )
-                    + ", finest tile " + to_string( field.tilesize, 0 ) + " m, " + std::to_string( field.levels ) + " levels"
+                    + "  tile " + metres( field.tilesize )
-                    + ", loaded out to " + to_string( field.range, 0 ) + " m, detail " + to_string( field.detail, 1 ) + " px",
+                    + "  " + std::to_string( field.levels ) + " levels"
+                    + "  range " + metres( field.range )
+                    + "  detail " + to_string( field.detail, 1 ) + " px",
                     Global.UITextColor );
                 Output.emplace_back(
-                    "  tiles: " + std::to_string( stats.chosen ) + " chosen, "
+                    "  drawn " + std::to_string( stats.drawn ) + "/" + std::to_string( stats.chosen )
-                    + std::to_string( stats.drawn ) + " drawn (" + std::to_string( stats.triangles / 1000 ) + "k triangles), "
+                    + "  " + std::to_string( stats.triangles / 1000 ) + "k tris"
-                    + std::to_string( stats.resident ) + " on gpu (" + std::to_string( stats.gpubytes / 1048576 ) + " MB), "
+                    + "  reach " + metres( stats.furthest )
-                    + std::to_string( stats.wanted ) + " wanted, " + std::to_string( field.backlog ) + " in loader, "
+                    + "   gpu " + std::to_string( stats.resident ) + " tiles "
-                    + std::to_string( stats.uploaded ) + " uploaded, " + std::to_string( stats.dropped ) + " dropped stale",
+                    + std::to_string( stats.gpubytes / 1048576 ) + " MB"
+                    + "   queue " + std::to_string( stats.wanted ) + " wanted "
+                    + std::to_string( field.backlog ) + " reading"
+                    + "   " + std::to_string( stats.uploaded ) + " up "
+                    + std::to_string( stats.dropped ) + " dropped",
                     Global.UITextColor );
-                std::string levels { "  levels:" };
+                Output.emplace_back( "  lvl     tile      off      tri     from    tiles     tris", Global.UITextColor );
                 for( std::uint32_t level = 0; ( level < field.levels ) && ( level < stats.perlevel.size() ); ++level ) {
-                    levels +=
+                    auto const finest { level + 1 == field.levels };
-                        " L" + std::to_string( level )
+                    auto const from {
-                        + " (" + to_string( field.leveltile[ level ], 0 ) + " m tile, "
+                        finest ? 0.0 : terrain_level_from(
-                        + to_string( field.levelerror[ level ], 2 ) + " m off): "
+                            field.levelerror[ level ], field.leveledge[ level ], field.pixelsperunit, field.detail ) };
-                        + std::to_string( stats.perlevel[ level ] );
+                    auto const column = []( std::string Text, std::size_t const Width ) {
-                    if( level + 1 < field.levels ) { levels += " |"; }
+                        while( Text.size() < Width ) { Text = " " + Text; }
+                        return Text; };
+                    Output.emplace_back(
+                        "  " + column( std::to_string( level ), 3 )
+                        + column( metres( field.leveltile[ level ] ), 9 )
+                        + column( to_string( field.levelerror[ level ], 1 ) + " m", 9 )
+                        + column( to_string( field.leveledge[ level ], 0 ) + " m", 9 )
+                        + column( metres( from ), 9 )
+                        + column( std::to_string( stats.perlevel[ level ] ), 9 )
+                        + column( std::to_string( stats.trianglesperlevel[ level ] / 1000 ) + "k", 9 ),
+                        Global.UITextColor );
                 }
-                Output.emplace_back( levels, Global.UITextColor );
             }
 
             textline += "\nRendering mode: ";

rendering/opengl33renderer.cpp

@@ -2917,9 +2917,12 @@ void opengl33_renderer::Render_terrain()
 			    Global.TerrainDetail);
 			field->update(viewpoint);
 		}
-		field->render(viewpoint, [this](glm::dvec3 const &Centre, float const Radius) {
+		field->render(
-			return m_renderpass.pass_camera.visible(scene::bounding_area(Centre, Radius));
+		    viewpoint,
-		});
+		    [this](glm::dvec3 const &Centre, float const Radius) {
+			    return m_renderpass.pass_camera.visible(scene::bounding_area(Centre, Radius));
+		    },
+		    m_current_viewport->main);
 	}
 
 	if (m_current_viewport->main)
@@ -2938,10 +2941,14 @@ void opengl33_renderer::Render_terrain()
 				{
 					status.leveltile[level] = field.leveltile(level);
 					status.levelerror[level] = field.levelerror(level);
+					status.leveledge[level] = field.leveledge(level);
 				}
 			}
 			status.range = field.range();
 			status.detail = field.detailpixels();
+			status.pixelsperunit =
+			    static_cast<float>(m_current_viewport->height) /
+			    (2.f * std::tan(glm::radians(Global.FieldOfView / Global.ZoomFactor) * 0.5f));
 			status.backlog = field.backlog();
 			status.stats = field.stats();
 		}

rendering/terrainclipmap.cpp

@@ -57,7 +57,7 @@ terrain_clipmap::open( std::string const &Path ) {
         close();
         return false;
     }
-    if( false == m_arena.create() ) {
+    if( ( false == m_arena.create( false ) ) || ( false == m_widearena.create( true ) ) ) {
         ErrorLog( "Terrain: cannot allocate the terrain buffers" );
         close();
         return false;
@@ -65,8 +65,10 @@ terrain_clipmap::open( std::string const &Path ) {
 
     try {
         gl::shader vertex( "terrainmesh.vert" );
+        gl::shader geometry( "terrainmesh.geom" );
         gl::shader fragment( "terrainmesh.frag" );
-        m_shader.emplace( std::vector<std::reference_wrapper<gl::shader const>>( { vertex, fragment } ) );
+        m_shader.emplace(
+            std::vector<std::reference_wrapper<gl::shader const>>( { vertex, geometry, fragment } ) );
     }
     catch( gl::shader_exception const &error ) {
         ErrorLog( std::string( "Terrain: shader failed, " ) + error.what() );
@@ -96,7 +98,7 @@ terrain_clipmap::open( std::string const &Path ) {
         "Terrain: " + Path + ", " + std::to_string( m_levels ) + " levels, finest tile "
         + std::to_string( static_cast<int>( m_finesttile ) ) + " m over "
         + std::to_string( static_cast<int>( m_table.side ) ) + " m, "
-        + std::to_string( m_reader.present().size() ) + " tiles, "
+        + std::to_string( m_reader.tilecount() ) + " tiles, "
         + std::to_string( m_table.materials.size() ) + " ground materials" );
     return true;
 }
@@ -124,8 +126,9 @@ terrain_clipmap::close() {
     m_wanted.clear();
     m_arrived.clear();
     m_asked.clear();
-    m_readiness.clear();
+    m_maydescend.clear();
     m_arena.destroy();
+    m_widearena.destroy();
     m_ground.destroy();
     m_shader.reset();
     m_reader.close();
@@ -149,6 +152,12 @@ terrain_clipmap::levelerror( std::uint32_t const Level ) const {
     return m_table.error( Level );
 }
 
+float
+terrain_clipmap::leveledge( std::uint32_t const Level ) const {
+
+    return m_table.edge( Level );
+}
+
 void
 terrain_clipmap::detail( double const Pixelsperunit, double const Pixels ) {
 
@@ -164,8 +173,15 @@ terrain_clipmap::square( quantizedmesh::tile_address const &Tile, double &West,
     North = m_table.north + Tile.z * Side;
 }
 
-// the distance to the nearest point of the tile's square, not to its centre: a coarse tile is
+// The distance to the nearest point of the tile's box, not to its centre: a coarse tile is
-// kilometres across, and the one the camera stands on has its centre far outside any draw range
+// kilometres across, and the one the camera stands on has its centre far outside any draw range.
+//
+// The height counts as much as the other two. Measuring on the ground plane alone makes the tile
+// underfoot nought metres away from a camera a kilometre above it, and nought metres asks for the
+// finest level there is - so a view from the air drew the whole ground at full detail, which is
+// more tiles than the buffers hold, and what would not fit came out as holes. The tile's own height
+// range is not known until it is read, so the terrain's is used for all of them: too near rather
+// than too far, which errs towards detail.
 double
 terrain_clipmap::distance_to( quantizedmesh::tile_address const &Tile, glm::dvec3 const &Viewpoint ) const {
 
@@ -173,35 +189,47 @@ terrain_clipmap::distance_to( quantizedmesh::tile_address const &Tile, glm::dvec
     square( Tile, west, north, side );
     auto const nearestx { std::clamp( Viewpoint.x, west, west + side ) };
     auto const nearestz { std::clamp( Viewpoint.z, north, north + side ) };
-    return glm::length( glm::dvec2 { nearestx - Viewpoint.x, nearestz - Viewpoint.z } );
+    auto const nearesty { std::clamp( Viewpoint.y, m_table.lowest, m_table.highest ) };
+    return glm::length( glm::dvec3 {
+        nearestx - Viewpoint.x, nearesty - Viewpoint.y, nearestz - Viewpoint.z } );
 }
 
 bool
 terrain_clipmap::present( quantizedmesh::tile_address const &Tile ) const {
 
-    return m_reader.present().contains( quantizedmesh::tile_key( Tile ) );
+    return m_reader.contains( Tile );
 }
 
 // Whether the ground a tile stands for is close enough to the real thing. The tile's error, at its
 // distance, covers so many pixels; while that is fewer than the detail asks for there is nothing to
 // be had by splitting it. The tile the camera stands on is always split, whatever its error, so that
 // what is underfoot is the ground as the scenery drew it.
+//
+// What is asked is the tile's own error and its own triangles, out of the archive's index, not the
+// worst in its level. The two are far apart: a level's figure is set by whichever tile in it holds
+// the steepest ground, and with that one figure standing for all of them a flat field twenty
+// kilometres out was split as finely as the valley wall that set it.
 bool
 terrain_clipmap::good_enough( quantizedmesh::tile_address const &Tile, double const Distance ) const {
 
     if( Tile.level + 1 >= m_levels ) { return true; }
     if( Distance <= 0.0 ) { return false; }
-    return terrain_level_enough( m_table.error( Tile.level ), Distance, m_pixelsperunit, m_detail );
+    auto const measure { m_reader.measure( Tile ) };
+    return terrain_level_enough(
+        measure.error, measure.edge, Distance, m_pixelsperunit, m_detail );
 }
 
 // Walks down to the levels the screen wants and asks the loader for whatever of them is not on the
-// card, deepest tile of the wanted cut included - not only the tiles one level down. Asking one level
+// card - every tile on the way down, not only the deepest. Asking one level at a time is what a first
-// at a time is what a first attempt at this did, and it stops dead: a tile whose children are all
+// attempt at this did, and it stops dead: a tile whose children are all resident asks for nothing,
-// resident asks for nothing, while what it is waiting for is their children.
+// while what it is waiting for is their children. And a level fetched only once its children turn out
+// to be late is fetched too late to cover anything.
 //
-// Returns whether the cut under this tile, and the tile itself where the cut ends at it, is there. A
+// Returns whether this tile's ground can be drawn at all: either everything below it is there, or the
-// tile whose subtree is not ready yet is asked for as well, so that something coarse covers the
+// tile itself is, and then it stands in for the lot. That distinction is the whole point. Answering
-// ground rather than a hole while the rest is read.
+// only "is the subtree complete" makes one missing tile twenty kilometres away answer no for its
+// parent, and so for its parent's parent, all the way to the root - and then the entire terrain is
+// drawn as the root tile until it arrives. Which is what flickering while the camera moved was.
 bool
 terrain_clipmap::need( quantizedmesh::tile_address const &Tile, glm::dvec3 const &Viewpoint ) {
 
@@ -212,23 +240,22 @@ terrain_clipmap::need( quantizedmesh::tile_address const &Tile, glm::dvec3 const
 
     auto const key { quantizedmesh::tile_key( Tile ) };
     auto const resident { m_tiles.contains( key ) };
+    if( false == resident ) { ask_for( Tile, distance ); }
 
     if( true == good_enough( Tile, distance ) ) {
-        if( false == resident ) { ask_for( Tile, distance ); }
+        m_maydescend.emplace( key, false );
-        m_readiness.emplace( key, resident );
         return resident;
     }
 
-    auto ready { true };
+    auto descend { true };
     for( std::int32_t stepz = 0; stepz < 2; ++stepz ) {
         for( std::int32_t stepx = 0; stepx < 2; ++stepx ) {
             // every child is asked, so the whole cut is queued at once rather than a level a frame
-            ready = need( { Tile.level + 1, Tile.x * 2 + stepx, Tile.z * 2 + stepz }, Viewpoint ) && ready;
+            descend = need( { Tile.level + 1, Tile.x * 2 + stepx, Tile.z * 2 + stepz }, Viewpoint ) && descend;
         }
     }
-    if( ( false == ready ) && ( false == resident ) ) { ask_for( Tile, distance ); }
+    m_maydescend.emplace( key, descend );
-    m_readiness.emplace( key, ready );
+    return descend || resident;
-    return ready;
 }
 
 void
@@ -240,10 +267,12 @@ terrain_clipmap::choose( quantizedmesh::tile_address const &Tile, glm::dvec3 con
     if( distance > m_range ) { return; }
 
     auto const key { quantizedmesh::tile_key( Tile ) };
-    auto const ready { m_readiness.find( key ) };
+    auto const descend { m_maydescend.find( key ) };
-    if( ( ready != m_readiness.end() ) && ( true == ready->second )
+    if( ( descend != m_maydescend.end() ) && ( true == descend->second )
      && ( false == good_enough( Tile, distance ) ) ) {
-        // the whole cut under this tile is on the card: draw that instead
+        // the whole cut under this tile is on the card: draw that instead, and hold on to this one,
+        // which is what covers this ground again as soon as the camera moves
+        m_kept.push_back( key );
         for( std::int32_t stepz = 0; stepz < 2; ++stepz ) {
             for( std::int32_t stepx = 0; stepx < 2; ++stepx ) {
                 choose( { Tile.level + 1, Tile.x * 2 + stepx, Tile.z * 2 + stepz }, Viewpoint );
@@ -271,21 +300,40 @@ terrain_clipmap::walk( glm::dvec3 const &Viewpoint ) {
     m_walkdetail = m_detail;
     m_walked = true;
     m_chosen.clear();
+    m_kept.clear();
     m_wishes.clear();
     m_asked.clear();
-    m_readiness.clear();
+    m_maydescend.clear();
 
     need( { 0, 0, 0 }, Viewpoint );
     choose( { 0, 0, 0 }, Viewpoint );
 
-    // nearest first, so the ground under the camera is never waiting behind the horizon
+    // Coarsest first, and within a level nearest first. What covers ground has to arrive before what
+    // refines it: a fine tile read ahead of the level standing in for it leaves a hole for as long as
+    // the queue takes, and the ground under the camera is still first among equals.
     std::sort( m_wishes.begin(), m_wishes.end(),
-        []( auto const &Left, auto const &Right ) { return Left.first < Right.first; } );
+        []( auto const &Left, auto const &Right ) {
+            if( Left.second.level != Right.second.level ) { return Left.second.level < Right.second.level; }
+            return Left.first < Right.first; } );
     m_wanted.clear();
     m_wanted.reserve( m_wishes.size() );
     for( auto const &wish : m_wishes ) { m_wanted.push_back( wish.second ); }
     m_loader.want( m_wanted );
 
+    // A walk that settles on a handful of tiles where the one before had a hundred is the ground going
+    // coarse for a frame: somewhere a tile is missing and every level between it and the root is
+    // missing too, so the root is the only thing left to stand in. Rare, and worth knowing about, so it
+    // says which tiles it was waiting for.
+    if( ( m_stats.chosen > 8 ) && ( m_chosen.size() * 4 < m_stats.chosen ) && ( false == m_wishes.empty() ) ) {
+        std::string waiting;
+        for( std::size_t index = 0; ( index < m_wishes.size() ) && ( index < 6 ); ++index ) {
+            auto const &tile { m_wishes[ index ].second };
+            waiting += " " + std::to_string( tile.level ) + "/" + std::to_string( tile.x ) + "/"
+                + std::to_string( tile.z ) + " at " + std::to_string( static_cast<int>( m_wishes[ index ].first ) ) + " m";
+        }
+        WriteLog( "Terrain: the walk fell from " + std::to_string( m_stats.chosen ) + " tiles to "
+            + std::to_string( m_chosen.size() ) + ", waiting for" + waiting );
+    }
     m_stats.chosen = m_chosen.size();
     m_stats.wanted = m_wanted.size();
 }
@@ -301,11 +349,14 @@ terrain_clipmap::put_in( terrain_tile_loader::payload const &Tile ) {
         return;
     }
 
-    auto piece { m_arena.put( Tile.data ) };
+    // the narrow arena first; only what will not fit in it costs four bytes an index
+    auto const wide { Tile.data.vertices.size() > m_arena.vertexlimit() };
+    auto piece { wide ? m_widearena.put( Tile.data ) : m_arena.put( Tile.data ) };
     if( false == piece.held() ) { return; }
 
     resident_tile tile;
     tile.piece = piece;
+    tile.wide = wide;
     tile.centre = glm::dvec3 { Tile.data.centrex, Tile.data.centrey, Tile.data.centrez };
     tile.radius = static_cast<float>( Tile.data.radius );
     tile.level = Tile.data.tile.level;
@@ -321,7 +372,12 @@ terrain_clipmap::put_in( terrain_tile_loader::payload const &Tile ) {
 void
 terrain_clipmap::retire_unused() {
 
-    if( m_arena.bytes() <= m_budget ) { return; }
+    // Measured against what the tiles take up, not against what the buffers were allocated at. The
+    // allocation only ever grows - gl cannot hand a buffer back in pieces - so gating on it means
+    // nothing is let go until the buffers reach the cap, and from then on everything is let go every
+    // frame and read again. That is how three thousand tiles came to be held for a working set of
+    // forty, and why it stopped exactly at the cap.
+    if( m_arena.usedbytes() + m_widearena.usedbytes() <= m_budget ) { return; }
 
     std::vector<std::pair<std::uint64_t, std::int64_t>> aged;
     aged.reserve( m_tiles.size() );
@@ -331,13 +387,23 @@ terrain_clipmap::retire_unused() {
     if( true == aged.empty() ) { return; }
     std::sort( aged.begin(), aged.end() );
 
-    // a fifth of what is held, so that this does not run every frame
+    // down to well under the cap rather than just under it, so that this settles instead of running
-    auto const letgo { std::max<std::size_t>( 1, m_tiles.size() / 5 ) };
+    // every frame
-    for( std::size_t index = 0; ( index < letgo ) && ( index < aged.size() ); ++index ) {
+    auto const target { m_budget * 3 / 4 };
-        auto const found { m_tiles.find( aged[ index ].second ) };
+    std::size_t letgo { 0 };
+    for( auto const &[ when, key ] : aged ) {
+        if( m_arena.usedbytes() + m_widearena.usedbytes() <= target ) { break; }
+        auto const found { m_tiles.find( key ) };
         if( found == m_tiles.end() ) { continue; }
-        m_arena.take( found->second.piece );
+        if( true == found->second.wide ) { m_widearena.take( found->second.piece ); }
+        else                             { m_arena.take( found->second.piece ); }
         m_tiles.erase( found );
+        ++letgo;
+    }
+    if( letgo > 0 ) {
+        WriteLog( "Terrain: over budget, let go of " + std::to_string( letgo ) + " tiles, now holding "
+            + std::to_string( ( m_arena.usedbytes() + m_widearena.usedbytes() ) / 1048576 ) + " MB in "
+            + std::to_string( ( m_arena.roombytes() + m_widearena.roombytes() ) / 1048576 ) + " MB of buffers" );
     }
 }
 
@@ -368,18 +434,22 @@ terrain_clipmap::update( glm::dvec3 const &Viewpoint ) {
         walk( Viewpoint );
     }
 
-    // what the walk settled on is wanted now, and is what eviction spares
+    // what the walk settled on is wanted now, and is what eviction spares. how many of them end up
-    m_stats.perlevel.fill( 0 );
+    // drawn at each level is counted by the draw itself, which is the one that knows
     for( auto const key : m_chosen ) {
         auto const found { m_tiles.find( key ) };
         if( found == m_tiles.end() ) { continue; }
         found->second.lastused = m_frame;
-        ++m_stats.perlevel[ std::min<std::size_t>( found->second.level, m_stats.perlevel.size() - 1 ) ];
+    }
+    // the levels above what is drawn are wanted too, as the cover for wherever the camera turns next
+    for( auto const key : m_kept ) {
+        auto const found { m_tiles.find( key ) };
+        if( found != m_tiles.end() ) { found->second.lastused = m_frame; }
     }
 
     retire_unused();
     m_stats.resident = m_tiles.size();
-    m_stats.gpubytes = m_arena.bytes();
+    m_stats.gpubytes = m_arena.usedbytes() + m_widearena.usedbytes();
     report_residency();
 }
 
@@ -408,7 +478,8 @@ terrain_clipmap::report_residency() {
         + std::to_string( m_stats.gpubytes / 1048576 ) + " MB of mesh, "
         + std::to_string( m_stats.uploaded ) + " put in so far, "
         + std::to_string( m_stats.drawn ) + " of " + std::to_string( m_stats.chosen ) + " chosen drawn, "
-        + std::to_string( m_stats.triangles / 1000 ) + "k triangles, levels "
+        + std::to_string( m_stats.triangles / 1000 ) + "k triangles out to "
+        + std::to_string( static_cast<int>( m_stats.furthest ) ) + " m, levels "
         + [ this ]() {
             std::string counts;
             for( std::uint32_t level = 0; level < m_levels; ++level ) {
@@ -418,32 +489,47 @@ terrain_clipmap::report_residency() {
 }
 
 void
-terrain_clipmap::render( glm::dvec3 const &Viewpoint, visibility const &Visible ) {
+terrain_clipmap::render( glm::dvec3 const &Viewpoint, visibility const &Visible, bool const Mainview ) {
 
     if( ( false == m_ready ) || ( true == m_chosen.empty() ) ) { return; }
 
     // the ground textures the engine has finished loading since the last frame
     m_ground.update();
 
-    m_counts.clear();
+    for( auto &list : m_counts ) { list.clear(); }
-    m_offsets.clear();
+    for( auto &list : m_offsets ) { list.clear(); }
-    m_bases.clear();
+    for( auto &list : m_bases ) { list.clear(); }
     std::size_t triangles { 0 };
+    std::array<std::size_t, terrain_maxlevels> pertile {};
+    std::array<std::size_t, terrain_maxlevels> pertriangle {};
+    auto furthest { 0.0 };
     for( auto const key : m_chosen ) {
         auto const found { m_tiles.find( key ) };
         if( found == m_tiles.end() ) { continue; }
         auto const &tile { found->second };
         if( false == Visible( tile.centre, tile.radius ) ) { continue; }
-        m_counts.push_back( static_cast<std::int32_t>( tile.piece.indices ) );
+        auto const level { std::min<std::size_t>( tile.level, terrain_maxlevels - 1 ) };
-        m_offsets.push_back(
+        ++pertile[ level ];
-            reinterpret_cast<void const *>(
+        pertriangle[ level ] += tile.piece.indices / 3;
-                static_cast<std::uintptr_t>( tile.piece.firstindex ) * sizeof( std::uint32_t ) ) );
+        furthest = std::max(
-        m_bases.push_back( static_cast<std::int32_t>( tile.piece.firstvertex ) );
+            furthest,
+            glm::length( glm::dvec2 { tile.centre.x - Viewpoint.x, tile.centre.z - Viewpoint.z } ) );
+        auto const which { tile.wide ? 1 : 0 };
+        auto const width { tile.wide ? sizeof( std::uint32_t ) : sizeof( std::uint16_t ) };
+        m_counts[ which ].push_back( static_cast<std::int32_t>( tile.piece.indices ) );
+        m_offsets[ which ].push_back(
+            reinterpret_cast<void const *>( static_cast<std::uintptr_t>( tile.piece.firstindex ) * width ) );
+        m_bases[ which ].push_back( static_cast<std::int32_t>( tile.piece.firstvertex ) );
         triangles += tile.piece.indices / 3;
     }
-    m_stats.drawn = m_counts.size();
+    if( true == Mainview ) {
-    m_stats.triangles = triangles;
+        m_stats.drawn = m_counts[ 0 ].size() + m_counts[ 1 ].size();
-    if( true == m_counts.empty() ) { return; }
+        m_stats.triangles = triangles;
+        m_stats.perlevel = pertile;
+        m_stats.trianglesperlevel = pertriangle;
+        m_stats.furthest = furthest;
+    }
+    if( ( true == m_counts[ 0 ].empty() ) && ( true == m_counts[ 1 ].empty() ) ) { return; }
 
     m_shader->bind();
     glm::vec3 const origin {
@@ -468,12 +554,22 @@ terrain_clipmap::render( glm::dvec3 const &Viewpoint, visibility const &Visible
         -1.f - 2.f * static_cast<float>( m_depthrank ),
         -2.f - 4.f * static_cast<float>( m_depthrank ) );
 
-    ::glBindVertexArray( m_arena.vertexarray() );
+    if( false == m_counts[ 0 ].empty() ) {
-    ::glMultiDrawElementsBaseVertex(
+        m_arena.bind();
-        GL_TRIANGLES, m_counts.data(), GL_UNSIGNED_INT,
+        ::glMultiDrawElementsBaseVertex(
-        const_cast<void const **>( m_offsets.data() ),
+            GL_TRIANGLES, m_counts[ 0 ].data(), GL_UNSIGNED_SHORT,
-        static_cast<GLsizei>( m_counts.size() ), m_bases.data() );
+            const_cast<void const **>( m_offsets[ 0 ].data() ),
-    ::glBindVertexArray( 0 );
+            static_cast<GLsizei>( m_counts[ 0 ].size() ), m_bases[ 0 ].data() );
+        m_arena.unbind();
+    }
+    if( false == m_counts[ 1 ].empty() ) {
+        m_widearena.bind();
+        ::glMultiDrawElementsBaseVertex(
+            GL_TRIANGLES, m_counts[ 1 ].data(), GL_UNSIGNED_INT,
+            const_cast<void const **>( m_offsets[ 1 ].data() ),
+            static_cast<GLsizei>( m_counts[ 1 ].size() ), m_bases[ 1 ].data() );
+        m_widearena.unbind();
+    }
 
     ::glDisable( GL_POLYGON_OFFSET_FILL );
     ::glActiveTexture( GL_TEXTURE1 );

rendering/terrainclipmap.cppm

@@ -61,8 +61,10 @@ public:
     // not been wanted for a while. safe to call every frame: the reading happens on the loader's
     // thread
     void update( glm::dvec3 const &Viewpoint );
-    // draws the tiles the walk settled on that the test says can be seen
+    // draws the tiles the walk settled on that the test says can be seen. Mainview says whether this
-    void render( glm::dvec3 const &Viewpoint, visibility const &Visible );
+    // is the view residency is decided by, and so whether what it drew is worth counting: a shadow
+    // pass sees the same tiles through a frustum of its own
+    void render( glm::dvec3 const &Viewpoint, visibility const &Visible, bool Mainview );
 
     // how far from the viewpoint terrain is drawn, in metres
     void range( float const Range ) { m_range = Range; }
@@ -86,11 +88,13 @@ public:
     // side of a tile at each level, and how far that level's ground is off, for the debug panel
     float leveltile( std::uint32_t const Level ) const;
     float levelerror( std::uint32_t const Level ) const;
+    float leveledge( std::uint32_t const Level ) const;
 
 private:
     // one tile held in the buffers
     struct resident_tile {
         terrain_mesh_arena::piece piece;
+        bool wide { false };   // which of the two arenas it is in
         // for culling, in world space
         glm::dvec3 centre { 0.0 };
         float radius { 0.f };
@@ -101,7 +105,7 @@ private:
     // the tiles the walk settled on, in the order they will be drawn
     void walk( glm::dvec3 const &Viewpoint );
     // asks the loader for every tile the wanted cut of the quadtree is missing, and says whether this
-    // tile and everything under it is already there
+    // tile's ground can be drawn at all - at the level wanted, or by the tile itself standing in
     bool need( quantizedmesh::tile_address const &Tile, glm::dvec3 const &Viewpoint );
     // whether the level this tile stands for is close enough to the ground below it
     bool good_enough( quantizedmesh::tile_address const &Tile, double Distance ) const;
@@ -118,7 +122,11 @@ private:
 
     quantizedmesh::reader m_reader;      // the table and which tiles there are; reading is the loader's
     terrain_tile_loader m_loader;
+    // Nearly every tile fits two-byte indices and lives in the first; a tile of more than sixty-five
+    // thousand vertices - a patch of survey data, usually - goes in the second, which pays four bytes
+    // for them. One draw call each, and the second is empty on most sceneries.
     terrain_mesh_arena m_arena;
+    terrain_mesh_arena m_widearena;
     terrain_ground m_ground;
 
     quantizedmesh::terrain_table m_table;
@@ -128,17 +136,24 @@ private:
 
     std::unordered_map<std::int64_t, resident_tile> m_tiles;
     std::vector<std::int64_t> m_chosen;
+    // the tiles the walk passed through on its way down. They are not drawn - what is below them is -
+    // but they are held all the same, because the moment the camera turns towards ground whose fine
+    // tiles have not arrived, one of them is what covers it. Without them that ground is a hole until
+    // the fine tiles come, which is what flickering while the camera moves was.
+    std::vector<std::int64_t> m_kept;
     std::vector<std::pair<double, quantizedmesh::tile_address>> m_wishes;
     std::vector<quantizedmesh::tile_address> m_wanted;
     std::vector<terrain_tile_loader::payload> m_arrived;
     std::unordered_set<std::int64_t> m_asked;    // what the last walk asked the loader for
-    // whether a subtree could be drawn, worked out once per walk
+    // per tile of the last walk, whether all four of its children can be drawn, and so whether the
-    std::unordered_map<std::int64_t, bool> m_readiness;
+    // walk may go past it. Worked out once and read again when the tiles to draw are picked
+    std::unordered_map<std::int64_t, bool> m_maydescend;
 
-    // what the draw hands to gl, kept between frames so that a frame allocates nothing
+    // what the draw hands to gl, kept between frames so that a frame allocates nothing. one set for
-    std::vector<std::int32_t> m_counts;
+    // each arena
-    std::vector<void const *> m_offsets;
+    std::vector<std::int32_t> m_counts[ 2 ];
-    std::vector<std::int32_t> m_bases;
+    std::vector<void const *> m_offsets[ 2 ];
+    std::vector<std::int32_t> m_bases[ 2 ];
 
     glm::dvec3 m_walkpoint { 0.0 };
     bool m_walked { false };
@@ -165,7 +180,10 @@ private:
     unsigned m_depthrank { 0 };
     // what the buffers may hold before tiles nobody asked for lately are let go
     std::size_t m_budget { 192u << 20 };
-    std::size_t m_putsperframe { 16 };
+    // Tiles put on the card in one frame. Generous on purpose: what is being absorbed is a read that
+    // has already happened, and the sooner the levels between a new tile and the root are all there,
+    // the shorter the window in which the ground has to be drawn coarse
+    std::size_t m_putsperframe { 64 };
     bool m_ready { false };
     std::string m_path;
 };

rendering/terrainground.cpp

@@ -225,16 +225,24 @@ terrain_ground::update() {
             entry.copied = true;
             --m_pending;
             ++copied;
+            // rare, and it changes gl state for a moment: worth a line, so that anything odd seen on
+            // screen can be told apart from what the streamer is doing
+            WriteLog( "Terrain: ground texture " + std::to_string( index ) + " copied into the array, "
+                + std::to_string( m_pending ) + " still waiting" );
         }
 
         gl::vao::unbind();
         gl::program::bind( 0 );
     }
-    opengl_texture::reset_unit_cache();
 
     if( copied > 0 ) {
         ::glBindTexture( GL_TEXTURE_2D_ARRAY, m_array );
         ::glGenerateMipmap( GL_TEXTURE_2D_ARRAY );
+        ::glBindTexture( GL_TEXTURE_2D_ARRAY, 0 );
         upload_table();
     }
+    // last of all, and after the mipmaps rather than before them: the engine remembers what it last
+    // put on each texture unit and skips a bind it believes is already in place, so leaving a bind of
+    // ours behind its back costs it a frame drawn with the wrong texture
+    opengl_texture::reset_unit_cache();
 }

rendering/terrainmesharena.cpp

@@ -12,22 +12,30 @@ module;
 #include <cstddef>
 #include <cstdint>
 #include <map>
+#include <memory>
+#include <optional>
 #include <vector>
 #include <glad/glad.h>
 #include "scene/quantizedmeshreader.h"
 
 module eu07.rendering.terrainmesharena;
+import eu07.gl.buffer;
 import eu07.utilities.logs;
+import eu07.gl.vao;
 
 namespace {
 
-// what the buffers start at, in vertices and indices, and the most they may reach. a vertex is 24
+// what the buffers start at, in vertices and indices, and the most they may reach. a vertex is 16
-// bytes and an index 4, so the ceiling is about 400 MB of terrain on the card
+// bytes and an index 4, so the ceiling is about 500 MB of terrain on the card
 constexpr std::uint32_t firstvertices { 1u << 19 };
 constexpr std::uint32_t firstindices { 1u << 21 };
 constexpr std::uint32_t mostvertices { 1u << 24 };
 constexpr std::uint32_t mostindices { 1u << 26 };
 
+constexpr int attribute_position { 0 };
+constexpr int attribute_material { 1 };
+constexpr int attribute_normal { 2 };
+
 } // anonymous namespace
 
 void
@@ -72,33 +80,44 @@ terrain_mesh_arena::~terrain_mesh_arena() {
     destroy();
 }
 
+// The material is handed over as an integer that the vertex stage takes as a float: gl::vao sets
+// attributes up one way, and going around it to ask for an integer attribute would put the engine's
+// idea of what is bound out of step with what is.
+void
+terrain_mesh_arena::describe_vertices() {
+
+    auto const stride { static_cast<int>( sizeof( quantizedmesh::render_vertex ) ) };
+    m_vao->setup_attrib(
+        *m_vertexbuffer, attribute_position, 3, GL_FLOAT, stride,
+        static_cast<int>( offsetof( quantizedmesh::render_vertex, x ) ) );
+    m_vao->setup_attrib(
+        *m_vertexbuffer, attribute_material, 1, GL_UNSIGNED_SHORT, stride,
+        static_cast<int>( offsetof( quantizedmesh::render_vertex, material ) ) );
+    // the normal, still folded onto its octahedron; the vertex stage unfolds it
+    m_vao->setup_attrib(
+        *m_vertexbuffer, attribute_normal, 2, GL_UNSIGNED_BYTE, stride,
+        static_cast<int>( offsetof( quantizedmesh::render_vertex, normal0 ) ) );
+}
+
 bool
-terrain_mesh_arena::create() {
+terrain_mesh_arena::create( bool const Wideindices ) {
 
     destroy();
-    ::glGenVertexArrays( 1, &m_vao );
+    m_wideindices = Wideindices;
-    ::glGenBuffers( 1, &m_vertexbuffer );
+    m_vao.emplace();
-    ::glGenBuffers( 1, &m_indexbuffer );
+    m_vertexbuffer = std::make_unique<gl::buffer>();
-    if( ( m_vao == 0 ) || ( m_vertexbuffer == 0 ) || ( m_indexbuffer == 0 ) ) { destroy(); return false; }
+    m_indexbuffer = std::make_unique<gl::buffer>();
 
-    ::glBindVertexArray( m_vao );
+    m_vao->bind();
-    ::glBindBuffer( GL_ARRAY_BUFFER, m_vertexbuffer );
+    m_vertexbuffer->allocate(
-    ::glBufferData( GL_ARRAY_BUFFER,
+        gl::buffer::ARRAY_BUFFER,
-        static_cast<GLsizeiptr>( firstvertices ) * sizeof( quantizedmesh::render_vertex ), nullptr, GL_DYNAMIC_DRAW );
+        static_cast<GLsizeiptr>( firstvertices ) * sizeof( quantizedmesh::render_vertex ), GL_DYNAMIC_DRAW );
-    ::glBindBuffer( GL_ELEMENT_ARRAY_BUFFER, m_indexbuffer );
+    m_indexbuffer->allocate(
-    ::glBufferData( GL_ELEMENT_ARRAY_BUFFER,
+        gl::buffer::ELEMENT_ARRAY_BUFFER,
-        static_cast<GLsizeiptr>( firstindices ) * sizeof( std::uint32_t ), nullptr, GL_DYNAMIC_DRAW );
+        static_cast<GLsizeiptr>( firstindices ) * indexwidth(), GL_DYNAMIC_DRAW );
-
+    describe_vertices();
-    auto const stride { static_cast<GLsizei>( sizeof( quantizedmesh::render_vertex ) ) };
+    m_vao->setup_ebo( *m_indexbuffer );
-    ::glEnableVertexAttribArray( 0 );
+    unbind();
-    ::glVertexAttribPointer( 0, 3, GL_FLOAT, GL_FALSE, stride,
-        reinterpret_cast<void const *>( offsetof( quantizedmesh::render_vertex, x ) ) );
-    ::glEnableVertexAttribArray( 1 );
-    ::glVertexAttribIPointer( 1, 1, GL_UNSIGNED_SHORT, stride,
-        reinterpret_cast<void const *>( offsetof( quantizedmesh::render_vertex, material ) ) );
-    ::glBindVertexArray( 0 );
-    ::glBindBuffer( GL_ARRAY_BUFFER, 0 );
-    ::glBindBuffer( GL_ELEMENT_ARRAY_BUFFER, 0 );
 
     m_vertexroom = firstvertices;
     m_indexroom = firstindices;
@@ -113,17 +132,37 @@ terrain_mesh_arena::create() {
 void
 terrain_mesh_arena::destroy() {
 
-    if( m_vao != 0 ) { ::glDeleteVertexArrays( 1, &m_vao ); m_vao = 0; }
+    if( true == m_vao.has_value() ) { m_vao->unbind(); }
-    if( m_vertexbuffer != 0 ) { ::glDeleteBuffers( 1, &m_vertexbuffer ); m_vertexbuffer = 0; }
+    gl::buffer::unbind( gl::buffer::ARRAY_BUFFER );
-    if( m_indexbuffer != 0 ) { ::glDeleteBuffers( 1, &m_indexbuffer ); m_indexbuffer = 0; }
+    gl::buffer::unbind( gl::buffer::ELEMENT_ARRAY_BUFFER );
+    m_vao.reset();
+    m_vertexbuffer.reset();
+    m_indexbuffer.reset();
     m_vertexroom = m_indexroom = 0;
     m_freevertices.clear();
     m_freeindices.clear();
     m_verticesheld = 0;
+    m_indicesheld = 0;
+}
+
+void
+terrain_mesh_arena::bind() {
+
+    if( false == m_vao.has_value() ) { return; }
+    m_vao->bind();
+}
+
+void
+terrain_mesh_arena::unbind() {
+
+    if( true == m_vao.has_value() ) { m_vao->unbind(); }
+    gl::buffer::unbind( gl::buffer::ARRAY_BUFFER );
+    gl::buffer::unbind( gl::buffer::COPY_READ_BUFFER );
+    gl::buffer::unbind( gl::buffer::COPY_WRITE_BUFFER );
 }
 
 // Growing copies what is there into a larger buffer on the card: the pieces keep their places, so
-// nothing has to be read back or re-uploaded.
+// nothing has to be read back or sent again.
 bool
 terrain_mesh_arena::grow_vertices( std::uint32_t const Wanted ) {
 
@@ -131,32 +170,27 @@ terrain_mesh_arena::grow_vertices( std::uint32_t const Wanted ) {
     while( ( room < m_vertexroom + Wanted ) && ( room < mostvertices ) ) { room *= 2; }
     if( room <= m_vertexroom ) { return false; }
 
-    std::uint32_t grown { 0 };
+    auto grown { std::make_unique<gl::buffer>() };
-    ::glGenBuffers( 1, &grown );
+    grown->allocate(
-    if( grown == 0 ) { return false; }
+        gl::buffer::COPY_WRITE_BUFFER,
-    ::glBindBuffer( GL_ARRAY_BUFFER, grown );
+        static_cast<GLsizeiptr>( room ) * sizeof( quantizedmesh::render_vertex ), GL_DYNAMIC_DRAW );
-    ::glBufferData( GL_ARRAY_BUFFER,
+    m_vertexbuffer->bind( gl::buffer::COPY_READ_BUFFER );
-        static_cast<GLsizeiptr>( room ) * sizeof( quantizedmesh::render_vertex ), nullptr, GL_DYNAMIC_DRAW );
-    ::glBindBuffer( GL_COPY_READ_BUFFER, m_vertexbuffer );
-    ::glBindBuffer( GL_COPY_WRITE_BUFFER, grown );
     ::glCopyBufferSubData( GL_COPY_READ_BUFFER, GL_COPY_WRITE_BUFFER, 0, 0,
         static_cast<GLsizeiptr>( m_vertexroom ) * sizeof( quantizedmesh::render_vertex ) );
-    ::glBindBuffer( GL_COPY_READ_BUFFER, 0 );
+    gl::buffer::unbind( gl::buffer::COPY_READ_BUFFER );
-    ::glBindBuffer( GL_COPY_WRITE_BUFFER, 0 );
+    gl::buffer::unbind( gl::buffer::COPY_WRITE_BUFFER );
-    ::glDeleteBuffers( 1, &m_vertexbuffer );
-    m_vertexbuffer = grown;
 
-    ::glBindVertexArray( m_vao );
+    // the tracker has to forget the buffer before it is destroyed: gl reuses the name of a deleted
-    ::glBindBuffer( GL_ARRAY_BUFFER, m_vertexbuffer );
+    // buffer, so the next one can come back with the same number, and a bind of it would be skipped as
-    auto const stride { static_cast<GLsizei>( sizeof( quantizedmesh::render_vertex ) ) };
+    // already in place while nothing is actually bound
-    ::glVertexAttribPointer( 0, 3, GL_FLOAT, GL_FALSE, stride,
+    gl::buffer::unbind( gl::buffer::ARRAY_BUFFER );
-        reinterpret_cast<void const *>( offsetof( quantizedmesh::render_vertex, x ) ) );
+    m_vertexbuffer = std::move( grown );
-    ::glVertexAttribIPointer( 1, 1, GL_UNSIGNED_SHORT, stride,
+    m_vao->bind();
-        reinterpret_cast<void const *>( offsetof( quantizedmesh::render_vertex, material ) ) );
+    describe_vertices();
-    ::glBindVertexArray( 0 );
-    ::glBindBuffer( GL_ARRAY_BUFFER, 0 );
 
     m_freevertices.give( m_vertexroom, room - m_vertexroom );
+    WriteLog( "Terrain: vertex buffer grown from " + std::to_string( m_vertexroom ) + " to "
+        + std::to_string( room ) + " vertices" );
     m_vertexroom = room;
     return true;
 }
@@ -168,27 +202,24 @@ terrain_mesh_arena::grow_indices( std::uint32_t const Wanted ) {
     while( ( room < m_indexroom + Wanted ) && ( room < mostindices ) ) { room *= 2; }
     if( room <= m_indexroom ) { return false; }
 
-    std::uint32_t grown { 0 };
+    auto grown { std::make_unique<gl::buffer>() };
-    ::glGenBuffers( 1, &grown );
+    grown->allocate(
-    if( grown == 0 ) { return false; }
+        gl::buffer::COPY_WRITE_BUFFER,
-    ::glBindBuffer( GL_ARRAY_BUFFER, grown );
+        static_cast<GLsizeiptr>( room ) * indexwidth(), GL_DYNAMIC_DRAW );
-    ::glBufferData( GL_ARRAY_BUFFER,
+    m_indexbuffer->bind( gl::buffer::COPY_READ_BUFFER );
-        static_cast<GLsizeiptr>( room ) * sizeof( std::uint32_t ), nullptr, GL_DYNAMIC_DRAW );
-    ::glBindBuffer( GL_COPY_READ_BUFFER, m_indexbuffer );
-    ::glBindBuffer( GL_COPY_WRITE_BUFFER, grown );
     ::glCopyBufferSubData( GL_COPY_READ_BUFFER, GL_COPY_WRITE_BUFFER, 0, 0,
-        static_cast<GLsizeiptr>( m_indexroom ) * sizeof( std::uint32_t ) );
+        static_cast<GLsizeiptr>( m_indexroom ) * indexwidth() );
-    ::glBindBuffer( GL_COPY_READ_BUFFER, 0 );
+    gl::buffer::unbind( gl::buffer::COPY_READ_BUFFER );
-    ::glBindBuffer( GL_COPY_WRITE_BUFFER, 0 );
+    gl::buffer::unbind( gl::buffer::COPY_WRITE_BUFFER );
-    ::glDeleteBuffers( 1, &m_indexbuffer );
-    m_indexbuffer = grown;
 
-    ::glBindVertexArray( m_vao );
+    gl::buffer::unbind( gl::buffer::ELEMENT_ARRAY_BUFFER );
-    ::glBindBuffer( GL_ELEMENT_ARRAY_BUFFER, m_indexbuffer );
+    m_indexbuffer = std::move( grown );
-    ::glBindVertexArray( 0 );
+    m_vao->bind();
-    ::glBindBuffer( GL_ELEMENT_ARRAY_BUFFER, 0 );
+    m_vao->setup_ebo( *m_indexbuffer );
 
     m_freeindices.give( m_indexroom, room - m_indexroom );
+    WriteLog( "Terrain: index buffer grown from " + std::to_string( m_indexroom ) + " to "
+        + std::to_string( room ) + " indices" );
     m_indexroom = room;
     return true;
 }
@@ -197,9 +228,10 @@ terrain_mesh_arena::piece
 terrain_mesh_arena::put( quantizedmesh::reader::tile_data const &Tile ) {
 
     piece where;
-    if( ( m_vao == 0 ) || ( true == Tile.vertices.empty() ) || ( true == Tile.indices.empty() ) ) {
+    if( ( false == m_vao.has_value() ) || ( nullptr == m_vertexbuffer ) || ( true == Tile.vertices.empty() ) || ( true == Tile.indices.empty() ) ) {
         return where;
     }
+    if( Tile.vertices.size() > vertexlimit() ) { return where; }
     auto const vertices { static_cast<std::uint32_t>( Tile.vertices.size() ) };
     auto const indices { static_cast<std::uint32_t>( Tile.indices.size() ) };
 
@@ -220,27 +252,36 @@ terrain_mesh_arena::put( quantizedmesh::reader::tile_data const &Tile ) {
         }
     }
 
-    ::glBindBuffer( GL_ARRAY_BUFFER, m_vertexbuffer );
+    // this vertex array first: which element buffer is bound belongs to it, and writing indices
-    ::glBufferSubData( GL_ARRAY_BUFFER,
+    // with the scenery's array bound would take its own indices away
-        static_cast<GLintptr>( atvertex ) * sizeof( quantizedmesh::render_vertex ),
+    m_vao->bind();
-        static_cast<GLsizeiptr>( vertices ) * sizeof( quantizedmesh::render_vertex ),
+    m_vertexbuffer->upload(
-        Tile.vertices.data() );
+        gl::buffer::ARRAY_BUFFER, Tile.vertices.data(),
-    ::glBindBuffer( GL_ARRAY_BUFFER, 0 );
+        static_cast<int>( atvertex * sizeof( quantizedmesh::render_vertex ) ),
-
+        static_cast<GLsizeiptr>( vertices ) * sizeof( quantizedmesh::render_vertex ) );
-    // the indices are stored as the tile had them, counted from its own first vertex; where the
+    // the indices are stored as the tile had them, counted from its own first vertex; where the tile
-    // tile sits in the buffer is given to the draw as its base vertex
+    // sits in the buffer is given to the draw as its base vertex
-    ::glBindBuffer( GL_ELEMENT_ARRAY_BUFFER, m_indexbuffer );
+    if( true == m_wideindices ) {
-    ::glBufferSubData( GL_ELEMENT_ARRAY_BUFFER,
+        m_indexbuffer->upload(
-        static_cast<GLintptr>( atindex ) * sizeof( std::uint32_t ),
+            gl::buffer::ELEMENT_ARRAY_BUFFER, Tile.indices.data(),
-        static_cast<GLsizeiptr>( indices ) * sizeof( std::uint32_t ),
+            static_cast<int>( atindex * sizeof( std::uint32_t ) ),
-        Tile.indices.data() );
+            static_cast<GLsizeiptr>( indices ) * sizeof( std::uint32_t ) );
-    ::glBindBuffer( GL_ELEMENT_ARRAY_BUFFER, 0 );
+    }
+    else {
+        m_narrow.assign( Tile.indices.begin(), Tile.indices.end() );
+        m_indexbuffer->upload(
+            gl::buffer::ELEMENT_ARRAY_BUFFER, m_narrow.data(),
+            static_cast<int>( atindex * sizeof( std::uint16_t ) ),
+            static_cast<GLsizeiptr>( indices ) * sizeof( std::uint16_t ) );
+    }
+    unbind();
 
     where.firstvertex = atvertex;
     where.vertices = vertices;
     where.firstindex = atindex;
     where.indices = indices;
     m_verticesheld += vertices;
+    m_indicesheld += indices;
     return where;
 }
 
@@ -251,5 +292,6 @@ terrain_mesh_arena::take( piece &Piece ) {
     m_freevertices.give( Piece.firstvertex, Piece.vertices );
     m_freeindices.give( Piece.firstindex, Piece.indices );
     m_verticesheld -= Piece.vertices;
+    m_indicesheld -= Piece.indices;
     Piece = piece {};
 }

rendering/terrainmesharena.cppm

@@ -11,10 +11,14 @@ module;
 #include <cstddef>
 #include <cstdint>
 #include <map>
+#include <memory>
+#include <optional>
 #include <vector>
 #include "scene/quantizedmeshreader.h"
 
 export module eu07.rendering.terrainmesharena;
+import eu07.gl.buffer;
+import eu07.gl.vao;
 
 export {
 
@@ -26,7 +30,18 @@ export {
 // bind and a draw per tile. Vertices are measured from one point for the whole terrain, which is
 // what makes that possible.
 //
+// Indices are two bytes wide unless the arena is asked for four. An index is a quarter of a vertex but
+// there are some six of them per vertex, so they are the larger half of what the terrain costs on the
+// card - and a tile of fewer than sixty-five thousand vertices, which is nearly every tile, has no use
+// for four bytes. The few that need more get an arena of their own and a draw call of their own.
+//
 // The buffers grow when a tile will not fit and nothing can be freed; they never shrink.
+//
+// Everything here goes through gl::vao and gl::buffer rather than through glBindVertexArray and
+// glBindBuffer. The engine remembers what it last bound and skips a bind it believes is already in
+// place, so a raw bind here leaves it drawing the scenery's geometry through this vertex array. And
+// the element buffer binding belongs to whichever vertex array is bound, so writing indices without
+// binding this one first overwrites the scenery's.
 class terrain_mesh_arena {
 
 public:
@@ -44,19 +59,29 @@ public:
     terrain_mesh_arena( terrain_mesh_arena const & ) = delete;
     terrain_mesh_arena &operator=( terrain_mesh_arena const & ) = delete;
 
-    bool create();
+    // Wideindices asks for four-byte indices, for tiles of more than 65536 vertices
+    bool create( bool Wideindices );
     void destroy();
+    bool wideindices() const { return m_wideindices; }
+    // the most vertices a tile may have to fit this arena
+    std::uint32_t vertexlimit() const { return m_wideindices ? ( 1u << 24 ) : 65536u; }
 
     // puts a tile's mesh in. returns a piece that is not held when it did not fit
     piece put( quantizedmesh::reader::tile_data const &Tile );
     void take( piece &Piece );
 
     // bound for drawing; the index buffer goes with it
-    std::uint32_t vertexarray() const { return m_vao; }
+    void bind();
+    void unbind();
 
-    std::size_t vertexbytes() const { return m_vertexroom * sizeof( quantizedmesh::render_vertex ); }
+    // what the two buffers were allocated at. it only grows: gl has no way of giving a buffer back
-    std::size_t indexbytes() const { return m_indexroom * sizeof( std::uint32_t ); }
+    // except by making a new one, and the pieces already in it would have to move
-    std::size_t bytes() const { return vertexbytes() + indexbytes(); }
+    std::size_t indexwidth() const { return m_wideindices ? sizeof( std::uint32_t ) : sizeof( std::uint16_t ); }
+    std::size_t roombytes() const {
+        return m_vertexroom * sizeof( quantizedmesh::render_vertex ) + m_indexroom * indexwidth(); }
+    // and what the tiles in them actually take up, which is what a budget has to be measured against
+    std::size_t usedbytes() const {
+        return m_verticesheld * sizeof( quantizedmesh::render_vertex ) + m_indicesheld * indexwidth(); }
     std::size_t verticesheld() const { return m_verticesheld; }
 
 private:
@@ -74,14 +99,22 @@ private:
     bool grow_vertices( std::uint32_t Wanted );
     bool grow_indices( std::uint32_t Wanted );
 
-    std::uint32_t m_vao { 0 };
+    // attribute places the shader reads, and what one vertex is made of
-    std::uint32_t m_vertexbuffer { 0 };
+    void describe_vertices();
-    std::uint32_t m_indexbuffer { 0 };
+
+    // gl::buffer cannot be moved - it deletes what it holds - so growing one means putting a new one
+    // in its place rather than handing it over
+    std::optional<gl::vao> m_vao;
+    std::unique_ptr<gl::buffer> m_vertexbuffer;
+    std::unique_ptr<gl::buffer> m_indexbuffer;
     std::uint32_t m_vertexroom { 0 };
     std::uint32_t m_indexroom { 0 };
     span_list m_freevertices;
     span_list m_freeindices;
     std::size_t m_verticesheld { 0 };
+    std::size_t m_indicesheld { 0 };
+    bool m_wideindices { false };
+    std::vector<std::uint16_t> m_narrow;   // scratch for the common case, so a put allocates nothing
 };
 
 }  // export

rendering/terrainstatus.cppm

@@ -30,10 +30,33 @@ export {
 // This is the format's own rule, and the reason a tile carries its error at all.
 inline constexpr std::size_t terrain_maxlevels { 16 };
 
+// A triangle may be about this much of the error budget across before it is worth splitting the tile. The error alone will not do the choosing: it is the worst departure anywhere in the tile and
+// every level above inherits it, so one steep corner sets the same figure for the whole pyramid and
+// the levels become indistinguishable - a terrain of sixty triangles and one of fifteen thousand ask
+// to be drawn from the same distance. What tells them apart is how large the triangles are.
+inline constexpr double terrain_edge_share { 0.1 };
+
+// what a level asks for, in metres: the worse of how far its ground is off and how large its
+// triangles are
+inline double
+terrain_level_demand( double const Error, double const Edge ) {
+    return std::max( Error, Edge * terrain_edge_share );
+}
+
 inline bool
 terrain_level_enough(
-    double const Error, double const Distance, double const Pixelsperunit, double const Detail ) {
+    double const Error, double const Edge, double const Distance,
-    return Error * Pixelsperunit <= std::max( 0.1, Detail ) * std::max( 1.0, Distance );
+    double const Pixelsperunit, double const Detail ) {
+    return terrain_level_demand( Error, Edge ) * Pixelsperunit
+        <= std::max( 0.1, Detail ) * std::max( 1.0, Distance );
+}
+
+// the other way round: how far away a level has to be before it is good enough. what the detail
+// setting means in metres, which is the only form it can be judged in
+inline double
+terrain_level_from(
+    double const Error, double const Edge, double const Pixelsperunit, double const Detail ) {
+    return terrain_level_demand( Error, Edge ) * Pixelsperunit / std::max( 0.1, Detail );
 }
 
 // how far the camera may move before which tiles are wanted is worked out again, as a share of the
@@ -50,7 +73,9 @@ struct terrain_statistics {
     std::size_t uploaded { 0 };     // tiles put in the buffers since opening
     std::size_t dropped { 0 };      // arrivals discarded as no longer wanted
     std::size_t gpubytes { 0 };     // what the two buffers hold
-    std::array<std::size_t, terrain_maxlevels> perlevel {}; // tiles drawn at each level
+    std::array<std::size_t, terrain_maxlevels> perlevel {};          // tiles drawn at each level
+    std::array<std::size_t, terrain_maxlevels> trianglesperlevel {}; // and the triangles in them
+    double furthest { 0.0 };        // how far the furthest tile drawn reaches, in metres
 };
 
 // one terrain as the debug panel shows it. the renderer writes it once a frame from the main view,
@@ -60,10 +85,12 @@ struct terrain_field_status {
     float tilesize { 0.f };         // side of a tile at the finest level, in metres
     float range { 0.f };
     float detail { 0.f };           // pixels an error may cover before a finer level is used
+    float pixelsperunit { 0.f };    // what a metre at a metre's distance spans on this screen
     std::uint32_t levels { 0 };
     std::size_t backlog { 0 };      // tiles the loader has queued or finished and not handed over
     std::array<float, terrain_maxlevels> leveltile {};   // tile side at each level
     std::array<float, terrain_maxlevels> levelerror {};  // metres a level departs from the finest
+    std::array<float, terrain_maxlevels> leveledge {};   // and how long its triangles typically are
     terrain_statistics stats;
 };
 

scene/pakformat.h

@@ -1,179 +0,0 @@
-/*
-This Source Code Form is subject to the
-terms of the Mozilla Public License, v.
-2.0. If a copy of the MPL was not
-distributed with this file, You can
-obtain one at
-http://mozilla.org/MPL/2.0/.
-*/
-
-#pragma once
-
-// One file holding a scenery's cooked terrain tiles.
-//
-// A terrain is tens of thousands of small files, which is the one thing a filesystem is worst at:
-// opening them costs more than reading them. They go into an archive instead, each entry
-// compressed on its own so that a tile can be read without touching the rest, and the index is
-// written last so that cooking never has to hold the tiles in memory.
-
-#include <cstdint>
-#include <cstring>
-#include <fstream>
-#include <string>
-#include <unordered_map>
-#include <vector>
-
-#include <zstd.h>
-
-namespace pak {
-
-inline constexpr char magic[ 8 ] { 'E', 'U', '0', '7', 'P', 'A', 'K', '\0' };
-inline constexpr std::uint32_t version { 2 };
-inline constexpr std::size_t namelength { 64 };
-
-#pragma pack( push, 1 )
-
-struct file_header {
-    char magic[ 8 ];
-    std::uint32_t version;
-    std::uint32_t entrycount;
-    std::uint64_t indexoffset;
-};
-
-struct index_entry {
-    char name[ namelength ];
-    std::uint64_t offset;
-    std::uint64_t stored;    // bytes in the file
-    std::uint64_t plain;     // bytes once decompressed; equal to stored when kept as it is
-};
-
-#pragma pack( pop )
-
-// Writes entries straight through to disk as they are added.
-class writer {
-
-public:
-    bool open( std::string const &Path ) {
-        m_file.open( Path, std::ios::binary | std::ios::trunc );
-        if( false == m_file.good() ) { return false; }
-        file_header header {};
-        m_file.write( reinterpret_cast<char const *>( &header ), sizeof( header ) );
-        m_at = sizeof( header );
-        return m_file.good();
-    }
-
-    bool add( std::string const &Name, std::vector<std::uint8_t> const &Data ) {
-        if( ( false == m_file.good() ) || ( Name.size() >= namelength ) ) { return false; }
-        index_entry entry {};
-        std::memcpy( entry.name, Name.c_str(), Name.size() );
-        entry.offset = m_at;
-        entry.plain = Data.size();
-
-        m_packed.resize( ZSTD_compressBound( Data.size() ) );
-        auto const packed {
-            ZSTD_compress( m_packed.data(), m_packed.size(), Data.data(), Data.size(), 3 ) };
-        auto const *bytes { reinterpret_cast<char const *>( Data.data() ) };
-        auto size { Data.size() };
-        if( ( 0 == ZSTD_isError( packed ) ) && ( packed < Data.size() ) ) {
-            bytes = reinterpret_cast<char const *>( m_packed.data() );
-            size = packed;
-        }
-        entry.stored = size;
-        m_file.write( bytes, static_cast<std::streamsize>( size ) );
-        m_at += size;
-        m_index.push_back( entry );
-        return m_file.good();
-    }
-
-    bool close() {
-        if( false == m_file.good() ) { return false; }
-        file_header header {};
-        std::memcpy( header.magic, magic, sizeof( magic ) );
-        header.version = version;
-        header.entrycount = static_cast<std::uint32_t>( m_index.size() );
-        header.indexoffset = m_at;
-        for( auto const &entry : m_index ) {
-            m_file.write( reinterpret_cast<char const *>( &entry ), sizeof( entry ) );
-        }
-        m_file.seekp( 0 );
-        m_file.write( reinterpret_cast<char const *>( &header ), sizeof( header ) );
-        auto const good { m_file.good() };
-        m_file.close();
-        return good;
-    }
-
-    std::size_t entrycount() const { return m_index.size(); }
-    std::uint64_t bytes() const { return m_at + m_index.size() * sizeof( index_entry ); }
-
-private:
-    std::ofstream m_file;
-    std::uint64_t m_at { 0 };
-    std::vector<index_entry> m_index;
-    std::vector<std::uint8_t> m_packed;
-};
-
-// Reads entries back. One reader is used from one thread at a time: it keeps a handle and seeks.
-class reader {
-
-public:
-    bool open( std::string const &Path ) {
-        close();
-        m_file.open( Path, std::ios::binary );
-        if( false == m_file.good() ) { return false; }
-        file_header header {};
-        m_file.read( reinterpret_cast<char *>( &header ), sizeof( header ) );
-        if( false == m_file.good() ) { return false; }
-        if( 0 != std::memcmp( header.magic, magic, sizeof( magic ) ) ) { return false; }
-        if( header.version != version ) { return false; }
-        m_file.seekg( static_cast<std::streamoff>( header.indexoffset ) );
-        for( std::uint32_t entry = 0; entry < header.entrycount; ++entry ) {
-            index_entry record {};
-            m_file.read( reinterpret_cast<char *>( &record ), sizeof( record ) );
-            if( false == m_file.good() ) { return false; }
-            record.name[ namelength - 1 ] = '\0';
-            m_index.emplace( record.name, record );
-        }
-        return true;
-    }
-
-    void close() {
-        m_index.clear();
-        if( true == m_file.is_open() ) { m_file.close(); }
-        m_file.clear();
-    }
-
-    bool exists( std::string const &Name ) const { return m_index.contains( Name ); }
-    std::size_t entrycount() const { return m_index.size(); }
-
-    std::vector<std::string> names() const {
-        std::vector<std::string> names;
-        names.reserve( m_index.size() );
-        for( auto const &[ name, entry ] : m_index ) { names.push_back( name ); }
-        return names;
-    }
-
-    bool read( std::string const &Name, std::vector<std::uint8_t> &Out ) {
-        auto const found { m_index.find( Name ) };
-        if( found == m_index.end() ) { return false; }
-        auto const &entry { found->second };
-        m_stored.resize( entry.stored );
-        m_file.seekg( static_cast<std::streamoff>( entry.offset ) );
-        m_file.read( reinterpret_cast<char *>( m_stored.data() ), static_cast<std::streamsize>( entry.stored ) );
-        if( false == m_file.good() ) { m_file.clear(); return false; }
-        if( entry.stored == entry.plain ) {
-            Out.swap( m_stored );
-            return true;
-        }
-        Out.resize( entry.plain );
-        auto const plain {
-            ZSTD_decompress( Out.data(), Out.size(), m_stored.data(), m_stored.size() ) };
-        return ( 0 == ZSTD_isError( plain ) ) && ( plain == entry.plain );
-    }
-
-private:
-    std::ifstream m_file;
-    std::unordered_map<std::string, index_entry> m_index;
-    std::vector<std::uint8_t> m_stored;
-};
-
-} // namespace pak

scene/quantizedmesharchive.h

@@ -0,0 +1,283 @@
+/*
+This Source Code Form is subject to the
+terms of the Mozilla Public License, v.
+2.0. If a copy of the MPL was not
+distributed with this file, You can
+obtain one at
+http://mozilla.org/MPL/2.0/.
+*/
+
+#pragma once
+
+// One file holding a scenery's cooked terrain tiles.
+//
+// A terrain is hundreds of thousands of small files, which is the one thing a filesystem is worst at:
+// opening them costs more than reading them. They go into an archive instead, each entry compressed on
+// its own so that a tile can be read without touching the rest, and the index written last so that
+// cooking never has to hold the tiles in memory.
+//
+// A tile is found by where it sits in the quadtree, not by a name. Galicja cooks to three quarters of
+// a million tiles: a name of its own for each would be seventy megabytes of index to read and a hash
+// table of that many strings to build before the first tile can be drawn. The index here is a sorted
+// run of thirty-two byte entries, read in one go and searched in place.
+//
+// An entry carries what the tile's ground asks for as well as where it lies, because which level to
+// draw has to be settled for tiles that are not on the card yet - that is the whole of the question -
+// and the index is the only thing about them that is read up front.
+
+#include <algorithm>
+#include <cstdint>
+#include <cstring>
+#include <fstream>
+#include <string>
+#include <vector>
+
+#include <zstd.h>
+
+#include "scene/quantizedmeshformat.h"
+
+namespace quantizedmesh {
+
+inline constexpr char archive_magic[ 8 ] { 'E', 'U', '0', '7', 'T', 'A', 'R', '\0' };
+inline constexpr std::uint32_t archive_version { 2 };
+
+#pragma pack( push, 1 )
+
+struct archive_header {
+    char magic[ 8 ];
+    std::uint32_t version;
+    // the cook's rules, so that an archive this build cannot make sense of is refused at the door
+    std::uint32_t rules;
+    std::uint64_t entrycount;
+    std::uint64_t indexoffset;
+    // the terrain's table, which every tile needs before it can be placed
+    std::uint64_t tableoffset;
+    std::uint32_t tablestored;
+    std::uint32_t tableplain;
+};
+
+struct archive_entry {
+    std::int64_t key;          // tile_key of the tile this holds
+    std::uint64_t offset;
+    std::uint32_t stored;      // bytes in the file
+    std::uint32_t plain;       // bytes once decompressed; equal to stored when kept as it is
+    // what this tile's own ground asks for, as the cook measured it
+    float error;
+    float edge;
+
+    bool operator<( archive_entry const &Other ) const { return key < Other.key; }
+};
+
+#pragma pack( pop )
+
+static_assert( sizeof( archive_entry ) == 32 );
+
+namespace detail {
+
+// zstd, or the bytes as they are when compressing them does not pay
+inline void
+pack( std::vector<std::uint8_t> const &Data, std::vector<std::uint8_t> &Room,
+    std::uint8_t const *&Bytes, std::size_t &Size ) {
+
+    Bytes = Data.data();
+    Size = Data.size();
+    if( true == Data.empty() ) { return; }
+    Room.resize( ZSTD_compressBound( Data.size() ) );
+    auto const packed { ZSTD_compress( Room.data(), Room.size(), Data.data(), Data.size(), 3 ) };
+    if( ( 0 == ZSTD_isError( packed ) ) && ( packed < Data.size() ) ) {
+        Bytes = Room.data();
+        Size = packed;
+    }
+}
+
+} // namespace detail
+
+// Writes tiles straight through to disk as they are cooked.
+class archive_writer {
+
+public:
+    bool open( std::string const &Path, std::uint32_t const Rules ) {
+        m_rules = Rules;
+        m_file.open( Path, std::ios::binary | std::ios::trunc );
+        if( false == m_file.good() ) { return false; }
+        archive_header header {};
+        m_file.write( reinterpret_cast<char const *>( &header ), sizeof( header ) );
+        m_at = sizeof( header );
+        return m_file.good();
+    }
+
+    bool add( tile_address const &Tile, std::vector<std::uint8_t> const &Data,
+        tile_measure const &Measure ) {
+        if( false == m_file.good() ) { return false; }
+        std::uint8_t const *bytes { nullptr };
+        std::size_t size { 0 };
+        detail::pack( Data, m_room, bytes, size );
+        archive_entry entry {};
+        entry.key = tile_key( Tile );
+        entry.offset = m_at;
+        entry.stored = static_cast<std::uint32_t>( size );
+        entry.plain = static_cast<std::uint32_t>( Data.size() );
+        entry.error = Measure.error;
+        entry.edge = Measure.edge;
+        m_file.write( reinterpret_cast<char const *>( bytes ), static_cast<std::streamsize>( size ) );
+        m_at += size;
+        m_index.push_back( entry );
+        return m_file.good();
+    }
+
+    // the terrain's table, written once. it has a place of its own in the header rather than an entry
+    bool table( std::vector<std::uint8_t> const &Data ) {
+        if( false == m_file.good() ) { return false; }
+        std::uint8_t const *bytes { nullptr };
+        std::size_t size { 0 };
+        detail::pack( Data, m_room, bytes, size );
+        m_tableoffset = m_at;
+        m_tablestored = static_cast<std::uint32_t>( size );
+        m_tableplain = static_cast<std::uint32_t>( Data.size() );
+        m_file.write( reinterpret_cast<char const *>( bytes ), static_cast<std::streamsize>( size ) );
+        m_at += size;
+        return m_file.good();
+    }
+
+    bool close() {
+        if( false == m_file.good() ) { return false; }
+        // in order, so that the index can be searched where it lies and so that cooking the same
+        // scenery twice writes the same file
+        std::sort( m_index.begin(), m_index.end() );
+
+        archive_header header {};
+        std::memcpy( header.magic, archive_magic, sizeof( archive_magic ) );
+        header.version = archive_version;
+        header.rules = m_rules;
+        header.entrycount = m_index.size();
+        header.indexoffset = m_at;
+        header.tableoffset = m_tableoffset;
+        header.tablestored = m_tablestored;
+        header.tableplain = m_tableplain;
+        for( auto const &entry : m_index ) {
+            m_file.write( reinterpret_cast<char const *>( &entry ), sizeof( entry ) );
+        }
+        m_file.seekp( 0 );
+        m_file.write( reinterpret_cast<char const *>( &header ), sizeof( header ) );
+        auto const good { m_file.good() };
+        m_file.close();
+        return good;
+    }
+
+    std::size_t tilecount() const { return m_index.size(); }
+    std::uint64_t bytes() const { return m_at + m_index.size() * sizeof( archive_entry ); }
+
+private:
+    std::ofstream m_file;
+    std::uint64_t m_at { 0 };
+    std::uint32_t m_rules { 0 };
+    std::uint64_t m_tableoffset { 0 };
+    std::uint32_t m_tablestored { 0 };
+    std::uint32_t m_tableplain { 0 };
+    std::vector<archive_entry> m_index;
+    std::vector<std::uint8_t> m_room;
+};
+
+// Reads tiles back. One reader is used from one thread at a time: it keeps a handle and seeks.
+class archive_reader {
+
+public:
+    bool open( std::string const &Path, std::uint32_t const Rules ) {
+        close();
+        m_file.open( Path, std::ios::binary );
+        if( false == m_file.good() ) { return false; }
+        archive_header header {};
+        m_file.read( reinterpret_cast<char *>( &header ), sizeof( header ) );
+        if( false == m_file.good() ) { return false; }
+        if( 0 != std::memcmp( header.magic, archive_magic, sizeof( archive_magic ) ) ) { return false; }
+        if( header.version != archive_version ) { return false; }
+        if( header.rules != Rules ) { return false; }
+        if( header.entrycount > ( 1ull << 32 ) ) { return false; }
+
+        m_index.resize( static_cast<std::size_t>( header.entrycount ) );
+        if( false == m_index.empty() ) {
+            m_file.seekg( static_cast<std::streamoff>( header.indexoffset ) );
+            m_file.read(
+                reinterpret_cast<char *>( m_index.data() ),
+                static_cast<std::streamsize>( m_index.size() * sizeof( archive_entry ) ) );
+            if( false == m_file.good() ) { close(); return false; }
+            // written in order, but a file from anywhere is not to be believed about that
+            if( false == std::is_sorted( m_index.begin(), m_index.end() ) ) {
+                std::sort( m_index.begin(), m_index.end() );
+            }
+        }
+        m_tableoffset = header.tableoffset;
+        m_tablestored = header.tablestored;
+        m_tableplain = header.tableplain;
+        m_open = true;
+        return true;
+    }
+
+    void close() {
+        m_index.clear();
+        m_index.shrink_to_fit();
+        if( true == m_file.is_open() ) { m_file.close(); }
+        m_file.clear();
+        m_open = false;
+    }
+
+    bool contains( tile_address const &Tile ) const { return nullptr != find( tile_key( Tile ) ); }
+    // what the tile asks for, or nothing when the archive does not hold it
+    bool measure( tile_address const &Tile, tile_measure &Out ) const {
+        auto const *entry { find( tile_key( Tile ) ) };
+        if( nullptr == entry ) { return false; }
+        Out.error = entry->error;
+        Out.edge = entry->edge;
+        return true;
+    }
+    std::size_t tilecount() const { return m_index.size(); }
+    // the tiles there are, in order, for anything that wants to walk them
+    std::vector<archive_entry> const &index() const { return m_index; }
+
+    bool read_table( std::vector<std::uint8_t> &Out ) {
+        if( ( false == m_open ) || ( m_tableplain == 0 ) ) { return false; }
+        return fetch( m_tableoffset, m_tablestored, m_tableplain, Out );
+    }
+
+    bool read_tile( tile_address const &Tile, std::vector<std::uint8_t> &Out ) {
+        if( false == m_open ) { return false; }
+        auto const *entry { find( tile_key( Tile ) ) };
+        if( nullptr == entry ) { return false; }
+        return fetch( entry->offset, entry->stored, entry->plain, Out );
+    }
+
+private:
+    archive_entry const *find( std::int64_t const Key ) const {
+        archive_entry wanted {};
+        wanted.key = Key;
+        auto const found { std::lower_bound( m_index.begin(), m_index.end(), wanted ) };
+        if( ( found == m_index.end() ) || ( found->key != Key ) ) { return nullptr; }
+        return &( *found );
+    }
+
+    bool fetch( std::uint64_t const Offset, std::uint32_t const Stored, std::uint32_t const Plain,
+        std::vector<std::uint8_t> &Out ) {
+
+        m_stored.resize( Stored );
+        m_file.seekg( static_cast<std::streamoff>( Offset ) );
+        m_file.read( reinterpret_cast<char *>( m_stored.data() ), static_cast<std::streamsize>( Stored ) );
+        if( false == m_file.good() ) { m_file.clear(); return false; }
+        if( Stored == Plain ) {
+            Out = m_stored;
+            return true;
+        }
+        Out.resize( Plain );
+        auto const plain { ZSTD_decompress( Out.data(), Out.size(), m_stored.data(), m_stored.size() ) };
+        return ( 0 == ZSTD_isError( plain ) ) && ( plain == Plain );
+    }
+
+    std::ifstream m_file;
+    std::vector<archive_entry> m_index;
+    std::vector<std::uint8_t> m_stored;
+    std::uint64_t m_tableoffset { 0 };
+    std::uint32_t m_tablestored { 0 };
+    std::uint32_t m_tableplain { 0 };
+    bool m_open { false };
+};
+
+} // namespace quantizedmesh

scene/quantizedmeshcooker.cpp

@@ -36,23 +36,50 @@ constexpr double weldepsilon { 1e-4 };
 constexpr std::uint32_t maxlevels { 13 };
 // The finest tile is this many typical triangle edges across, and never outside these bounds. A tile
 // has to be large against the triangles in it: a triangle crossing a border is cut, and the pieces
-// cost vertices neither side had. It also has to stay small enough to be worth loading and culling
+// cost vertices neither side had. It also has to stay small enough to be worth loading and culling on
-// on its own, which is what the bounds are for.
+// its own, which is what the bounds are for.
+//
+// The typical edge is averaged over the ground weighted by how much of it each triangle covers, not
+// over the triangles counted one each. A scenery holds both kinds of ground - a dense patch from a
+// survey by the track, and big triangles drawn by hand for the rest - and counting triangles one each
+// lets a patch of a few hundred metres decide the tile for forty kilometres. What the cut costs
+// depends on how much ground the large triangles cover, which is what this measures.
 constexpr double tileedges { 16.0 };
 constexpr double smallesttileside { 128.0 };
 constexpr double largesttileside { 1024.0 };
-// how many edge lengths are kept to find the typical one. the first hundred thousand triangles say
-// as much about a scenery as all of them
-constexpr std::size_t edgesample { 100000 };
 // a tile is never collapsed below this, however far away it is drawn: past a point the triangles
 // saved are not worth the ground losing its shape
 constexpr std::size_t smallesttile { 64 };
-// A level may move the ground by this share of its tile's side, and no further. A level is used from
+// A level may move the ground by this share of its tile's side, and no further.
-// about twice its tile's side away, where a share this small is well under a pixel; letting it go
+//
-// further would buy triangles that no distance is far enough to use. Ground already so coarse that a
+// The number decides how far away a level has to be before it may be used, and so how long the level
-// level cannot be thinned within it simply comes out nearly a copy of the level below, which is the
+// below it has to be drawn. Too generous and the finest mesh - which is the whole of the scenery's
-// right answer: there was nothing there to save.
+// ground - is drawn out to several kilometres, and only three or four levels are ever in view at once.
-constexpr double errorshare { 128.0 };
+// A third of what it was buys coarse levels faithful enough to be used three times closer, which is
+// both fewer triangles in the end and more of the ladder visible at any moment.
+constexpr double errorshare { 384.0 };
+// How far the finest level may move the ground, in metres. Not a share of the tile like the levels
+// above: this is the one that is stood on, so what it may lose is set by what the eye can catch from
+// there rather than by how large the tile happens to be. Three centimetres of ground twenty metres
+// ahead covers about a pixel and a half.
+//
+// What it takes off is redundancy rather than detail - a field drawn as a grid a metre across is a
+// grid a metre across even where it is flat, and survey data is mostly flat. Nothing else in the
+// pyramid can remove it, because every level above is built from this one.
+constexpr double finesterror { 0.03 };
+// How wide the ground may change over where two materials meet, as a share of the tile's side.
+//
+// The fragment stage blends the three materials of a triangle by how near the fragment is to each
+// corner, so a transition is one triangle wide - and a triangle is exactly what a coarser level makes
+// larger. Left alone the ground would fade over four metres near to and over sixty far away, and the
+// change would be seen as the level changed.
+//
+// Tying it to the tile rather than to a fixed number of metres holds it steady where it counts, which
+// is on screen: a level is drawn from about as far away as its tiles are wide, so a share of the tile
+// is a roughly constant number of pixels. It also lets material detail finer than this merge away at
+// the levels where it could not be seen anyway - without that, ground drawn in stripes a few metres
+// across can never be simplified at all, because every edge in it crosses a boundary.
+constexpr double blendshare { 64.0 };
 
 // the area of a triangle in space, whichever way it faces
 double
@@ -111,6 +138,7 @@ sides_of( double const X, double const Z, double const West, double const North,
 // a vertex of a polygon being cut along the tile grid
 struct clipvertex {
     double x, y, z;
+    double nx, ny, nz;
 };
 
 clipvertex
@@ -118,7 +146,10 @@ between( clipvertex const &From, clipvertex const &To, double const Share ) {
     return {
         From.x + ( To.x - From.x ) * Share,
         From.y + ( To.y - From.y ) * Share,
-        From.z + ( To.z - From.z ) * Share };
+        From.z + ( To.z - From.z ) * Share,
+        From.nx + ( To.nx - From.nx ) * Share,
+        From.ny + ( To.ny - From.ny ) * Share,
+        From.nz + ( To.nz - From.nz ) * Share };
 }
 
 // Sutherland and Hodgman against one side of the tile's square. Axis 0 is x and 1 is z; the
@@ -224,14 +255,19 @@ cooker::add( corner const &A, corner const &B, corner const &C, std::string_view
     }
 
     corner const corners[ 3 ] { A, B, C };
-    if( m_edges.size() < edgesample ) {
+    {
+        auto edges { 0.0 };
         for( std::size_t index = 0; index < 3; ++index ) {
             auto const &from { corners[ index ] };
             auto const &to { corners[ ( index + 1 ) % 3 ] };
-            auto const length {
+            edges += std::sqrt(
-                std::sqrt( ( to.x - from.x ) * ( to.x - from.x ) + ( to.z - from.z ) * ( to.z - from.z ) ) };
+                ( to.x - from.x ) * ( to.x - from.x ) + ( to.z - from.z ) * ( to.z - from.z ) );
-            if( length > 0.0 ) { m_edges.push_back( length ); }
         }
+        auto const plan {
+            0.5 * std::abs(
+                ( B.x - A.x ) * ( C.z - A.z ) - ( C.x - A.x ) * ( B.z - A.z ) ) };
+        m_edgesum += plan * edges / 3.0;
+        m_areasum += plan;
     }
     spilled record {};
     record.material = material_index( Material );
@@ -263,16 +299,13 @@ cooker::add( corner const &A, corner const &B, corner const &C, std::string_view
 void
 cooker::measure() {
 
-    // the middle edge length of the sample, and the tile that suits it, rounded to a power of two so
+    // the typical edge and the tile that suits it, rounded to a power of two so that the same scenery
-    // that the same scenery always comes out the same way
+    // always comes out the same way
-    if( false == m_edges.empty() ) {
+    if( m_areasum > 0.0 ) {
-        auto const middle { m_edges.begin() + m_edges.size() / 2 };
+        auto const typical { m_edgesum / m_areasum };
-        std::nth_element( m_edges.begin(), middle, m_edges.end() );
+        auto const wanted { std::clamp( tileedges * typical, smallesttileside, largesttileside ) };
-        auto const wanted { std::clamp( tileedges * *middle, smallesttileside, largesttileside ) };
+        m_finesttile = std::clamp(
-        m_finesttile = std::exp2( std::round( std::log2( wanted ) ) );
+            std::exp2( std::round( std::log2( wanted ) ) ), smallesttileside, largesttileside );
-        m_finesttile = std::clamp( m_finesttile, smallesttileside, largesttileside );
-        m_edges.clear();
-        m_edges.shrink_to_fit();
     }
 
     m_west = std::floor( m_minx / m_finesttile ) * m_finesttile;
@@ -326,10 +359,13 @@ cooker::finish( sink const &Sink, terrain_table &Table ) {
         }
         auto const first = [ this ]( double const Value, double const Origin ) {
             return static_cast<std::int32_t>( std::floor( ( Value - Origin ) / m_finesttile ) ); };
-        auto const fromx { std::clamp( first( minx, m_west ), 0, acrossfinest - 1 ) };
+        // a hair past the triangle either way: one that merely touches a border belongs to the tile
-        auto const tox { std::clamp( first( maxx, m_west ), 0, acrossfinest - 1 ) };
+        // beyond it as well, and that is what lets two tiles agree on the normal of a vertex they
-        auto const fromz { std::clamp( first( minz, m_north ), 0, acrossfinest - 1 ) };
+        // share. it contributes nothing to that tile's own triangles - there is nothing of it inside
-        auto const toz { std::clamp( first( maxz, m_north ), 0, acrossfinest - 1 ) };
+        auto const fromx { std::clamp( first( minx - weldepsilon, m_west ), 0, acrossfinest - 1 ) };
+        auto const tox { std::clamp( first( maxx + weldepsilon, m_west ), 0, acrossfinest - 1 ) };
+        auto const fromz { std::clamp( first( minz - weldepsilon, m_north ), 0, acrossfinest - 1 ) };
+        auto const toz { std::clamp( first( maxz + weldepsilon, m_north ), 0, acrossfinest - 1 ) };
         for( auto x = fromx; x <= tox; ++x ) {
             for( auto z = fromz; z <= toz; ++z ) {
                 m_placement.emplace_back(
@@ -340,6 +376,8 @@ cooker::finish( sink const &Sink, terrain_table &Table ) {
     std::sort( m_placement.begin(), m_placement.end() );
 
     m_errors.assign( m_levels, 0.0 );
+    m_levelarea.assign( m_levels, 0.0 );
+    m_leveltriangles.assign( m_levels, 0 );
     cook( { 0, 0, 0 }, Sink );
     m_report.levels = m_levels;
 
@@ -351,6 +389,14 @@ cooker::finish( sink const &Sink, terrain_table &Table ) {
     Table.highest = m_highest;
     Table.levels = m_levels;
     Table.errors.assign( m_errors.begin(), m_errors.end() );
+    // the side of the square a triangle of this level covers on average: what it is worth on screen
+    Table.edges.clear();
+    for( std::uint32_t level = 0; level < m_levels; ++level ) {
+        auto const triangles { m_leveltriangles[ level ] };
+        auto const typical {
+            triangles > 0 ? std::sqrt( 2.0 * m_levelarea[ level ] / static_cast<double>( triangles ) ) : 0.0 };
+        Table.edges.push_back( static_cast<float>( typical ) );
+    }
     Table.materials.clear();
     for( std::size_t index = 0; index < m_materials.size(); ++index ) {
         // a material the measurement never caught falls back to something plausible rather than
@@ -375,23 +421,55 @@ cooker::cook( tile_address const &Tile, sink const &Sink ) {
         ( Tile.level + 1 == m_levels ) ? lay( Tile ) : merge( Tile, Sink ) };
     if( true == result.empty() ) { return result; }
 
-    if( Tile.level + 1 < m_levels ) {
+    {
-        // The finest level is the ground as the scenery drew it and is left alone. A level above it
-        // holds what four tiles below it held, and is collapsed back to what one of them held: the
-        // cost of a level is then a quarter of the level below, and the whole pyramid costs a third
-        // more than the finest alone. A count fixed in advance cannot do that - a tile that never
-        // reaches it would be copied rather than simplified - so it only serves as a ceiling.
         double west { 0.0 }, north { 0.0 }, side { 0.0 };
         square( Tile, west, north, side );
         auto const merged { result.indices.size() / 3 };
-        simplify( result, std::clamp( merged / 4, smallesttile, m_tilebudget ), side / errorshare );
+        if( Tile.level + 1 < m_levels ) {
+            // A level holds what four tiles below it held, and is collapsed back to what one of them
+            // held: the cost of a level is then a quarter of the level below, and the whole pyramid
+            // costs a third more than the finest alone. A count fixed in advance cannot do that - a
+            // tile that never reaches it would be copied rather than simplified - so it only serves
+            // as a ceiling.
+            simplify(
+                result, std::clamp( merged / 4, smallesttile, m_tilebudget ),
+                side / errorshare, side / blendshare, side / blendshare );
+        }
+        else {
+            // the finest level keeps its shape to within a fraction of nothing, and is let go of
+            // only where it says the same thing twice
+            // Its outline is left exactly where the scenery put it. Ground drawn in separate patches
+            // meets at T-junctions - a vertex of one lying partway along an edge of the other - and the
+            // two outlines there are not the same set of vertices, so moving one and not the other opens
+            // a crack. Far away that is a fraction of a pixel and worth the triangles; on the ground the
+            // camera stands on it is a line of sky through the field.
+            simplify( result, smallesttile, finesterror, side / blendshare, 0.0 );
+        }
     }
 
     m_errors[ Tile.level ] = std::max( m_errors[ Tile.level ], result.error );
+    // the ground this tile covers and how many triangles it spends on it, from which the side of the
+    // square one of them covers on average - what a triangle of this tile is worth on screen
+    double area { 0.0 };
+    std::size_t triangles { 0 };
+    for( std::size_t index = 0; index + 2 < result.indices.size(); index += 3 ) {
+        auto const &a { result.vertices[ result.indices[ index + 0 ] ] };
+        auto const &b { result.vertices[ result.indices[ index + 1 ] ] };
+        auto const &c { result.vertices[ result.indices[ index + 2 ] ] };
+        area += 0.5 * std::abs( ( b.x - a.x ) * ( c.z - a.z ) - ( c.x - a.x ) * ( b.z - a.z ) );
+        ++triangles;
+    }
+    m_levelarea[ Tile.level ] += area;
+    m_leveltriangles[ Tile.level ] += triangles;
+
+    tile_measure measure {};
+    measure.error = static_cast<float>( result.error );
+    measure.edge = static_cast<float>(
+        triangles > 0 ? std::sqrt( 2.0 * area / static_cast<double>( triangles ) ) : 0.0 );
 
     std::vector<std::uint8_t> bytes;
     encode( Tile, result, bytes );
-    Sink( Tile, bytes );
+    Sink( Tile, bytes, measure );
 
     ++m_report.tiles;
     m_report.triangles += result.indices.size() / 3;
@@ -406,9 +484,14 @@ cooker::cook( tile_address const &Tile, sink const &Sink ) {
 
 namespace {
 
-// Welds vertices that land in the same place and belong to the same material. Everything the cut
+// Welds vertices that land in the same place, whatever material the triangle bringing them there is
-// produces is either a vertex of the source or a point on a tile border worked out the same way
+// drawn with. That is what lets one triangle have a different material at each corner, which is what
-// from both sides, so the tolerance only has to cover the last bits of the arithmetic.
+// the fragment stage blends across: keeping a vertex per material instead would put a hard edge along
+// every boundary, on the triangle's own edges. The vertex keeps the material of whichever triangle
+// reached it first, and a boundary therefore moves by at most half a triangle.
+//
+// Everything the cut produces is either a vertex of the source or a point on a tile border worked out
+// the same way from both sides, so the tolerance only has to cover the last bits of the arithmetic.
 class welder {
 
 public:
@@ -427,7 +510,6 @@ public:
 private:
     struct key {
         std::int64_t x, y, z;
-        std::uint16_t material;
         bool operator==( key const & ) const = default;
     };
     struct keyhash {
@@ -437,7 +519,6 @@ private:
                 hash ^= static_cast<std::size_t>( Value );
                 hash *= 1099511628211ull; };
             mix( Key.x ); mix( Key.y ); mix( Key.z );
-            mix( Key.material );
             return hash;
         }
     };
@@ -445,7 +526,7 @@ private:
     static key make_key( cooker::meshvertex const &Vertex ) {
         auto const grid = []( double const Value ) {
             return static_cast<std::int64_t>( std::llround( Value / weldepsilon ) ); };
-        return { grid( Vertex.x ), grid( Vertex.y ), grid( Vertex.z ), Vertex.material };
+        return { grid( Vertex.x ), grid( Vertex.y ), grid( Vertex.z ) };
     }
 
     std::vector<cooker::meshvertex> &m_vertices;
@@ -470,19 +551,68 @@ cooker::lay( tile_address const &Tile ) {
             std::pair<std::int64_t, std::uint32_t> { wanted, std::numeric_limits<std::uint32_t>::max() } ) };
     if( from == to ) { return result; }
 
-    welder weld { result.vertices };
+    // read once into memory: the triangles are walked twice, first for the normals and then for the
-    std::vector<clipvertex> polygon;
+    // cutting, and a tile holds few enough of them for that to be cheaper than seeking again
+    std::vector<spilled> records;
+    records.reserve( static_cast<std::size_t>( to - from ) );
     for( auto entry = from; entry != to; ++entry ) {
-
         m_scratch.seekg( static_cast<std::streamoff>( entry->second ) * sizeof( spilled ) );
         spilled record {};
         m_scratch.read( reinterpret_cast<char *>( &record ), sizeof( record ) );
         if( false == m_scratch.good() ) { m_scratch.clear(); continue; }
+        records.push_back( record );
+    }
+    if( true == records.empty() ) { return result; }
+
+    // The normal at a vertex is the area-weighted sum of the faces meeting there, taken from the
+    // scenery's triangles as they were written rather than from the pieces the cut leaves. Both tiles
+    // sharing a border read the same triangles around a vertex on it, so both work out the same
+    // normal and the light runs across the border without a line.
+    std::unordered_map<std::int64_t, std::array<double, 3>> normals;
+    auto const place_key = []( double const X, double const Y, double const Z ) {
+        auto const grid = []( double const Value ) {
+            return static_cast<std::int64_t>( std::llround( Value / weldepsilon ) ); };
+        std::int64_t hash { 1469598103934665603ll };
+        for( auto const value : { grid( X ), grid( Y ), grid( Z ) } ) {
+            hash ^= value;
+            hash *= 1099511628211ll;
+        }
+        return hash; };
+    for( auto const &record : records ) {
+        auto const &corners { record.corners };
+        double const ax { corners[ 0 ] }, ay { corners[ 1 ] }, az { corners[ 2 ] };
+        double const bx { corners[ 3 ] }, by { corners[ 4 ] }, bz { corners[ 5 ] };
+        double const cx { corners[ 6 ] }, cy { corners[ 7 ] }, cz { corners[ 8 ] };
+        // twice the area times the unit normal, which is the weighting wanted
+        std::array<double, 3> const face {
+            ( by - ay ) * ( cz - az ) - ( bz - az ) * ( cy - ay ),
+            ( bz - az ) * ( cx - ax ) - ( bx - ax ) * ( cz - az ),
+            ( bx - ax ) * ( cy - ay ) - ( by - ay ) * ( cx - ax ) };
+        for( std::size_t corner = 0; corner < 3; ++corner ) {
+            auto &sum {
+                normals[ place_key(
+                    corners[ corner * 3 + 0 ], corners[ corner * 3 + 1 ], corners[ corner * 3 + 2 ] ) ] };
+            sum[ 0 ] += face[ 0 ]; sum[ 1 ] += face[ 1 ]; sum[ 2 ] += face[ 2 ];
+        }
+    }
+    // the sum as it stands, not a unit vector: its length is the surface behind it, and that is what
+    // makes adding two of them an average rather than a guess
+    auto const normal_at = [ & ]( double const X, double const Y, double const Z ) {
+        auto const found { normals.find( place_key( X, Y, Z ) ) };
+        if( found == normals.end() ) { return std::array<double, 3> { 0.0, 1.0, 0.0 }; }
+        return found->second; };
+
+    welder weld { result.vertices };
+    std::vector<clipvertex> polygon;
+    for( auto const &record : records ) {
 
         polygon.clear();
         for( std::size_t at = 0; at < 3; ++at ) {
-            polygon.push_back(
+            auto const x { record.corners[ at * 3 + 0 ] };
-                { record.corners[ at * 3 + 0 ], record.corners[ at * 3 + 1 ], record.corners[ at * 3 + 2 ] } );
+            auto const y { record.corners[ at * 3 + 1 ] };
+            auto const z { record.corners[ at * 3 + 2 ] };
+            auto const normal { normal_at( x, y, z ) };
+            polygon.push_back( { x, y, z, normal[ 0 ], normal[ 1 ], normal[ 2 ] } );
         }
         clip_side( polygon, 0, west, true );
         clip_side( polygon, 0, west + side, false );
@@ -491,8 +621,8 @@ cooker::lay( tile_address const &Tile ) {
         if( polygon.size() < 3 ) { continue; }
 
         // A wall lying along a tile border is inside both tiles: the cut keeps it whole on either
-        // side, since it has nothing to cut. Whichever border it sits on, the tile past it is the
+        // side, since it has nothing to cut. Whichever border it sits on, the tile past it is the one
-        // one that keeps it, so that it is laid exactly once.
+        // that keeps it, so that it is laid exactly once.
         {
             auto onborderonly { false };
             for( int axis = 0; axis < 2; ++axis ) {
@@ -511,8 +641,8 @@ cooker::lay( tile_address const &Tile ) {
         }
 
         auto const place = [ & ]( clipvertex const &Vertex ) {
-            return weld( { Vertex.x, Vertex.y, Vertex.z, record.material,
+            return weld( { Vertex.x, Vertex.y, Vertex.z, Vertex.nx, Vertex.ny, Vertex.nz,
-                           sides_of( Vertex.x, Vertex.z, west, north, side ) } ); };
+                           record.material, sides_of( Vertex.x, Vertex.z, west, north, side ) } ); };
 
         auto const first { place( polygon[ 0 ] ) };
         for( std::size_t corner = 1; corner + 1 < polygon.size(); ++corner ) {
@@ -520,6 +650,16 @@ cooker::lay( tile_address const &Tile ) {
             auto const second { place( polygon[ corner ] ) };
             auto const third { place( polygon[ corner + 1 ] ) };
             if( ( first == second ) || ( second == third ) || ( third == first ) ) { continue; }
+            // a triangle that only touched this tile leaves nothing of itself inside it
+            auto const &a { result.vertices[ first ] };
+            auto const &b { result.vertices[ second ] };
+            auto const &c { result.vertices[ third ] };
+            auto const twicearea {
+                std::sqrt(
+                    std::pow( ( b.y - a.y ) * ( c.z - a.z ) - ( b.z - a.z ) * ( c.y - a.y ), 2.0 )
+                  + std::pow( ( b.z - a.z ) * ( c.x - a.x ) - ( b.x - a.x ) * ( c.z - a.z ), 2.0 )
+                  + std::pow( ( b.x - a.x ) * ( c.y - a.y ) - ( b.y - a.y ) * ( c.x - a.x ), 2.0 ) ) };
+            if( 0.5 * twicearea < nullarea ) { continue; }
             result.indices.push_back( first );
             result.indices.push_back( second );
             result.indices.push_back( third );
@@ -565,7 +705,9 @@ cooker::merge( tile_address const &Tile, sink const &Sink ) {
 // tile shares it, simply is never the one that goes. That is the whole of the crack handling:
 // levels meet because the border is literally the same set of vertices in both.
 void
-cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
+cooker::simplify(
+    mesh &Mesh, std::size_t const Budget, double const Allowed,
+    double const Boundaryreach, double const Outlinereach ) {
 
     auto triangles { Mesh.indices.size() / 3 };
     if( triangles <= Budget ) { return; }
@@ -624,12 +766,14 @@ cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
         }
     }
 
-    // Where the ground ends - the outline of a hole the scenery left, or the edge of the terrain -
+    // Where the ground ends - the outline of a hole the scenery left, the edge of the terrain, or the
-    // only one triangle meets the edge. Nothing in the quadrics above holds such an edge in
+    // seam between two patches drawn at different densities - only one triangle meets the edge. Those
-    // place, so a plane standing on it, weighted heavily, is added to both its ends: the outline
+    // edges are what the rules below protect, and a plane standing on each of them, weighted heavily,
-    // then survives into the coarse levels instead of being eaten away.
+    // is added to both its ends so that the outline is expensive to disturb as well as forbidden to
+    // break.
+    std::map<std::pair<std::uint32_t, std::uint32_t>, std::size_t> uses;
+    std::vector<bool> onmeshedge( vertices.size(), false );
     {
-        std::map<std::pair<std::uint32_t, std::uint32_t>, std::size_t> uses;
         for( std::size_t triangle = 0; triangle < triangles; ++triangle ) {
             for( std::size_t corner = 0; corner < 3; ++corner ) {
                 auto const from { cornerof( triangle, corner ) };
@@ -668,8 +812,14 @@ cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
                 quadrics[ to ].add_plane( px, py, pz, offset, weight );
             }
         }
+        for( auto const &[ edge, count ] : uses ) {
+            if( count != 1 ) { continue; }
+            onmeshedge[ edge.first ] = true;
+            onmeshedge[ edge.second ] = true;
+        }
     }
 
+
     struct candidate {
         double cost;
         std::uint32_t from, to;
@@ -720,13 +870,47 @@ cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
         // nothing in the plan at all. what changes there is the height where the vertex went
         return std::abs( was[ 1 ] - local[ To ][ 1 ] ); };
 
+    // vertices with a neighbour of another material: the ground changes colour around them, and how
+    // large the triangles there are is how wide that change looks
+    std::vector<bool> onboundary( vertices.size(), false );
+    for( std::size_t triangle = 0; triangle < triangles; ++triangle ) {
+        for( std::size_t corner = 0; corner < 3; ++corner ) {
+            auto const here { cornerof( triangle, corner ) };
+            auto const next { cornerof( triangle, ( corner + 1 ) % 3 ) };
+            if( vertices[ here ].material != vertices[ next ].material ) {
+                onboundary[ here ] = true;
+                onboundary[ next ] = true;
+            }
+        }
+    }
+
+    auto const meshedge = [ &uses ]( std::uint32_t const From, std::uint32_t const To ) {
+        auto const found { uses.find( { std::min( From, To ), std::max( From, To ) } ) };
+        return ( found != uses.end() ) && ( found->second == 1 ); };
+
     auto const allowed = [ & ]( std::uint32_t const From, std::uint32_t const To ) {
         if( ( true == removed[ From ] ) || ( true == removed[ To ] ) ) { return false; }
         // a vertex a neighbouring tile shares stays where it is
         if( true == vertices[ From ].locked() ) { return false; }
-        // and a material boundary stays where it is too, or the ground would change colour by
+        // Where the ground ends, a vertex may only travel along that end, onto another vertex of it, and
-        // level
+        // no further than this level reaches. What is lost then is the thin wedge between the old
-        if( vertices[ From ].material != vertices[ To ].material ) { return false; }
+        // outline and the new one. Letting it collapse inward instead takes whole triangles with it and
+        // leaves holes the size of them - and since the height barely changes there, nothing in the
+        // error measure objects: forty per cent of the ground went that way before this rule. Forbidding
+        // it outright is no good either, since a narrow strip of terrain is almost all outline, and then
+        // no level above the finest can be thinned at all
+        if( true == onmeshedge[ From ] ) {
+            if( Outlinereach <= 0.0 ) { return false; }
+            if( false == meshedge( From, To ) ) { return false; }
+            auto const &from { local[ From ] };
+            auto const &to { local[ To ] };
+            auto const reach {
+                std::sqrt( ( from[ 0 ] - to[ 0 ] ) * ( from[ 0 ] - to[ 0 ] )
+                         + ( from[ 1 ] - to[ 1 ] ) * ( from[ 1 ] - to[ 1 ] )
+                         + ( from[ 2 ] - to[ 2 ] ) * ( from[ 2 ] - to[ 2 ] ) ) };
+            if( reach > Outlinereach ) { return false; }
+        }
+
         // nothing that turns a triangle over or squashes it flat
         for( auto const triangle : incident[ From ] ) {
             if( true == gone[ triangle ] ) { continue; }
@@ -738,7 +922,10 @@ cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
                 // that can be one is between To and the third corner
                 for( auto const third : corners ) {
                     if( ( third == From ) || ( third == To ) ) { continue; }
+                    // a piece of a tile's border, or of the ground's own edge: either way this triangle
+                    // is the only thing holding it, and it is about to go
                     if( ( vertices[ To ].sides & vertices[ third ].sides ) != 0 ) { return false; }
+                    if( true == meshedge( To, third ) ) { return false; }
                 }
                 continue;
             }
@@ -765,6 +952,20 @@ cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
         }
         // and nothing that moves the ground further than this level is allowed to
         if( drop_of( From, To ) > Allowed ) { return false; }
+        // Anything touching a boundary between materials - crossing it, or standing next to it - may
+        // only reach so far. Short of that the boundary is left where the scenery drew it and the
+        // change stays narrow; past it there is nothing worth defending, since at the distance this
+        // level is drawn from the whole thing is a pixel or two wide.
+        if( ( true == onboundary[ From ] ) || ( true == onboundary[ To ] )
+         || ( vertices[ From ].material != vertices[ To ].material ) ) {
+            auto const &from { local[ From ] };
+            auto const &to { local[ To ] };
+            auto const reach {
+                std::sqrt( ( from[ 0 ] - to[ 0 ] ) * ( from[ 0 ] - to[ 0 ] )
+                         + ( from[ 1 ] - to[ 1 ] ) * ( from[ 1 ] - to[ 1 ] )
+                         + ( from[ 2 ] - to[ 2 ] ) * ( from[ 2 ] - to[ 2 ] ) ) };
+            if( reach > Boundaryreach ) { return false; }
+        }
         return true; };
 
     auto const offer = [ & ]( std::uint32_t const Vertex ) {
@@ -797,6 +998,11 @@ cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
 
         removed[ best.from ] = true;
         quadrics[ best.to ] += quadrics[ best.from ];
+        // the survivor now stands for both, and its normal has to say so, or a coarse level ends up
+        // lit by whichever slope happened to keep its vertex
+        vertices[ best.to ].nx += vertices[ best.from ].nx;
+        vertices[ best.to ].ny += vertices[ best.from ].ny;
+        vertices[ best.to ].nz += vertices[ best.from ].nz;
 
         for( auto const triangle : incident[ best.from ] ) {
             if( true == gone[ triangle ] ) { continue; }
@@ -923,7 +1129,7 @@ cooker::encode( tile_address const &Tile, mesh const &Mesh, std::vector<std::uin
     header.horizon_x = header.horizon_y = header.horizon_z = 0.0;
 
     Out.clear();
-    Out.reserve( order.size() * 8 + indices.size() * 2 + sizeof( header ) + 64 );
+    Out.reserve( order.size() * 10 + indices.size() * 2 + sizeof( header ) + 64 );
     append( Out, header );
     append( Out, static_cast<std::uint32_t>( order.size() ) );
 
@@ -970,6 +1176,17 @@ cooker::encode( tile_address const &Tile, mesh const &Mesh, std::vector<std::uin
         else               { append_edge<std::uint16_t>( Out, edge ); }
     }
 
+    // the format's own: a normal per vertex, two bytes each
+    append( Out, static_cast<std::uint8_t>( ext_normals ) );
+    append( Out, static_cast<std::uint32_t>( order.size() * 2 ) );
+    for( auto const index : order ) {
+        auto const &vertex { Mesh.vertices[ index ] };
+        std::uint8_t first { 128 }, second { 128 };
+        encode_normal( vertex.nx, vertex.ny, vertex.nz, first, second );
+        append( Out, first );
+        append( Out, second );
+    }
+
     // ours: which ground material each vertex belongs to. the format has nowhere for it
     append( Out, static_cast<std::uint8_t>( ext_ground ) );
     append( Out, static_cast<std::uint32_t>( order.size() * sizeof( std::uint16_t ) ) );

scene/quantizedmeshcooker.h

@@ -46,8 +46,9 @@ public:
         float u, v;
     };
 
-    // one finished tile, handed over as the bytes of its file
+    // one finished tile, handed over as the bytes of its file and what its ground asks for
-    using sink = std::function<void( tile_address const &, std::vector<std::uint8_t> const & )>;
+    using sink = std::function<
+        void( tile_address const &, std::vector<std::uint8_t> const &, tile_measure const & )>;
 
     cooker() = default;
     ~cooker();
@@ -91,6 +92,12 @@ public:
     // a vertex of a tile's mesh while it is being worked on, in metres
     struct meshvertex {
         double x, y, z;
+        // The ground's normal here, taken from the scenery's own triangles before they were cut, and
+        // kept unnormalised: what is stored is the sum of the faces meeting here weighted by their
+        // area, so the length carries the weight. When a vertex is collapsed away its sum is added to
+        // the survivor's, which makes a coarse level's normals the average of the fine surface they
+        // stand for rather than a sample of it. It is normalised on the way out.
+        double nx { 0.0 }, ny { 1.0 }, nz { 0.0 };
         std::uint16_t material;
         // which of the tile's four sides it sits on, one bit each. a vertex on a side is shared
         // with the tile beyond it, so it survives into every level; and an edge whose two ends
@@ -130,8 +137,10 @@ private:
     // the four tiles below this one, welded into one mesh
     mesh merge( tile_address const &Tile, sink const &Sink );
     // collapses edges until the mesh fits the budget, or until the ground would move further than
-    // Allowed, whichever comes first. locked vertices are left where they are
+    // Allowed, whichever comes first. locked vertices are left where they are; a collapse touching a
-    void simplify( mesh &Mesh, std::size_t Budget, double Allowed );
+    // boundary between materials may not reach further than Boundaryreach; and one that moves the edge
+    // of the ground itself may not reach further than Outlinereach, which at zero forbids it outright
+    void simplify( mesh &Mesh, std::size_t Budget, double Allowed, double Boundaryreach, double Outlinereach );
     // the bytes of a tile's file
     void encode( tile_address const &Tile, mesh const &Mesh, std::vector<std::uint8_t> &Out ) const;
     // where a tile's square lies
@@ -144,8 +153,10 @@ private:
 
     double m_finesttile { 256.0 };
     std::size_t m_tilebudget { 8192 };
-    // a sample of how long a triangle's edges are in the plan, which decides the finest tile
+    // how long a triangle's edges are in the plan, averaged over the ground and weighted by how much
-    std::vector<double> m_edges;
+    // of it each triangle covers. that is what decides the finest tile
+    double m_edgesum { 0.0 };
+    double m_areasum { 0.0 };
 
     std::vector<std::string> m_materials;
     std::unordered_map<std::string, std::uint16_t> m_materiallookup;
@@ -168,8 +179,10 @@ private:
     std::vector<std::pair<std::int64_t, std::uint32_t>> m_placement;
 
     std::vector<std::uint32_t> m_refusals;
-    // the worst error any tile of each level came out with
+    // the worst error any tile of each level came out with, and how much ground its triangles cover
     std::vector<double> m_errors;
+    std::vector<double> m_levelarea;
+    std::vector<std::size_t> m_leveltriangles;
     report m_report;
 };
 

scene/quantizedmeshformat.h

@@ -40,6 +40,7 @@ http://mozilla.org/MPL/2.0/.
 //    the gzip the format expects from an HTTP layer. The bytes of a tile are the format's.
 
 #include <algorithm>
+#include <cmath>
 #include <cstdint>
 #include <cstring>
 #include <string>
@@ -52,7 +53,17 @@ inline constexpr std::uint16_t quantum_max { 32767 };
 
 // what the cook counted as ground, and how it cut and simplified it. tiles cooked under other
 // rules are baked again rather than read
-inline constexpr std::uint32_t cook_rules { 2 };
+inline constexpr std::uint32_t cook_rules { 3 };
+
+// What one tile's ground asks for, measured when it was cooked: how far it departs from the finest
+// mesh, and how long its triangles are. Which level to draw follows from these two, and they belong
+// to the tile rather than to its level. The level's figure is the worst tile in it, so a single
+// steep corner made the whole pyramid ask to be drawn from as far away as a mountain - and flat
+// ground twenty kilometres out was split as finely as a valley wall underfoot.
+struct tile_measure {
+    float error { 0.f };
+    float edge { 0.f };
+};
 
 #pragma pack( push, 1 )
 
@@ -71,6 +82,7 @@ static_assert( sizeof( file_header ) == 88 );
 
 // extension ids. 1, 2 and 4 are taken by the format for normals, the water mask and metadata
 enum extension_id : std::uint8_t {
+    // the format's own: one normal per vertex, octahedron-encoded into two bytes
     ext_normals = 1,
     ext_watermask = 2,
     ext_metadata = 4,
@@ -89,6 +101,34 @@ zigzag_decode( std::uint16_t const Stored ) {
     return ( Stored >> 1 ) ^ -static_cast<std::int32_t>( Stored & 1 );
 }
 
+// A unit vector into two bytes, as the format's normals extension has it: the sphere folded onto an
+// octahedron, unfolded into a square, and the square's coordinates stored as unsigned bytes. Two bytes
+// hold a normal to well under a degree, which is what the lighting needs and no more.
+//
+// Normals are stored rather than worked out while drawing. A normal taken from how the position
+// changes across a triangle is that triangle's own, so the ground reads as facets, and - worse - a
+// coarser level's facets face differently from the ones they stand for, so the light over a hillside
+// changes as the level does. A normal carried on the vertex comes from the ground as the scenery drew
+// it and stays the same at every level.
+inline void
+encode_normal( double const X, double const Y, double const Z, std::uint8_t &Out0, std::uint8_t &Out1 ) {
+
+    auto const length { std::abs( X ) + std::abs( Y ) + std::abs( Z ) };
+    if( length < 1e-12 ) { Out0 = Out1 = 128; return; }
+    auto first { X / length };
+    auto second { Z / length };
+    if( Y < 0.0 ) {
+        auto const was { first };
+        first = ( 1.0 - std::abs( second ) ) * ( was >= 0.0 ? 1.0 : -1.0 );
+        second = ( 1.0 - std::abs( was ) ) * ( second >= 0.0 ? 1.0 : -1.0 );
+    }
+    auto const tobyte = []( double const Value ) {
+        return static_cast<std::uint8_t>(
+            std::clamp( std::round( ( Value + 1.0 ) * 0.5 * 255.0 ), 0.0, 255.0 ) ); };
+    Out0 = tobyte( first );
+    Out1 = tobyte( second );
+}
+
 // a quantized coordinate back to metres, against the range it was quantized in
 inline double
 dequantize( std::uint16_t const Stored, double const Low, double const Span ) {
@@ -175,9 +215,13 @@ struct terrain_table {
     double lowest { 0.0 }, highest { 0.0 };
     // levels cooked, root included: addresses run from 0 to levels - 1
     std::uint32_t levels { 1 };
-    // metres the ground of a tile at each level departs from the finest one. what decides which
+    // metres the ground of a tile at each level departs from the finest one, and how long a triangle
-    // level to draw: a level is good enough while its error covers few enough pixels
+    // of it typically is. Between them they decide which level to draw. The error alone cannot: it is
+    // the worst departure anywhere in the tile and is inherited by every level above, so one steep
+    // corner sets it for the whole pyramid and the levels become indistinguishable. What actually
+    // tells a level of sixty triangles from one of fifteen thousand is how large the triangles are.
     std::vector<float> errors;
+    std::vector<float> edges;
     // one ground texture: its name, and how many metres of ground one repeat of it covers, as
     // measured from the texture coordinates the scenery drew it with
     struct material {
@@ -194,6 +238,9 @@ struct terrain_table {
     float error( std::uint32_t const Level ) const {
         return Level < errors.size() ? errors[ Level ] : 0.f;
     }
+    float edge( std::uint32_t const Level ) const {
+        return Level < edges.size() ? edges[ Level ] : 0.f;
+    }
 };
 
 } // namespace quantizedmesh

scene/quantizedmeshreader.h

@@ -23,22 +23,22 @@ http://mozilla.org/MPL/2.0/.
 #include <filesystem>
 #include <fstream>
 #include <string>
-#include <unordered_set>
 #include <vector>
 
-#include "scene/pakformat.h"
+#include "scene/quantizedmesharchive.h"
 #include "scene/quantizedmeshformat.h"
 
 namespace quantizedmesh {
 
 // one vertex as the renderer wants it: measured from the terrain's origin, so that the whole
 // terrain fits in floats and every tile of it can go into one vertex buffer. there are no texture
-// coordinates - the ground tiles in world space, so the shader works them out from the position
+// coordinates - the ground tiles in world space, so the shader works them out from the position -
+// and the normal stays octahedron-encoded, which the shader unfolds. sixteen bytes either way
 #pragma pack( push, 1 )
 struct render_vertex {
     float x, y, z;
     std::uint16_t material;
-    std::uint16_t padding;
+    std::uint8_t normal0, normal1;
 };
 #pragma pack( pop )
 
@@ -76,20 +76,26 @@ public:
     // that every tile can go into the same vertex buffer and be drawn in one call
     bool read_tile( tile_address const &Tile, tile_data &Out,
         double Originx, double Originy, double Originz );
-    // the tiles the source holds, by tile_key. a tile the cook found nothing for is not there
+    // whether the source holds this tile. a tile the cook found no ground for is not there, and most
-    std::unordered_set<std::int64_t> const &present() const { return m_present; }
+    // of a quadtree is like that, so this is the commonest question the renderer asks
-    bool contains( tile_address const &Tile ) const { return m_present.contains( tile_key( Tile ) ); }
+    bool contains( tile_address const &Tile ) const;
+    // What this tile's ground asks for. An archive carries it per tile; a directory of loose tiles
+    // has nowhere to keep it, so there the level's figure stands in, which is what the whole terrain
+    // used before the index carried anything better
+    tile_measure measure( tile_address const &Tile ) const;
+    std::size_t tilecount() const;
 
 private:
-    bool entry( std::string const &Name, std::vector<std::uint8_t> &Out );
+    bool loose_file( std::string const &Name, std::vector<std::uint8_t> &Out ) const;
     bool read_table();
-    void find_tiles();
+    void find_loose_tiles();
 
     std::string m_path;
-    bool m_archive { false };
+    bool m_isarchive { false };
-    pak::reader m_pak;
+    archive_reader m_archive;
     terrain_table m_table;
-    std::unordered_set<std::int64_t> m_present;
+    // only for a directory of loose tiles; an archive is its own index
+    std::vector<std::int64_t> m_loose;
     bool m_ready { false };
     std::vector<std::uint8_t> m_buffer;
 };
@@ -103,18 +109,18 @@ reader::open( std::string const &Path ) {
     m_path = Path;
     std::error_code error;
     if( true == std::filesystem::is_directory( Path, error ) ) {
-        m_archive = false;
+        m_isarchive = false;
     }
     else if( true == std::filesystem::is_regular_file( Path, error ) ) {
-        m_archive = true;
+        m_isarchive = true;
-        if( false == m_pak.open( Path ) ) { close(); return false; }
+        if( false == m_archive.open( Path, cook_rules ) ) { close(); return false; }
     }
     else {
         close();
         return false;
     }
     if( false == read_table() ) { close(); return false; }
-    find_tiles();
+    if( false == m_isarchive ) { find_loose_tiles(); }
     m_ready = true;
     return true;
 }
@@ -122,18 +128,38 @@ reader::open( std::string const &Path ) {
 inline void
 reader::close() {
 
-    m_pak.close();
+    m_archive.close();
     m_table = {};
-    m_present.clear();
+    m_loose.clear();
+    m_loose.shrink_to_fit();
     m_path.clear();
-    m_archive = false;
+    m_isarchive = false;
     m_ready = false;
 }
 
 inline bool
-reader::entry( std::string const &Name, std::vector<std::uint8_t> &Out ) {
+reader::contains( tile_address const &Tile ) const {
 
-    if( true == m_archive ) { return m_pak.read( Name, Out ); }
+    if( true == m_isarchive ) { return m_archive.contains( Tile ); }
+    return std::binary_search( m_loose.begin(), m_loose.end(), tile_key( Tile ) );
+}
+
+inline tile_measure
+reader::measure( tile_address const &Tile ) const {
+
+    tile_measure measure { m_table.error( Tile.level ), m_table.edge( Tile.level ) };
+    if( true == m_isarchive ) { m_archive.measure( Tile, measure ); }
+    return measure;
+}
+
+inline std::size_t
+reader::tilecount() const {
+
+    return m_isarchive ? m_archive.tilecount() : m_loose.size();
+}
+
+inline bool
+reader::loose_file( std::string const &Name, std::vector<std::uint8_t> &Out ) const {
 
     std::ifstream file( std::filesystem::path( m_path ) / Name, std::ios::binary );
     if( false == file.good() ) { return false; }
@@ -147,13 +173,37 @@ reader::entry( std::string const &Name, std::vector<std::uint8_t> &Out ) {
     return static_cast<std::size_t>( file.gcount() ) == size;
 }
 
+// Which tiles a directory holds is worked out once. Asking the filesystem for a tile that is not there
+// is the commonest thing the renderer does - most of a quadtree is empty - and it must not cost a
+// system call.
+inline void
+reader::find_loose_tiles() {
+
+    std::error_code error;
+    auto const base { std::filesystem::path( m_path ) };
+    for( auto const &found : std::filesystem::recursive_directory_iterator( base, error ) ) {
+        if( false == found.is_regular_file( error ) ) { continue; }
+        auto const relative { std::filesystem::relative( found.path(), base, error ) };
+        if( error ) { continue; }
+        auto const name { relative.generic_string() };
+        if( false == name.ends_with( ".terrain" ) ) { continue; }
+        tile_address tile {};
+        if( 3 != std::sscanf( name.c_str(), "%u/%d/%d.terrain", &tile.level, &tile.x, &tile.z ) ) { continue; }
+        if( name != tile_name( tile ) ) { continue; }
+        m_loose.push_back( tile_key( tile ) );
+    }
+    std::sort( m_loose.begin(), m_loose.end() );
+}
+
 // the table is text: a terrain outlives the build that cooked it, and a line of it should be
 // readable without a tool
 inline bool
 reader::read_table() {
 
     std::vector<std::uint8_t> bytes;
-    if( false == entry( tablename, bytes ) ) { return false; }
+    auto const got {
+        m_isarchive ? m_archive.read_table( bytes ) : loose_file( tablename, bytes ) };
+    if( false == got ) { return false; }
     std::string const text( bytes.begin(), bytes.end() );
 
     auto found { false };
@@ -190,6 +240,13 @@ reader::read_table() {
                     { rest.substr( 0, gap ), static_cast<float>( std::strtod( rest.c_str() + gap + 1, nullptr ) ) } );
             }
         }
+        else if( what == "edge" ) {
+            char const *cursor { rest.c_str() };
+            auto const level { std::strtoul( cursor, const_cast<char **>( &cursor ), 10 ) };
+            auto const metres { std::strtod( cursor, const_cast<char **>( &cursor ) ) };
+            if( m_table.edges.size() <= level ) { m_table.edges.resize( level + 1, 0.f ); }
+            m_table.edges[ level ] = static_cast<float>( metres );
+        }
         else if( what == "error" ) {
             char const *cursor { rest.c_str() };
             auto const level { std::strtoul( cursor, const_cast<char **>( &cursor ), 10 ) };
@@ -204,40 +261,15 @@ reader::read_table() {
         && ( m_table.levels >= 1 );
 }
 
-// Which tiles there are is worked out once. Asking the filesystem, or even the archive's index,
-// for a tile that is not there is the commonest thing the renderer does - most of a quadtree is
-// empty - and it must not cost anything.
-inline void
-reader::find_tiles() {
-
-    auto const remember = [ this ]( std::string const &Name ) {
-        if( false == Name.ends_with( ".terrain" ) ) { return; }
-        tile_address tile {};
-        if( 3 != std::sscanf( Name.c_str(), "%u/%d/%d.terrain", &tile.level, &tile.x, &tile.z ) ) { return; }
-        if( Name != tile_name( tile ) ) { return; }
-        m_present.insert( tile_key( tile ) ); };
-
-    if( true == m_archive ) {
-        for( auto const &name : m_pak.names() ) { remember( name ); }
-        return;
-    }
-    std::error_code error;
-    auto const base { std::filesystem::path( m_path ) };
-    for( auto const &found : std::filesystem::recursive_directory_iterator( base, error ) ) {
-        if( false == found.is_regular_file( error ) ) { continue; }
-        auto const relative { std::filesystem::relative( found.path(), base, error ) };
-        if( error ) { continue; }
-        remember( relative.generic_string() );
-    }
-}
-
 inline bool
 reader::read_tile( tile_address const &Tile, tile_data &Out,
     double const Originx, double const Originy, double const Originz ) {
 
     if( false == m_ready ) { return false; }
     if( Tile.level >= m_table.levels ) { return false; }
-    if( false == entry( tile_name( Tile ), m_buffer ) ) { return false; }
+    auto const got {
+        m_isarchive ? m_archive.read_tile( Tile, m_buffer ) : loose_file( tile_name( Tile ), m_buffer ) };
+    if( false == got ) { return false; }
 
     auto const *cursor { m_buffer.data() };
     auto const *end { cursor + m_buffer.size() };
@@ -307,8 +339,9 @@ reader::read_tile( tile_address const &Tile, tile_data &Out,
         cursor += count * indexsize;
     }
 
-    // which material each vertex belongs to, from the extension the cook writes
+    // the normals the format carries, and which material each vertex belongs to, which is ours
     std::vector<std::uint16_t> ground;
+    std::vector<std::uint8_t> normals;
     while( cursor < end ) {
         std::uint8_t id { 0 };
         std::uint32_t length { 0 };
@@ -319,6 +352,10 @@ reader::read_tile( tile_address const &Tile, tile_data &Out,
             ground.resize( vertexcount );
             std::memcpy( ground.data(), cursor, length );
         }
+        else if( ( id == ext_normals ) && ( length == vertexcount * 2u ) ) {
+            normals.resize( length );
+            std::memcpy( normals.data(), cursor, length );
+        }
         cursor += length;
     }
     if( ground.size() != vertexcount ) { return false; }
@@ -346,7 +383,10 @@ reader::read_tile( tile_address const &Tile, tile_data &Out,
         vertex.y = static_cast<float>( dequantize( quantized[ 2 ][ index ], header.lowest, heightspan ) - Out.originy );
         vertex.z = static_cast<float>( dequantize( quantized[ 1 ][ index ], north, side ) - Out.originz );
         vertex.material = ground[ index ];
-        vertex.padding = 0;
+        // a tile without the normals extension - one made by another tool - is lit as if the ground
+        // were level, which is wrong but is not a reason to refuse it
+        vertex.normal0 = ( normals.size() == vertexcount * 2u ) ? normals[ index * 2 + 0 ] : 128;
+        vertex.normal1 = ( normals.size() == vertexcount * 2u ) ? normals[ index * 2 + 1 ] : 128;
     }
     return true;
 }
@@ -368,6 +408,9 @@ write_table( terrain_table const &Table ) {
     for( std::size_t level = 0; level < Table.errors.size(); ++level ) {
         text += "error " + std::to_string( level ) + " " + number( Table.errors[ level ] ) + "\n";
     }
+    for( std::size_t level = 0; level < Table.edges.size(); ++level ) {
+        text += "edge " + std::to_string( level ) + " " + number( Table.edges[ level ] ) + "\n";
+    }
     for( auto const &material : Table.materials ) {
         text += "material " + material.name + " " + number( material.repeat ) + "\n";
     }

shaders/terrainmesh.frag

@@ -1,16 +1,21 @@
 // Terrain mesh, fragment stage.
 //
-// One triangle carries one material: the cook only welds vertices that share it, so nothing has to
+// The texture coordinate is the world position over the material's tiling, so the ground keeps the
-// be blended here and a material boundary follows the edges the scenery drew it with. The texture
+// scale it was authored at whatever level the tile is drawn at. Where a triangle has one material at
-// coordinate is the world position over the material's tiling, so the ground keeps the scale it was
+// every corner - which is nearly all of them - that is one texture sample; where it crosses a boundary
-// authored at whatever level the tile is drawn at.
+// the three are blended by how near the fragment is to each corner, so the ground changes over a
+// triangle rather than along its edge.
 //
-// The mesh carries no normals - that would be four more bytes on every vertex - so the surface
+// The normal comes from the vertex, not from how the view position changes across the triangle. A
-// normal comes from how the view position changes across the triangle, which is its face normal.
+// normal worked out here would be the triangle's own, so the ground would read as facets, and a coarser
+// level's facets face differently from the ones they stand for - which shows as the light over a
+// hillside changing when the level does.
 
 in vec4 f_pos;
+in vec3 f_normal;
 in vec2 f_ground;
-flat in uint f_material;
+in vec3 f_weights;
+flat in uvec3 f_materials;
 
 #include <common>
 #include <apply_fog.glsl>
@@ -26,16 +31,59 @@ layout(location = 0) out vec4 out_color;
 layout(location = 1) out vec4 out_motion;
 #endif
 
+// Value noise from the world position: the same everywhere the ground is, at any level, and costing
+// nothing to store. Two octaves are enough for what it is for.
+float ground_hash( vec2 Cell )
+{
+	return fract( sin( dot( Cell, vec2( 127.1, 311.7 ) ) ) * 43758.5453 );
+}
+
+float ground_noise( vec2 Where )
+{
+	vec2 cell = floor( Where );
+	vec2 part = fract( Where );
+	part = part * part * ( 3.0 - 2.0 * part );
+	return mix(
+		mix( ground_hash( cell ), ground_hash( cell + vec2( 1.0, 0.0 ) ), part.x ),
+		mix( ground_hash( cell + vec2( 0.0, 1.0 ) ), ground_hash( cell + vec2( 1.0, 1.0 ) ), part.x ),
+		part.y );
+}
+
+vec3 material_colour( uint Material )
+{
+	vec4 entry = texelFetch( materialtable, ivec2( int( Material ), 0 ), 0 );
+	if( entry.a < 0.0 ) { return entry.rgb; }
+	return texture( ground, vec3( f_ground / max( entry.a, 0.01 ), float( Material ) ) ).rgb;
+}
+
 void main()
 {
-	vec4 entry = texelFetch( materialtable, ivec2( int( f_material ), 0 ), 0 );
+	vec3 albedo;
-	vec3 albedo =
+	if( ( f_materials.x == f_materials.y ) && ( f_materials.x == f_materials.z ) ) {
-		( entry.a < 0.0 )
+		// the common case: one kind of ground across the whole triangle, one texture sample
-			? entry.rgb
+		albedo = material_colour( f_materials.x );
-			: texture( ground, vec3( f_ground / max( entry.a, 0.01 ), float( f_material ) ) ).rgb;
+	}
+	else {
+		// The weights say how near the fragment is to each corner, so left alone the ground would fade
+		// along a straight line across the triangle and the mesh would be visible in the texturing.
+		// Disturbing them with noise taken from the world position makes the two grounds interlock
+		// instead, which is what a boundary between them looks like - and since the noise is a function
+		// of where the fragment is, it does not change with the level the tile is drawn at.
+		float grain = ground_noise( f_ground * 0.6 ) * 0.62
+		            + ground_noise( f_ground * 2.9 ) * 0.28
+		            + ground_noise( f_ground * 11.0 ) * 0.10;
+		vec3 weights = max( f_weights + ( grain - 0.5 ) * 0.55, vec3( 0.0 ) );
+		// and sharpened a little, so the interlocking reads as one ground over the other rather than as
+		// a wash of both
+		weights = weights * weights;
+		weights /= max( weights.x + weights.y + weights.z, 1e-4 );
+		albedo = weights.x * material_colour( f_materials.x )
+		       + weights.y * material_colour( f_materials.y )
+		       + weights.z * material_colour( f_materials.z );
+	}
 
-	vec3 normal = normalize( cross( dFdx( f_pos.xyz ), dFdy( f_pos.xyz ) ) );
+	vec3 normal = normalize( f_normal );
-	// towards the camera, whichever way the derivatives came out
+	// a wall is drawn from either side, so the normal follows whichever way it is seen from
 	if( dot( normal, -f_pos.xyz ) < 0.0 ) { normal = -normal; }
 
 	// a plain lambert term against the first directional light, so that relief reads

shaders/terrainmesh.geom

@@ -0,0 +1,42 @@
+// Terrain mesh, geometry stage.
+//
+// It is here only to tell the fragment stage which three materials the triangle it is in has at its
+// corners, and how near that fragment is to each of them. A vertex carries one material; a triangle
+// crossing a boundary between two kinds of ground has different ones at its corners, and what makes
+// that read as a transition rather than as a line along the triangle's edge is blending the three by
+// their barycentric weight.
+//
+// Nothing else needs a whole stage: the weights cannot be worked out in the fragment stage on its own,
+// since an indexed mesh shares a vertex between triangles and a vertex cannot say which corner of which
+// triangle it is. Desktop GL 4.6 has gl_BaryCoordEXT for exactly this; 3.3 does not.
+
+#include <common>
+
+layout(triangles) in;
+layout(triangle_strip, max_vertices = 3) out;
+
+in vec4 v_pos[];
+in vec3 v_normal[];
+in vec2 v_ground[];
+in uint v_material[];
+
+out vec4 f_pos;
+out vec3 f_normal;
+out vec2 f_ground;
+out vec3 f_weights;
+flat out uvec3 f_materials;
+
+void main()
+{
+	uvec3 materials = uvec3( v_material[ 0 ], v_material[ 1 ], v_material[ 2 ] );
+	for( int corner = 0; corner < 3; ++corner ) {
+		f_pos = v_pos[ corner ];
+		f_normal = v_normal[ corner ];
+		f_ground = v_ground[ corner ];
+		f_weights = vec3( corner == 0 ? 1.0 : 0.0, corner == 1 ? 1.0 : 0.0, corner == 2 ? 1.0 : 0.0 );
+		f_materials = materials;
+		gl_Position = gl_in[ corner ].gl_Position;
+		EmitVertex();
+	}
+	EndPrimitive();
+}

shaders/terrainmesh.vert

@@ -1,32 +1,52 @@
 // Terrain mesh, vertex stage.
 //
-// A vertex is a position and the ground material it belongs to, nothing else. The position is
+// A vertex is a position, the ground material it belongs to, and the ground's normal there, folded onto
-// measured from the middle of the terrain rather than from the scenery's origin, so that the whole
+// an octahedron and kept in two bytes. The position is measured from the middle of the terrain rather
-// of a map fits in floats and every resident tile can live in one vertex buffer - which is what
+// than from the scenery's origin, so that the whole of a map fits in floats and every resident tile can
-// lets a frame of terrain be one draw call. terrainorigin is where that middle is relative to the
+// live in one vertex buffer - which is what lets a frame of terrain be one draw call. terrainorigin is
-// camera, terrainworld where it is in the world.
+// where that middle is relative to the camera, terrainworld where it is in the world.
 //
-// There are no texture coordinates: ground textures tile in world space, so the coordinate follows
+// There are no texture coordinates: ground textures tile in world space, so the coordinate follows from
-// from where the vertex is and from how many metres one repeat of its material covers, which the
+// where the vertex is and from how many metres one repeat of its material covers, which the cook
-// cook measured from the scenery and left in the material table.
+// measured from the scenery and left in the material table.
+//
+// Everything arrives as a float rather than as an integer. The engine's vertex array wrapper sets
+// attributes up that way, and going around it would leave what the engine thinks is bound out of step
+// with what is.
 
 #include <common>
 
 layout(location = 0) in vec3 in_position;
-layout(location = 1) in uint in_material;
+layout(location = 1) in float in_material;
+layout(location = 2) in vec2 in_normal;     // octahedron-encoded, 0..255 each
 
 uniform vec3 terrainorigin;   // the terrain's middle, relative to the camera
 uniform vec2 terrainworld;    // the terrain's middle, in the world
 
-out vec4 f_pos;
+out vec4 v_pos;
-out vec2 f_ground;            // world position on the ground, for texturing
+out vec3 v_normal;
-flat out uint f_material;
+out vec2 v_ground;            // world position on the ground, for texturing
+out uint v_material;
+
+// the square unfolded back onto the sphere. y is up, so it is y whose sign the fold carries
+vec3 unfold_normal( vec2 Encoded )
+{
+	vec2 folded = Encoded / 255.0 * 2.0 - 1.0;
+	vec3 normal = vec3( folded.x, 1.0 - abs( folded.x ) - abs( folded.y ), folded.y );
+	if( normal.y < 0.0 ) {
+		float wasx = normal.x;
+		normal.x = ( 1.0 - abs( normal.z ) ) * ( wasx >= 0.0 ? 1.0 : -1.0 );
+		normal.z = ( 1.0 - abs( wasx ) ) * ( normal.z >= 0.0 ? 1.0 : -1.0 );
+	}
+	return normalize( normal );
+}
 
 void main()
 {
 	vec3 local = in_position + terrainorigin;
-	f_ground = terrainworld + in_position.xz;
+	v_ground = terrainworld + in_position.xz;
-	f_material = in_material;
+	v_material = uint( in_material + 0.5 );
-	f_pos = modelview * vec4( local, 1.0 );
+	v_normal = normalize( modelviewnormal * unfold_normal( in_normal ) );
-	gl_Position = projection * f_pos;
+	v_pos = modelview * vec4( local, 1.0 );
+	gl_Position = projection * v_pos;
 }

simulation/terrainbake.cpp

@@ -21,7 +21,7 @@ module;
 #include <unordered_map>
 #include <vector>
 #include "utilities/Globals_macros.h"
-#include "scene/pakformat.h"
+#include "scene/quantizedmesharchive.h"
 #include "scene/quantizedmeshcooker.h"
 #include "scene/quantizedmeshreader.h"
 
@@ -374,6 +374,14 @@ void
 finish( std::string const &Sceneryfile, std::vector<std::string> const &Included ) {
 
     if( true == planning() ) {
+        // let the note go: it is only wanted while the scenery is being read, and on a large map it is
+        // hundreds of megabytes - galicja's is two hundred and seventy
+        auto const letgo {
+            []() {
+                decltype( state.note ) empty;
+                state.note.swap( empty );
+                std::vector<std::uint8_t> nomasks;
+                state.masks.swap( nomasks ); } };
         WriteLog(
             "Terrain cook: " + std::to_string( state.skipped ) + " triangle nodes left to the tiles, "
             + std::to_string( state.filtered ) + " drawn in part, " + std::to_string( state.drawn )
@@ -384,6 +392,7 @@ finish( std::string const &Sceneryfile, std::vector<std::string> const &Included
             WriteLog( "Terrain cook: the scenery includes different files than when it was cooked; cooking again on the next start" );
             discard( Sceneryfile );
         }
+        letgo();
         return;
     }
     if( false == state.collecting ) { return; }
@@ -401,8 +410,8 @@ finish( std::string const &Sceneryfile, std::vector<std::string> const &Included
     // the cook itself, apart from the load that fed it: what a second start saves is the cook
     auto const cookstarted { std::chrono::steady_clock::now() };
     auto const archive { archive_path( Sceneryfile ) };
-    pak::writer writer;
+    quantizedmesh::archive_writer writer;
-    if( false == writer.open( archive ) ) {
+    if( false == writer.open( archive, quantizedmesh::cook_rules ) ) {
         ErrorLog( "Terrain cook: cannot write \"" + archive + "\"" );
         return;
     }
@@ -410,8 +419,9 @@ finish( std::string const &Sceneryfile, std::vector<std::string> const &Included
     auto written { true };
     quantizedmesh::terrain_table table;
     auto const sink {
-        [ &writer, &written ]( quantizedmesh::tile_address const &Tile, std::vector<std::uint8_t> const &Bytes ) {
+        [ &writer, &written ]( quantizedmesh::tile_address const &Tile, std::vector<std::uint8_t> const &Bytes,
-            written = writer.add( quantizedmesh::tile_name( Tile ), Bytes ) && written; } };
+            quantizedmesh::tile_measure const &Measure ) {
+            written = writer.add( Tile, Bytes, Measure ) && written; } };
 
     if( ( false == state.cooker.finish( sink, table ) ) || ( false == written ) ) {
         ErrorLog( "Terrain cook: cooking failed, no tiles written" );
@@ -419,7 +429,7 @@ finish( std::string const &Sceneryfile, std::vector<std::string> const &Included
         discard( Sceneryfile );
         return;
     }
-    if( false == writer.add( quantizedmesh::tablename, quantizedmesh::write_table( table ) ) ) {
+    if( false == writer.table( quantizedmesh::write_table( table ) ) ) {
         written = false;
     }
     if( ( false == writer.close() ) || ( false == written ) ) {

tools/terraincook/main.cpp

@@ -33,6 +33,7 @@ http://mozilla.org/MPL/2.0/.
 #include "scene/heightfieldformat.h"
 #include "scene/heightfieldreader.h"
 #include "scene/terraincooker.h"
+#include "scene/quantizedmesharchive.h"
 #include "scene/quantizedmeshcooker.h"
 #include "scene/quantizedmeshreader.h"
 
@@ -885,9 +886,9 @@ main( int argc, char *argv[] ) {
     if( cooking ) {
         std::printf( "\n-- cooking terrain mesh tiles\n" );
 
-        pak::writer writer;
+        quantizedmesh::archive_writer writer;
         auto const archivepath { ( outputdirectory / archive ).string() };
-        if( false == writer.open( archivepath ) ) {
+        if( false == writer.open( archivepath, quantizedmesh::cook_rules ) ) {
             std::cerr << "cannot write " << archivepath << "\n";
             return 1;
         }
@@ -895,13 +896,14 @@ main( int argc, char *argv[] ) {
         quantizedmesh::terrain_table table;
         auto const sink {
             [ &writer, &written ](
-                quantizedmesh::tile_address const &Tile, std::vector<std::uint8_t> const &Bytes ) {
+                quantizedmesh::tile_address const &Tile, std::vector<std::uint8_t> const &Bytes,
-                written = writer.add( quantizedmesh::tile_name( Tile ), Bytes ) && written; } };
+                quantizedmesh::tile_measure const &Measure ) {
+                written = writer.add( Tile, Bytes, Measure ) && written; } };
         if( ( false == cooker.finish( sink, table ) ) || ( false == written ) ) {
             std::cerr << "cooking failed\n";
             return 1;
         }
-        written = writer.add( quantizedmesh::tablename, quantizedmesh::write_table( table ) ) && written;
+        written = writer.table( quantizedmesh::write_table( table ) ) && written;
         if( ( false == writer.close() ) || ( false == written ) ) {
             std::cerr << "cannot finish " << archivepath << "\n";
             return 1;

tools/terraincook/makepoligon.py

@@ -0,0 +1,234 @@
+#!/usr/bin/env python3
+"""Builds the terrain test bed for the td scenery.
+
+It stands in for td's own ground, which was sixty-two huge triangles of grass, and gives the cook and
+the renderer the three things a real scenery has and a synthetic one usually lacks: ground drawn at
+several densities, boundaries between materials of several shapes, and enough length that travelling
+along it loads tiles and lets them go.
+
+Run it with the game's directory as its argument. It writes scenery/td/tinpoligon.scm, which
+scenery/$td.scn includes.
+"""
+import math
+import os
+import sys
+
+FIELDX0, FIELDX1 = -1200.0, 2200.0
+FIELDZ0, FIELDZ1 = -1400.0, 3000.0
+CORRIDORX0, CORRIDORX1 = -200.0, 200.0
+CORRIDORZ1 = 120000.0
+COARSE, FINE, DENSE = 40.0, 4.0, 1.0
+TILING = 5.0
+
+# where the ground is drawn small, as survey-derived ground is: one by the track and one far along, to
+# look at again after travelling
+FINESTRETCHES = ( ( -120.0, 400.0 ), ( 20000.0, 20400.0 ) )
+# A range of hills the corridor runs through, six kilometres across and ten long, with a valley down
+# the middle to travel along. Peaks four to six hundred metres: high enough to be read as hills from
+# twenty kilometres away, which is what makes the coarse levels worth looking at - a level that keeps
+# the height but loses the ridges shows it at once from that distance.
+MOUNTAINS = ( -3000.0, 3000.0, 24000.0, 34000.0 )
+
+# And a far larger range at the end of the corridor, to be looked at from a hundred kilometres. Peaks
+# of two thousand metres: at that distance one still covers some twenty pixels, so whether a coarse
+# level keeps the ridges or flattens them into a lump is plain to see from where the corridor starts.
+FARRANGE = ( -12000.0, 12000.0, 88000.0, 112000.0 )
+
+# and where it is drawn at the density an airborne scan gives, a metre between points. Two of them:
+# one by the track, and one far along the corridor, so the same kind of ground can be looked at after
+# travelling as well as from the start
+DENSEPATCHES = (
+    ( -200.0, 200.0, 600.0, 1000.0 ),
+    ( -200.0, 200.0, 10000.0, 10300.0 ),
+)
+
+
+def inside( x, z, rectangle ):
+    x0, x1, z0, z1 = rectangle
+    return ( x0 <= x < x1 ) and ( z0 <= z < z1 )
+
+
+def dense_here( x, z ):
+    for patch in DENSEPATCHES:
+        if inside( x, z, patch ):
+            return patch
+    return None
+
+
+def far_height( x, z ):
+    # the same shape at five times the size, so that it reads as a range from a hundred kilometres
+    across = min( 1.0, max( 0.0, ( abs( x ) - 400.0 ) / 7000.0 ) )
+    def ridged( value ):
+        return 1.0 - abs( math.sin( value ) )
+    crests = ( 1200.0 * ridged( z / 4200.0 + x / 9000.0 )
+             + 600.0 * ridged( z / 1700.0 - x / 5200.0 )
+             + 240.0 * ridged( z / 700.0 + x / 2100.0 ) )
+    rolling = 180.0 * math.sin( z / 9000.0 ) * math.cos( x / 7000.0 )
+    x0, x1, z0, z1 = FARRANGE
+    edge = min( x - x0, x1 - x, z - z0, z1 - z )
+    return ( across ** 1.5 ) * ( crests + rolling ) * min( 1.0, edge / 3000.0 )
+
+
+def mountain_height( x, z ):
+    # a valley along the middle, rising to ridges either side
+    across = min( 1.0, max( 0.0, ( abs( x ) - 250.0 ) / 2000.0 ) )
+    # ridged waves: sharp crests where a plain sine would give round tops
+    def ridged( value ):
+        return 1.0 - abs( math.sin( value ) )
+    crests = ( 220.0 * ridged( z / 900.0 + x / 2600.0 )
+             + 130.0 * ridged( z / 380.0 - x / 1500.0 )
+             + 60.0 * ridged( z / 160.0 + x / 520.0 ) )
+    rolling = 40.0 * math.sin( z / 2200.0 ) * math.cos( x / 1700.0 )
+    # faded to nothing at the edges of the range, so it meets the flat ground around it
+    x0, x1, z0, z1 = MOUNTAINS
+    edge = min( x - x0, x1 - x, z - z0, z1 - z )
+    return ( across ** 1.6 ) * ( crests + rolling ) * min( 1.0, edge / 600.0 )
+
+
+def height( x, z ):
+    if inside( x, z, FARRANGE ):
+        return far_height( x, z )
+    if inside( x, z, MOUNTAINS ):
+        return mountain_height( x, z )
+    # near nothing where the track runs, so the rails still sit on the ground
+    away = min( 1.0, ( abs( x ) / 120.0 ) ** 2 )
+    rolling = ( 3.0 * math.sin( z / 260.0 ) + 2.0 * math.cos( x / 140.0 )
+              + 8.0 * math.sin( z / 1700.0 ) * math.cos( x / 400.0 ) )
+    ridge = 6.0 * math.exp( -( ( x - 60.0 ) ** 2 + ( z - 200.0 ) ** 2 ) / 6000.0 )
+    shape = away * ( rolling + ridge )
+    # Detail only a dense mesh can hold, and only where the mesh is dense enough to hold it. Sampling
+    # a wave four metres long with cells forty metres apart gives every corner an arbitrary height,
+    # and the ground comes out as noise rather than as terrain.
+    patch = dense_here( x, z )
+    if patch is not None:
+        dx0, dx1, dz0, dz1 = patch
+        grain = 0.25 * math.sin( x * 1.7 ) * math.sin( z * 1.9 ) + 0.4 * math.sin( ( x + z ) / 7.0 )
+        # faded at the patch's own edge, so it meets the coarse ground around it evenly
+        grain *= min( 1.0, min( x - dx0, dx1 - x, z - dz0, dz1 - z ) / 10.0 )
+        return shape + grain
+    return shape
+
+
+def material( x, z ):
+    if inside( x, z, FARRANGE ):
+        above = far_height( x, z )
+        if above > 1100.0:
+            return "concrete1"
+        if above > 500.0:
+            return "rockmossy1"
+        if above > 150.0:
+            return "ziemia"
+        return "grassdarkgreen2"
+    if inside( x, z, MOUNTAINS ):
+        # by height, as ground cover goes: grass in the valley, bare earth on the slopes, rock on top
+        above = mountain_height( x, z )
+        if above > 260.0:
+            return "rockmossy1"
+        if above > 90.0:
+            return "ziemia"
+        return "grassdarkgreen2"
+    if dense_here( x, z ) is not None:
+        # patches a few metres across: material detail finer than the levels that will be drawn from
+        # far away, which is what the cook has to be able to merge
+        return "grass" if int( ( x + z ) // 6.0 ) % 2 == 0 else "ziemia"
+    for from_, to in FINESTRETCHES:
+        if from_ <= z < to:
+            local = z - from_
+            if ( x - 60.0 ) ** 2 + ( local - 150.0 ) ** 2 < 30.0 ** 2:
+                return "concrete1"
+            if local > 200.0 and ( x + 200.0 ) > ( local - 200.0 ) * 2.0:
+                return "piach_plaza"
+            return "grass" if x < 0.0 else "ziemia"
+    band = int( ( z + 1400.0 ) // 500.0 )
+    return [ "grass", "grassdarkgreen2", "ziemia", "piach_plaza", "asphaltgray1", "concrete1" ][ band % 6 ]
+
+
+def hole( x, z ):
+    # gaps the scenery leaves, as it does for water or a building, every two kilometres - but not
+    # through the hills, where a hole would only be confusing
+    if inside( x, z, MOUNTAINS ) or inside( x, z, FARRANGE ):
+        return False
+    return ( z > 3200.0 ) and ( math.fmod( z - 3000.0, 2000.0 ) < 80.0 ) and ( -80.0 < x < 80.0 )
+
+
+lines = [
+    "// Poligon doswiadczalny terenu TIN. Zastepuje wlasny plaski teren td.",
+    "// Pisany przez tools/terraincook/makepoligon.py - nie poprawiac recznie.",
+    "//",
+    "// Plat 3400 x 4400 m wokol toru, a od z = 3000 korytarz 400 m szerokosci do z = 43000, czyli",
+    "// 40 km - dwa razy wiecej niz zasieg wczytywania terenu, zeby bylo widac doladowywanie kafli.",
+    "//",
+    "// Trzy gestosci siatki, bo teren w sceneriach jest rysowany na trzy sposoby:",
+    "//   duze trojkaty co 40 m         - tlo, jak z 3ds Maxa",
+    "//   drobne co 4 m                 - x -200..200, z -120..400 przy torze, i z 20000..20400",
+    "//   gesta siatka co 1 m jak z NMT - x -200..200, z 600..1000 oraz z 10000..10300",
+    "// Granice materialow: prosta, skosna, okragla, a w gestym placie plamy co 6 m.",
+    "// Sciana nasypu na x 160, z 100..300. Dziury w ziemi co 2 km wzdluz korytarza.",
+    "// Gory na x -3000..3000, z 24000..34000: dolina posrodku, grzbiety do 400 m.",
+    "// Wielkie pasmo na x -12000..12000, z 88000..112000, szczyty do 2000 m - do ogladania ze 100 km.",
+    "// Korytarz siega z = 120000, czyli 120 km od toru.",
+    "",
+]
+
+
+def emit( triangle, name ):
+    lines.append( "node -1 0 none triangles " + name )
+    for index, ( x, y, z ) in enumerate( triangle ):
+        tail = " end" if index < 2 else ""
+        lines.append( "{:.3f} {:.3f} {:.3f} 0 1 0 {:.4f} {:.4f}{}".format(
+            x, y, z, x / TILING, z / TILING, tail ) )
+    lines.append( "endtri" )
+
+
+def quad( x0, z0, step ):
+    x1, z1 = x0 + step, z0 + step
+    if hole( x0 + step * 0.5, z0 + step * 0.5 ):
+        return
+    corner = { ( cx, cz ): ( cx, height( cx, cz ), cz ) for cx in ( x0, x1 ) for cz in ( z0, z1 ) }
+    name = material( x0 + step * 0.5, z0 + step * 0.5 )
+    emit( [ corner[ ( x0, z0 ) ], corner[ ( x0, z1 ) ], corner[ ( x1, z1 ) ] ], name )
+    emit( [ corner[ ( x0, z0 ) ], corner[ ( x1, z1 ) ], corner[ ( x1, z0 ) ] ], name )
+
+
+def lay( x0, x1, z0, z1, step, skip = () ):
+    # each patch is a rectangle on the coarse grid, and the coarse ground is not laid where a finer
+    # patch takes over. Where two densities meet the source has a T-junction, and so the cook
+    # faithfully has one - which is what scenery drawn in pieces looks like
+    z = z0
+    while z < z1 - 0.001:
+        x = x0
+        while x < x1 - 0.001:
+            if not any( inside( x + step * 0.5, z + step * 0.5, rectangle ) for rectangle in skip ):
+                quad( x, z, step )
+            x += step
+        z += step
+
+
+FINEA = ( -200.0, 200.0, -120.0, 400.0 )
+FINEB = ( -200.0, 200.0, 20000.0, 20400.0 )
+
+lay( FIELDX0, FIELDX1, FIELDZ0, FIELDZ1, COARSE, ( FINEA, ) + DENSEPATCHES )
+lay( FINEA[ 0 ], FINEA[ 1 ], FINEA[ 2 ], FINEA[ 3 ], FINE )
+lay( CORRIDORX0, CORRIDORX1, FIELDZ1, CORRIDORZ1, COARSE, ( FINEB, MOUNTAINS, FARRANGE ) + DENSEPATCHES )
+lay( MOUNTAINS[ 0 ], MOUNTAINS[ 1 ], MOUNTAINS[ 2 ], MOUNTAINS[ 3 ], COARSE )
+# the far range is drawn with cells of its own size: it is never seen from near, and a hundred
+# thousand triangles there would be a hundred thousand nobody ever looks at closely
+lay( FARRANGE[ 0 ], FARRANGE[ 1 ], FARRANGE[ 2 ], FARRANGE[ 3 ], 120.0 )
+lay( FINEB[ 0 ], FINEB[ 1 ], FINEB[ 2 ], FINEB[ 3 ], FINE )
+for patch in DENSEPATCHES:
+    lay( patch[ 0 ], patch[ 1 ], patch[ 2 ], patch[ 3 ], DENSE )
+
+# an embankment side: upright triangles, which are ground as much as anything else
+step, wallheight, wallx = 8.0, 6.0, 160.0
+z = 100.0
+while z < 300.0 - 0.001:
+    z2 = z + step
+    emit( [ ( wallx, height( wallx, z ), z ), ( wallx, height( wallx, z2 ), z2 ),
+            ( wallx, height( wallx, z2 ) + wallheight, z2 ) ], "concrete1" )
+    emit( [ ( wallx, height( wallx, z ), z ), ( wallx, height( wallx, z2 ) + wallheight, z2 ),
+            ( wallx, height( wallx, z ) + wallheight, z ) ], "concrete1" )
+    z += step
+
+out = os.path.join( sys.argv[ 1 ], "scenery", "td", "tinpoligon.scm" )
+open( out, "w" ).write( "\n".join( lines ) + "\n" )
+print( "{}: {} triangles".format( out, sum( 1 for line in lines if line.startswith( "node" ) ) ) )

tools/terraincook/meshtest.cpp

@@ -8,12 +8,14 @@
 #include <cstdio>
 #include <filesystem>
 #include <set>
+#include <cstdlib>
 #include <system_error>
 #include <map>
 #include <tuple>
 #include <set>
 #include <vector>
 
+#include "scene/quantizedmesharchive.h"
 #include "scene/quantizedmeshcooker.h"
 #include "scene/quantizedmeshreader.h"
 
@@ -66,18 +68,19 @@ int main() {
     // one big triangle spanning many tiles, as 3ds Max terrain has
     cooker.add( { -600.0, -5.0, 700.0, 0.f, 0.f }, { 600.0, -5.0, 700.0, 24.f, 0.f }, { 0.0, -5.0, 1100.0, 12.f, 16.f }, "grass" );
 
-    pak::writer writer;
+    qm::archive_writer writer;
-    check( writer.open( archive ), "pak open" );
+    check( writer.open( archive, qm::cook_rules ), "archive opened for writing" );
     qm::terrain_table table;
     std::map<std::int64_t, qm::tile_address> written;
-    auto const sink = [&]( qm::tile_address const &tile, std::vector<std::uint8_t> const &bytes ) {
+    auto const sink = [&]( qm::tile_address const &tile, std::vector<std::uint8_t> const &bytes,
+        qm::tile_measure const &measure ) {
         written.emplace( qm::tile_key( tile ), tile );
-        check( writer.add( qm::tile_name( tile ), bytes ), "pak add" );
+        check( writer.add( tile, bytes, measure ), "tile written" );
     };
     check( cooker.finish( sink, table ), "finish" );
     auto const tabletext = qm::write_table( table );
-    check( writer.add( qm::tablename, tabletext ), "table add" );
+    check( writer.table( tabletext ), "table written" );
-    check( writer.close(), "pak close" );
+    check( writer.close(), "archive closed" );
 
     auto const &report = cooker.result();
     std::printf( "added %zu, laid %zu triangles in %zu tiles, %u levels, %zu vertices, %zu kB, refused %zu\n",
@@ -100,8 +103,23 @@ int main() {
         std::printf( "  level %u error %.3f m, tile %.0f m\n", level,
             reader.table().error( level ), reader.table().tilesize( level ) );
     }
-    check( reader.table().error( reader.table().levels - 1 ) == 0.f, "the finest level is the ground as drawn" );
+    // the finest level is let go of only where it says the same thing twice, so it is off by
-    check( reader.table().error( 0 ) > reader.table().error( reader.table().levels - 2 ), "a coarser level is further off" );
+    // centimetres rather than by nothing
+    check( reader.table().error( reader.table().levels - 1 ) < 0.05f, "the finest level is off by centimetres" );
+    // every level is at least as far off as the one below it, and the coarsest is further off than
+    // the finest - which is what makes the ladder worth walking down
+    for( std::uint32_t level = 0; level + 1 < reader.table().levels; ++level ) {
+        check( reader.table().error( level ) >= reader.table().error( level + 1 ) - 1e-4f,
+            "a coarser level is at least as far off as the one below it" );
+    }
+    check( reader.table().error( 0 ) > reader.table().error( reader.table().levels - 1 ),
+        "the coarsest level is further off than the finest" );
+
+    // A coordinate is quantized to fifteen bits across its tile, so how closely a position comes back
+    // follows the tile's size: a thousand-metre tile holds it to a few centimetres. Everything below
+    // compares against that rather than against a fixed figure.
+    auto const quantum = [ & ]( std::uint32_t level ) { return reader.table().tilesize( level ) / 32767.0; };
+
 
     // every tile reads back, and its vertices lie inside its own square
     // per level and border point, the heights each tile put there. with walls in the mesh one
@@ -121,12 +139,14 @@ int main() {
         auto const north = reader.table().north + tile.z * side;
         for( auto const &v : data.vertices ) {
             double const x = data.originx + v.x, z = data.originz + v.z, y = data.originy + v.y;
-            check( x >= west - 0.01 && x <= west + side + 0.01, "vertex inside its tile in x" );
+            auto const slack = 2.0 * quantum( tile.level );
-            check( z >= north - 0.01 && z <= north + side + 0.01, "vertex inside its tile in z" );
+            check( x >= west - slack && x <= west + side + slack, "vertex inside its tile in x" );
-            check( y >= data.lowest - 0.01 && y <= data.highest + 0.01, "vertex inside its height range" );
+            check( z >= north - slack && z <= north + side + slack, "vertex inside its tile in z" );
+            check( y >= data.lowest - slack && y <= data.highest + slack, "vertex inside its height range" );
             // remember heights on tile borders, to compare with the neighbour later
-            auto const onborder = std::abs( x - west ) < 0.01 || std::abs( x - west - side ) < 0.01
+            auto const close = 2.0 * quantum( tile.level );
-                               || std::abs( z - north ) < 0.01 || std::abs( z - north - side ) < 0.01;
+            auto const onborder = std::abs( x - west ) < close || std::abs( x - west - side ) < close
+                               || std::abs( z - north ) < close || std::abs( z - north - side ) < close;
             if( onborder ) {
                 auto const gx = static_cast<long long>( std::llround( x * 100.0 ) );
                 auto const gz = static_cast<long long>( std::llround( z * 100.0 ) );
@@ -156,8 +176,10 @@ int main() {
     }
     std::printf( "%zu of %zu shared border points carry more than one surface\n", manysurfaced, shared );
     std::printf( "%zu border points shared by two tiles\n", shared );
-    std::printf( "worst height disagreement on a shared border: %.4f mm\n", worst * 1000.0 );
+    std::printf( "worst height disagreement on a shared border: %.1f mm\n", worst * 1000.0 );
-    check( worst < 0.01, "shared border vertices agree within a centimetre" );
+    // heights are quantized against each tile's own range, as the format has it, so two tiles round a
+    // shared vertex a fraction of a quantum apart
+    check( worst < 2.0 * quantum( reader.table().levels - 1 ), "shared border vertices agree to a quantum" );
 
     // the coarse levels must actually be coarser
     if( perlevel.size() >= 2 ) {
@@ -166,7 +188,7 @@ int main() {
         check( next < finest, "a coarser level has fewer triangles" );
     }
 
-    // the wall along the tile border must be there, and exactly once: 8 triangles, not 16
+    // the wall along the tile border must be there, and exactly once
     {
         std::size_t wallpieces = 0;
         std::set<std::pair<long long,long long>> wallcentres;
@@ -178,7 +200,7 @@ int main() {
                 bool wall = true;
                 for( int c = 0; c < 3; ++c ) {
                     auto const &v = data.vertices[ data.indices[ i + c ] ];
-                    if( std::abs( data.originz + v.z + 512.0 ) > 0.01 ) { wall = false; }
+                    if( std::abs( data.originz + v.z + 512.0 ) > 2.0 * quantum( tile.level ) ) { wall = false; }
                 }
                 if( wall ) {
                     ++wallpieces;
@@ -215,7 +237,7 @@ int main() {
     std::printf( "triangles inside the hole: %zu\n", inhole );
     check( inhole == 0, "the hole was not filled in" );
 
-    std::filesystem::remove_all( directory, ignored );
+    if( nullptr == getenv( "KEEP" ) ) { std::filesystem::remove_all( directory, ignored ); }
     std::printf( failures ? "\n%d checks failed\n" : "\nall checks passed\n", failures );
     return failures ? 1 : 0;
 }