eu07/maszyna
17
0
Fork 0
mirror of https://github.com/MaSzyna-EU07/maszyna.git synced 2026-11-01 04:39:07 +01:00
Code Issues 156 Wiki Activity

Give the terrain its own archive, its normals, and a level per tile

Browse Source 
The tiles moved out of the scenery's pak into an archive of their own: a
sorted index of fixed-size entries found by where a tile sits in the
quadtree rather than by a name, each entry compressed on its own so a
tile can be read without touching the rest. A name apiece would have been
seventy megabytes of index to read before the first tile could be drawn.

An entry carries what its tile's ground asks for - how far it departs
from the finest mesh, and how long its triangles are - because which
level to draw has to be settled for tiles that are not on the card yet,
and the index is the only thing about them that is read up front. Until
now the figure came from the tile's level, which is the worst tile in it,
so one steep corner spoke for every tile that shared its level.

The mesh carries a normal per vertex, octahedron-encoded into two bytes,
in place of the face normal the fragment stage worked out from how the
view position changed across a triangle. A face normal makes the ground
read as facets, and a coarser level's facets face differently from the
ones they stand for, so the light over a hillside changed as the level
did. A geometry stage hands the fragment stage the three materials of the
triangle it is in and the barycentric weights, so ground changes over a
triangle rather than along its edge; 3.3 has no gl_BaryCoordEXT.

And the level a tile is drawn at now counts the camera's height. The
distance was measured on the ground plane alone, which puts the tile
underfoot nought metres away from a camera a kilometre above it - and
nought metres asks for the finest level there is, so a view from the air
wanted the whole ground at full detail, more tiles than the buffers hold,
and what would not fit came out as holes.

Archives written under the old rules are refused at the door and baked
again.
This commit is contained in:
maj00r
2026-09-20 21:22:03 +02:00
parent b877bc0772
commit 9408216ae7
21 changed files with 1604 additions and 550 deletions
Download Patch File Download Diff File Expand all files Collapse all files

179
scene/pakformat.h
View File

@@ -1,179 +0,0 @@
/*
This Source Code Form is subject to the
terms of the Mozilla Public License, v.
2.0. If a copy of the MPL was not
distributed with this file, You can
obtain one at
http://mozilla.org/MPL/2.0/.
*/
#pragma once
// One file holding a scenery's cooked terrain tiles.
//
// A terrain is tens of thousands of small files, which is the one thing a filesystem is worst at:
// opening them costs more than reading them. They go into an archive instead, each entry
// compressed on its own so that a tile can be read without touching the rest, and the index is
// written last so that cooking never has to hold the tiles in memory.
#include <cstdint>
#include <cstring>
#include <fstream>
#include <string>
#include <unordered_map>
#include <vector>
#include <zstd.h>
namespace pak {
inline constexpr char magic[ 8 ] { 'E', 'U', '0', '7', 'P', 'A', 'K', '\0' };
inline constexpr std::uint32_t version { 2 };
inline constexpr std::size_t namelength { 64 };
#pragma pack( push, 1 )
struct file_header {
char magic[ 8 ];
std::uint32_t version;
std::uint32_t entrycount;
std::uint64_t indexoffset;
};
struct index_entry {
char name[ namelength ];
std::uint64_t offset;
std::uint64_t stored; // bytes in the file
std::uint64_t plain; // bytes once decompressed; equal to stored when kept as it is
};
#pragma pack( pop )
// Writes entries straight through to disk as they are added.
class writer {
public:
bool open( std::string const &Path ) {
m_file.open( Path, std::ios::binary | std::ios::trunc );
if( false == m_file.good() ) { return false; }
file_header header {};
m_file.write( reinterpret_cast<char const *>( &header ), sizeof( header ) );
m_at = sizeof( header );
return m_file.good();
}
bool add( std::string const &Name, std::vector<std::uint8_t> const &Data ) {
if( ( false == m_file.good() ) || ( Name.size() >= namelength ) ) { return false; }
index_entry entry {};
std::memcpy( entry.name, Name.c_str(), Name.size() );
entry.offset = m_at;
entry.plain = Data.size();
m_packed.resize( ZSTD_compressBound( Data.size() ) );
auto const packed {
ZSTD_compress( m_packed.data(), m_packed.size(), Data.data(), Data.size(), 3 ) };
auto const *bytes { reinterpret_cast<char const *>( Data.data() ) };
auto size { Data.size() };
if( ( 0 == ZSTD_isError( packed ) ) && ( packed < Data.size() ) ) {
bytes = reinterpret_cast<char const *>( m_packed.data() );
size = packed;
}
entry.stored = size;
m_file.write( bytes, static_cast<std::streamsize>( size ) );
m_at += size;
m_index.push_back( entry );
return m_file.good();
}
bool close() {
if( false == m_file.good() ) { return false; }
file_header header {};
std::memcpy( header.magic, magic, sizeof( magic ) );
header.version = version;
header.entrycount = static_cast<std::uint32_t>( m_index.size() );
header.indexoffset = m_at;
for( auto const &entry : m_index ) {
m_file.write( reinterpret_cast<char const *>( &entry ), sizeof( entry ) );
}
m_file.seekp( 0 );
m_file.write( reinterpret_cast<char const *>( &header ), sizeof( header ) );
auto const good { m_file.good() };
m_file.close();
return good;
}
std::size_t entrycount() const { return m_index.size(); }
std::uint64_t bytes() const { return m_at + m_index.size() * sizeof( index_entry ); }
private:
std::ofstream m_file;
std::uint64_t m_at { 0 };
std::vector<index_entry> m_index;
std::vector<std::uint8_t> m_packed;
};
// Reads entries back. One reader is used from one thread at a time: it keeps a handle and seeks.
class reader {
public:
bool open( std::string const &Path ) {
close();
m_file.open( Path, std::ios::binary );
if( false == m_file.good() ) { return false; }
file_header header {};
m_file.read( reinterpret_cast<char *>( &header ), sizeof( header ) );
if( false == m_file.good() ) { return false; }
if( 0 != std::memcmp( header.magic, magic, sizeof( magic ) ) ) { return false; }
if( header.version != version ) { return false; }
m_file.seekg( static_cast<std::streamoff>( header.indexoffset ) );
for( std::uint32_t entry = 0; entry < header.entrycount; ++entry ) {
index_entry record {};
m_file.read( reinterpret_cast<char *>( &record ), sizeof( record ) );
if( false == m_file.good() ) { return false; }
record.name[ namelength - 1 ] = '\0';
m_index.emplace( record.name, record );
}
return true;
}
void close() {
m_index.clear();
if( true == m_file.is_open() ) { m_file.close(); }
m_file.clear();
}
bool exists( std::string const &Name ) const { return m_index.contains( Name ); }
std::size_t entrycount() const { return m_index.size(); }
std::vector<std::string> names() const {
std::vector<std::string> names;
names.reserve( m_index.size() );
for( auto const &[ name, entry ] : m_index ) { names.push_back( name ); }
return names;
}
bool read( std::string const &Name, std::vector<std::uint8_t> &Out ) {
auto const found { m_index.find( Name ) };
if( found == m_index.end() ) { return false; }
auto const &entry { found->second };
m_stored.resize( entry.stored );
m_file.seekg( static_cast<std::streamoff>( entry.offset ) );
m_file.read( reinterpret_cast<char *>( m_stored.data() ), static_cast<std::streamsize>( entry.stored ) );
if( false == m_file.good() ) { m_file.clear(); return false; }
if( entry.stored == entry.plain ) {
Out.swap( m_stored );
return true;
}
Out.resize( entry.plain );
auto const plain {
ZSTD_decompress( Out.data(), Out.size(), m_stored.data(), m_stored.size() ) };
return ( 0 == ZSTD_isError( plain ) ) && ( plain == entry.plain );
}
private:
std::ifstream m_file;
std::unordered_map<std::string, index_entry> m_index;
std::vector<std::uint8_t> m_stored;
};
} // namespace pak

283
scene/quantizedmesharchive.h Normal file
View File

@@ -0,0 +1,283 @@
/*
This Source Code Form is subject to the
terms of the Mozilla Public License, v.
2.0. If a copy of the MPL was not
distributed with this file, You can
obtain one at
http://mozilla.org/MPL/2.0/.
*/
#pragma once
// One file holding a scenery's cooked terrain tiles.
//
// A terrain is hundreds of thousands of small files, which is the one thing a filesystem is worst at:
// opening them costs more than reading them. They go into an archive instead, each entry compressed on
// its own so that a tile can be read without touching the rest, and the index written last so that
// cooking never has to hold the tiles in memory.
//
// A tile is found by where it sits in the quadtree, not by a name. Galicja cooks to three quarters of
// a million tiles: a name of its own for each would be seventy megabytes of index to read and a hash
// table of that many strings to build before the first tile can be drawn. The index here is a sorted
// run of thirty-two byte entries, read in one go and searched in place.
//
// An entry carries what the tile's ground asks for as well as where it lies, because which level to
// draw has to be settled for tiles that are not on the card yet - that is the whole of the question -
// and the index is the only thing about them that is read up front.
#include <algorithm>
#include <cstdint>
#include <cstring>
#include <fstream>
#include <string>
#include <vector>
#include <zstd.h>
#include "scene/quantizedmeshformat.h"
namespace quantizedmesh {
inline constexpr char archive_magic[ 8 ] { 'E', 'U', '0', '7', 'T', 'A', 'R', '\0' };
inline constexpr std::uint32_t archive_version { 2 };
#pragma pack( push, 1 )
struct archive_header {
char magic[ 8 ];
std::uint32_t version;
// the cook's rules, so that an archive this build cannot make sense of is refused at the door
std::uint32_t rules;
std::uint64_t entrycount;
std::uint64_t indexoffset;
// the terrain's table, which every tile needs before it can be placed
std::uint64_t tableoffset;
std::uint32_t tablestored;
std::uint32_t tableplain;
};
struct archive_entry {
std::int64_t key; // tile_key of the tile this holds
std::uint64_t offset;
std::uint32_t stored; // bytes in the file
std::uint32_t plain; // bytes once decompressed; equal to stored when kept as it is
// what this tile's own ground asks for, as the cook measured it
float error;
float edge;
bool operator<( archive_entry const &Other ) const { return key < Other.key; }
};
#pragma pack( pop )
static_assert( sizeof( archive_entry ) == 32 );
namespace detail {
// zstd, or the bytes as they are when compressing them does not pay
inline void
pack( std::vector<std::uint8_t> const &Data, std::vector<std::uint8_t> &Room,
std::uint8_t const *&Bytes, std::size_t &Size ) {
Bytes = Data.data();
Size = Data.size();
if( true == Data.empty() ) { return; }
Room.resize( ZSTD_compressBound( Data.size() ) );
auto const packed { ZSTD_compress( Room.data(), Room.size(), Data.data(), Data.size(), 3 ) };
if( ( 0 == ZSTD_isError( packed ) ) && ( packed < Data.size() ) ) {
Bytes = Room.data();
Size = packed;
}
}
} // namespace detail
// Writes tiles straight through to disk as they are cooked.
class archive_writer {
public:
bool open( std::string const &Path, std::uint32_t const Rules ) {
m_rules = Rules;
m_file.open( Path, std::ios::binary | std::ios::trunc );
if( false == m_file.good() ) { return false; }
archive_header header {};
m_file.write( reinterpret_cast<char const *>( &header ), sizeof( header ) );
m_at = sizeof( header );
return m_file.good();
}
bool add( tile_address const &Tile, std::vector<std::uint8_t> const &Data,
tile_measure const &Measure ) {
if( false == m_file.good() ) { return false; }
std::uint8_t const *bytes { nullptr };
std::size_t size { 0 };
detail::pack( Data, m_room, bytes, size );
archive_entry entry {};
entry.key = tile_key( Tile );
entry.offset = m_at;
entry.stored = static_cast<std::uint32_t>( size );
entry.plain = static_cast<std::uint32_t>( Data.size() );
entry.error = Measure.error;
entry.edge = Measure.edge;
m_file.write( reinterpret_cast<char const *>( bytes ), static_cast<std::streamsize>( size ) );
m_at += size;
m_index.push_back( entry );
return m_file.good();
}
// the terrain's table, written once. it has a place of its own in the header rather than an entry
bool table( std::vector<std::uint8_t> const &Data ) {
if( false == m_file.good() ) { return false; }
std::uint8_t const *bytes { nullptr };
std::size_t size { 0 };
detail::pack( Data, m_room, bytes, size );
m_tableoffset = m_at;
m_tablestored = static_cast<std::uint32_t>( size );
m_tableplain = static_cast<std::uint32_t>( Data.size() );
m_file.write( reinterpret_cast<char const *>( bytes ), static_cast<std::streamsize>( size ) );
m_at += size;
return m_file.good();
}
bool close() {
if( false == m_file.good() ) { return false; }
// in order, so that the index can be searched where it lies and so that cooking the same
// scenery twice writes the same file
std::sort( m_index.begin(), m_index.end() );
archive_header header {};
std::memcpy( header.magic, archive_magic, sizeof( archive_magic ) );
header.version = archive_version;
header.rules = m_rules;
header.entrycount = m_index.size();
header.indexoffset = m_at;
header.tableoffset = m_tableoffset;
header.tablestored = m_tablestored;
header.tableplain = m_tableplain;
for( auto const &entry : m_index ) {
m_file.write( reinterpret_cast<char const *>( &entry ), sizeof( entry ) );
}
m_file.seekp( 0 );
m_file.write( reinterpret_cast<char const *>( &header ), sizeof( header ) );
auto const good { m_file.good() };
m_file.close();
return good;
}
std::size_t tilecount() const { return m_index.size(); }
std::uint64_t bytes() const { return m_at + m_index.size() * sizeof( archive_entry ); }
private:
std::ofstream m_file;
std::uint64_t m_at { 0 };
std::uint32_t m_rules { 0 };
std::uint64_t m_tableoffset { 0 };
std::uint32_t m_tablestored { 0 };
std::uint32_t m_tableplain { 0 };
std::vector<archive_entry> m_index;
std::vector<std::uint8_t> m_room;
};
// Reads tiles back. One reader is used from one thread at a time: it keeps a handle and seeks.
class archive_reader {
public:
bool open( std::string const &Path, std::uint32_t const Rules ) {
close();
m_file.open( Path, std::ios::binary );
if( false == m_file.good() ) { return false; }
archive_header header {};
m_file.read( reinterpret_cast<char *>( &header ), sizeof( header ) );
if( false == m_file.good() ) { return false; }
if( 0 != std::memcmp( header.magic, archive_magic, sizeof( archive_magic ) ) ) { return false; }
if( header.version != archive_version ) { return false; }
if( header.rules != Rules ) { return false; }
if( header.entrycount > ( 1ull << 32 ) ) { return false; }
m_index.resize( static_cast<std::size_t>( header.entrycount ) );
if( false == m_index.empty() ) {
m_file.seekg( static_cast<std::streamoff>( header.indexoffset ) );
m_file.read(
reinterpret_cast<char *>( m_index.data() ),
static_cast<std::streamsize>( m_index.size() * sizeof( archive_entry ) ) );
if( false == m_file.good() ) { close(); return false; }
// written in order, but a file from anywhere is not to be believed about that
if( false == std::is_sorted( m_index.begin(), m_index.end() ) ) {
std::sort( m_index.begin(), m_index.end() );
}
}
m_tableoffset = header.tableoffset;
m_tablestored = header.tablestored;
m_tableplain = header.tableplain;
m_open = true;
return true;
}
void close() {
m_index.clear();
m_index.shrink_to_fit();
if( true == m_file.is_open() ) { m_file.close(); }
m_file.clear();
m_open = false;
}
bool contains( tile_address const &Tile ) const { return nullptr != find( tile_key( Tile ) ); }
// what the tile asks for, or nothing when the archive does not hold it
bool measure( tile_address const &Tile, tile_measure &Out ) const {
auto const *entry { find( tile_key( Tile ) ) };
if( nullptr == entry ) { return false; }
Out.error = entry->error;
Out.edge = entry->edge;
return true;
}
std::size_t tilecount() const { return m_index.size(); }
// the tiles there are, in order, for anything that wants to walk them
std::vector<archive_entry> const &index() const { return m_index; }
bool read_table( std::vector<std::uint8_t> &Out ) {
if( ( false == m_open ) || ( m_tableplain == 0 ) ) { return false; }
return fetch( m_tableoffset, m_tablestored, m_tableplain, Out );
}
bool read_tile( tile_address const &Tile, std::vector<std::uint8_t> &Out ) {
if( false == m_open ) { return false; }
auto const *entry { find( tile_key( Tile ) ) };
if( nullptr == entry ) { return false; }
return fetch( entry->offset, entry->stored, entry->plain, Out );
}
private:
archive_entry const *find( std::int64_t const Key ) const {
archive_entry wanted {};
wanted.key = Key;
auto const found { std::lower_bound( m_index.begin(), m_index.end(), wanted ) };
if( ( found == m_index.end() ) || ( found->key != Key ) ) { return nullptr; }
return &( *found );
}
bool fetch( std::uint64_t const Offset, std::uint32_t const Stored, std::uint32_t const Plain,
std::vector<std::uint8_t> &Out ) {
m_stored.resize( Stored );
m_file.seekg( static_cast<std::streamoff>( Offset ) );
m_file.read( reinterpret_cast<char *>( m_stored.data() ), static_cast<std::streamsize>( Stored ) );
if( false == m_file.good() ) { m_file.clear(); return false; }
if( Stored == Plain ) {
Out = m_stored;
return true;
}
Out.resize( Plain );
auto const plain { ZSTD_decompress( Out.data(), Out.size(), m_stored.data(), m_stored.size() ) };
return ( 0 == ZSTD_isError( plain ) ) && ( plain == Plain );
}
std::ifstream m_file;
std::vector<archive_entry> m_index;
std::vector<std::uint8_t> m_stored;
std::uint64_t m_tableoffset { 0 };
std::uint32_t m_tablestored { 0 };
std::uint32_t m_tableplain { 0 };
bool m_open { false };
};
} // namespace quantizedmesh

343
scene/quantizedmeshcooker.cpp
View File

@@ -36,23 +36,50 @@ constexpr double weldepsilon { 1e-4 };
constexpr std::uint32_t maxlevels { 13 };
// The finest tile is this many typical triangle edges across, and never outside these bounds. A tile
// has to be large against the triangles in it: a triangle crossing a border is cut, and the pieces
// cost vertices neither side had. It also has to stay small enough to be worth loading and culling
// on its own, which is what the bounds are for.
// cost vertices neither side had. It also has to stay small enough to be worth loading and culling on
// its own, which is what the bounds are for.
//
// The typical edge is averaged over the ground weighted by how much of it each triangle covers, not
// over the triangles counted one each. A scenery holds both kinds of ground - a dense patch from a
// survey by the track, and big triangles drawn by hand for the rest - and counting triangles one each
// lets a patch of a few hundred metres decide the tile for forty kilometres. What the cut costs
// depends on how much ground the large triangles cover, which is what this measures.
constexpr double tileedges { 16.0 };
constexpr double smallesttileside { 128.0 };
constexpr double largesttileside { 1024.0 };
// how many edge lengths are kept to find the typical one. the first hundred thousand triangles say
// as much about a scenery as all of them
constexpr std::size_t edgesample { 100000 };
// a tile is never collapsed below this, however far away it is drawn: past a point the triangles
// saved are not worth the ground losing its shape
constexpr std::size_t smallesttile { 64 };
// A level may move the ground by this share of its tile's side, and no further. A level is used from
// about twice its tile's side away, where a share this small is well under a pixel; letting it go
// further would buy triangles that no distance is far enough to use. Ground already so coarse that a
// level cannot be thinned within it simply comes out nearly a copy of the level below, which is the
// right answer: there was nothing there to save.
constexpr double errorshare { 128.0 };
// A level may move the ground by this share of its tile's side, and no further.
//
// The number decides how far away a level has to be before it may be used, and so how long the level
// below it has to be drawn. Too generous and the finest mesh - which is the whole of the scenery's
// ground - is drawn out to several kilometres, and only three or four levels are ever in view at once.
// A third of what it was buys coarse levels faithful enough to be used three times closer, which is
// both fewer triangles in the end and more of the ladder visible at any moment.
constexpr double errorshare { 384.0 };
// How far the finest level may move the ground, in metres. Not a share of the tile like the levels
// above: this is the one that is stood on, so what it may lose is set by what the eye can catch from
// there rather than by how large the tile happens to be. Three centimetres of ground twenty metres
// ahead covers about a pixel and a half.
//
// What it takes off is redundancy rather than detail - a field drawn as a grid a metre across is a
// grid a metre across even where it is flat, and survey data is mostly flat. Nothing else in the
// pyramid can remove it, because every level above is built from this one.
constexpr double finesterror { 0.03 };
// How wide the ground may change over where two materials meet, as a share of the tile's side.
//
// The fragment stage blends the three materials of a triangle by how near the fragment is to each
// corner, so a transition is one triangle wide - and a triangle is exactly what a coarser level makes
// larger. Left alone the ground would fade over four metres near to and over sixty far away, and the
// change would be seen as the level changed.
//
// Tying it to the tile rather than to a fixed number of metres holds it steady where it counts, which
// is on screen: a level is drawn from about as far away as its tiles are wide, so a share of the tile
// is a roughly constant number of pixels. It also lets material detail finer than this merge away at
// the levels where it could not be seen anyway - without that, ground drawn in stripes a few metres
// across can never be simplified at all, because every edge in it crosses a boundary.
constexpr double blendshare { 64.0 };
// the area of a triangle in space, whichever way it faces
double
@@ -111,6 +138,7 @@ sides_of( double const X, double const Z, double const West, double const North,
// a vertex of a polygon being cut along the tile grid
struct clipvertex {
double x, y, z;
double nx, ny, nz;
};
clipvertex
@@ -118,7 +146,10 @@ between( clipvertex const &From, clipvertex const &To, double const Share ) {
return {
From.x + ( To.x - From.x ) * Share,
From.y + ( To.y - From.y ) * Share,
From.z + ( To.z - From.z ) * Share };
From.z + ( To.z - From.z ) * Share,
From.nx + ( To.nx - From.nx ) * Share,
From.ny + ( To.ny - From.ny ) * Share,
From.nz + ( To.nz - From.nz ) * Share };
}
// Sutherland and Hodgman against one side of the tile's square. Axis 0 is x and 1 is z; the
@@ -224,14 +255,19 @@ cooker::add( corner const &A, corner const &B, corner const &C, std::string_view
}
corner const corners[ 3 ] { A, B, C };
if( m_edges.size() < edgesample ) {
{
auto edges { 0.0 };
for( std::size_t index = 0; index < 3; ++index ) {
auto const &from { corners[ index ] };
auto const &to { corners[ ( index + 1 ) % 3 ] };
auto const length {
std::sqrt( ( to.x - from.x ) * ( to.x - from.x ) + ( to.z - from.z ) * ( to.z - from.z ) ) };
if( length > 0.0 ) { m_edges.push_back( length ); }
edges += std::sqrt(
( to.x - from.x ) * ( to.x - from.x ) + ( to.z - from.z ) * ( to.z - from.z ) );
}
auto const plan {
0.5 * std::abs(
( B.x - A.x ) * ( C.z - A.z ) - ( C.x - A.x ) * ( B.z - A.z ) ) };
m_edgesum += plan * edges / 3.0;
m_areasum += plan;
}
spilled record {};
record.material = material_index( Material );
@@ -263,16 +299,13 @@ cooker::add( corner const &A, corner const &B, corner const &C, std::string_view
void
cooker::measure() {
// the middle edge length of the sample, and the tile that suits it, rounded to a power of two so
// that the same scenery always comes out the same way
if( false == m_edges.empty() ) {
auto const middle { m_edges.begin() + m_edges.size() / 2 };
std::nth_element( m_edges.begin(), middle, m_edges.end() );
auto const wanted { std::clamp( tileedges * *middle, smallesttileside, largesttileside ) };
m_finesttile = std::exp2( std::round( std::log2( wanted ) ) );
m_finesttile = std::clamp( m_finesttile, smallesttileside, largesttileside );
m_edges.clear();
m_edges.shrink_to_fit();
// the typical edge and the tile that suits it, rounded to a power of two so that the same scenery
// always comes out the same way
if( m_areasum > 0.0 ) {
auto const typical { m_edgesum / m_areasum };
auto const wanted { std::clamp( tileedges * typical, smallesttileside, largesttileside ) };
m_finesttile = std::clamp(
std::exp2( std::round( std::log2( wanted ) ) ), smallesttileside, largesttileside );
}
m_west = std::floor( m_minx / m_finesttile ) * m_finesttile;
@@ -326,10 +359,13 @@ cooker::finish( sink const &Sink, terrain_table &Table ) {
}
auto const first = [ this ]( double const Value, double const Origin ) {
return static_cast<std::int32_t>( std::floor( ( Value - Origin ) / m_finesttile ) ); };
auto const fromx { std::clamp( first( minx, m_west ), 0, acrossfinest - 1 ) };
auto const tox { std::clamp( first( maxx, m_west ), 0, acrossfinest - 1 ) };
auto const fromz { std::clamp( first( minz, m_north ), 0, acrossfinest - 1 ) };
auto const toz { std::clamp( first( maxz, m_north ), 0, acrossfinest - 1 ) };
// a hair past the triangle either way: one that merely touches a border belongs to the tile
// beyond it as well, and that is what lets two tiles agree on the normal of a vertex they
// share. it contributes nothing to that tile's own triangles - there is nothing of it inside
auto const fromx { std::clamp( first( minx - weldepsilon, m_west ), 0, acrossfinest - 1 ) };
auto const tox { std::clamp( first( maxx + weldepsilon, m_west ), 0, acrossfinest - 1 ) };
auto const fromz { std::clamp( first( minz - weldepsilon, m_north ), 0, acrossfinest - 1 ) };
auto const toz { std::clamp( first( maxz + weldepsilon, m_north ), 0, acrossfinest - 1 ) };
for( auto x = fromx; x <= tox; ++x ) {
for( auto z = fromz; z <= toz; ++z ) {
m_placement.emplace_back(
@@ -340,6 +376,8 @@ cooker::finish( sink const &Sink, terrain_table &Table ) {
std::sort( m_placement.begin(), m_placement.end() );
m_errors.assign( m_levels, 0.0 );
m_levelarea.assign( m_levels, 0.0 );
m_leveltriangles.assign( m_levels, 0 );
cook( { 0, 0, 0 }, Sink );
m_report.levels = m_levels;
@@ -351,6 +389,14 @@ cooker::finish( sink const &Sink, terrain_table &Table ) {
Table.highest = m_highest;
Table.levels = m_levels;
Table.errors.assign( m_errors.begin(), m_errors.end() );
// the side of the square a triangle of this level covers on average: what it is worth on screen
Table.edges.clear();
for( std::uint32_t level = 0; level < m_levels; ++level ) {
auto const triangles { m_leveltriangles[ level ] };
auto const typical {
triangles > 0 ? std::sqrt( 2.0 * m_levelarea[ level ] / static_cast<double>( triangles ) ) : 0.0 };
Table.edges.push_back( static_cast<float>( typical ) );
}
Table.materials.clear();
for( std::size_t index = 0; index < m_materials.size(); ++index ) {
// a material the measurement never caught falls back to something plausible rather than
@@ -375,23 +421,55 @@ cooker::cook( tile_address const &Tile, sink const &Sink ) {
( Tile.level + 1 == m_levels ) ? lay( Tile ) : merge( Tile, Sink ) };
if( true == result.empty() ) { return result; }
if( Tile.level + 1 < m_levels ) {
// The finest level is the ground as the scenery drew it and is left alone. A level above it
// holds what four tiles below it held, and is collapsed back to what one of them held: the
// cost of a level is then a quarter of the level below, and the whole pyramid costs a third
// more than the finest alone. A count fixed in advance cannot do that - a tile that never
// reaches it would be copied rather than simplified - so it only serves as a ceiling.
{
double west { 0.0 }, north { 0.0 }, side { 0.0 };
square( Tile, west, north, side );
auto const merged { result.indices.size() / 3 };
simplify( result, std::clamp( merged / 4, smallesttile, m_tilebudget ), side / errorshare );
if( Tile.level + 1 < m_levels ) {
// A level holds what four tiles below it held, and is collapsed back to what one of them
// held: the cost of a level is then a quarter of the level below, and the whole pyramid
// costs a third more than the finest alone. A count fixed in advance cannot do that - a
// tile that never reaches it would be copied rather than simplified - so it only serves
// as a ceiling.
simplify(
result, std::clamp( merged / 4, smallesttile, m_tilebudget ),
side / errorshare, side / blendshare, side / blendshare );
}
else {
// the finest level keeps its shape to within a fraction of nothing, and is let go of
// only where it says the same thing twice
// Its outline is left exactly where the scenery put it. Ground drawn in separate patches
// meets at T-junctions - a vertex of one lying partway along an edge of the other - and the
// two outlines there are not the same set of vertices, so moving one and not the other opens
// a crack. Far away that is a fraction of a pixel and worth the triangles; on the ground the
// camera stands on it is a line of sky through the field.
simplify( result, smallesttile, finesterror, side / blendshare, 0.0 );
}
}
m_errors[ Tile.level ] = std::max( m_errors[ Tile.level ], result.error );
// the ground this tile covers and how many triangles it spends on it, from which the side of the
// square one of them covers on average - what a triangle of this tile is worth on screen
double area { 0.0 };
std::size_t triangles { 0 };
for( std::size_t index = 0; index + 2 < result.indices.size(); index += 3 ) {
auto const &a { result.vertices[ result.indices[ index + 0 ] ] };
auto const &b { result.vertices[ result.indices[ index + 1 ] ] };
auto const &c { result.vertices[ result.indices[ index + 2 ] ] };
area += 0.5 * std::abs( ( b.x - a.x ) * ( c.z - a.z ) - ( c.x - a.x ) * ( b.z - a.z ) );
++triangles;
}
m_levelarea[ Tile.level ] += area;
m_leveltriangles[ Tile.level ] += triangles;
tile_measure measure {};
measure.error = static_cast<float>( result.error );
measure.edge = static_cast<float>(
triangles > 0 ? std::sqrt( 2.0 * area / static_cast<double>( triangles ) ) : 0.0 );
std::vector<std::uint8_t> bytes;
encode( Tile, result, bytes );
Sink( Tile, bytes );
Sink( Tile, bytes, measure );
++m_report.tiles;
m_report.triangles += result.indices.size() / 3;
@@ -406,9 +484,14 @@ cooker::cook( tile_address const &Tile, sink const &Sink ) {
namespace {
// Welds vertices that land in the same place and belong to the same material. Everything the cut
// produces is either a vertex of the source or a point on a tile border worked out the same way
// from both sides, so the tolerance only has to cover the last bits of the arithmetic.
// Welds vertices that land in the same place, whatever material the triangle bringing them there is
// drawn with. That is what lets one triangle have a different material at each corner, which is what
// the fragment stage blends across: keeping a vertex per material instead would put a hard edge along
// every boundary, on the triangle's own edges. The vertex keeps the material of whichever triangle
// reached it first, and a boundary therefore moves by at most half a triangle.
//
// Everything the cut produces is either a vertex of the source or a point on a tile border worked out
// the same way from both sides, so the tolerance only has to cover the last bits of the arithmetic.
class welder {
public:
@@ -427,7 +510,6 @@ public:
private:
struct key {
std::int64_t x, y, z;
std::uint16_t material;
bool operator==( key const & ) const = default;
};
struct keyhash {
@@ -437,7 +519,6 @@ private:
hash ^= static_cast<std::size_t>( Value );
hash *= 1099511628211ull; };
mix( Key.x ); mix( Key.y ); mix( Key.z );
mix( Key.material );
return hash;
}
};
@@ -445,7 +526,7 @@ private:
static key make_key( cooker::meshvertex const &Vertex ) {
auto const grid = []( double const Value ) {
return static_cast<std::int64_t>( std::llround( Value / weldepsilon ) ); };
return { grid( Vertex.x ), grid( Vertex.y ), grid( Vertex.z ), Vertex.material };
return { grid( Vertex.x ), grid( Vertex.y ), grid( Vertex.z ) };
}
std::vector<cooker::meshvertex> &m_vertices;
@@ -470,19 +551,68 @@ cooker::lay( tile_address const &Tile ) {
std::pair<std::int64_t, std::uint32_t> { wanted, std::numeric_limits<std::uint32_t>::max() } ) };
if( from == to ) { return result; }
welder weld { result.vertices };
std::vector<clipvertex> polygon;
// read once into memory: the triangles are walked twice, first for the normals and then for the
// cutting, and a tile holds few enough of them for that to be cheaper than seeking again
std::vector<spilled> records;
records.reserve( static_cast<std::size_t>( to - from ) );
for( auto entry = from; entry != to; ++entry ) {
m_scratch.seekg( static_cast<std::streamoff>( entry->second ) * sizeof( spilled ) );
spilled record {};
m_scratch.read( reinterpret_cast<char *>( &record ), sizeof( record ) );
if( false == m_scratch.good() ) { m_scratch.clear(); continue; }
records.push_back( record );
}
if( true == records.empty() ) { return result; }
// The normal at a vertex is the area-weighted sum of the faces meeting there, taken from the
// scenery's triangles as they were written rather than from the pieces the cut leaves. Both tiles
// sharing a border read the same triangles around a vertex on it, so both work out the same
// normal and the light runs across the border without a line.
std::unordered_map<std::int64_t, std::array<double, 3>> normals;
auto const place_key = []( double const X, double const Y, double const Z ) {
auto const grid = []( double const Value ) {
return static_cast<std::int64_t>( std::llround( Value / weldepsilon ) ); };
std::int64_t hash { 1469598103934665603ll };
for( auto const value : { grid( X ), grid( Y ), grid( Z ) } ) {
hash ^= value;
hash *= 1099511628211ll;
}
return hash; };
for( auto const &record : records ) {
auto const &corners { record.corners };
double const ax { corners[ 0 ] }, ay { corners[ 1 ] }, az { corners[ 2 ] };
double const bx { corners[ 3 ] }, by { corners[ 4 ] }, bz { corners[ 5 ] };
double const cx { corners[ 6 ] }, cy { corners[ 7 ] }, cz { corners[ 8 ] };
// twice the area times the unit normal, which is the weighting wanted
std::array<double, 3> const face {
( by - ay ) * ( cz - az ) - ( bz - az ) * ( cy - ay ),
( bz - az ) * ( cx - ax ) - ( bx - ax ) * ( cz - az ),
( bx - ax ) * ( cy - ay ) - ( by - ay ) * ( cx - ax ) };
for( std::size_t corner = 0; corner < 3; ++corner ) {
auto &sum {
normals[ place_key(
corners[ corner * 3 + 0 ], corners[ corner * 3 + 1 ], corners[ corner * 3 + 2 ] ) ] };
sum[ 0 ] += face[ 0 ]; sum[ 1 ] += face[ 1 ]; sum[ 2 ] += face[ 2 ];
}
}
// the sum as it stands, not a unit vector: its length is the surface behind it, and that is what
// makes adding two of them an average rather than a guess
auto const normal_at = [ & ]( double const X, double const Y, double const Z ) {
auto const found { normals.find( place_key( X, Y, Z ) ) };
if( found == normals.end() ) { return std::array<double, 3> { 0.0, 1.0, 0.0 }; }
return found->second; };
welder weld { result.vertices };
std::vector<clipvertex> polygon;
for( auto const &record : records ) {
polygon.clear();
for( std::size_t at = 0; at < 3; ++at ) {
polygon.push_back(
{ record.corners[ at * 3 + 0 ], record.corners[ at * 3 + 1 ], record.corners[ at * 3 + 2 ] } );
auto const x { record.corners[ at * 3 + 0 ] };
auto const y { record.corners[ at * 3 + 1 ] };
auto const z { record.corners[ at * 3 + 2 ] };
auto const normal { normal_at( x, y, z ) };
polygon.push_back( { x, y, z, normal[ 0 ], normal[ 1 ], normal[ 2 ] } );
}
clip_side( polygon, 0, west, true );
clip_side( polygon, 0, west + side, false );
@@ -491,8 +621,8 @@ cooker::lay( tile_address const &Tile ) {
if( polygon.size() < 3 ) { continue; }
// A wall lying along a tile border is inside both tiles: the cut keeps it whole on either
// side, since it has nothing to cut. Whichever border it sits on, the tile past it is the
// one that keeps it, so that it is laid exactly once.
// side, since it has nothing to cut. Whichever border it sits on, the tile past it is the one
// that keeps it, so that it is laid exactly once.
{
auto onborderonly { false };
for( int axis = 0; axis < 2; ++axis ) {
@@ -511,8 +641,8 @@ cooker::lay( tile_address const &Tile ) {
}
auto const place = [ & ]( clipvertex const &Vertex ) {
return weld( { Vertex.x, Vertex.y, Vertex.z, record.material,
sides_of( Vertex.x, Vertex.z, west, north, side ) } ); };
return weld( { Vertex.x, Vertex.y, Vertex.z, Vertex.nx, Vertex.ny, Vertex.nz,
record.material, sides_of( Vertex.x, Vertex.z, west, north, side ) } ); };
auto const first { place( polygon[ 0 ] ) };
for( std::size_t corner = 1; corner + 1 < polygon.size(); ++corner ) {
@@ -520,6 +650,16 @@ cooker::lay( tile_address const &Tile ) {
auto const second { place( polygon[ corner ] ) };
auto const third { place( polygon[ corner + 1 ] ) };
if( ( first == second ) || ( second == third ) || ( third == first ) ) { continue; }
// a triangle that only touched this tile leaves nothing of itself inside it
auto const &a { result.vertices[ first ] };
auto const &b { result.vertices[ second ] };
auto const &c { result.vertices[ third ] };
auto const twicearea {
std::sqrt(
std::pow( ( b.y - a.y ) * ( c.z - a.z ) - ( b.z - a.z ) * ( c.y - a.y ), 2.0 )
+ std::pow( ( b.z - a.z ) * ( c.x - a.x ) - ( b.x - a.x ) * ( c.z - a.z ), 2.0 )
+ std::pow( ( b.x - a.x ) * ( c.y - a.y ) - ( b.y - a.y ) * ( c.x - a.x ), 2.0 ) ) };
if( 0.5 * twicearea < nullarea ) { continue; }
result.indices.push_back( first );
result.indices.push_back( second );
result.indices.push_back( third );
@@ -565,7 +705,9 @@ cooker::merge( tile_address const &Tile, sink const &Sink ) {
// tile shares it, simply is never the one that goes. That is the whole of the crack handling:
// levels meet because the border is literally the same set of vertices in both.
void
cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
cooker::simplify(
mesh &Mesh, std::size_t const Budget, double const Allowed,
double const Boundaryreach, double const Outlinereach ) {
auto triangles { Mesh.indices.size() / 3 };
if( triangles <= Budget ) { return; }
@@ -624,12 +766,14 @@ cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
}
}
// Where the ground ends - the outline of a hole the scenery left, or the edge of the terrain -
// only one triangle meets the edge. Nothing in the quadrics above holds such an edge in
// place, so a plane standing on it, weighted heavily, is added to both its ends: the outline
// then survives into the coarse levels instead of being eaten away.
// Where the ground ends - the outline of a hole the scenery left, the edge of the terrain, or the
// seam between two patches drawn at different densities - only one triangle meets the edge. Those
// edges are what the rules below protect, and a plane standing on each of them, weighted heavily,
// is added to both its ends so that the outline is expensive to disturb as well as forbidden to
// break.
std::map<std::pair<std::uint32_t, std::uint32_t>, std::size_t> uses;
std::vector<bool> onmeshedge( vertices.size(), false );
{
std::map<std::pair<std::uint32_t, std::uint32_t>, std::size_t> uses;
for( std::size_t triangle = 0; triangle < triangles; ++triangle ) {
for( std::size_t corner = 0; corner < 3; ++corner ) {
auto const from { cornerof( triangle, corner ) };
@@ -668,8 +812,14 @@ cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
quadrics[ to ].add_plane( px, py, pz, offset, weight );
}
}
for( auto const &[ edge, count ] : uses ) {
if( count != 1 ) { continue; }
onmeshedge[ edge.first ] = true;
onmeshedge[ edge.second ] = true;
}
}
struct candidate {
double cost;
std::uint32_t from, to;
@@ -720,13 +870,47 @@ cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
// nothing in the plan at all. what changes there is the height where the vertex went
return std::abs( was[ 1 ] - local[ To ][ 1 ] ); };
// vertices with a neighbour of another material: the ground changes colour around them, and how
// large the triangles there are is how wide that change looks
std::vector<bool> onboundary( vertices.size(), false );
for( std::size_t triangle = 0; triangle < triangles; ++triangle ) {
for( std::size_t corner = 0; corner < 3; ++corner ) {
auto const here { cornerof( triangle, corner ) };
auto const next { cornerof( triangle, ( corner + 1 ) % 3 ) };
if( vertices[ here ].material != vertices[ next ].material ) {
onboundary[ here ] = true;
onboundary[ next ] = true;
}
}
}
auto const meshedge = [ &uses ]( std::uint32_t const From, std::uint32_t const To ) {
auto const found { uses.find( { std::min( From, To ), std::max( From, To ) } ) };
return ( found != uses.end() ) && ( found->second == 1 ); };
auto const allowed = [ & ]( std::uint32_t const From, std::uint32_t const To ) {
if( ( true == removed[ From ] ) || ( true == removed[ To ] ) ) { return false; }
// a vertex a neighbouring tile shares stays where it is
if( true == vertices[ From ].locked() ) { return false; }
// and a material boundary stays where it is too, or the ground would change colour by
// level
if( vertices[ From ].material != vertices[ To ].material ) { return false; }
// Where the ground ends, a vertex may only travel along that end, onto another vertex of it, and
// no further than this level reaches. What is lost then is the thin wedge between the old
// outline and the new one. Letting it collapse inward instead takes whole triangles with it and
// leaves holes the size of them - and since the height barely changes there, nothing in the
// error measure objects: forty per cent of the ground went that way before this rule. Forbidding
// it outright is no good either, since a narrow strip of terrain is almost all outline, and then
// no level above the finest can be thinned at all
if( true == onmeshedge[ From ] ) {
if( Outlinereach <= 0.0 ) { return false; }
if( false == meshedge( From, To ) ) { return false; }
auto const &from { local[ From ] };
auto const &to { local[ To ] };
auto const reach {
std::sqrt( ( from[ 0 ] - to[ 0 ] ) * ( from[ 0 ] - to[ 0 ] )
+ ( from[ 1 ] - to[ 1 ] ) * ( from[ 1 ] - to[ 1 ] )
+ ( from[ 2 ] - to[ 2 ] ) * ( from[ 2 ] - to[ 2 ] ) ) };
if( reach > Outlinereach ) { return false; }
}
// nothing that turns a triangle over or squashes it flat
for( auto const triangle : incident[ From ] ) {
if( true == gone[ triangle ] ) { continue; }
@@ -738,7 +922,10 @@ cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
// that can be one is between To and the third corner
for( auto const third : corners ) {
if( ( third == From ) || ( third == To ) ) { continue; }
// a piece of a tile's border, or of the ground's own edge: either way this triangle
// is the only thing holding it, and it is about to go
if( ( vertices[ To ].sides & vertices[ third ].sides ) != 0 ) { return false; }
if( true == meshedge( To, third ) ) { return false; }
}
continue;
}
@@ -765,6 +952,20 @@ cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
}
// and nothing that moves the ground further than this level is allowed to
if( drop_of( From, To ) > Allowed ) { return false; }
// Anything touching a boundary between materials - crossing it, or standing next to it - may
// only reach so far. Short of that the boundary is left where the scenery drew it and the
// change stays narrow; past it there is nothing worth defending, since at the distance this
// level is drawn from the whole thing is a pixel or two wide.
if( ( true == onboundary[ From ] ) || ( true == onboundary[ To ] )
|| ( vertices[ From ].material != vertices[ To ].material ) ) {
auto const &from { local[ From ] };
auto const &to { local[ To ] };
auto const reach {
std::sqrt( ( from[ 0 ] - to[ 0 ] ) * ( from[ 0 ] - to[ 0 ] )
+ ( from[ 1 ] - to[ 1 ] ) * ( from[ 1 ] - to[ 1 ] )
+ ( from[ 2 ] - to[ 2 ] ) * ( from[ 2 ] - to[ 2 ] ) ) };
if( reach > Boundaryreach ) { return false; }
}
return true; };
auto const offer = [ & ]( std::uint32_t const Vertex ) {
@@ -797,6 +998,11 @@ cooker::simplify( mesh &Mesh, std::size_t const Budget, double const Allowed ) {
removed[ best.from ] = true;
quadrics[ best.to ] += quadrics[ best.from ];
// the survivor now stands for both, and its normal has to say so, or a coarse level ends up
// lit by whichever slope happened to keep its vertex
vertices[ best.to ].nx += vertices[ best.from ].nx;
vertices[ best.to ].ny += vertices[ best.from ].ny;
vertices[ best.to ].nz += vertices[ best.from ].nz;
for( auto const triangle : incident[ best.from ] ) {
if( true == gone[ triangle ] ) { continue; }
@@ -923,7 +1129,7 @@ cooker::encode( tile_address const &Tile, mesh const &Mesh, std::vector<std::uin
header.horizon_x = header.horizon_y = header.horizon_z = 0.0;
Out.clear();
Out.reserve( order.size() * 8 + indices.size() * 2 + sizeof( header ) + 64 );
Out.reserve( order.size() * 10 + indices.size() * 2 + sizeof( header ) + 64 );
append( Out, header );
append( Out, static_cast<std::uint32_t>( order.size() ) );
@@ -970,6 +1176,17 @@ cooker::encode( tile_address const &Tile, mesh const &Mesh, std::vector<std::uin
else { append_edge<std::uint16_t>( Out, edge ); }
}
// the format's own: a normal per vertex, two bytes each
append( Out, static_cast<std::uint8_t>( ext_normals ) );
append( Out, static_cast<std::uint32_t>( order.size() * 2 ) );
for( auto const index : order ) {
auto const &vertex { Mesh.vertices[ index ] };
std::uint8_t first { 128 }, second { 128 };
encode_normal( vertex.nx, vertex.ny, vertex.nz, first, second );
append( Out, first );
append( Out, second );
}
// ours: which ground material each vertex belongs to. the format has nowhere for it
append( Out, static_cast<std::uint8_t>( ext_ground ) );
append( Out, static_cast<std::uint32_t>( order.size() * sizeof( std::uint16_t ) ) );

27
scene/quantizedmeshcooker.h
View File

@@ -46,8 +46,9 @@ public:
float u, v;
};
// one finished tile, handed over as the bytes of its file
using sink = std::function<void( tile_address const &, std::vector<std::uint8_t> const & )>;
// one finished tile, handed over as the bytes of its file and what its ground asks for
using sink = std::function<
void( tile_address const &, std::vector<std::uint8_t> const &, tile_measure const & )>;
cooker() = default;
~cooker();
@@ -91,6 +92,12 @@ public:
// a vertex of a tile's mesh while it is being worked on, in metres
struct meshvertex {
double x, y, z;
// The ground's normal here, taken from the scenery's own triangles before they were cut, and
// kept unnormalised: what is stored is the sum of the faces meeting here weighted by their
// area, so the length carries the weight. When a vertex is collapsed away its sum is added to
// the survivor's, which makes a coarse level's normals the average of the fine surface they
// stand for rather than a sample of it. It is normalised on the way out.
double nx { 0.0 }, ny { 1.0 }, nz { 0.0 };
std::uint16_t material;
// which of the tile's four sides it sits on, one bit each. a vertex on a side is shared
// with the tile beyond it, so it survives into every level; and an edge whose two ends
@@ -130,8 +137,10 @@ private:
// the four tiles below this one, welded into one mesh
mesh merge( tile_address const &Tile, sink const &Sink );
// collapses edges until the mesh fits the budget, or until the ground would move further than
// Allowed, whichever comes first. locked vertices are left where they are
void simplify( mesh &Mesh, std::size_t Budget, double Allowed );
// Allowed, whichever comes first. locked vertices are left where they are; a collapse touching a
// boundary between materials may not reach further than Boundaryreach; and one that moves the edge
// of the ground itself may not reach further than Outlinereach, which at zero forbids it outright
void simplify( mesh &Mesh, std::size_t Budget, double Allowed, double Boundaryreach, double Outlinereach );
// the bytes of a tile's file
void encode( tile_address const &Tile, mesh const &Mesh, std::vector<std::uint8_t> &Out ) const;
// where a tile's square lies
@@ -144,8 +153,10 @@ private:
double m_finesttile { 256.0 };
std::size_t m_tilebudget { 8192 };
// a sample of how long a triangle's edges are in the plan, which decides the finest tile
std::vector<double> m_edges;
// how long a triangle's edges are in the plan, averaged over the ground and weighted by how much
// of it each triangle covers. that is what decides the finest tile
double m_edgesum { 0.0 };
double m_areasum { 0.0 };
std::vector<std::string> m_materials;
std::unordered_map<std::string, std::uint16_t> m_materiallookup;
@@ -168,8 +179,10 @@ private:
std::vector<std::pair<std::int64_t, std::uint32_t>> m_placement;
std::vector<std::uint32_t> m_refusals;
// the worst error any tile of each level came out with
// the worst error any tile of each level came out with, and how much ground its triangles cover
std::vector<double> m_errors;
std::vector<double> m_levelarea;
std::vector<std::size_t> m_leveltriangles;
report m_report;
};

53
scene/quantizedmeshformat.h
View File

@@ -40,6 +40,7 @@ http://mozilla.org/MPL/2.0/.
// the gzip the format expects from an HTTP layer. The bytes of a tile are the format's.
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <cstring>
#include <string>
@@ -52,7 +53,17 @@ inline constexpr std::uint16_t quantum_max { 32767 };
// what the cook counted as ground, and how it cut and simplified it. tiles cooked under other
// rules are baked again rather than read
inline constexpr std::uint32_t cook_rules { 2 };
inline constexpr std::uint32_t cook_rules { 3 };
// What one tile's ground asks for, measured when it was cooked: how far it departs from the finest
// mesh, and how long its triangles are. Which level to draw follows from these two, and they belong
// to the tile rather than to its level. The level's figure is the worst tile in it, so a single
// steep corner made the whole pyramid ask to be drawn from as far away as a mountain - and flat
// ground twenty kilometres out was split as finely as a valley wall underfoot.
struct tile_measure {
float error { 0.f };
float edge { 0.f };
};
#pragma pack( push, 1 )
@@ -71,6 +82,7 @@ static_assert( sizeof( file_header ) == 88 );
// extension ids. 1, 2 and 4 are taken by the format for normals, the water mask and metadata
enum extension_id : std::uint8_t {
// the format's own: one normal per vertex, octahedron-encoded into two bytes
ext_normals = 1,
ext_watermask = 2,
ext_metadata = 4,
@@ -89,6 +101,34 @@ zigzag_decode( std::uint16_t const Stored ) {
return ( Stored >> 1 ) ^ -static_cast<std::int32_t>( Stored & 1 );
}
// A unit vector into two bytes, as the format's normals extension has it: the sphere folded onto an
// octahedron, unfolded into a square, and the square's coordinates stored as unsigned bytes. Two bytes
// hold a normal to well under a degree, which is what the lighting needs and no more.
//
// Normals are stored rather than worked out while drawing. A normal taken from how the position
// changes across a triangle is that triangle's own, so the ground reads as facets, and - worse - a
// coarser level's facets face differently from the ones they stand for, so the light over a hillside
// changes as the level does. A normal carried on the vertex comes from the ground as the scenery drew
// it and stays the same at every level.
inline void
encode_normal( double const X, double const Y, double const Z, std::uint8_t &Out0, std::uint8_t &Out1 ) {
auto const length { std::abs( X ) + std::abs( Y ) + std::abs( Z ) };
if( length < 1e-12 ) { Out0 = Out1 = 128; return; }
auto first { X / length };
auto second { Z / length };
if( Y < 0.0 ) {
auto const was { first };
first = ( 1.0 - std::abs( second ) ) * ( was >= 0.0 ? 1.0 : -1.0 );
second = ( 1.0 - std::abs( was ) ) * ( second >= 0.0 ? 1.0 : -1.0 );
}
auto const tobyte = []( double const Value ) {
return static_cast<std::uint8_t>(
std::clamp( std::round( ( Value + 1.0 ) * 0.5 * 255.0 ), 0.0, 255.0 ) ); };
Out0 = tobyte( first );
Out1 = tobyte( second );
}
// a quantized coordinate back to metres, against the range it was quantized in
inline double
dequantize( std::uint16_t const Stored, double const Low, double const Span ) {
@@ -175,9 +215,13 @@ struct terrain_table {
double lowest { 0.0 }, highest { 0.0 };
// levels cooked, root included: addresses run from 0 to levels - 1
std::uint32_t levels { 1 };
// metres the ground of a tile at each level departs from the finest one. what decides which
// level to draw: a level is good enough while its error covers few enough pixels
// metres the ground of a tile at each level departs from the finest one, and how long a triangle
// of it typically is. Between them they decide which level to draw. The error alone cannot: it is
// the worst departure anywhere in the tile and is inherited by every level above, so one steep
// corner sets it for the whole pyramid and the levels become indistinguishable. What actually
// tells a level of sixty triangles from one of fifteen thousand is how large the triangles are.
std::vector<float> errors;
std::vector<float> edges;
// one ground texture: its name, and how many metres of ground one repeat of it covers, as
// measured from the texture coordinates the scenery drew it with
struct material {
@@ -194,6 +238,9 @@ struct terrain_table {
float error( std::uint32_t const Level ) const {
return Level < errors.size() ? errors[ Level ] : 0.f;
}
float edge( std::uint32_t const Level ) const {
return Level < edges.size() ? edges[ Level ] : 0.f;
}
};
} // namespace quantizedmesh

147
scene/quantizedmeshreader.h
View File

@@ -23,22 +23,22 @@ http://mozilla.org/MPL/2.0/.
#include <filesystem>
#include <fstream>
#include <string>
#include <unordered_set>
#include <vector>
#include "scene/pakformat.h"
#include "scene/quantizedmesharchive.h"
#include "scene/quantizedmeshformat.h"
namespace quantizedmesh {
// one vertex as the renderer wants it: measured from the terrain's origin, so that the whole
// terrain fits in floats and every tile of it can go into one vertex buffer. there are no texture
// coordinates - the ground tiles in world space, so the shader works them out from the position
// coordinates - the ground tiles in world space, so the shader works them out from the position -
// and the normal stays octahedron-encoded, which the shader unfolds. sixteen bytes either way
#pragma pack( push, 1 )
struct render_vertex {
float x, y, z;
std::uint16_t material;
std::uint16_t padding;
std::uint8_t normal0, normal1;
};
#pragma pack( pop )
@@ -76,20 +76,26 @@ public:
// that every tile can go into the same vertex buffer and be drawn in one call
bool read_tile( tile_address const &Tile, tile_data &Out,
double Originx, double Originy, double Originz );
// the tiles the source holds, by tile_key. a tile the cook found nothing for is not there
std::unordered_set<std::int64_t> const &present() const { return m_present; }
bool contains( tile_address const &Tile ) const { return m_present.contains( tile_key( Tile ) ); }
// whether the source holds this tile. a tile the cook found no ground for is not there, and most
// of a quadtree is like that, so this is the commonest question the renderer asks
bool contains( tile_address const &Tile ) const;
// What this tile's ground asks for. An archive carries it per tile; a directory of loose tiles
// has nowhere to keep it, so there the level's figure stands in, which is what the whole terrain
// used before the index carried anything better
tile_measure measure( tile_address const &Tile ) const;
std::size_t tilecount() const;
private:
bool entry( std::string const &Name, std::vector<std::uint8_t> &Out );
bool loose_file( std::string const &Name, std::vector<std::uint8_t> &Out ) const;
bool read_table();
void find_tiles();
void find_loose_tiles();
std::string m_path;
bool m_archive { false };
pak::reader m_pak;
bool m_isarchive { false };
archive_reader m_archive;
terrain_table m_table;
std::unordered_set<std::int64_t> m_present;
// only for a directory of loose tiles; an archive is its own index
std::vector<std::int64_t> m_loose;
bool m_ready { false };
std::vector<std::uint8_t> m_buffer;
};
@@ -103,18 +109,18 @@ reader::open( std::string const &Path ) {
m_path = Path;
std::error_code error;
if( true == std::filesystem::is_directory( Path, error ) ) {
m_archive = false;
m_isarchive = false;
}
else if( true == std::filesystem::is_regular_file( Path, error ) ) {
m_archive = true;
if( false == m_pak.open( Path ) ) { close(); return false; }
m_isarchive = true;
if( false == m_archive.open( Path, cook_rules ) ) { close(); return false; }
}
else {
close();
return false;
}
if( false == read_table() ) { close(); return false; }
find_tiles();
if( false == m_isarchive ) { find_loose_tiles(); }
m_ready = true;
return true;
}
@@ -122,18 +128,38 @@ reader::open( std::string const &Path ) {
inline void
reader::close() {
m_pak.close();
m_archive.close();
m_table = {};
m_present.clear();
m_loose.clear();
m_loose.shrink_to_fit();
m_path.clear();
m_archive = false;
m_isarchive = false;
m_ready = false;
}
inline bool
reader::entry( std::string const &Name, std::vector<std::uint8_t> &Out ) {
reader::contains( tile_address const &Tile ) const {
if( true == m_archive ) { return m_pak.read( Name, Out ); }
if( true == m_isarchive ) { return m_archive.contains( Tile ); }
return std::binary_search( m_loose.begin(), m_loose.end(), tile_key( Tile ) );
}
inline tile_measure
reader::measure( tile_address const &Tile ) const {
tile_measure measure { m_table.error( Tile.level ), m_table.edge( Tile.level ) };
if( true == m_isarchive ) { m_archive.measure( Tile, measure ); }
return measure;
}
inline std::size_t
reader::tilecount() const {
return m_isarchive ? m_archive.tilecount() : m_loose.size();
}
inline bool
reader::loose_file( std::string const &Name, std::vector<std::uint8_t> &Out ) const {
std::ifstream file( std::filesystem::path( m_path ) / Name, std::ios::binary );
if( false == file.good() ) { return false; }
@@ -147,13 +173,37 @@ reader::entry( std::string const &Name, std::vector<std::uint8_t> &Out ) {
return static_cast<std::size_t>( file.gcount() ) == size;
}
// Which tiles a directory holds is worked out once. Asking the filesystem for a tile that is not there
// is the commonest thing the renderer does - most of a quadtree is empty - and it must not cost a
// system call.
inline void
reader::find_loose_tiles() {
std::error_code error;
auto const base { std::filesystem::path( m_path ) };
for( auto const &found : std::filesystem::recursive_directory_iterator( base, error ) ) {
if( false == found.is_regular_file( error ) ) { continue; }
auto const relative { std::filesystem::relative( found.path(), base, error ) };
if( error ) { continue; }
auto const name { relative.generic_string() };
if( false == name.ends_with( ".terrain" ) ) { continue; }
tile_address tile {};
if( 3 != std::sscanf( name.c_str(), "%u/%d/%d.terrain", &tile.level, &tile.x, &tile.z ) ) { continue; }
if( name != tile_name( tile ) ) { continue; }
m_loose.push_back( tile_key( tile ) );
}
std::sort( m_loose.begin(), m_loose.end() );
}
// the table is text: a terrain outlives the build that cooked it, and a line of it should be
// readable without a tool
inline bool
reader::read_table() {
std::vector<std::uint8_t> bytes;
if( false == entry( tablename, bytes ) ) { return false; }
auto const got {
m_isarchive ? m_archive.read_table( bytes ) : loose_file( tablename, bytes ) };
if( false == got ) { return false; }
std::string const text( bytes.begin(), bytes.end() );
auto found { false };
@@ -190,6 +240,13 @@ reader::read_table() {
{ rest.substr( 0, gap ), static_cast<float>( std::strtod( rest.c_str() + gap + 1, nullptr ) ) } );
}
}
else if( what == "edge" ) {
char const *cursor { rest.c_str() };
auto const level { std::strtoul( cursor, const_cast<char **>( &cursor ), 10 ) };
auto const metres { std::strtod( cursor, const_cast<char **>( &cursor ) ) };
if( m_table.edges.size() <= level ) { m_table.edges.resize( level + 1, 0.f ); }
m_table.edges[ level ] = static_cast<float>( metres );
}
else if( what == "error" ) {
char const *cursor { rest.c_str() };
auto const level { std::strtoul( cursor, const_cast<char **>( &cursor ), 10 ) };
@@ -204,40 +261,15 @@ reader::read_table() {
&& ( m_table.levels >= 1 );
}
// Which tiles there are is worked out once. Asking the filesystem, or even the archive's index,
// for a tile that is not there is the commonest thing the renderer does - most of a quadtree is
// empty - and it must not cost anything.
inline void
reader::find_tiles() {
auto const remember = [ this ]( std::string const &Name ) {
if( false == Name.ends_with( ".terrain" ) ) { return; }
tile_address tile {};
if( 3 != std::sscanf( Name.c_str(), "%u/%d/%d.terrain", &tile.level, &tile.x, &tile.z ) ) { return; }
if( Name != tile_name( tile ) ) { return; }
m_present.insert( tile_key( tile ) ); };
if( true == m_archive ) {
for( auto const &name : m_pak.names() ) { remember( name ); }
return;
}
std::error_code error;
auto const base { std::filesystem::path( m_path ) };
for( auto const &found : std::filesystem::recursive_directory_iterator( base, error ) ) {
if( false == found.is_regular_file( error ) ) { continue; }
auto const relative { std::filesystem::relative( found.path(), base, error ) };
if( error ) { continue; }
remember( relative.generic_string() );
}
}
inline bool
reader::read_tile( tile_address const &Tile, tile_data &Out,
double const Originx, double const Originy, double const Originz ) {
if( false == m_ready ) { return false; }
if( Tile.level >= m_table.levels ) { return false; }
if( false == entry( tile_name( Tile ), m_buffer ) ) { return false; }
auto const got {
m_isarchive ? m_archive.read_tile( Tile, m_buffer ) : loose_file( tile_name( Tile ), m_buffer ) };
if( false == got ) { return false; }
auto const *cursor { m_buffer.data() };
auto const *end { cursor + m_buffer.size() };
@@ -307,8 +339,9 @@ reader::read_tile( tile_address const &Tile, tile_data &Out,
cursor += count * indexsize;
}
// which material each vertex belongs to, from the extension the cook writes
// the normals the format carries, and which material each vertex belongs to, which is ours
std::vector<std::uint16_t> ground;
std::vector<std::uint8_t> normals;
while( cursor < end ) {
std::uint8_t id { 0 };
std::uint32_t length { 0 };
@@ -319,6 +352,10 @@ reader::read_tile( tile_address const &Tile, tile_data &Out,
ground.resize( vertexcount );
std::memcpy( ground.data(), cursor, length );
}
else if( ( id == ext_normals ) && ( length == vertexcount * 2u ) ) {
normals.resize( length );
std::memcpy( normals.data(), cursor, length );
}
cursor += length;
}
if( ground.size() != vertexcount ) { return false; }
@@ -346,7 +383,10 @@ reader::read_tile( tile_address const &Tile, tile_data &Out,
vertex.y = static_cast<float>( dequantize( quantized[ 2 ][ index ], header.lowest, heightspan ) - Out.originy );
vertex.z = static_cast<float>( dequantize( quantized[ 1 ][ index ], north, side ) - Out.originz );
vertex.material = ground[ index ];
vertex.padding = 0;
// a tile without the normals extension - one made by another tool - is lit as if the ground
// were level, which is wrong but is not a reason to refuse it
vertex.normal0 = ( normals.size() == vertexcount * 2u ) ? normals[ index * 2 + 0 ] : 128;
vertex.normal1 = ( normals.size() == vertexcount * 2u ) ? normals[ index * 2 + 1 ] : 128;
}
return true;
}
@@ -368,6 +408,9 @@ write_table( terrain_table const &Table ) {
for( std::size_t level = 0; level < Table.errors.size(); ++level ) {
text += "error " + std::to_string( level ) + " " + number( Table.errors[ level ] ) + "\n";
}
for( std::size_t level = 0; level < Table.edges.size(); ++level ) {
text += "edge " + std::to_string( level ) + " " + number( Table.edges[ level ] ) + "\n";
}
for( auto const &material : Table.materials ) {
text += "material " + material.name + " " + number( material.repeat ) + "\n";
}