eu07/maszyna
17
0
Fork 0
mirror of https://github.com/MaSzyna-EU07/maszyna.git synced 2026-11-03 07:20:22 +01:00
Code Issues 157 Wiki Activity

Stream the scenery: a baked terrain heightfield, a logic container, and prefabs

Browse Source 
Today's work on the binary scene format plan (binary-scene-format-plan.md), from the
first measurement to a terrain that bakes itself on first run and streams from a
thread. The step-by-step history is kept on backup/scene-streaming-format-presquash-0916.

Measuring first. -loadprofile times a scenery load by stage and by node kind and reports
node counts and resident memory. On l204 terrain is 95% of the bytes and 99.8% of the
nodes; on quarkmce2007 vehicles, not models, dominate what is left. The terrain turned
out to be a heightfield in all but name, a ribbon covering 12% of its bounding box.

The heightfield. scene/heightfieldformat.h is the .ehf layout: Morton-ordered tiles
of 128 samples with one shared edge sample, a point-sampled mip chain, zstd per tile,
a nodata sentinel, a material palette with a measured tiling per material, and a stamp
of the rules that decided what counted as terrain. scene/heightfieldreader.h reads it
for the tool, the renderer and the loader alike, and scene/terraincooker.h writes it,
shared by the terraincook tool and the engine so a first-run bake is the file the tool
would have made. The cooker rasterises every triangle where it is drawn (through
scene::place_in_world, the placement the region applies too), keeps an upper surface
laid within half a metre over another and the lower of two further apart, and then
asks each triangle whether the heightfield shows it: laid, reaching a sample, keeping
nearly all it claimed, and with the ground around it drawn at every level out to 2 km.

Baking on first run. simulation/terrainbake bakes <scenario>.ehf the first time a
scenery is opened, choosing the grid step from the first 200,000 vertices, and on later
loads leaves out the triangles its node table says the heightfield shows. Everything
else - trees, grass tufts, gantries, bridge decks - is drawn as written, so a bad bake
can at worst draw something twice, and the heightfield sits 2 cm under the triangles so
the original wins there. Nodes are keyed by file, line, include parameters and
placement; an edited file changes its keys and those nodes are drawn again. A bake under
older rules, or without its table, is thrown away and redone. A launcher copy named
$scenario shares the scenario's files. -nobaketerrain shows the triangles as written.
A scenery can still name hand-cooked fields, several at different steps.

Drawing and streaming. rendering/terrainclipmap draws resident tiles at the mip level
their distance earns, deriving the grid from gl_VertexID and texturing in world space
with the scenery's own materials. rendering/terraintileloader reads and decodes tiles on
a thread of its own; the renderer rescans what is in range only after moving a quarter
of a tile, asks for what is missing nearest first, uploads at most 32 tiles a frame,
drops arrivals the camera has moved past, and keeps drawing a tile at the level it has
until the right one arrives. The terrain has its own range, terrainrange in the ini and
a slider in the debug panel, 20 km by default and independent of basedrawrange; the far
plane of an ordinary view is pushed out to it. The debug panel shows the range, tiles in
view, on the gpu, wanted and queued, and how many sit at each level.

Events. The event queue moved out of the events into a binary heap with a deterministic
tie-break, and an event is handed its activator instead of storing it, so basic_event
is a definition and nothing else.

The logic container. scene/logicformat.h and scene/logiccontainer write a scenery's
events, their targets and sibling chains as flat, deterministic arrays with a schema
hash that refuses a mismatched build. simulation/logiccook cooks it from the loaded
state whenever it is missing, stale or unreadable, reads it straight back to check it,
and resolves each target to a position in the registry its event kind searches.
-cooklogic forces a cook, -verifylogic only checks. Nothing reads it at load yet.

Prefabs. scene/prefab describes a signal by the aspects it can show - lights, speeds,
animation channels, a sound - instead of pasting an .inc per variant, and expands an
instance to ordinary scenery text. Tried on a light signal and a shape signal.

Screenshots without a person present: -shot x,y,z[,yaw[,pitch]], -shotsettle, -shotstay.
This commit is contained in:
maj00r
2026-09-16 23:18:09 +02:00
parent 366645ef59
commit 2660d14565
47 changed files with 7117 additions and 240 deletions
Download Patch File Download Diff File Expand all files Collapse all files

89
rendering/opengl33renderer.cpp
View File

@@ -1627,7 +1627,15 @@ void opengl33_renderer::setup_pass(viewport_config &Viewport, renderpass_config
Config.draw_range = std::max( 2000.f, Config.draw_range );
// projection
auto const zfar = ( Zfar > 1.f ? Zfar : Config.draw_range * Zfar );
auto zfar = ( Zfar > 1.f ? Zfar : Config.draw_range * Zfar );
if ((false == Ignoredebug) && (Zfar == 1.f))
{
// the far plane of an ordinary view reaches as far as the terrain is streamed, which
// is its own setting rather than the draw range. only the projection moves: what
// else gets drawn is still decided by draw_range. reverse z on a float depth
// buffer keeps the precision this costs negligible
zfar = std::max(zfar, Global.TerrainRange);
}
auto const znear = ( Znear > 1.f ? Znear : Znear > 0.f ? Znear * zfar : 0.1f * Global.ZoomFactor);
// the editor's top-down plan view drops the perspective, so distances read the same wherever
@@ -2467,6 +2475,7 @@ void opengl33_renderer::Render(scene::basic_region *Region)
{
case rendermode::color:
{
Render_terrain();
Render(std::begin(m_sectionqueue), std::end(m_sectionqueue));
// draw queue is filled while rendering sections
if (EditorModeFlag && m_current_viewport->main)
@@ -2851,6 +2860,84 @@ void opengl33_renderer::Render(cell_sequence::iterator First, cell_sequence::ite
m_frame_instance_buckets.clear();
}
// The heightfield draws itself; what it needs from here is the state every other draw in
// this pass gets - the camera-relative modelview in the model block - and the viewpoint
// the world is being rendered around, so it can place its tiles relative to it.
void opengl33_renderer::Render_terrain()
{
// opened on first use rather than at init: a scenery names its heightfields in
// terrain directives, which are not read until the scenario is parsed, long after the
// renderer starts. attempted once either way, so a bad path does not retry per frame
if (false == m_terrainattempted)
{
m_terrainattempted = true;
for (auto const &path : Global.terrain_heightfields)
{
auto field{std::make_unique<terrain_clipmap>()};
if (true == field->open(path))
{
m_terrain.emplace_back(std::move(field));
}
}
// finest first: drawn in this order, the depth test keeps the better surface
// where two fields describe the same ground
std::sort(std::begin(m_terrain), std::end(m_terrain),
[](auto const &Left, auto const &Right) { return Left->gridstep() < Right->gridstep(); });
for (std::size_t index = 0; index < m_terrain.size(); ++index)
{
// every field sits a little under the triangles, so where a baked heightfield and
// the geometry it was baked from are both drawn, the geometry is what shows. a
// coarser field is sunk further still, far enough that rounding cannot let it
// through, and little enough to be invisible at the spacing it is drawn at
m_terrain[index]->depthbias(index == 0 ? 0.02f : 0.1f);
}
}
if (true == m_terrain.empty())
{
return;
}
glm::dvec3 const viewpoint{m_renderpass.pass_camera.position()};
model_ubs.set_modelview(OpenGLMatrices.data(GL_MODELVIEW));
model_ubo->update(model_ubs);
for (auto const &field : m_terrain)
{
// what stays on the gpu is decided by the main view alone. any other colour pass
// draws what is there: letting a second viewport with its own camera and range
// steer residency would evict what the main view is looking at
if (m_current_viewport->main)
{
// the terrain has a range of its own, not the pass draw range
field->range(Global.TerrainRange);
field->update(viewpoint);
}
field->render(viewpoint);
}
if (m_current_viewport->main)
{
TerrainStatus.resize(m_terrain.size());
for (std::size_t index = 0; index < m_terrain.size(); ++index)
{
auto const &field{*m_terrain[index]};
auto &status{TerrainStatus[index]};
if (status.path.empty())
{
status.path = field.path();
status.gridstep = field.gridstep();
status.tilesize = field.tilesize();
status.levels = field.levels();
}
status.range = field.range();
status.backlog = field.backlog();
status.stats = field.stats();
}
}
}
void opengl33_renderer::Draw_Geometry(std::vector<gfx::geometrybank_handle>::iterator begin, std::vector<gfx::geometrybank_handle>::iterator end)
{
m_geometry.draw(begin, end);

9
rendering/opengl33renderer.cppm
View File

@@ -36,6 +36,8 @@ import eu07.rendering.openglcamera;
import eu07.rendering.opengl33light;
import eu07.rendering.opengl33particles;
import eu07.rendering.opengl33skydome;
import eu07.rendering.terrainclipmap;
import eu07.rendering.terrainstatus;
import eu07.rendering.opengl33precipitation;
import eu07.simulation.simulationenvironment;
import eu07.simcore;
@@ -46,6 +48,7 @@ import eu07.gl.framebuffer;
import eu07.gl.renderbuffer;
import eu07.gl.postfx;
import eu07.gl.shader;
import eu07.rendering.terrainclipmap;
import eu07.gl.cubemap;
import eu07.gl.glsl_common;
import eu07.gl.pbo;
@@ -301,6 +304,8 @@ class opengl33_renderer : public gfx_renderer {
void setup_sunlight_intensity( float const Factor = 1.f);
// runs jobs needed to generate graphics for specified render pass
void Render_pass(viewport_config &vp, rendermode const Mode);
// draws the cooked terrain heightfield, if one is open
void Render_terrain();
// creates dynamic environment cubemap
bool Render_reflections(viewport_config &vp);
bool Render(world_environment *Environment);
@@ -379,6 +384,10 @@ class opengl33_renderer : public gfx_renderer {
double m_environmentupdatetime{0}; // time of the most recent environment map update
glm::dvec3 m_environmentupdatelocation; // coordinates of most recent environment map update
opengl33_skydome m_skydomerenderer;
// cooked terrain heightfields, ordered finest first. a scenery carries a coarse
// regional model and a fine corridor per line, and they overlap
std::vector<std::unique_ptr<terrain_clipmap>> m_terrain;
bool m_terrainattempted { false };
opengl33_precipitation m_precipitationrenderer;
opengl33_particles m_particlerenderer; // particle visualization subsystem

516
rendering/terrainclipmap.cpp Normal file
View File

@@ -0,0 +1,516 @@
/*
This Source Code Form is subject to the
terms of the Mozilla Public License, v.
2.0. If a copy of the MPL was not
distributed with this file, You can
obtain one at
http://mozilla.org/MPL/2.0/.
*/
module;
#include <algorithm>
#include <cmath>
#include <array>
#include <cstdint>
#include <functional>
#include <memory>
#include <optional>
#include <string>
#include <unordered_map>
#include <vector>
#include "glad/glad.h"
#include "scene/heightfieldformat.h"
#include "global_include/interfaces/ITexture_macros.h"
#include "scene/heightfieldreader.h"
module eu07.rendering.terrainclipmap;
import eu07.glm;
import eu07.utilities.logs;
import eu07.rendering.renderer;
import eu07.model.material;
namespace {
// NOTE: glTexImage2D rather than glTexStorage2D. Immutable storage arrives with OpenGL
// 4.2 and this renderer holds a 3.3 context, where the entry point is simply not there -
// calling it jumps through a null pointer instead of failing in any visible way
GLuint
create_texture( GLenum const Internalformat, GLenum const Type, GLsizei const Side, void const *Data ) {
GLuint texture { 0 };
::glGenTextures( 1, &texture );
::glBindTexture( GL_TEXTURE_2D, texture );
// integer samples carry the cooker's values, no-data included, so nothing may filter them
::glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST );
::glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST );
::glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE );
::glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE );
::glTexImage2D( GL_TEXTURE_2D, 0, Internalformat, Side, Side, 0, GL_RED_INTEGER, Type, Data );
return texture;
}
} // namespace
terrain_clipmap::~terrain_clipmap() {
close();
}
bool
terrain_clipmap::open( std::string const &Path ) {
close();
if( ( false == m_reader.open( Path ) )
|| ( false == m_loader.open( Path ) ) ) {
ErrorLog( "Terrain: cannot open heightfield \"" + Path + "\"" );
close();
return false;
}
m_path = Path;
build_indices();
build_palette();
resolve_materials();
try {
gl::shader vertex( "terrainclipmap.vert" );
gl::shader fragment( "terrainclipmap.frag" );
m_shader.emplace( std::vector<std::reference_wrapper<gl::shader const>>( { vertex, fragment } ) );
}
catch( gl::shader_exception const &error ) {
ErrorLog( std::string( "Terrain: shader failed, " ) + error.what() );
close();
return false;
}
m_vao.emplace();
resolve_uniforms();
m_ready = true;
WriteLog(
"Terrain: " + Path + ", " + std::to_string( m_reader.tiles().size() ) + " tiles, "
+ std::to_string( m_reader.levels() ) + " levels, grid "
+ std::to_string( m_reader.gridstep() ) + " m, tile " + std::to_string( m_reader.tilesize() ) + " m" );
return true;
}
// uniform locations do not change once the program is linked, so they are looked up
// here rather than per tile per frame
void
terrain_clipmap::resolve_uniforms() {
auto const program { static_cast<GLuint>( *m_shader ) };
m_uniforms.tileorigin = ::glGetUniformLocation( program, "tileorigin" );
m_uniforms.samplestep = ::glGetUniformLocation( program, "samplestep" );
m_uniforms.heightbias = ::glGetUniformLocation( program, "heightbias" );
m_uniforms.heightscale = ::glGetUniformLocation( program, "heightscale" );
m_uniforms.nodata = ::glGetUniformLocation( program, "nodata" );
m_uniforms.side = ::glGetUniformLocation( program, "side" );
m_uniforms.heights = ::glGetUniformLocation( program, "heights" );
m_uniforms.materials = ::glGetUniformLocation( program, "materials" );
m_uniforms.palette = ::glGetUniformLocation( program, "palette" );
m_uniforms.groundtexture = ::glGetUniformLocation( program, "ground" );
m_uniforms.groundscale = ::glGetUniformLocation( program, "groundscale" );
m_uniforms.drawnmaterial = ::glGetUniformLocation( program, "drawnmaterial" );
m_uniforms.tileworld = ::glGetUniformLocation( program, "tileworld" );
m_uniforms.hastexture = ::glGetUniformLocation( program, "hastexture" );
m_uniforms.fallback = ::glGetUniformLocation( program, "fallback" );
}
void
terrain_clipmap::close() {
// the loader first, so nothing arrives for tiles that are being torn down
m_loader.close();
m_wanted.clear();
m_arrived.clear();
m_inrange.clear();
m_wantedlevel.clear();
m_scanrange = -1.f;
for( auto & [ tilekey, tile ] : m_tiles ) { retire( tile ); }
m_tiles.clear();
m_indices.clear();
m_indexcounts.clear();
m_vao.reset();
m_shader.reset();
m_palette.clear();
m_reader.close();
m_stats = {};
m_ready = false;
}
// index buffers, one per mip level. the grid they describe is the same shape at every
// level, only smaller, and the vertices themselves never exist in memory: the shader
// derives a sample's row and column from gl_VertexID
void
terrain_clipmap::build_indices() {
m_indices.resize( m_reader.levels() );
m_indexcounts.resize( m_reader.levels() );
std::vector<std::uint32_t> indices;
for( std::uint32_t level = 0; level < m_reader.levels(); ++level ) {
auto const side { m_reader.samples_per_side( level ) };
indices.clear();
indices.reserve( static_cast<std::size_t>( side - 1 ) * ( side - 1 ) * 6 );
for( std::uint32_t row = 0; row + 1 < side; ++row ) {
for( std::uint32_t column = 0; column + 1 < side; ++column ) {
auto const topleft { row * side + column };
auto const topright { topleft + 1 };
auto const bottomleft { topleft + side };
auto const bottomright { bottomleft + 1 };
indices.push_back( topleft ); indices.push_back( bottomleft ); indices.push_back( topright );
indices.push_back( topright ); indices.push_back( bottomleft ); indices.push_back( bottomright );
}
}
m_indexcounts[ level ] = static_cast<std::uint32_t>( indices.size() );
m_indices[ level ] = std::make_unique<gl::buffer>();
m_indices[ level ]->allocate(
gl::buffer::ELEMENT_ARRAY_BUFFER, indices.size() * sizeof( std::uint32_t ), GL_STATIC_DRAW );
m_indices[ level ]->upload(
gl::buffer::ELEMENT_ARRAY_BUFFER, indices.data(), 0, indices.size() * sizeof( std::uint32_t ) );
}
}
// placeholder colours, one per cooked material, so that the shape of the terrain and the
// division into surfaces can be judged before the real materials are wired in
void
terrain_clipmap::build_palette() {
m_palette.clear();
m_palette.reserve( std::max<std::size_t>( 1, m_reader.materials().size() ) );
for( std::size_t index = 0; index < m_reader.materials().size(); ++index ) {
auto const hue { static_cast<float>( index ) * 0.61803399f };
auto const phase { ( hue - std::floor( hue ) ) * 6.f };
auto const rising { phase - std::floor( phase ) };
glm::vec3 colour { 0.45f, 0.5f, 0.4f };
switch( static_cast<int>( phase ) ) {
case 0: colour = { 0.55f, 0.45f + 0.2f * rising, 0.3f }; break;
case 1: colour = { 0.55f - 0.2f * rising, 0.6f, 0.3f }; break;
case 2: colour = { 0.35f, 0.6f, 0.3f + 0.2f * rising }; break;
case 3: colour = { 0.35f, 0.6f - 0.15f * rising, 0.5f }; break;
case 4: colour = { 0.35f + 0.2f * rising, 0.45f, 0.5f }; break;
default: colour = { 0.55f, 0.45f, 0.5f - 0.15f * rising }; break;
}
m_palette.push_back( colour );
}
if( true == m_palette.empty() ) { m_palette.push_back( { 0.45f, 0.5f, 0.4f } ); }
}
std::uint32_t
terrain_clipmap::level_for( double const Distance ) const {
if( Distance <= terrain_finest_range ) { return 0; }
auto const steps { static_cast<std::uint32_t>( std::log2( Distance / terrain_finest_range ) ) + 1 };
return std::min( steps, m_reader.levels() - 1 );
}
void
terrain_clipmap::upload( terrain_tile_loader::payload const &Tile ) {
auto const tilekey { key( Tile.tile.x, Tile.tile.z ) };
auto const side { m_reader.samples_per_side( Tile.tile.level ) };
auto const existing { m_tiles.find( tilekey ) };
if( existing != m_tiles.end() ) {
// a level change means new textures: the sample count differs
retire( existing->second );
m_tiles.erase( existing );
}
resident_tile tile;
tile.x = Tile.tile.x;
tile.z = Tile.tile.z;
tile.level = Tile.tile.level;
tile.side = side;
tile.lastused = m_frame;
tile.heighttexture = create_texture(
GL_R16UI, GL_UNSIGNED_SHORT, static_cast<GLsizei>( side ), Tile.heights.data() );
tile.materialtexture = create_texture(
GL_R8UI, GL_UNSIGNED_BYTE, static_cast<GLsizei>( side ), Tile.materials.data() );
// the tile is drawn once per material it contains, so the set is worked out here
// rather than rescanned every frame
std::array<bool, 256> present {};
for( auto const material : Tile.materials ) { present[ material ] = true; }
for( std::size_t material = 0; material < present.size(); ++material ) {
if( true == present[ material ] ) { tile.materials.push_back( static_cast<std::uint8_t>( material ) ); }
}
if( auto const *description { m_reader.find( Tile.tile.x, Tile.tile.z ) } ) {
tile.minheight = description->minheight;
tile.maxheight = description->maxheight;
}
m_stats.texturebytes += static_cast<std::size_t>( side ) * side * 3u;
++m_stats.uploaded;
m_tiles.emplace( tilekey, std::move( tile ) );
}
void
terrain_clipmap::retire( resident_tile &Tile ) {
if( Tile.heighttexture != 0 ) { ::glDeleteTextures( 1, &Tile.heighttexture ); Tile.heighttexture = 0; }
if( Tile.materialtexture != 0 ) { ::glDeleteTextures( 1, &Tile.materialtexture ); Tile.materialtexture = 0; }
auto const bytes { static_cast<std::size_t>( Tile.side ) * Tile.side * 3u };
m_stats.texturebytes -= std::min( m_stats.texturebytes, bytes );
}
void
terrain_clipmap::scan( glm::dvec3 const &Viewpoint ) {
m_scanpoint = Viewpoint;
m_scanrange = m_range;
m_inrange.clear();
m_wantedlevel.clear();
auto const tilesize { static_cast<double>( m_reader.tilesize() ) };
auto const reach { std::max( 1, static_cast<std::int32_t>( m_range / tilesize ) + 1 ) };
auto const centrex { m_reader.tile_x( Viewpoint.x ) };
auto const centrez { m_reader.tile_z( Viewpoint.z ) };
std::vector<std::pair<double, terrain_tile_loader::request>> wishes;
for( std::int32_t z = centrez - reach; z <= centrez + reach; ++z ) {
for( std::int32_t x = centrex - reach; x <= centrex + reach; ++x ) {
if( false == m_reader.contains( x, z ) ) { continue; }
// distance to the nearest point of the tile, not to its centre. a regional
// field has tiles kilometres across, and a tile the camera is standing on
// can have its centre far outside the draw range
auto const west { x * tilesize };
auto const north { z * tilesize };
auto const nearestx { std::clamp( Viewpoint.x, west, west + tilesize ) };
auto const nearestz { std::clamp( Viewpoint.z, north, north + tilesize ) };
auto const distance {
glm::length( glm::dvec2 { nearestx - Viewpoint.x, nearestz - Viewpoint.z } ) };
if( distance > m_range ) { continue; }
auto const tilekey { key( x, z ) };
auto const level { level_for( distance ) };
m_inrange.push_back( { tilekey, level } );
m_wantedlevel.emplace( tilekey, level );
auto const resident { m_tiles.find( tilekey ) };
if( ( resident == m_tiles.end() ) || ( resident->second.level != level ) ) {
wishes.push_back( { distance, { x, z, level } } );
}
}
}
// nearest first, so the ground under the camera is never waiting behind the horizon
std::sort( wishes.begin(), wishes.end(),
[]( auto const &Left, auto const &Right ) { return Left.first < Right.first; } );
m_wanted.clear();
for( auto const &wish : wishes ) { m_wanted.push_back( wish.second ); }
m_loader.want( m_wanted );
}
void
terrain_clipmap::update( glm::dvec3 const &Viewpoint ) {
if( false == m_ready ) { return; }
++m_frame;
auto const moved {
glm::length( glm::dvec2 { Viewpoint.x - m_scanpoint.x, Viewpoint.z - m_scanpoint.z } ) };
if( ( m_range != m_scanrange )
|| ( moved > m_reader.tilesize() * 0.25 ) ) {
scan( Viewpoint );
}
// the tiles in range are touched every frame, which is what render() draws and what
// eviction spares
m_stats.inview = 0;
m_stats.wanted = 0;
m_stats.perlevel.fill( 0 );
for( auto const &tile : m_inrange ) {
auto const resident { m_tiles.find( tile.key ) };
if( resident == m_tiles.end() ) {
++m_stats.wanted;
continue;
}
// still wanted, at whatever level it has until the right one arrives
resident->second.lastused = m_frame;
++m_stats.inview;
++m_stats.perlevel[ std::min<std::size_t>( resident->second.level, m_stats.perlevel.size() - 1 ) ];
if( resident->second.level != tile.level ) { ++m_stats.wanted; }
}
m_arrived.clear();
m_loader.collect( m_arrived, m_uploadsperframe );
for( auto const &tile : m_arrived ) {
if( false == tile.valid ) { continue; }
// a tile read for where the camera was is put up only if nothing of it is on the gpu
// yet - some level is better than a hole. otherwise it is dropped, and the level
// wanted now is already queued
auto const tilekey { key( tile.tile.x, tile.tile.z ) };
auto const wanted { m_wantedlevel.find( tilekey ) };
auto const stillwanted { ( wanted != m_wantedlevel.end() ) && ( wanted->second == tile.tile.level ) };
if( stillwanted || ( false == m_tiles.contains( tilekey ) ) ) {
upload( tile );
// put up this frame, so drawn this frame
m_tiles.find( tilekey )->second.lastused = m_frame;
}
else {
++m_stats.dropped;
}
}
if( ( m_tiles.size() > m_budget ) && ( m_tiles.size() > m_stats.inview ) ) {
// over budget, and not everything is in view: drop whatever has gone longest without
// being asked for
std::vector<std::pair<std::uint64_t, std::int64_t>> aged;
for( auto const & [ tilekey, tile ] : m_tiles ) {
if( tile.lastused != m_frame ) { aged.emplace_back( tile.lastused, tilekey ); }
}
auto const excess { std::min( aged.size(), m_tiles.size() - m_budget ) };
std::partial_sort( aged.begin(), aged.begin() + excess, aged.end() );
for( std::size_t index = 0; index < excess; ++index ) {
auto const victim { m_tiles.find( aged[ index ].second ) };
retire( victim->second );
m_tiles.erase( victim );
}
}
m_stats.resident = m_tiles.size();
report_residency();
}
// the cooked material names are the ones the scenery used, so the engine resolves them
// exactly as it does for any other geometry. only the diffuse texture is taken: the
// terrain has its own shader and none of the rest of a material applies to it
void
terrain_clipmap::resolve_materials() {
m_materials.clear();
auto const &names { m_reader.materials() };
auto const &scales { m_reader.material_scales() };
m_materials.reserve( names.size() );
std::size_t resolved { 0 };
std::string unresolved;
for( std::size_t index = 0; index < names.size(); ++index ) {
material_binding binding;
binding.scale = ( index < scales.size() ? scales[ index ] : 4.f );
if( binding.scale < 0.01f ) { binding.scale = 4.f; }
auto const material { GfxRenderer->Fetch_Material( names[ index ] ) };
if( material != null_handle ) {
binding.texture = GfxRenderer->Material( material )->GetTexture( 0 );
}
if( binding.texture != null_handle ) {
++resolved;
}
else {
// drawn in its placeholder colour instead, so a missing texture shows up as
// an obvious flat patch rather than as black ground
unresolved += ( unresolved.empty() ? "" : ", " ) + names[ index ];
}
m_materials.push_back( binding );
}
WriteLog(
"Terrain: " + std::to_string( resolved ) + " of " + std::to_string( names.size() )
+ " materials resolved to textures" );
if( false == unresolved.empty() ) {
WriteLog( "Terrain: no texture for " + unresolved );
}
}
// residency settles within a second of the camera stopping and then barely moves, so a
// line is written when it shifts by a noticeable amount rather than every frame. that
// makes the warm figure - what the terrain actually costs - readable from the log
void
terrain_clipmap::report_residency() {
auto const difference {
m_stats.resident > m_reportedresidency ?
m_stats.resident - m_reportedresidency :
m_reportedresidency - m_stats.resident };
if( ( m_reported == true ) && ( difference < 16 ) ) { return; }
m_reported = true;
m_reportedresidency = m_stats.resident;
WriteLog(
"Terrain: " + std::to_string( m_stats.resident ) + " tiles resident, "
+ std::to_string( m_stats.texturebytes / 1048576 ) + " MB of samples, "
+ std::to_string( m_stats.uploaded ) + " uploads so far" );
}
void
terrain_clipmap::render( glm::dvec3 const &Viewpoint ) {
if( ( false == m_ready ) || ( true == m_tiles.empty() ) ) { return; }
m_shader->bind();
m_vao->bind();
::glUniform1f( m_uniforms.heightbias, m_reader.height( 0 ) );
::glUniform1f( m_uniforms.heightscale, m_reader.height( 1 ) - m_reader.height( 0 ) );
::glUniform1ui( m_uniforms.nodata, heightfield::nodata );
::glUniform1i( m_uniforms.heights, 0 );
::glUniform1i( m_uniforms.materials, 1 );
::glUniform1i( m_uniforms.groundtexture, 2 );
::glUniform3fv(
m_uniforms.palette, static_cast<GLsizei>( m_palette.size() ), &m_palette.front().x );
auto const tilesize { static_cast<double>( m_reader.tilesize() ) };
auto const gridstep { m_reader.gridstep() };
for( auto const & [ tilekey, tile ] : m_tiles ) {
if( tile.lastused != m_frame ) { continue; }
// tile position relative to the viewpoint, so that the shader never sees a
// coordinate large enough for single precision to matter
glm::vec3 const origin {
static_cast<float>( tile.x * tilesize - Viewpoint.x ),
static_cast<float>( -Viewpoint.y ) - m_depthbias,
static_cast<float>( tile.z * tilesize - Viewpoint.z ) };
::glUniform3fv( m_uniforms.tileorigin, 1, &origin.x );
::glUniform1f( m_uniforms.samplestep, gridstep * static_cast<float>( 1u << tile.level ) );
::glUniform1i( m_uniforms.side, static_cast<GLint>( tile.side ) );
::glUniform2f(
m_uniforms.tileworld,
static_cast<float>( tile.x * tilesize ), static_cast<float>( tile.z * tilesize ) );
::glActiveTexture( GL_TEXTURE0 );
::glBindTexture( GL_TEXTURE_2D, tile.heighttexture );
::glActiveTexture( GL_TEXTURE1 );
::glBindTexture( GL_TEXTURE_2D, tile.materialtexture );
m_indices[ tile.level ]->bind( gl::buffer::ELEMENT_ARRAY_BUFFER );
// one pass per material in the tile: each wants a different texture bound, and
// a fragment belonging to any other material is dropped
for( auto const material : tile.materials ) {
auto const &binding {
material < m_materials.size() ? m_materials[ material ] : material_binding {} };
GfxRenderer->Bind_Texture( 2, binding.texture );
::glUniform1f( m_uniforms.groundscale, binding.scale );
::glUniform1ui( m_uniforms.drawnmaterial, material );
::glUniform1i( m_uniforms.hastexture, binding.texture != null_handle ? 1 : 0 );
auto const &fallback {
m_palette[ std::min<std::size_t>( material, m_palette.size() - 1 ) ] };
::glUniform3fv( m_uniforms.fallback, 1, &fallback.x );
::glDrawElements(
GL_TRIANGLES, static_cast<GLsizei>( m_indexcounts[ tile.level ] ), GL_UNSIGNED_INT, nullptr );
}
}
::glActiveTexture( GL_TEXTURE0 );
m_vao->unbind();
gl::program::bind( 0 );
}

179
rendering/terrainclipmap.cppm Normal file
View File

@@ -0,0 +1,179 @@
/*
This Source Code Form is subject to the
terms of the Mozilla Public License, v.
2.0. If a copy of the MPL was not
distributed with this file, You can
obtain one at
http://mozilla.org/MPL/2.0/.
*/
module;
#include <array>
#include <cstdint>
#include <memory>
#include <optional>
#include <string>
#include <unordered_map>
#include <vector>
#include "global_include/interfaces/ITexture_macros.h"
#include "scene/heightfieldreader.h"
export module eu07.rendering.terrainclipmap;
import eu07.gl.shader;
import eu07.gl.buffer;
import eu07.gl.vao;
import eu07.glm;
import eu07.rendering.terraintileloader;
import eu07.rendering.terrainstatus;
import eu07.model.material;
import eu07.global_include.interfaces.itexture;
export {
// Draws the cooked terrain heightfield.
//
// Not a toroidal clipmap, despite what the plan called it. The measurements say the
// terrain is a ribbon: 390 km2 of surface inside a 51 x 46 km rectangle, twelve percent
// of the tile bounding box occupied. Clipmap rings sized for that rectangle would be
// mostly empty, so this draws the cooked tiles themselves, each at the mip level its
// distance earns. The tiles were laid out for exactly this - sorted by morton code, one
// shared sample of overlap so neighbouring levels meet without skirts.
//
// The class owns its gl resources and the file handle, and knows nothing about the
// scenery, the camera class or the rest of the renderer: it is given a viewpoint and a
// pair of matrices, and draws.
class terrain_clipmap {
public:
terrain_clipmap() = default;
~terrain_clipmap();
terrain_clipmap( terrain_clipmap const & ) = delete;
terrain_clipmap &operator=( terrain_clipmap const & ) = delete;
// opens a cooked heightfield and builds the gl resources. returns false when the
// file cannot be read, leaving the object unusable but safe to destroy
bool open( std::string const &Path );
bool ready() const { return m_ready; }
void close();
// asks for the tiles around the viewpoint, uploads a bounded number of those that have
// arrived, and retires the ones that left range. safe to call every frame: the reading
// and decoding happen on the loader's thread, and a tile waiting for a finer level keeps
// being drawn at the level it has
void update( glm::dvec3 const &Viewpoint );
// draws the resident tiles. the caller is expected to have the scene and model
// uniform blocks already set for a camera-relative pass, the way the rest of the
// renderer does: tile positions reach the shader relative to the viewpoint, so
// nothing here needs a matrix of its own
void render( glm::dvec3 const &Viewpoint );
// how far from the viewpoint terrain is drawn, in metres
void range( float const Range ) { m_range = Range; }
float range() const { return m_range; }
// metres between samples at the finest level; a scenery's fields are drawn from the
// finest to the coarsest so that the depth test keeps the better surface
float gridstep() const { return m_reader.gridstep(); }
// sinks this field by the given amount, so a coarse field cannot win fragments from
// a finer one describing the same ground
void depthbias( float const Metres ) { m_depthbias = Metres; }
terrain_statistics const &stats() const { return m_stats; }
std::string const &path() const { return m_path; }
float tilesize() const { return static_cast<float>( m_reader.tilesize() ); }
std::uint32_t levels() const { return m_reader.levels(); }
std::size_t backlog() const { return m_loader.backlog(); }
private:
// one tile resident on the gpu. heights are unsigned so that the no-data value stays
// exactly what the cooker wrote instead of being normalised into a float
struct resident_tile {
std::int32_t x { 0 }, z { 0 };
std::uint32_t level { 0 }; // mip level currently uploaded
std::uint32_t side { 0 }; // samples per side at that level
std::uint32_t heighttexture { 0 };
std::uint32_t materialtexture { 0 };
float minheight { 0.f }, maxheight { 0.f };
std::uint64_t lastused { 0 };
// materials appearing in this tile. a tile is drawn once per entry, since each
// needs its own texture bound, and most tiles carry only one or two
std::vector<std::uint8_t> materials;
};
// mip level a tile at the given distance is drawn at
std::uint32_t level_for( double const Distance ) const;
// works out which tiles are in range and at what level, and tells the loader what is
// missing. the answer only changes once the camera has moved a fair part of a tile
void scan( glm::dvec3 const &Viewpoint );
// puts a tile the loader has read on the gpu, replacing whatever level it had before
void upload( terrain_tile_loader::payload const &Tile );
void retire( resident_tile &Tile );
void build_indices();
void build_palette();
// resolves the cooked material names to the engine's materials, once
void resolve_materials();
void resolve_uniforms();
void report_residency();
static std::int64_t key( std::int32_t const X, std::int32_t const Z ) { return heightfield::tile_key( X, Z ); }
// for the file's layout only: tile coordinates, levels, materials. tiles themselves are
// read by the loader, on its own handle
heightfield::reader m_reader;
terrain_tile_loader m_loader;
std::vector<terrain_tile_loader::request> m_wanted;
std::vector<terrain_tile_loader::payload> m_arrived;
// one tile within range, and the level its distance earns
struct in_range {
std::int64_t key;
std::uint32_t level;
};
// what the last scan found in range, and the level wanted for each tile, to tell a useful
// arrival from one the camera has already moved past
std::vector<in_range> m_inrange;
std::unordered_map<std::int64_t, std::uint32_t> m_wantedlevel;
glm::dvec3 m_scanpoint { 0.0 };
float m_scanrange { -1.f };
std::unordered_map<std::int64_t, resident_tile> m_tiles;
// one index buffer per mip level, over a grid of that level's sample count. vertices
// are computed in the shader from gl_VertexID, so there is no vertex buffer at all.
// held by pointer because a gl::buffer owns a name it cannot hand to a copy
std::vector<std::unique_ptr<gl::buffer>> m_indices;
std::vector<std::uint32_t> m_indexcounts;
std::optional<gl::vao> m_vao;
std::optional<gl::program> m_shader;
// resolved once at link time; -1 for a uniform the compiler dropped, which gl ignores
struct uniform_locations {
int tileorigin { -1 }, samplestep { -1 };
int heightbias { -1 }, heightscale { -1 }, nodata { -1 }, side { -1 };
int heights { -1 }, materials { -1 }, palette { -1 };
int groundtexture { -1 }, groundscale { -1 }, drawnmaterial { -1 }, tileworld { -1 };
int hastexture { -1 }, fallback { -1 };
} m_uniforms;
std::vector<glm::vec3> m_palette;
// one entry per cooked material: the engine texture to bind and how much ground one
// repeat of it covers
struct material_binding {
texture_handle texture { null_handle };
float scale { 4.f };
};
std::vector<material_binding> m_materials;
terrain_statistics m_stats;
std::uint64_t m_frame { 0 };
std::size_t m_reportedresidency { 0 };
bool m_reported { false };
float m_range { 6000.f };
float m_depthbias { 0.f };
// tiles allowed to stay resident; at 129 samples a tile costs about 50 kB of texture
std::size_t m_budget { 2048 };
// tiles put on the gpu per frame. a tile is about 50 kB of texture, so this is not about
// the upload itself but about never spending a frame on terrain the camera will reach
// later anyway
std::size_t m_uploadsperframe { 32 };
bool m_ready { false };
std::string m_path;
};
} // export

57
rendering/terrainstatus.cppm Normal file
View File

@@ -0,0 +1,57 @@
/*
This Source Code Form is subject to the
terms of the Mozilla Public License, v.
2.0. If a copy of the MPL was not
distributed with this file, You can
obtain one at
http://mozilla.org/MPL/2.0/.
*/
module;
#include <array>
#include <cmath>
#include <cstddef>
#include <cstdint>
#include <string>
#include <vector>
export module eu07.rendering.terrainstatus;
export {
// distance out to which a terrain tile is drawn at its finest level; each further level
// doubles it
inline constexpr double terrain_finest_range { 512.0 };
// the distance up to which a level is used
inline double
terrain_level_distance( std::uint32_t const Level ) {
return terrain_finest_range * std::pow( 2.0, Level );
}
// what one terrain field's streamer is doing, kept by the field and read by the debug panel
struct terrain_statistics {
std::size_t resident { 0 }; // tiles holding gl textures
std::size_t inview { 0 }; // resident tiles within range in the last update
std::size_t wanted { 0 }; // tiles in range missing, or at the wrong level
std::size_t uploaded { 0 }; // uploads since opening
std::size_t dropped { 0 }; // arrivals discarded as stale since opening
std::size_t texturebytes { 0 };
std::array<std::size_t, 8> perlevel {}; // tiles in view at each level
};
// one field as the debug panel shows it. the renderer writes it once a frame from the main
// view, the panel reads it; neither needs to know the other's types
struct terrain_field_status {
std::string path;
float gridstep { 0.f };
float tilesize { 0.f };
float range { 0.f };
std::uint32_t levels { 0 };
std::size_t backlog { 0 }; // tiles the loader has queued or finished and not handed over
terrain_statistics stats;
};
inline std::vector<terrain_field_status> TerrainStatus;
}

129
rendering/terraintileloader.cpp Normal file
View File

@@ -0,0 +1,129 @@
/*
This Source Code Form is subject to the
terms of the Mozilla Public License, v.
2.0. If a copy of the MPL was not
distributed with this file, You can
obtain one at
http://mozilla.org/MPL/2.0/.
*/
module;
#include <algorithm>
#include <cstddef>
#include <mutex>
#include <string>
#include <thread>
#include <utility>
#include <vector>
#include "scene/heightfieldreader.h"
module eu07.rendering.terraintileloader;
terrain_tile_loader::~terrain_tile_loader() {
close();
}
bool
terrain_tile_loader::open( std::string const &Path ) {
close();
if( false == m_reader.open( Path ) ) { return false; }
m_stop = false;
m_thread = std::thread( &terrain_tile_loader::work, this );
return true;
}
void
terrain_tile_loader::close() {
if( true == m_thread.joinable() ) {
{
std::lock_guard<std::mutex> lock( m_mutex );
m_stop = true;
}
m_wake.notify_all();
m_thread.join();
}
std::lock_guard<std::mutex> lock( m_mutex );
m_pending.clear();
m_done.clear();
m_running.reset();
m_reader.close();
}
bool
terrain_tile_loader::underway( request const &Tile ) const {
if( ( m_running.has_value() ) && ( *m_running == Tile ) ) { return true; }
return std::any_of(
m_done.begin(), m_done.end(),
[ &Tile ]( payload const &Done ) { return Done.tile == Tile; } );
}
void
terrain_tile_loader::want( std::vector<request> const &Requests ) {
{
std::lock_guard<std::mutex> lock( m_mutex );
m_pending.clear();
for( auto const &tile : Requests ) {
if( false == underway( tile ) ) {
m_pending.push_back( tile );
}
}
}
m_wake.notify_one();
}
std::size_t
terrain_tile_loader::collect( std::vector<payload> &Out, std::size_t const Count ) {
std::unique_lock<std::mutex> lock( m_mutex );
std::size_t moved { 0 };
while( ( moved < Count ) && ( false == m_done.empty() ) ) {
Out.emplace_back( std::move( m_done.front() ) );
m_done.pop_front();
++moved;
}
lock.unlock();
if( moved > 0 ) {
// room was made; the thread may be waiting for it
m_wake.notify_one();
}
return moved;
}
std::size_t
terrain_tile_loader::backlog() const {
std::lock_guard<std::mutex> lock( m_mutex );
return m_pending.size() + m_done.size() + ( m_running.has_value() ? 1 : 0 );
}
void
terrain_tile_loader::work() {
while( true ) {
request tile;
{
std::unique_lock<std::mutex> lock( m_mutex );
m_wake.wait( lock, [ this ]() {
return m_stop || ( ( false == m_pending.empty() ) && ( m_done.size() < donelimit ) ); } );
if( true == m_stop ) { return; }
tile = m_pending.front();
m_pending.pop_front();
m_running = tile;
}
// the read itself happens outside the lock, so the render thread can replace the
// wish list while a tile is being decoded
payload result;
result.tile = tile;
result.valid = m_reader.read_level( tile.x, tile.z, tile.level, result.heights, result.materials );
std::lock_guard<std::mutex> lock( m_mutex );
m_running.reset();
m_done.emplace_back( std::move( result ) );
}
}

90
rendering/terraintileloader.cppm Normal file
View File

@@ -0,0 +1,90 @@
/*
This Source Code Form is subject to the
terms of the Mozilla Public License, v.
2.0. If a copy of the MPL was not
distributed with this file, You can
obtain one at
http://mozilla.org/MPL/2.0/.
*/
module;
#include <condition_variable>
#include <cstddef>
#include <cstdint>
#include <deque>
#include <mutex>
#include <optional>
#include <string>
#include <thread>
#include <vector>
#include "scene/heightfieldreader.h"
export module eu07.rendering.terraintileloader;
export {
// Reads and decodes heightfield tiles on a thread of its own.
//
// Reading a tile means a seek, a read and a zstd decode, which is too much to do on the render
// thread for the dozens of tiles a moving camera asks for at once. This does that part and
// nothing else: it knows no gl, keeps its own file handle, and hands back plain sample arrays
// for the renderer to upload at a pace it chooses.
//
// The renderer says each frame what it wants, nearest first, and that list replaces whatever
// was asked for before and has not been started. A tile the camera has already left is never
// read, and the nearest tile is always the next one read.
class terrain_tile_loader {
public:
struct request {
std::int32_t x { 0 }, z { 0 };
std::uint32_t level { 0 };
bool operator==( request const & ) const = default;
};
struct payload {
request tile;
bool valid { false };
std::vector<std::uint16_t> heights;
std::vector<std::uint8_t> materials;
};
terrain_tile_loader() = default;
~terrain_tile_loader();
terrain_tile_loader( terrain_tile_loader const & ) = delete;
terrain_tile_loader &operator=( terrain_tile_loader const & ) = delete;
// opens the file on the loader's own handle and starts the thread
bool open( std::string const &Path );
// stops the thread; anything not collected is dropped
void close();
// replaces the list of wanted tiles, nearest first. tiles being read or already read and
// not yet collected are left out of it, so asking again every frame costs nothing
void want( std::vector<request> const &Requests );
// moves up to Count finished tiles into Out, returning how many
std::size_t collect( std::vector<payload> &Out, std::size_t const Count );
// tiles queued or finished and not yet collected
std::size_t backlog() const;
private:
void work();
// whether the tile is being read or waits to be collected. caller holds the lock
bool underway( request const &Tile ) const;
heightfield::reader m_reader;
std::thread m_thread;
mutable std::mutex m_mutex;
std::condition_variable m_wake;
std::deque<request> m_pending;
std::optional<request> m_running;
std::deque<payload> m_done;
bool m_stop { false };
// finished tiles allowed to wait for collection. past this the thread stops reading:
// the renderer uploads at its own pace, and a backlog read for where the camera was a
// second ago is work spent on tiles nobody wants any more
static constexpr std::size_t donelimit { 32 };
};
}