start /wait gives the exit code but not the output; a .bat file waits for the windowed simulator and passes its output on.
Scenario tests
End-to-end checks that run the real simulator on a scenery and look at how the run ended and what a vehicle did: did it start, did it keep to the speed limit, did it brake and stop, did it pass a signal at stop.
Scenery and vehicle data are not in this repo, so these run against a local game install only, not in CI.
The simulator's test mode
The runner drives the simulator's test mode (eu07/maszyna#541), which works without the runner too and is meant for scenery authors as well:
eu07 --test -s <scenery.scn> [--test-timeout <seconds>] [-seed <number|word>] [-ini <settings file>]
With --test the simulator:
- needs
-s;-vis ignored and every train is driven by AI (ghostview), - draws nothing: the null renderer loads the 3D models (the physics needs some of their data) but no textures, and the window stays hidden,
- has no sound, no network, no python, no pause (not even one set by the scenery), and no crash dialog,
- takes no input from outside: keys and mouse sent to its window, gamepad, UART (a cab
desk), ZeroMQ and commands sent by other programs (
WM_COPYDATA) are all ignored, and motion telemetry is off, - gives nothing to the outside either: no keyboard leds, LPT or PoKeys (
feedbackmode), no messages to a multiplayer server (multiplayer), no program run at the end (execonexit), no console window of its own (showsystemconsole), and the game'simgui.inistays as it is, - runs the simulation as fast as it can in fixed steps of 0.03 s of simulation time per frame (three 10 ms physics steps), whatever the real time, the time speed or the machine,
- seeds its random numbers with a fixed seed unless
-seedorrandomseedgives one, so two runs of a scenery come out the same (the physicslogs of two runs are byte for byte equal), - quits when the test ends, and says how it ended in the exit code and in a
test: ...line on stdout and in the log. A command line it can't take, or a start that fails, ends the same way with theerrorcode.
On Windows the simulator is a windowed program, which the cmd prompt and PowerShell don't wait
for. In a .bat file eu07 --test ... > result.txt waits and sets %errorlevel%; at the prompt
start /wait eu07 --test ... waits too but doesn't pass the output on. In PowerShell use
(Start-Process eu07 '--test ...' -Wait -PassThru -RedirectStandardOutput result.txt).ExitCode.
Scripts (Python's subprocess, bash) wait for it as for any other program.
A scenery reports its result to the memory cell __test.result, which the simulator creates
(in every run, so a test scenery loads normally too). Write value 1 to it for a pass and 0
for a failure, with an UpdateValues event; other values are ignored, and the text, if
given, goes to the log as the reason:
event test_pass UpdateValues 0 __test.result reached_the_stop 1 * endevent
event test_fail UpdateValues 0 __test.result wrong_track 0 * endevent
(PutValues isn't the event for it: it hands a command to a vehicle, it doesn't write a cell.)
A good place for the pass event is event0 of a track the train should stop on: it runs
while a train stands there, and the length of the track is the margin of the stop.
The test ends with the first of these:
| exit code | result | when |
|---|---|---|
| 0 | pass | __test.result was set to 1 and then every vehicle stood still for 1 s, so a train still moving towards a signal at stop has to stop first (with test.no_movement_before_pass no the pass ends the test at once). Also the standstill end with test.pass_on_standstill yes |
| 1 | fail | __test.result was set to 0; ends at once |
| 2 | signal passed at danger | a train went past a signal showing stop (SetVelocity 0 0; SetVelocity 0 x on automatic block, which may be passed after a stop, doesn't count); ends at once, unless test.fail_if_stop_overran no |
| 3 | timeout | test.timeout seconds of simulation time went by (default 3600) |
| 4 | standstill | every vehicle stood still for test.standstill seconds (default 300) and nothing reported a result. Trains waiting at a station or for a signal count as standing, so the limit has to be longer than such waits |
| 5 | error | the run could not start: no -s, the scenery failed to load or has no vehicles, or the simulator threw |
A crash ends with the system's code instead (on Windows e.g. 0xC0000005, -1073741819).
The limits go in a config block of the scenery:
config
test.timeout 600
test.standstill 120
test.pass_on_standstill no
test.fail_if_stop_overran yes
test.no_movement_before_pass yes
endconfig
| key | default | meaning |
|---|---|---|
test.timeout |
3600 | seconds of simulation time before the test ends as timeout; 0 = never |
test.standstill |
300 | seconds with every vehicle still and no result before the test ends as standstill; 0 = never |
test.pass_on_standstill |
no | the standstill end passes the test instead |
test.fail_if_stop_overran |
yes | a signal passed at danger ends the test; with no it's only logged |
test.no_movement_before_pass |
yes | a pass waits until every vehicle stands still; with no it ends the test at once |
--test-timeout on the command line wins over test.timeout of the scenery.
A signal is passed at danger when the head of a train goes past a signal which its driver has in its speed table, showing stop. So it's the driver's view that counts: a train without a driver isn't watched, and neither is a signal the driver never scanned (a train placed at 100 km/h 150 m before a signal at stop got past it before its AI first looked ahead; at 60 km/h it was caught). Outside the test mode the same is written to the log, nothing more.
Running
python tests/scenarios/run.py --game-dir "C:/Steam/steamapps/common/MaSzyna" -j 4
Python 3.10+, no extra packages. Options:
EU07_GAME_DIRcan replace--game-dir.--exe path/to/eu07.exepicks the build to test; the default is the newestcmake-build-*/bin/Release/eu07*.exe. The build needs the--testand-inicommand line options.- name one or more cases to run only those, e.g.
run.py ... eu07_td_timetable. -j Nruns N cases at once. Each case prints its results when it is done. All 15 cases take about 30 s with-j 4, loading included.--watchruns a case the old way, in a visible window in real time (or faster with--timespeed N), closed after the case'sduration; the exit code isn't checked then. For watching what a case does.
Exit code is 0 when every case passed. Logs of each run (log.txt, errors.txt,
stdout.txt, the vehicle's physicslog) are moved to tests/scenarios/out/<case>/.
How it works
For each case the runner:
- writes
scenery/autotest_<case>.scninto the game dir: the base scenery with its own trainsets removed, the start time changed if the case says so, the case's track events added, and the case's.scnfragment added: its trainsets afterFirstInit, everything else (the case'stest_config, event launchers standing in for keys the scenery expects a player to press, extra events) before it, - starts the simulator with
--test -s autotest_<case>.scn -ini out/<case>/autotest.ini, plus--test-timeout <duration>when the case gives aduration; the extra ini turns onphysicslogand writes the logs tologs/log (autotest_<case>) <date>.txtrather than the sharedlog.txt(multiplelogs), - waits until the simulator quits by itself. It gives up early when the log reports
Bad vehicle/Bad trainset, when the scenario has loaded and no physicslog appeared, or after the case'stimeoutof wall-clock time, - compares the result from the exit code with the case's
expect, then runs the checks on the physicslog.
The lead vehicle is named autotest_<case>, so every case has its own physicslog and log
files and cases can run in parallel. The autotest_* sceneries (and the simulator's .sbt
geometry cache of them) stay in the game's scenery/ dir; the cache is dropped when the base
scenery is newer.
Writing a case
cases/<name>.json:
| key | meaning |
|---|---|
base_scenery |
scenery file in the game's scenery/ dir |
strip_trainsets |
drop the base scenery's trainsets (default true) |
time |
replace the scenery start time, HH:MM |
track_friction |
set the friction of every track of the base scenery (default in the sceneries is 0.15), e.g. for slippery rail |
track_events |
lines added to tracks of the base scenery, by track name, e.g. { "tor1k": "event0 test_pass" }; only tracks in the scenery file itself, not in its includes |
test_config |
lines of a config block added to the scenery, e.g. ["test.timeout 90"] |
fragment |
.scn file in cases/ added to the scenery: trainsets after FirstInit (before it they derail on unlinked track), the rest before it (launchers read after it never fire) |
vars |
values for {{name}} in the fragment; a list becomes one line per item |
expect |
the result the test mode should end with: pass, fail, signal_passed_at_danger, timeout, standstill or error. Default timeout for a case with duration, else pass |
duration |
seconds of simulation to record: the test timeout of the run (--test-timeout) |
timeout |
wall-clock seconds before giving up, loading included |
fail_on_errors |
fail when errors.txt is not empty (default false) |
checks |
list of checks, below |
{{vehicle}} is always set, also inside vars values: the node name of the lead vehicle,
the one whose physicslog is checked. Other vehicles of the consist take names made from it,
e.g. {{vehicle}}_s. The td.scn cases share two fragments, td_timetable.scn (route set,
train runs to Przystanek) and td_red_signal.scn (no route, exit signal at stop), and differ
only in vars.consist. td_red_signal_moving.scn is the red signal one with the train
starting at vars.velocity km/h; with track_friction 0.03 it locks the wheels when braking
and slides past the signal, so those cases set test.fail_if_stop_overran no.
td_red_signal_pass.scn adds the test_pass event, which test_mode_pass puts as event0
on tor1k, the track in front of the signal.
The test_mode_* cases show the ends of the test mode: test_mode_pass (exit 0, the pass
event), test_mode_signal_passed_at_danger (exit 2), test_mode_timeout (exit 3, from
test.timeout) and test_mode_standstill (exit 4, from test.standstill).
test_mode_pass_while_moving reports the pass from event2 of tor1k (the direction the train runs), as the train runs onto
it, with test.no_movement_before_pass no, so the test ends with the train still moving.
Checks (speeds in km/h, times in seconds of physicslog time, distances in metres). Most use the physicslog Velocity column, which is the wheel speed: it stays at 0 while the wheels are locked in a skid. The ones marked vehicle speed use the vehicle speed column at the end of the row instead, logged by builds from this branch on.
startsmin_speed,within: reachesmin_speedwithinwithinseconds.max_speedlimit,tolerance: never goes faster thanlimit + tolerance.brakesmin_pressure: after moving off, brake cylinder pressure reachesmin_pressurebar (physicslog pressures are in bar whatever its header says; the brake pipe runs at 5.0).stopsmoving_speed,stop_speed,hold: after reachingmoving_speed, stays at or belowstop_speedforholdseconds.stays_stoppedmoving_speed,stop_speed: after moving off and stopping, never goes abovestop_speedagain.distancemin,max: total distance travelled. The physicslog has no position, so the distance comes from integrating the speed; on td.scn every vehicle's first stop after Stacja1 came out at 1400 m (+-3), and the red signal stop at 140 m.slipswithin: the lead vehicle's wheels slip or lock at some point (withinwithinseconds if given), so a slippery rail case does what it's meant to.slip_clearsmoving_speed,stop_speed,within(vehicle speed): once stopped after moving off, the wheels stop slipping withinwithinseconds.holds_stillmoving_speed,stop_speed,max_distance(vehicle speed): once stopped after moving off, moves no more thanmax_distancemetres, e.g. no creeping on locked wheels.ends_movingmin_speed: the last row of the physicslog is atmin_speedor faster, e.g. the test ended without waiting for the train to stop.
New check types go in CHECKS in run.py; each gets the physicslog rows and the check's
parameters and returns (passed, detail).