mirror of
https://github.com/MaSzyna-EU07/maszyna.git
synced 2026-11-19 13:26:08 +01:00
main() tells a test run (--test) apart before anything runs. Only a test run catches every exception and installs a std::terminate handler, both ending the run with the error code and a test: line; any other run keeps master's exception handling, so on Linux an uncaught runtime_error aborts at the throw site again. A test run doesn't start the Discord RPC thread: it published the test scenery as the player's presence and held the closing up to 5 s. run.py drops a case scenery's .sbt cache after a run cut short (wall_timeout, error, a crash, killed by run.py), which could have been writing it. The -edit branch of the test command line parser had no effect (a token that isn't an option is ignored either way) and is gone; its test checks what can be seen instead.
250 lines
17 KiB
Markdown
250 lines
17 KiB
Markdown
# Scenario tests
|
|
|
|
End-to-end checks that run the real simulator on a scenery and look at how the run ended
|
|
and what a vehicle did: did it start, did it keep to the speed limit, did it brake and stop,
|
|
did it pass a signal at stop.
|
|
|
|
Scenery and vehicle data are not in this repo, so these run against a local game install
|
|
only, not in CI.
|
|
|
|
## The simulator's test mode
|
|
|
|
The runner drives the simulator's test mode (eu07/maszyna#541), which works without the
|
|
runner too and is meant for scenery authors as well:
|
|
|
|
```
|
|
eu07 --test -s <scenery.scn> [--test-timeout <seconds>] [--test-wall-timeout <seconds>] [-seed <number|word>] [-ini <settings file>]
|
|
```
|
|
|
|
With `--test` the simulator:
|
|
|
|
- needs `-s`; `-v` is ignored and every train is driven by AI (ghostview),
|
|
- reads the settings of the game folder's `eu07.ini` and of the files given with `-ini` (in that
|
|
order), not the player's own `eu07.ini` in `%APPDATA%\MaSzyna` (`~/.config/MaSzyna` on Linux),
|
|
so a test runs the same for whoever runs it,
|
|
- draws nothing: the null renderer loads the 3D models (the physics needs some of their
|
|
data) but no textures, and the window stays hidden,
|
|
- has no sound, no network, no python, no pause (not even one set by the scenery), and
|
|
no crash dialog,
|
|
- takes no input from outside: keys and mouse sent to its window, gamepad, UART (a cab
|
|
desk), ZeroMQ and commands sent by other programs (`WM_COPYDATA`) are all ignored, and
|
|
motion telemetry is off,
|
|
- gives nothing to the outside either: no keyboard leds, LPT or PoKeys (`feedbackmode`), no
|
|
messages to a multiplayer server (`multiplayer`), no program run at the end (`execonexit`),
|
|
no console window of its own (`showsystemconsole`), no Discord presence, and the game's `imgui.ini`
|
|
stays as it is,
|
|
- keeps all of the above whatever the `config` block of the scenery says: the settings of the
|
|
test mode are applied again after it,
|
|
- runs the simulation as fast as it can in fixed steps of 0.03 s of simulation time per
|
|
frame (three 10 ms physics steps), whatever the real time, the time speed or the machine,
|
|
- seeds its random numbers with a fixed seed unless `-seed` or `randomseed` gives one, so two
|
|
runs of a scenery come out the same (the physicslogs of two runs are byte for byte equal),
|
|
- quits when the test ends, and says how it ended in the exit code and in a `test: ...`
|
|
line on stdout and in the log. A command line it can't take, a start that fails, or an
|
|
exception thrown by the simulator (of any type, on Linux too) ends the same way, with the
|
|
`error` code. That line is all there is on stdout: the log goes to
|
|
the log file only (`debuglog` doesn't print it on the console in test mode).
|
|
|
|
On Windows the simulator is a windowed program, which the `cmd` prompt and PowerShell don't wait
|
|
for. In a `.bat` file `eu07 --test ... > result.txt` waits and sets `%errorlevel%`; at the prompt
|
|
`start /wait eu07 --test ...` waits too but doesn't pass the output on. In PowerShell use
|
|
`(Start-Process eu07 '--test ...' -Wait -PassThru -RedirectStandardOutput result.txt).ExitCode`.
|
|
Scripts (Python's `subprocess`, bash) wait for it as for any other program.
|
|
|
|
A scenery reports its result to the memory cell `__test.result`, which the simulator creates
|
|
(in every run, so a test scenery loads normally too). Write value 1 to it for a pass and 0
|
|
for a failure, with an `UpdateValues` event; other values are ignored, and the text, if
|
|
given, goes to the log as the reason:
|
|
|
|
```
|
|
event test_pass UpdateValues 0 __test.result reached_the_stop 1 * endevent
|
|
event test_fail UpdateValues 0 __test.result wrong_track 0 * endevent
|
|
```
|
|
|
|
(`PutValues` isn't the event for it: it hands a command to a vehicle, it doesn't write a cell.)
|
|
A good place for the pass event is `event0` of a track the train should stop on: it runs
|
|
while a train stands there, and the length of the track is the margin of the stop.
|
|
|
|
The test ends with the first of these:
|
|
|
|
| exit code | result | when |
|
|
| --- | --- | --- |
|
|
| 0 | pass | `__test.result` was set to 1 and then the tested trains stood still for 1 s, so a train still moving towards a signal at stop has to stop first (with `test.no_movement_before_pass no` the pass ends the test at once). Also the standstill end with `test.pass_on_standstill yes` |
|
|
| 10 | fail | `__test.result` was set to 0; ends at once |
|
|
| 11 | signal passed at danger | a train went past a signal showing stop (`SetVelocity 0 0`; `SetVelocity 0 x` on automatic block, which may be passed after a stop, doesn't count); ends at once, unless `test.fail_if_stop_overran no` |
|
|
| 12 | timeout | `test.timeout` seconds of simulation time went by (default 3600), or `test.wall_timeout` seconds of real time since the start (see below) |
|
|
| 13 | standstill | the tested trains stood still for `test.standstill` seconds (default 300) and nothing reported a result. Trains waiting at a station or for a signal count as standing, so the limit has to be longer than such waits |
|
|
| 14 | error | the run could not start: no `-s`, the scenery failed to load or has no vehicles, or the simulator threw |
|
|
|
|
A crash ends with the system's code instead (on Windows e.g. `0xC0000005`, -1073741819). The
|
|
codes of the test mode keep clear of the ones a process gets when it goes wrong on its own: 1
|
|
(killed with `TerminateProcess`, or a plain failure of most programs), 3 (`abort()` on Windows),
|
|
126-127 (a shell that can't run the program) and 128 + a signal number (killed, on Linux).
|
|
|
|
The limits go in a `config` block of the scenery:
|
|
|
|
```
|
|
config
|
|
test.timeout 600
|
|
test.wall_timeout 900
|
|
test.standstill 120
|
|
test.pass_on_standstill no
|
|
test.fail_if_stop_overran yes
|
|
test.no_movement_before_pass yes
|
|
test.vehicle ep07-424
|
|
endconfig
|
|
```
|
|
|
|
| key | default | meaning |
|
|
| --- | --- | --- |
|
|
| `test.timeout` | 3600 | seconds of simulation time before the test ends as `timeout`; 0 = never |
|
|
| `test.wall_timeout` | `test.timeout` + 600 | seconds of real time since the start before the test ends as `timeout`; 0 = never |
|
|
| `test.standstill` | 300 | seconds with the tested trains still and no result before the test ends as `standstill`; 0 = never |
|
|
| `test.pass_on_standstill` | no | the standstill end passes the test instead |
|
|
| `test.fail_if_stop_overran` | yes | a signal passed at danger ends the test; with `no` it's only logged |
|
|
| `test.no_movement_before_pass` | yes | a pass waits until the tested trains stand still; with `no` it ends the test at once |
|
|
| `test.vehicle` | none | a vehicle of a train under test, by its name in the scenery (once for each train under test). The tested trains are the consists of these vehicles as they are at the moment (a train the tested one couples to joins it), and only they count for the pass and the standstill: other trains, AI traffic, may run on. A name the scenery doesn't have ends the test with `error`. Not given, every vehicle of the scenery counts. A signal passed at danger counts for every train either way |
|
|
|
|
`--test-timeout` and `--test-wall-timeout` on the command line win over `test.timeout` and
|
|
`test.wall_timeout` of the scenery and the settings files.
|
|
|
|
`test.wall_timeout` is a watchdog in real time, for a run that hangs or crawls: when the
|
|
simulator still runs that many seconds after its start, loading included, it says `test: timeout`
|
|
and quits with the timeout code, whatever it is doing at the moment, with no dialog. Not given,
|
|
it is `test.timeout` + 600 s: ten minutes for the loading, and a simulation that runs at least as
|
|
fast as real time is never cut short by it. With `test.timeout 0` there is no watchdog unless
|
|
`test.wall_timeout` is given; `test.wall_timeout 0` turns it off. A test that has ended but whose
|
|
simulator doesn't close is ended by the watchdog too, with its own result.
|
|
|
|
A signal is passed at danger when the head of a train goes past a signal which its driver
|
|
has in its speed table, showing stop. So it's the driver's view that counts: a train without
|
|
a driver isn't watched, and neither is a signal the driver never scanned (a train placed at
|
|
100 km/h 150 m before a signal at stop got past it before its AI first looked ahead; at
|
|
60 km/h it was caught). Outside the test mode the same is written to the log, nothing more.
|
|
|
|
## Running
|
|
|
|
```
|
|
python tests/scenarios/run.py --game-dir "C:/Steam/steamapps/common/MaSzyna" -j 4
|
|
```
|
|
|
|
Python 3.10+, no extra packages. Options:
|
|
|
|
- `EU07_GAME_DIR` can replace `--game-dir`.
|
|
- `--exe path/to/eu07.exe` picks the build to test; the default is the newest
|
|
`cmake-build-*/bin/Release/eu07*.exe`. The build needs the `--test` and `-ini` command
|
|
line options.
|
|
- name one or more cases to run only those, e.g. `run.py ... eu07_td_timetable`.
|
|
- `-j N` runs N cases at once. Each case prints its results when it is done. All 17 cases
|
|
take about 30 s with `-j 4`, loading included.
|
|
- `--watch` runs a case the old way, in a visible window in real time (or faster with
|
|
`--timespeed N`), closed after the case's `duration`; the exit code isn't checked then. For
|
|
watching what a case does.
|
|
|
|
Exit code is 0 when every case passed. Logs of each run (`log.txt`, `errors.txt`,
|
|
`stdout.txt`, the physicslogs of the vehicle and of other trains of the case) are moved to
|
|
`tests/scenarios/out/<case>/`.
|
|
|
|
## How it works
|
|
|
|
For each case the runner:
|
|
|
|
1. writes `scenery/autotest_<case>.scn` into the game dir: the base scenery with its own
|
|
trainsets removed, the start time changed if the case says so, the case's track events
|
|
added, and the case's `.scn` fragment added: its trainsets after `FirstInit`, everything
|
|
else (the case's `test_config`, event launchers standing in for keys the scenery expects a
|
|
player to press, extra events) before it,
|
|
2. starts the simulator with `--test -s autotest_<case>.scn -ini out/<case>/autotest.ini
|
|
--test-wall-timeout <timeout>`, plus `--test-timeout <duration>` when the case gives a
|
|
`duration`; the extra ini turns on
|
|
`physicslog` and writes the logs to `logs/log (autotest_<case>) <date>.txt` rather than the
|
|
shared `log.txt` (`multiplelogs`),
|
|
3. waits until the simulator quits by itself. It gives up early when the log reports
|
|
`Bad vehicle` / `Bad trainset`, or when the scenario has loaded and no physicslog appeared.
|
|
After the case's `timeout` of wall-clock time the simulator ends itself with the timeout code;
|
|
one that doesn't is killed 30 s later,
|
|
4. compares the result from the exit code with the case's `expect`, then runs the checks on
|
|
the physicslog.
|
|
|
|
The lead vehicle is named `autotest_<case>`, so every case has its own physicslog and log
|
|
files and cases can run in parallel. The `autotest_*` sceneries (and the simulator's `.sbt`
|
|
geometry cache of them) stay in the game's `scenery/` dir; the cache is dropped when the base
|
|
scenery is newer, and after a run cut short (`wall_timeout`, `error`, a crash, or killed by run.py),
|
|
which could have been writing it.
|
|
|
|
## Writing a case
|
|
|
|
`cases/<name>.json`:
|
|
|
|
| key | meaning |
|
|
| --- | --- |
|
|
| `base_scenery` | scenery file in the game's `scenery/` dir |
|
|
| `strip_trainsets` | drop the base scenery's trainsets (default `true`) |
|
|
| `time` | replace the scenery start time, `HH:MM` |
|
|
| `track_friction` | set the friction of every track of the base scenery (default in the sceneries is 0.15), e.g. for slippery rail |
|
|
| `track_events` | lines added to tracks of the base scenery, by track name, e.g. `{ "tor1k": "event0 test_pass" }`; only tracks in the scenery file itself, not in its includes |
|
|
| `test_config` | lines of a `config` block added to the scenery, e.g. `["test.timeout 90"]`; the block always has `test.vehicle` with the case's lead vehicle, so only its train counts for the pass and the standstill |
|
|
| `fragment` | `.scn` file in `cases/` added to the scenery: trainsets after `FirstInit` (before it they derail on unlinked track), the rest before it (launchers read after it never fire) |
|
|
| `vars` | values for `{{name}}` in the fragment; a list becomes one line per item |
|
|
| `expect` | the result the test mode should end with: `pass`, `fail`, `signal_passed_at_danger`, `timeout`, `standstill` or `error`, or `wall_timeout` for the timeout of the real time watchdog (the timeout code with a verdict saying it was the real time; a case expecting `timeout` fails on it). Default `timeout` for a case with `duration`, else `pass` |
|
|
| `duration` | seconds of simulation to record: the test timeout of the run (`--test-timeout`) |
|
|
| `timeout` | wall-clock seconds before giving up, loading included: the simulator's `--test-wall-timeout` (default 600) |
|
|
| `fail_on_errors` | fail when `errors.txt` is not empty (default `false`) |
|
|
| `checks` | list of checks, below |
|
|
|
|
`{{vehicle}}` is always set, also inside `vars` values: the node name of the lead vehicle,
|
|
the one whose physicslog is checked, and the train under test (`test.vehicle`). Other vehicles
|
|
of the consist take names made from it, e.g. `{{vehicle}}_s`; another train with a driver is named
|
|
`<something>_{{vehicle}}`, so its physicslog is moved out of the game dir too. The td.scn cases share two fragments, `td_timetable.scn` (route set,
|
|
train runs to Przystanek) and `td_red_signal.scn` (no route, exit signal at stop), and differ
|
|
only in `vars.consist`. Przystanek is a technical stop in that timetable (`pt`), where the AI
|
|
doesn't wait for the departure time: it sets off as soon as it counts the train as standing at
|
|
the W4, which is below 1 km/h, so the train stands there for a few seconds at most, or doesn't
|
|
quite stop (since the AI braking of #264 the SN61 turns back at 0.4 km/h). The
|
|
timetable cases therefore check that the train comes down to 1 km/h, with no hold, 1380-1420 m
|
|
from the start (each of the seven trains stops at 1397-1401 m). `td_red_signal_moving.scn` is the red signal one with the train
|
|
starting at `vars.velocity` km/h; with `track_friction` 0.03 it locks the wheels when braking
|
|
and slides past the signal, so those cases set `test.fail_if_stop_overran no`.
|
|
`td_red_signal_pass.scn` adds the `test_pass` event, which `test_mode_pass` puts as `event0`
|
|
on `tor1k`, the track in front of the signal.
|
|
|
|
The `test_mode_*` cases show the ends of the test mode: `test_mode_pass` (exit 0, the pass
|
|
event), `test_mode_signal_passed_at_danger` (exit 11), `test_mode_timeout` (exit 12, from
|
|
`test.timeout`), `test_mode_standstill` (exit 13, from `test.standstill`) and
|
|
`test_mode_wall_timeout` (exit 12 from the real time watchdog, which ends it while it loads).
|
|
`test_mode_standstill_traffic` (fragment `td_traffic.scn`) has a train under test standing on
|
|
track 2 of Stacja1 and AI traffic, `traffic_{{vehicle}}`, leaving track 1 on the timetable: the
|
|
standstill of 100 s ends the test although the traffic runs on (counting every vehicle, it runs
|
|
into the test timeout).
|
|
`test_mode_pass_while_moving` reports the pass from `event2` of `tor1k` (the direction the train runs), as the train runs onto
|
|
it, with `test.no_movement_before_pass no`, so the test ends with the train still moving.
|
|
|
|
Checks (speeds in km/h, times in seconds of physicslog time, distances in metres). Most use
|
|
the physicslog Velocity column, which is the wheel speed: it stays at 0 while the wheels are
|
|
locked in a skid. The ones marked *vehicle speed* use the vehicle speed column at the end of
|
|
the row instead, logged by builds from this branch on.
|
|
|
|
- `starts` `min_speed`, `within`: reaches `min_speed` within `within` seconds.
|
|
- `max_speed` `limit`, `tolerance`: never goes faster than `limit + tolerance`.
|
|
- `brakes` `min_pressure`: after moving off, brake cylinder pressure reaches `min_pressure` bar
|
|
(physicslog pressures are in bar whatever its header says; the brake pipe runs at 5.0).
|
|
- `stops` `moving_speed`, `stop_speed`, `hold`, optional `min_distance`, `max_distance`: after
|
|
reaching `moving_speed`, stays at or below `stop_speed` for `hold` seconds; with the distances,
|
|
that stop has to be between them (metres travelled, as in `distance`).
|
|
- `stays_stopped` `moving_speed`, `stop_speed`: after moving off and stopping, never goes
|
|
above `stop_speed` again.
|
|
- `distance` `min`, `max`: total distance travelled. The physicslog has no position, so the
|
|
distance comes from integrating the speed; on td.scn every vehicle's first stop after
|
|
Stacja1 came out at 1400 m (+-4), and the red signal stop at 140 m.
|
|
- `slips` `within`: the lead vehicle's wheels slip or lock at some point (within `within`
|
|
seconds if given), so a slippery rail case does what it's meant to.
|
|
- `slip_clears` `moving_speed`, `stop_speed`, `within` (*vehicle speed*): once stopped after
|
|
moving off, the wheels stop slipping within `within` seconds.
|
|
- `holds_still` `moving_speed`, `stop_speed`, `max_distance` (*vehicle speed*): once stopped
|
|
after moving off, moves no more than `max_distance` metres, e.g. no creeping on locked wheels.
|
|
- `ends_moving` `min_speed`: the last row of the physicslog is at `min_speed` or faster, e.g.
|
|
the test ended without waiting for the train to stop.
|
|
|
|
New check types go in `CHECKS` in `run.py`; each gets the physicslog rows and the check's
|
|
parameters and returns `(passed, detail)`.
|