18
0
mirror of https://github.com/MaSzyna-EU07/maszyna.git synced 2026-11-19 04:05:58 +01:00

Merge pull request #266 from MaSzyna-EU07/feat/exe-test-mode

Test mode: `--test` runs a scenery unattended and exits with the result (#541)
This commit is contained in:
maj00r
2026-11-08 20:41:57 +01:00
committed by GitHub
47 changed files with 2597 additions and 33 deletions

2
tests/scenarios/.gitignore vendored Normal file
View File

@@ -0,0 +1,2 @@
out/
__pycache__/

249
tests/scenarios/README.md Normal file
View File

@@ -0,0 +1,249 @@
# Scenario tests
End-to-end checks that run the real simulator on a scenery and look at how the run ended
and what a vehicle did: did it start, did it keep to the speed limit, did it brake and stop,
did it pass a signal at stop.
Scenery and vehicle data are not in this repo, so these run against a local game install
only, not in CI.
## The simulator's test mode
The runner drives the simulator's test mode (eu07/maszyna#541), which works without the
runner too and is meant for scenery authors as well:
```
eu07 --test -s <scenery.scn> [--test-timeout <seconds>] [--test-wall-timeout <seconds>] [-seed <number|word>] [-ini <settings file>]
```
With `--test` the simulator:
- needs `-s`; `-v` is ignored and every train is driven by AI (ghostview),
- reads the settings of the game folder's `eu07.ini` and of the files given with `-ini` (in that
order), not the player's own `eu07.ini` in `%APPDATA%\MaSzyna` (`~/.config/MaSzyna` on Linux),
so a test runs the same for whoever runs it,
- draws nothing: the null renderer loads the 3D models (the physics needs some of their
data) but no textures, and the window stays hidden,
- has no sound, no network, no python, no pause (not even one set by the scenery), and
no crash dialog,
- takes no input from outside: keys and mouse sent to its window, gamepad, UART (a cab
desk), ZeroMQ and commands sent by other programs (`WM_COPYDATA`) are all ignored, and
motion telemetry is off,
- gives nothing to the outside either: no keyboard leds, LPT or PoKeys (`feedbackmode`), no
messages to a multiplayer server (`multiplayer`), no program run at the end (`execonexit`),
no console window of its own (`showsystemconsole`), no Discord presence, and the game's `imgui.ini`
stays as it is,
- keeps all of the above whatever the `config` block of the scenery says: the settings of the
test mode are applied again after it,
- runs the simulation as fast as it can in fixed steps of 0.03 s of simulation time per
frame (three 10 ms physics steps), whatever the real time, the time speed or the machine,
- seeds its random numbers with a fixed seed unless `-seed` or `randomseed` gives one, so two
runs of a scenery come out the same (the physicslogs of two runs are byte for byte equal),
- quits when the test ends, and says how it ended in the exit code and in a `test: ...`
line on stdout and in the log. A command line it can't take, a start that fails, or an
exception thrown by the simulator (of any type, on Linux too) ends the same way, with the
`error` code. That line is all there is on stdout: the log goes to
the log file only (`debuglog` doesn't print it on the console in test mode).
On Windows the simulator is a windowed program, which the `cmd` prompt and PowerShell don't wait
for. In a `.bat` file `eu07 --test ... > result.txt` waits and sets `%errorlevel%`; at the prompt
`start /wait eu07 --test ...` waits too but doesn't pass the output on. In PowerShell use
`(Start-Process eu07 '--test ...' -Wait -PassThru -RedirectStandardOutput result.txt).ExitCode`.
Scripts (Python's `subprocess`, bash) wait for it as for any other program.
A scenery reports its result to the memory cell `__test.result`, which the simulator creates
(in every run, so a test scenery loads normally too). Write value 1 to it for a pass and 0
for a failure, with an `UpdateValues` event; other values are ignored, and the text, if
given, goes to the log as the reason:
```
event test_pass UpdateValues 0 __test.result reached_the_stop 1 * endevent
event test_fail UpdateValues 0 __test.result wrong_track 0 * endevent
```
(`PutValues` isn't the event for it: it hands a command to a vehicle, it doesn't write a cell.)
A good place for the pass event is `event0` of a track the train should stop on: it runs
while a train stands there, and the length of the track is the margin of the stop.
The test ends with the first of these:
| exit code | result | when |
| --- | --- | --- |
| 0 | pass | `__test.result` was set to 1 and then the tested trains stood still for 1 s, so a train still moving towards a signal at stop has to stop first (with `test.no_movement_before_pass no` the pass ends the test at once). Also the standstill end with `test.pass_on_standstill yes` |
| 10 | fail | `__test.result` was set to 0; ends at once |
| 11 | signal passed at danger | a train went past a signal showing stop (`SetVelocity 0 0`; `SetVelocity 0 x` on automatic block, which may be passed after a stop, doesn't count); ends at once, unless `test.fail_if_stop_overran no` |
| 12 | timeout | `test.timeout` seconds of simulation time went by (default 3600), or `test.wall_timeout` seconds of real time since the start (see below) |
| 13 | standstill | the tested trains stood still for `test.standstill` seconds (default 300) and nothing reported a result. Trains waiting at a station or for a signal count as standing, so the limit has to be longer than such waits |
| 14 | error | the run could not start: no `-s`, the scenery failed to load or has no vehicles, or the simulator threw |
A crash ends with the system's code instead (on Windows e.g. `0xC0000005`, -1073741819). The
codes of the test mode keep clear of the ones a process gets when it goes wrong on its own: 1
(killed with `TerminateProcess`, or a plain failure of most programs), 3 (`abort()` on Windows),
126-127 (a shell that can't run the program) and 128 + a signal number (killed, on Linux).
The limits go in a `config` block of the scenery:
```
config
test.timeout 600
test.wall_timeout 900
test.standstill 120
test.pass_on_standstill no
test.fail_if_stop_overran yes
test.no_movement_before_pass yes
test.vehicle ep07-424
endconfig
```
| key | default | meaning |
| --- | --- | --- |
| `test.timeout` | 3600 | seconds of simulation time before the test ends as `timeout`; 0 = never |
| `test.wall_timeout` | `test.timeout` + 600 | seconds of real time since the start before the test ends as `timeout`; 0 = never |
| `test.standstill` | 300 | seconds with the tested trains still and no result before the test ends as `standstill`; 0 = never |
| `test.pass_on_standstill` | no | the standstill end passes the test instead |
| `test.fail_if_stop_overran` | yes | a signal passed at danger ends the test; with `no` it's only logged |
| `test.no_movement_before_pass` | yes | a pass waits until the tested trains stand still; with `no` it ends the test at once |
| `test.vehicle` | none | a vehicle of a train under test, by its name in the scenery (once for each train under test). The tested trains are the consists of these vehicles as they are at the moment (a train the tested one couples to joins it), and only they count for the pass and the standstill: other trains, AI traffic, may run on. A name the scenery doesn't have ends the test with `error`. Not given, every vehicle of the scenery counts. A signal passed at danger counts for every train either way |
`--test-timeout` and `--test-wall-timeout` on the command line win over `test.timeout` and
`test.wall_timeout` of the scenery and the settings files.
`test.wall_timeout` is a watchdog in real time, for a run that hangs or crawls: when the
simulator still runs that many seconds after its start, loading included, it says `test: timeout`
and quits with the timeout code, whatever it is doing at the moment, with no dialog. Not given,
it is `test.timeout` + 600 s: ten minutes for the loading, and a simulation that runs at least as
fast as real time is never cut short by it. With `test.timeout 0` there is no watchdog unless
`test.wall_timeout` is given; `test.wall_timeout 0` turns it off. A test that has ended but whose
simulator doesn't close is ended by the watchdog too, with its own result.
A signal is passed at danger when the head of a train goes past a signal which its driver
has in its speed table, showing stop. So it's the driver's view that counts: a train without
a driver isn't watched, and neither is a signal the driver never scanned (a train placed at
100 km/h 150 m before a signal at stop got past it before its AI first looked ahead; at
60 km/h it was caught). Outside the test mode the same is written to the log, nothing more.
## Running
```
python tests/scenarios/run.py --game-dir "C:/Steam/steamapps/common/MaSzyna" -j 4
```
Python 3.10+, no extra packages. Options:
- `EU07_GAME_DIR` can replace `--game-dir`.
- `--exe path/to/eu07.exe` picks the build to test; the default is the newest
`cmake-build-*/bin/Release/eu07*.exe`. The build needs the `--test` and `-ini` command
line options.
- name one or more cases to run only those, e.g. `run.py ... eu07_td_timetable`.
- `-j N` runs N cases at once. Each case prints its results when it is done. All 17 cases
take about 30 s with `-j 4`, loading included.
- `--watch` runs a case the old way, in a visible window in real time (or faster with
`--timespeed N`), closed after the case's `duration`; the exit code isn't checked then. For
watching what a case does.
Exit code is 0 when every case passed. Logs of each run (`log.txt`, `errors.txt`,
`stdout.txt`, the physicslogs of the vehicle and of other trains of the case) are moved to
`tests/scenarios/out/<case>/`.
## How it works
For each case the runner:
1. writes `scenery/autotest_<case>.scn` into the game dir: the base scenery with its own
trainsets removed, the start time changed if the case says so, the case's track events
added, and the case's `.scn` fragment added: its trainsets after `FirstInit`, everything
else (the case's `test_config`, event launchers standing in for keys the scenery expects a
player to press, extra events) before it,
2. starts the simulator with `--test -s autotest_<case>.scn -ini out/<case>/autotest.ini
--test-wall-timeout <timeout>`, plus `--test-timeout <duration>` when the case gives a
`duration`; the extra ini turns on
`physicslog` and writes the logs to `logs/log (autotest_<case>) <date>.txt` rather than the
shared `log.txt` (`multiplelogs`),
3. waits until the simulator quits by itself. It gives up early when the log reports
`Bad vehicle` / `Bad trainset`, or when the scenario has loaded and no physicslog appeared.
After the case's `timeout` of wall-clock time the simulator ends itself with the timeout code;
one that doesn't is killed 30 s later,
4. compares the result from the exit code with the case's `expect`, then runs the checks on
the physicslog.
The lead vehicle is named `autotest_<case>`, so every case has its own physicslog and log
files and cases can run in parallel. The `autotest_*` sceneries (and the simulator's `.sbt`
geometry cache of them) stay in the game's `scenery/` dir; the cache is dropped when the base
scenery is newer, and after a run cut short (`wall_timeout`, `error`, a crash, or killed by run.py),
which could have been writing it.
## Writing a case
`cases/<name>.json`:
| key | meaning |
| --- | --- |
| `base_scenery` | scenery file in the game's `scenery/` dir |
| `strip_trainsets` | drop the base scenery's trainsets (default `true`) |
| `time` | replace the scenery start time, `HH:MM` |
| `track_friction` | set the friction of every track of the base scenery (default in the sceneries is 0.15), e.g. for slippery rail |
| `track_events` | lines added to tracks of the base scenery, by track name, e.g. `{ "tor1k": "event0 test_pass" }`; only tracks in the scenery file itself, not in its includes |
| `test_config` | lines of a `config` block added to the scenery, e.g. `["test.timeout 90"]`; the block always has `test.vehicle` with the case's lead vehicle, so only its train counts for the pass and the standstill |
| `fragment` | `.scn` file in `cases/` added to the scenery: trainsets after `FirstInit` (before it they derail on unlinked track), the rest before it (launchers read after it never fire) |
| `vars` | values for `{{name}}` in the fragment; a list becomes one line per item |
| `expect` | the result the test mode should end with: `pass`, `fail`, `signal_passed_at_danger`, `timeout`, `standstill` or `error`, or `wall_timeout` for the timeout of the real time watchdog (the timeout code with a verdict saying it was the real time; a case expecting `timeout` fails on it). Default `timeout` for a case with `duration`, else `pass` |
| `duration` | seconds of simulation to record: the test timeout of the run (`--test-timeout`) |
| `timeout` | wall-clock seconds before giving up, loading included: the simulator's `--test-wall-timeout` (default 600) |
| `fail_on_errors` | fail when `errors.txt` is not empty (default `false`) |
| `checks` | list of checks, below |
`{{vehicle}}` is always set, also inside `vars` values: the node name of the lead vehicle,
the one whose physicslog is checked, and the train under test (`test.vehicle`). Other vehicles
of the consist take names made from it, e.g. `{{vehicle}}_s`; another train with a driver is named
`<something>_{{vehicle}}`, so its physicslog is moved out of the game dir too. The td.scn cases share two fragments, `td_timetable.scn` (route set,
train runs to Przystanek) and `td_red_signal.scn` (no route, exit signal at stop), and differ
only in `vars.consist`. Przystanek is a technical stop in that timetable (`pt`), where the AI
doesn't wait for the departure time: it sets off as soon as it counts the train as standing at
the W4, which is below 1 km/h, so the train stands there for a few seconds at most, or doesn't
quite stop (since the AI braking of #264 the SN61 turns back at 0.4 km/h). The
timetable cases therefore check that the train comes down to 1 km/h, with no hold, 1380-1420 m
from the start (each of the seven trains stops at 1397-1401 m). `td_red_signal_moving.scn` is the red signal one with the train
starting at `vars.velocity` km/h; with `track_friction` 0.03 it locks the wheels when braking
and slides past the signal, so those cases set `test.fail_if_stop_overran no`.
`td_red_signal_pass.scn` adds the `test_pass` event, which `test_mode_pass` puts as `event0`
on `tor1k`, the track in front of the signal.
The `test_mode_*` cases show the ends of the test mode: `test_mode_pass` (exit 0, the pass
event), `test_mode_signal_passed_at_danger` (exit 11), `test_mode_timeout` (exit 12, from
`test.timeout`), `test_mode_standstill` (exit 13, from `test.standstill`) and
`test_mode_wall_timeout` (exit 12 from the real time watchdog, which ends it while it loads).
`test_mode_standstill_traffic` (fragment `td_traffic.scn`) has a train under test standing on
track 2 of Stacja1 and AI traffic, `traffic_{{vehicle}}`, leaving track 1 on the timetable: the
standstill of 100 s ends the test although the traffic runs on (counting every vehicle, it runs
into the test timeout).
`test_mode_pass_while_moving` reports the pass from `event2` of `tor1k` (the direction the train runs), as the train runs onto
it, with `test.no_movement_before_pass no`, so the test ends with the train still moving.
Checks (speeds in km/h, times in seconds of physicslog time, distances in metres). Most use
the physicslog Velocity column, which is the wheel speed: it stays at 0 while the wheels are
locked in a skid. The ones marked *vehicle speed* use the vehicle speed column at the end of
the row instead, logged by builds from this branch on.
- `starts` `min_speed`, `within`: reaches `min_speed` within `within` seconds.
- `max_speed` `limit`, `tolerance`: never goes faster than `limit + tolerance`.
- `brakes` `min_pressure`: after moving off, brake cylinder pressure reaches `min_pressure` bar
(physicslog pressures are in bar whatever its header says; the brake pipe runs at 5.0).
- `stops` `moving_speed`, `stop_speed`, `hold`, optional `min_distance`, `max_distance`: after
reaching `moving_speed`, stays at or below `stop_speed` for `hold` seconds; with the distances,
that stop has to be between them (metres travelled, as in `distance`).
- `stays_stopped` `moving_speed`, `stop_speed`: after moving off and stopping, never goes
above `stop_speed` again.
- `distance` `min`, `max`: total distance travelled. The physicslog has no position, so the
distance comes from integrating the speed; on td.scn every vehicle's first stop after
Stacja1 came out at 1400 m (+-4), and the red signal stop at 140 m.
- `slips` `within`: the lead vehicle's wheels slip or lock at some point (within `within`
seconds if given), so a slippery rail case does what it's meant to.
- `slip_clears` `moving_speed`, `stop_speed`, `within` (*vehicle speed*): once stopped after
moving off, the wheels stop slipping within `within` seconds.
- `holds_still` `moving_speed`, `stop_speed`, `max_distance` (*vehicle speed*): once stopped
after moving off, moves no more than `max_distance` metres, e.g. no creeping on locked wheels.
- `ends_moving` `min_speed`: the last row of the physicslog is at `min_speed` or faster, e.g.
the test ended without waiting for the train to stop.
New check types go in `CHECKS` in `run.py`; each gets the physicslog rows and the check's
parameters and returns `(passed, detail)`.

View File

@@ -0,0 +1,21 @@
{
"description": "EN57 EMU (three cars) on td.scn, AI driver on the td/rozklad timetable: starts from Stacja1, keeps to the speed limit, brakes to a stop at Przystanek.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_timetable.scn",
"vars": {
"consist": [
"node -1 0 {{vehicle}} dynamic pkp\\en57_v2 EN57-1000ra EN57RA_GSO_WKM1_KM_NS_N 0 headdriver 255 0 enddynamic",
"node -1 0 {{vehicle}}_s dynamic pkp\\en57_v2 EN57-1000s EN57S_WKM1_KM_NS_S 0 nobody 255 0 enddynamic",
"node -1 0 {{vehicle}}_rb dynamic pkp\\en57_v2 EN57-1000rb EN57RB_GSO_WKM1_KM_NS 0 nobody 0 0 enddynamic"
]
},
"duration": 240,
"timeout": 600,
"checks": [
{ "type": "starts", "min_speed": 20, "within": 150 },
{ "type": "max_speed", "limit": 90, "tolerance": 3 },
{ "type": "brakes", "min_pressure": 0.5 },
{ "type": "stops", "moving_speed": 20, "stop_speed": 1.0, "hold": 0, "min_distance": 1380, "max_distance": 1420 }
]
}

View File

@@ -0,0 +1,17 @@
{
"description": "EP09 on td.scn, AI driver on the td/rozklad timetable: starts from Stacja1, keeps to the speed limit, brakes to a stop at Przystanek.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_timetable.scn",
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\ep09_v1 104e-005 104e_3 0 headdriver 0 0 enddynamic"
},
"duration": 240,
"timeout": 600,
"checks": [
{ "type": "starts", "min_speed": 20, "within": 150 },
{ "type": "max_speed", "limit": 90, "tolerance": 3 },
{ "type": "brakes", "min_pressure": 0.5 },
{ "type": "stops", "moving_speed": 20, "stop_speed": 1.0, "hold": 0, "min_distance": 1380, "max_distance": 1420 }
]
}

View File

@@ -0,0 +1,17 @@
{
"description": "ET22 on td.scn, AI driver on the td/rozklad timetable: starts from Stacja1, keeps to the speed limit, brakes to a stop at Przystanek.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_timetable.scn",
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\et22_v3 201E-1150 201E 0 headdriver 0 0 enddynamic"
},
"duration": 240,
"timeout": 600,
"checks": [
{ "type": "starts", "min_speed": 20, "within": 150 },
{ "type": "max_speed", "limit": 90, "tolerance": 3 },
{ "type": "brakes", "min_pressure": 0.5 },
{ "type": "stops", "moving_speed": 20, "stop_speed": 1.0, "hold": 0, "min_distance": 1380, "max_distance": 1420 }
]
}

View File

@@ -0,0 +1,39 @@
{
"description": "EU07 pushing three coaches on td.scn with slippery rail and no route set, starting at 30 km/h: brakes for the exit signal at stop and skids. Pushed by the train it doesn't roll back at the stop, so it used to stay skidding on locked wheels with the brakes capped by the sliding adhesion and creep (eu07/maszyna#479).",
"base_scenery": "td.scn",
"time": "10:34",
"track_friction": 0.03,
"fragment": "td_red_signal_moving.scn",
"test_config": [
"test.fail_if_stop_overran no"
],
"vars": {
"velocity": "30",
"consist": [
"node -1 0 {{vehicle}}_w1 dynamic pkp\\bdhpumn_v3 bdhpumn_50512608068-6 BDHPUMN_TAPICERKA_1 0 nobody 115 0 enddynamic",
"node -1 0 {{vehicle}}_w2 dynamic pkp\\bdhpumn_v3 bdhpumn_50512608011-6 BDHPUMN_NG_1 0 nobody 115 0 enddynamic",
"node -1 0 {{vehicle}}_w3 dynamic pkp\\bdhpumn_v3 bdhpumn_50512608068-6 BDHPUMN_TAPICERKA_1 0 nobody 115 0 enddynamic",
"node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
]
},
"duration": 240,
"timeout": 900,
"checks": [
{
"type": "slips",
"within": 60
},
{
"type": "slip_clears",
"moving_speed": 5,
"stop_speed": 0.05,
"within": 5
},
{
"type": "holds_still",
"moving_speed": 5,
"stop_speed": 0.05,
"max_distance": 0.01
}
]
}

View File

@@ -0,0 +1,39 @@
{
"description": "EU07 with three coaches on td.scn with slippery rail and no route set, starting at 30 km/h: brakes for the exit signal at stop and skids, then the skid ends and the train holds still.",
"base_scenery": "td.scn",
"time": "10:34",
"track_friction": 0.03,
"fragment": "td_red_signal_moving.scn",
"test_config": [
"test.fail_if_stop_overran no"
],
"vars": {
"velocity": "30",
"consist": [
"node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic",
"node -1 0 {{vehicle}}_w1 dynamic pkp\\bdhpumn_v3 bdhpumn_50512608068-6 BDHPUMN_TAPICERKA_1 0 nobody 115 0 enddynamic",
"node -1 0 {{vehicle}}_w2 dynamic pkp\\bdhpumn_v3 bdhpumn_50512608011-6 BDHPUMN_NG_1 0 nobody 115 0 enddynamic",
"node -1 0 {{vehicle}}_w3 dynamic pkp\\bdhpumn_v3 bdhpumn_50512608068-6 BDHPUMN_TAPICERKA_1 0 nobody 115 0 enddynamic"
]
},
"duration": 240,
"timeout": 900,
"checks": [
{
"type": "slips",
"within": 60
},
{
"type": "slip_clears",
"moving_speed": 5,
"stop_speed": 0.05,
"within": 5
},
{
"type": "holds_still",
"moving_speed": 5,
"stop_speed": 0.05,
"max_distance": 0.01
}
]
}

View File

@@ -0,0 +1,18 @@
{
"description": "EU07 on td.scn with no route set: pulls off, stops before the exit signal stacja1_j at stop, and stays there.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_red_signal.scn",
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
},
"duration": 240,
"timeout": 600,
"checks": [
{ "type": "starts", "min_speed": 5, "within": 150 },
{ "type": "max_speed", "limit": 40 },
{ "type": "stops", "moving_speed": 5, "stop_speed": 0.5, "hold": 5 },
{ "type": "stays_stopped", "moving_speed": 5, "stop_speed": 0.5 },
{ "type": "distance", "max": 150 }
]
}

View File

@@ -0,0 +1,17 @@
{
"description": "EU07 on td.scn, AI driver on the td/rozklad timetable: starts from Stacja1, keeps to the speed limit, brakes to a stop at Przystanek.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_timetable.scn",
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
},
"duration": 240,
"timeout": 600,
"checks": [
{ "type": "starts", "min_speed": 20, "within": 150 },
{ "type": "max_speed", "limit": 90, "tolerance": 3 },
{ "type": "brakes", "min_pressure": 0.5 },
{ "type": "stops", "moving_speed": 20, "stop_speed": 1.0, "hold": 0, "min_distance": 1380, "max_distance": 1420 }
]
}

View File

@@ -0,0 +1,17 @@
{
"description": "SM42 diesel shunter on td.scn, AI driver on the td/rozklad timetable: starts from Stacja1, keeps to the speed limit, brakes to a stop at Przystanek.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_timetable.scn",
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\sm42_v1 6D-468 6D 0 headdriver 0 0 enddynamic"
},
"duration": 240,
"timeout": 600,
"checks": [
{ "type": "starts", "min_speed": 20, "within": 150 },
{ "type": "max_speed", "limit": 90, "tolerance": 3 },
{ "type": "brakes", "min_pressure": 0.5 },
{ "type": "stops", "moving_speed": 20, "stop_speed": 1.0, "hold": 0, "min_distance": 1380, "max_distance": 1420 }
]
}

View File

@@ -0,0 +1,17 @@
{
"description": "SN61 diesel railcar on td.scn, AI driver on the td/rozklad timetable: starts from Stacja1, keeps to the speed limit, brakes to a stop at Przystanek.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_timetable.scn",
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\sn61_v2 SN61-13 SN61_v1 0 headdriver 0 0 enddynamic"
},
"duration": 240,
"timeout": 600,
"checks": [
{ "type": "starts", "min_speed": 20, "within": 150 },
{ "type": "max_speed", "limit": 90, "tolerance": 3 },
{ "type": "brakes", "min_pressure": 0.5 },
{ "type": "stops", "moving_speed": 20, "stop_speed": 1.0, "hold": 0, "min_distance": 1380, "max_distance": 1420 }
]
}

View File

@@ -0,0 +1,6 @@
// added by tests/scenarios/run.py to td.scn, whose own trainsets were removed.
// same start as td_timetable.scn, but nobody sets the route: the exit signal stacja1_j,
// about 150 m ahead of the train, stays at stop.
trainset td/rozklad tdo_n1734 0.0 0.1
{{consist}}
endtrainset

View File

@@ -0,0 +1,6 @@
// added by tests/scenarios/run.py to td.scn, whose own trainsets were removed.
// like td_red_signal.scn (no route, the exit signal stacja1_j about 150 m ahead stays at stop),
// but the train starts already moving at {{velocity}} km/h, so it has to brake right away.
trainset td/rozklad tdo_n1734 0.0 {{velocity}}
{{consist}}
endtrainset

View File

@@ -0,0 +1,8 @@
// added by tests/scenarios/run.py to td.scn, whose own trainsets were removed.
// like td_red_signal.scn (no route, the exit signal stacja1_j about 150 m ahead stays at stop),
// with the test mode's pass event: the case puts it as event0 on tor1k, the track in front of
// the signal, so it runs once the train stands there. the test passes when every vehicle stopped.
event test_pass UpdateValues 0 __test.result stopped_before_stacja1_j 1 * endevent
trainset td/rozklad tdo_n1734 0.0 0.1
{{consist}}
endtrainset

View File

@@ -0,0 +1,12 @@
// added by tests/scenarios/run.py to td.scn, whose own trainsets were removed:
// the trainset goes after FirstInit, the launcher before it.
// the case's consist stands where td.scn keeps its timetable train (track 1 of Stacja1)
// and gets the same timetable: Stacja1 10:35 -> Przystanek -> Stacja2.
trainset td/rozklad tdo_n1734 0.0 0.1
{{consist}}
endtrainset
// td.scn sets routes from the keyboard (shift+1..4 run events keyctrl01..04). keyctrl04 is the
// way out of track 1: switch zwr02 and the exit signal stacja1_j at 40.
// a launcher with a positive time fires once at that hour (HHMM), radius -1 means anywhere.
node -1 0 autotest_route eventlauncher 0 0 0 -1 none 1034 keyctrl04 end

View File

@@ -0,0 +1,13 @@
// added by tests/scenarios/run.py to td.scn, whose own trainsets were removed: two trains at Stacja1.
// the train under test (test.vehicle) stands on track 2, where its driver has nothing to do; the other one,
// AI traffic, has the td/rozklad timetable on track 1 and leaves for Przystanek at 10:35. the standstill of
// the test waits for the train under test only, so the traffic running on doesn't hold it up.
trainset none tor2k 30.0 0.0
node -1 0 {{vehicle}} dynamic pkp\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic
endtrainset
trainset td/rozklad tdo_n1734 0.0 0.1
node -1 0 traffic_{{vehicle}} dynamic pkp\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic
endtrainset
// the way out of track 1 for the traffic, see td_timetable.scn
node -1 0 autotest_route eventlauncher 0 0 0 -1 none 1034 keyctrl04 end

View File

@@ -0,0 +1,31 @@
{
"description": "Test mode, pass: EU07 on td.scn with no route set stops before the exit signal stacja1_j; event0 of the track there reports the pass, and the test ends as passed once the train stands.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_red_signal_pass.scn",
"track_events": {
"tor1k": "event0 test_pass"
},
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
},
"expect": "pass",
"timeout": 600,
"checks": [
{
"type": "starts",
"min_speed": 5,
"within": 150
},
{
"type": "stops",
"moving_speed": 5,
"stop_speed": 0.5,
"hold": 0.5
},
{
"type": "distance",
"max": 150
}
]
}

View File

@@ -0,0 +1,28 @@
{
"description": "Test mode, pass without waiting for a stop: like test_mode_pass, but event2 of the track in front of the exit signal reports the pass as the train runs onto it, and with test.no_movement_before_pass no the test ends at once, the train still moving.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_red_signal_pass.scn",
"track_events": {
"tor1k": "event2 test_pass"
},
"test_config": [
"test.no_movement_before_pass no"
],
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
},
"expect": "pass",
"timeout": 600,
"checks": [
{
"type": "starts",
"min_speed": 5,
"within": 150
},
{
"type": "ends_moving",
"min_speed": 3
}
]
}

View File

@@ -0,0 +1,22 @@
{
"description": "Test mode, signal passed at danger: EU07 on td.scn with no route set starts at 60 km/h about 150 m before the exit signal stacja1_j at stop. Its AI driver is still preparing the engine and doesn't brake, and the run ends when the train passes the signal.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_red_signal_moving.scn",
"vars": {
"velocity": "60",
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
},
"expect": "signal_passed_at_danger",
"timeout": 600,
"checks": [
{
"type": "max_speed",
"limit": 61
},
{
"type": "distance",
"min": 140
}
]
}

View File

@@ -0,0 +1,26 @@
{
"description": "Test mode, standstill: EU07 on td.scn with no route set stops before the exit signal stacja1_j at stop and stays; nothing reports a result, so the test ends once the train under test stood still for test.standstill 120 seconds (the train waits about 70 s before it pulls off, so less would end it before the start).",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_red_signal.scn",
"test_config": [
"test.standstill 120"
],
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
},
"expect": "standstill",
"timeout": 600,
"checks": [
{
"type": "stops",
"moving_speed": 5,
"stop_speed": 0.5,
"hold": 5
},
{
"type": "distance",
"max": 150
}
]
}

View File

@@ -0,0 +1,18 @@
{
"description": "Test mode, standstill with traffic: an EU07 stands on track 2 of Stacja1 on td.scn while another EU07, AI traffic, leaves track 1 on the td/rozklad timetable at 10:35 and runs on. Only the train under test (test.vehicle, the case's vehicle) counts, so the test ends on test.standstill 100 although the traffic is moving; counting every vehicle, it would run into the 300 s test timeout.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_traffic.scn",
"test_config": [
"test.standstill 100"
],
"expect": "standstill",
"duration": 300,
"timeout": 600,
"checks": [
{
"type": "max_speed",
"limit": 0.5
}
]
}

View File

@@ -0,0 +1,21 @@
{
"description": "Test mode, timeout: EU07 on td.scn with no route set pulls off towards the exit signal stacja1_j at stop; nothing reports a result and test.timeout 90 from the scenery's config block runs out first.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_red_signal.scn",
"test_config": [
"test.timeout 90"
],
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
},
"expect": "timeout",
"timeout": 600,
"checks": [
{
"type": "starts",
"min_speed": 5,
"within": 90
}
]
}

View File

@@ -0,0 +1,11 @@
{
"description": "Test mode, real time limit: EU07 on td.scn, with a limit of 2 s of real time (the case's timeout, which the runner passes as --test-wall-timeout). The scenery takes longer to load, so the simulator's watchdog ends the run while it loads, with the timeout code and a verdict saying it was the real time.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_red_signal.scn",
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
},
"expect": "wall_timeout",
"timeout": 2
}

View File

@@ -0,0 +1,17 @@
{
"description": "Vectron 1293-107 on td.scn, AI driver on the td/rozklad timetable: starts from Stacja1, keeps to the speed limit, brakes to a stop at Przystanek.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_timetable.scn",
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\vectron_v1 1293-107 wariant_7w 0 headdriver 0 0 enddynamic"
},
"duration": 240,
"timeout": 600,
"checks": [
{ "type": "starts", "min_speed": 20, "within": 150 },
{ "type": "max_speed", "limit": 90, "tolerance": 3 },
{ "type": "brakes", "min_pressure": 0.5 },
{ "type": "stops", "moving_speed": 20, "stop_speed": 1.0, "hold": 0, "min_distance": 1380, "max_distance": 1420 }
]
}

640
tests/scenarios/run.py Normal file
View File

@@ -0,0 +1,640 @@
#!/usr/bin/env python3
"""Scenario tests: run the real simulator on a scenery and check what a vehicle did.
A case (cases/<name>.json) names a base scenery from the game install, a fragment
added to it (usually a trainset with an AI driver), the result the simulator's test
mode should end with, and checks on the vehicle's physics log. The runner:
1. builds scenery/autotest_<case>.scn in the game dir: the base scenery with its own
trainsets removed, optionally a different start time, and the case fragment added,
2. starts eu07 in test mode (--test: every train driven by AI, nothing drawn, fixed
time steps as fast as it can) with that scenery and an extra settings file (-ini)
that turns on physicslog,
3. waits until the simulator quits by itself and reads the test result from its exit
code: pass, fail, signal passed at danger, timeout (after `duration` seconds of
simulation, if the case gives one), standstill or error,
4. compares the result with the one the case expects, evaluates the checks on the
physicslog and prints PASS/FAIL per check.
Scenery and vehicle data are not part of this repo, so this runs against a local game
install only (--game-dir or EU07_GAME_DIR), not in CI. See README.md.
Usage:
python tests/scenarios/run.py --game-dir "E:/.../MaSzyna" -j 4 # all cases
python tests/scenarios/run.py --game-dir ... eu07_td_timetable # chosen cases
"""
from __future__ import annotations
import argparse
import json
import os
import re
import shutil
import subprocess
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass, field
from pathlib import Path
HERE = Path(__file__).resolve().parent
REPO = HERE.parents[1]
CASES_DIR = HERE / "cases"
OUT_DIR = HERE / "out"
# physicslog columns, see TController::PhysicsLog() in vehicle/Driver.cpp.
# Velocity is in km/h and pressures in bar, despite the header saying m/s and MPa.
COL_TIME = 0
COL_VELOCITY = 1 # wheel speed: stays at 0 while the wheels are locked in a skid
COL_BRAKE_PRESS = 8
COL_COMMAND = 16 # text, read as None
COL_WHEELSLIP = 20
COL_VEHICLE_VELOCITY = 25 # signed vehicle speed, km/h; only in builds which log it
# exit codes of the simulator's test mode (eu07 --test), see simulation/simulationtestrules.h
EXIT_RESULTS = {
0: "pass",
10: "fail",
11: "signal_passed_at_danger",
12: "timeout",
13: "standstill",
14: "error",
}
# the timeout of the simulator's real time watchdog (test.wall_timeout) has the timeout code too; told
# apart by its verdict, so a run that hung doesn't pass a case that expects the simulation time to run out
WALL_TIMEOUT = "wall_timeout"
RESULTS = (*EXIT_RESULTS.values(), WALL_TIMEOUT)
# settings layered over the game's eu07.ini for a test run (test mode doesn't read the player's own
# in %APPDATA%); the window ones only matter
# with --watch, test mode keeps its window hidden and turns the sound off by itself
TEST_INI = """\
fullscreen no
fullscreenwindowed no
width 1280
height 720
inactivepause no
soundenabled no
physicslog yes
multiplelogs yes
timespeed {timespeed}
"""
# multiplelogs: log and errors go to logs/log (<scenery>) <date>.txt instead of the shared
# log.txt, so runs of different cases can go in parallel
# ---------------------------------------------------------------- scenery
def split_trainsets(text: str) -> tuple[str, str]:
"""Splits text into (everything else, the trainset ... endtrainset blocks)."""
rest, trainsets = [], []
inside = False
for line in text.splitlines(keepends=True):
tokens = line.split("//", 1)[0].split()
if not inside and tokens and tokens[0].lower() == "trainset":
inside = True
if inside:
trainsets.append(line)
if any(t.lower() == "endtrainset" for t in tokens):
inside = False
continue
rest.append(line)
return "".join(rest), "".join(trainsets)
def set_start_time(text: str, hhmm: str) -> str:
return re.sub(r"(?im)^(\s*time\s+)\d{1,2}:\d{2}", lambda m: m.group(1) + hhmm, text, count=1)
def add_track_events(text: str, events: dict) -> str:
"""Adds lines (e.g. "event0 test_pass") to tracks named in events, before their endtrack."""
for track, lines in events.items():
lines = [lines] if isinstance(lines, str) else lines
m = re.search(rf"(?im)^\s*node\s+\S+\s+\S+\s+{re.escape(track)}\s+track\b.*?^(\s*endtrack\b)", text, re.S)
if not m:
sys.exit(f"track_events: no track {track} in the base scenery itself (not in its includes)")
text = text[: m.start(1)] + "".join(line + "\n" for line in lines) + text[m.start(1):]
return text
def set_track_friction(text: str, friction: float) -> str:
"""Sets the friction of every track: node <range> <range> <name> track <type> <length> <gauge> <friction> ..."""
return re.sub(r"(?im)^(\s*node\s+\S+\s+\S+\s+\S+\s+track\s+\S+\s+\S+\s+\S+\s+)[-\d.]+",
lambda m: m.group(1) + f"{friction}", text)
def fill_vars(fragment: str, values: dict, keep=frozenset()) -> str:
"""Replaces {{name}} with the case's value; a list value becomes one line per item."""
def value(m):
if m.group(1) in keep:
return m.group(0)
v = values.get(m.group(1))
if v is None:
sys.exit(f"fragment uses {{{{{m.group(1)}}}}} but the case sets no such var")
return "\n".join(v) if isinstance(v, list) else str(v)
return re.sub(r"\{\{(\w+)\}\}", value, fragment)
def scenery_cache(game_dir: Path, case_name: str) -> Path:
"""The simulator's geometry cache of the case scenery."""
return game_dir / "scenery" / f"autotest_{case_name}.sbt"
def build_scenery(case: dict, game_dir: Path, case_name: str, vehicle: str) -> str:
"""Writes the case scenery into the game's scenery dir and returns its file name."""
scenery_dir = game_dir / "scenery"
# sceneries are cp1250; latin-1 round-trips every byte, so nothing gets mangled
text = (scenery_dir / case["base_scenery"]).read_bytes().decode("latin-1")
if case.get("strip_trainsets", True):
text, _ = split_trainsets(text)
if "time" in case:
text = set_start_time(text, case["time"])
if "track_friction" in case:
text = set_track_friction(text, case["track_friction"])
if "track_events" in case:
text = add_track_events(text, case["track_events"])
fragment = (CASES_DIR / case["fragment"]).read_bytes().decode("latin-1")
# vars first: {{vehicle}} inside a var's value gets filled in the second pass
fragment = fill_vars(fragment, case.get("vars", {}), keep={"vehicle"})
fragment = fill_vars(fragment, {"vehicle": vehicle})
# FirstInit binds event launchers to their events and links the tracks. so launchers and
# events of the case go in before it (read after it they never fire), trainsets after it
# (placed before it they stand on unlinked track and derail at once)
before, trainsets = split_trainsets(fragment)
# settings of the test mode, read like any other config block: the case's vehicle is the train under test,
# whose consist the pass and the standstill wait for, and the case's own (test.timeout and so on)
config = [f"test.vehicle {vehicle}", *case.get("test_config", [])]
before = "config\n" + "".join(line + "\n" for line in config) + "endconfig\n" + before
header = f"\n// ---- autotest: {case_name} ----\n"
firstinit = re.search(r"(?im)^\s*firstinit\b", text)
if firstinit:
text = text[: firstinit.start()] + header + before + "\n" + text[firstinit.start():]
else:
text += header + before + "\n"
text += header + trainsets + "\n"
name = f"autotest_{case_name}.scn"
(scenery_dir / name).write_bytes(text.encode("latin-1"))
# the simulator caches the scenery's static geometry in <name>.sbt and reads it instead of
# the terrain in the .scn (tracks, events and trainsets always come from the .scn), so
# drop the cache once the base scenery has changed
cache = scenery_cache(game_dir, case_name)
if cache.exists() and cache.stat().st_mtime < (scenery_dir / case["base_scenery"]).stat().st_mtime:
cache.unlink()
return name
# ---------------------------------------------------------------- run
def read_physicslog(path: Path) -> list[list[float]]:
"""Returns rows of columns, None for the text ones; skips the header and partial lines."""
rows = []
try:
lines = path.read_text(errors="replace").splitlines()
except FileNotFoundError:
return rows
for line in lines[1:]:
parts = line.split()
if len(parts) <= COL_BRAKE_PRESS:
continue # line still being written
try:
row = [float(p) for p in parts[: COL_BRAKE_PRESS + 1]]
except ValueError:
continue
for p in parts[COL_BRAKE_PRESS + 1:]:
try:
row.append(float(p))
except ValueError:
row.append(None)
rows.append(row)
return rows
def find_logs(game_dir: Path, scenery: str, since: float) -> dict[str, Path]:
"""The log.txt / errors.txt of one run; with multiplelogs each scenery gets its own files."""
stem = scenery.rsplit(".", 1)[0]
found = {}
for kind in ("log", "errors"):
files = [p for p in (game_dir / "logs").glob(f"{kind} ({stem}) *.txt") if p.stat().st_mtime >= since]
if files:
found[kind] = max(files, key=lambda p: p.stat().st_mtime)
return found
def read_text(path: Path | None) -> str:
try:
return path.read_text(errors="replace") if path else ""
except FileNotFoundError:
return ""
# load problems that leave the case without its vehicle; no point waiting for the timeout
FATAL_LOG_LINES = ("Bad vehicle:", "Bad trainset:")
# wall seconds from start to the first physicslog row before giving up. "Scenario loading
# time" in the log is not the end of loading (the trainsets after FirstInit still follow),
# and with several simulators loading at once the first rows took well over 15 s
STARTUP_TIMEOUT = 120.0
# wall seconds past a case's timeout before the runner kills a simulator that didn't end itself
KILL_GRACE = 30.0
@dataclass
class RunResult:
rows: list[list[float]] = field(default_factory=list)
exit_code: int | None = None # None: still running when we closed it (the normal end with --watch)
verdict: str = "" # the simulator's own line about the test result
failure: str = "" # why the run was cut short
wall_seconds: float = 0.0
logs: dict[str, Path] = field(default_factory=dict)
def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog: Path,
duration: float | None, timeout: float, watch: bool, say) -> RunResult:
"""Runs the simulator on the scenery. In test mode it quits by itself and the exit code
is the result; with watch it runs in a window and gets closed after `duration`."""
physicslog.unlink(missing_ok=True)
if watch:
cmd = [str(exe), "-s", scenery, "-v", "ghostview", "-ini", str(ini)]
else:
# the simulator's own watchdog ends a run that takes too long, with the timeout code and its verdict
cmd = [str(exe), "--test", "-s", scenery, "-ini", str(ini), "--test-wall-timeout", f"{timeout:g}"]
if duration is not None:
cmd += ["--test-timeout", f"{duration:g}"]
say(" $ " + " ".join(f'"{c}"' if " " in c else c for c in cmd), progress=True)
since = time.time() - 1
started = time.monotonic()
# the test result line goes to stdout; a file, not a pipe, so a chatty run can't block on it
stdout = (ini.parent / "stdout.txt").open("w+b")
proc = subprocess.Popen(cmd, cwd=game_dir, stdout=stdout, stderr=subprocess.STDOUT)
result = RunResult()
last_report = 0.0
try:
while True:
time.sleep(0.25) # at timespeed 20 that is 5 s of simulation
wall = time.monotonic() - started
result.rows = read_physicslog(physicslog)
sim_t = result.rows[-1][COL_TIME] if result.rows else 0.0
if wall - last_report >= 15:
v = result.rows[-1][COL_VELOCITY] if result.rows else 0.0
say(f" ... {wall:5.0f}s wall, sim {sim_t:6.1f}s, v={v:6.1f} km/h", progress=True)
last_report = wall
if proc.poll() is not None:
result.exit_code = proc.returncode
break
if watch and sim_t >= duration:
break
# in test mode a little later than the simulator's own limit, for a run that can't even end itself
if wall > timeout + (0 if watch else KILL_GRACE):
result.failure = f"timed out after {wall:.0f}s before {duration}s of simulation"
break
if not result.rows:
result.logs = find_logs(game_dir, scenery, since)
fatal = [l.strip() for l in read_text(result.logs.get("errors")).splitlines()
if any(k in l for k in FATAL_LOG_LINES)]
if fatal:
result.failure = "vehicle failed to load: " + " | ".join(fatal)
break
if wall > STARTUP_TIMEOUT:
result.failure = f"no physicslog rows after {STARTUP_TIMEOUT:.0f}s"
break
finally:
if proc.poll() is None:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
result.wall_seconds = time.monotonic() - started
stdout.seek(0)
result.verdict = next((line.strip() for line in stdout.read().decode(errors="replace").splitlines()
if line.startswith("test:")), "")
stdout.close()
result.rows = read_physicslog(physicslog)
result.logs = find_logs(game_dir, scenery, since)
return result
# ---------------------------------------------------------------- checks
def check_starts(rows, min_speed, within):
for r in rows:
if r[COL_TIME] > within:
break
if r[COL_VELOCITY] >= min_speed:
return True, f"reached {min_speed} km/h at t={r[COL_TIME]:.1f}s"
top = max((r[COL_VELOCITY] for r in rows if r[COL_TIME] <= within), default=0.0)
return False, f"top speed in the first {within}s was {top:.1f} km/h, wanted {min_speed}"
def check_max_speed(rows, limit, tolerance=0.0):
worst = max(rows, key=lambda r: r[COL_VELOCITY])
v, t = worst[COL_VELOCITY], worst[COL_TIME]
ok = v <= limit + tolerance
return ok, f"top speed {v:.1f} km/h at t={t:.1f}s (limit {limit} +{tolerance})"
def _first_moving(rows, moving_speed):
return next((i for i, r in enumerate(rows) if r[COL_VELOCITY] >= moving_speed), None)
def _distances(rows):
"""Metres travelled up to each row, from the speed; the physicslog has no position."""
out, d = [0.0], 0.0
for a, b in zip(rows, rows[1:]):
d += (a[COL_VELOCITY] + b[COL_VELOCITY]) / 2 / 3.6 * (b[COL_TIME] - a[COL_TIME])
out.append(d)
return out
def check_brakes(rows, min_pressure, moving_speed=20):
start = _first_moving(rows, moving_speed)
if start is None:
return False, "never moved, so never braked"
for r in rows[start:]:
if r[COL_BRAKE_PRESS] >= min_pressure:
return True, f"brake cylinder {r[COL_BRAKE_PRESS]:.2f} bar at t={r[COL_TIME]:.1f}s, v={r[COL_VELOCITY]:.1f} km/h"
return False, f"brake cylinder pressure never reached {min_pressure} bar after moving off"
def check_stops(rows, moving_speed, stop_speed, hold, min_distance=None, max_distance=None):
start = _first_moving(rows, moving_speed)
if start is None:
return False, f"never reached {moving_speed} km/h"
dist = _distances(rows)
stopped_at = None
for i in range(start, len(rows)):
r = rows[i]
if r[COL_VELOCITY] <= stop_speed:
if stopped_at is None:
stopped_at = i
if r[COL_TIME] - rows[stopped_at][COL_TIME] >= hold:
where = dist[stopped_at]
detail = f"came to a stop at t={rows[stopped_at][COL_TIME]:.1f}s after {where:.0f} m"
if (min_distance is not None and where < min_distance) or (max_distance is not None and where > max_distance):
return False, f"{detail}, wanted between {min_distance} and {max_distance} m"
return True, detail
else:
stopped_at = None
return False, f"did not stay below {stop_speed} km/h for {hold}s after moving off"
def check_stays_stopped(rows, moving_speed, stop_speed):
"""Once stopped after moving off, never moves again (e.g. waiting at a red signal)."""
start = _first_moving(rows, moving_speed)
if start is None:
return False, f"never reached {moving_speed} km/h"
stop = next((i for i in range(start, len(rows)) if rows[i][COL_VELOCITY] <= stop_speed), None)
if stop is None:
return False, f"never came down to {stop_speed} km/h"
moved = next((r for r in rows[stop:] if r[COL_VELOCITY] > stop_speed), None)
if moved is not None:
return False, f"stopped at t={rows[stop][COL_TIME]:.1f}s but moved again at t={moved[COL_TIME]:.1f}s, v={moved[COL_VELOCITY]:.1f} km/h"
return True, f"stopped at t={rows[stop][COL_TIME]:.1f}s and stayed until t={rows[-1][COL_TIME]:.1f}s"
def check_distance(rows, max=None, min=None):
"""Total distance travelled, in metres."""
d = _distances(rows)[-1]
ok = (max is None or d <= max) and (min is None or d >= min)
bounds = " ".join(s for s in (f">= {min}" if min is not None else "", f"<= {max}" if max is not None else "") if s)
return ok, f"travelled {d:.0f} m (wanted {bounds})"
def check_slips(rows, within=None):
"""The lead vehicle's wheels slip (or lock in a skid) at some point, e.g. to make sure a slippery rail case does what it's meant to."""
for r in rows:
if within is not None and r[COL_TIME] > within:
break
if len(r) > COL_WHEELSLIP and r[COL_WHEELSLIP]:
return True, f"wheels slipping at t={r[COL_TIME]:.1f}s, v={r[COL_VELOCITY]:.1f} km/h"
return False, "the wheels never slipped"
def check_holds_still(rows, moving_speed, stop_speed, max_distance):
"""Once stopped after moving off, the vehicle doesn't creep. Uses the vehicle speed rather than the
wheel speed, so it also catches a vehicle sliding on locked wheels."""
if not rows or len(rows[0]) <= COL_VEHICLE_VELOCITY or rows[0][COL_VEHICLE_VELOCITY] is None:
return False, "the physicslog has no vehicle speed column, the build is too old"
speed = [abs(r[COL_VEHICLE_VELOCITY]) for r in rows]
start = next((i for i, v in enumerate(speed) if v >= moving_speed), None)
if start is None:
return False, f"never reached {moving_speed} km/h"
stop = next((i for i in range(start, len(rows)) if speed[i] <= stop_speed), None)
if stop is None:
return False, f"never came down to {stop_speed} km/h"
d, top, top_t = 0.0, 0.0, rows[stop][COL_TIME]
for i in range(stop + 1, len(rows)):
d += (speed[i - 1] + speed[i]) / 2 / 3.6 * (rows[i][COL_TIME] - rows[i - 1][COL_TIME])
if speed[i] > top:
top, top_t = speed[i], rows[i][COL_TIME]
held = rows[-1][COL_TIME] - rows[stop][COL_TIME]
detail = f"stopped at t={rows[stop][COL_TIME]:.1f}s, then moved {d:.3f} m in {held:.0f}s, top {top:.4f} km/h at t={top_t:.1f}s"
return d <= max_distance, detail
def check_ends_moving(rows, min_speed):
"""The run ended with the vehicle still moving, e.g. a pass that didn't wait for the trains to stop."""
if not rows:
return False, "the physicslog is empty"
last = rows[-1]
ok = last[COL_VELOCITY] >= min_speed
return ok, f"last row t={last[COL_TIME]:.1f}s, v={last[COL_VELOCITY]:.1f} km/h (wanted >= {min_speed})"
def check_slip_clears(rows, moving_speed, stop_speed, within):
"""Once the vehicle has stopped after moving off, its wheels stop slipping within `within` seconds
(a vehicle left skidding on locked wheels keeps its brakes capped by the sliding adhesion)."""
if not rows or len(rows[0]) <= COL_VEHICLE_VELOCITY or rows[0][COL_VEHICLE_VELOCITY] is None:
return False, "the physicslog has no vehicle speed column, the build is too old"
start = next((i for i, r in enumerate(rows) if abs(r[COL_VEHICLE_VELOCITY]) >= moving_speed), None)
if start is None:
return False, f"never reached {moving_speed} km/h"
stop = next((i for i in range(start, len(rows)) if abs(rows[i][COL_VEHICLE_VELOCITY]) <= stop_speed), None)
if stop is None:
return False, f"never came down to {stop_speed} km/h"
deadline = rows[stop][COL_TIME] + within
late = [r for r in rows[stop:] if r[COL_TIME] > deadline and r[COL_WHEELSLIP]]
if late:
return False, f"stopped at t={rows[stop][COL_TIME]:.1f}s, wheels still slipping at t={late[0][COL_TIME]:.1f}s ({len(late)} rows after the {within}s grace)"
return True, f"stopped at t={rows[stop][COL_TIME]:.1f}s, no slipping after t={deadline:.1f}s"
CHECKS = {
"slips": check_slips,
"slip_clears": check_slip_clears,
"holds_still": check_holds_still,
"starts": check_starts,
"max_speed": check_max_speed,
"brakes": check_brakes,
"stops": check_stops,
"stays_stopped": check_stays_stopped,
"distance": check_distance,
"ends_moving": check_ends_moving,
}
# ---------------------------------------------------------------- main
def find_exe(explicit: str | None) -> Path:
if explicit:
return Path(explicit).resolve()
candidates = sorted(REPO.glob("cmake-build-*/bin/Release/eu07*.exe"), key=lambda p: p.stat().st_mtime)
if not candidates:
sys.exit("no eu07 executable found under cmake-build-*/bin/Release, pass --exe")
return candidates[-1]
def count_lines(path: Path) -> int:
try:
return sum(1 for line in path.read_text(errors="replace").splitlines() if line.strip())
except FileNotFoundError:
return 0
def describe_exit(code: int | None) -> str:
if code is None:
return "none, still running"
name = EXIT_RESULTS.get(code)
return f"{code} ({name})" if name else f"{code} (0x{code & 0xFFFFFFFF:08X}, a crash?)"
def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, watch: bool, say) -> bool:
case = json.loads((CASES_DIR / f"{name}.json").read_text(encoding="utf-8"))
say(f"\n=== {name}: {case.get('description', '')}")
out = OUT_DIR / name
shutil.rmtree(out, ignore_errors=True)
out.mkdir(parents=True)
# the lead vehicle is named after the case, so parallel runs never share a physicslog
vehicle = f"autotest_{name}"
scenery = build_scenery(case, game_dir, name, vehicle)
ini = out / "autotest.ini"
ini.write_text(TEST_INI.format(timespeed=timespeed), encoding="ascii")
physicslog = game_dir / "physicslog" / f"{vehicle}.dat"
duration = float(case["duration"]) if "duration" in case else None
# a case that only records `duration` seconds ends on the test timeout; one that says
# how the test ends (a pass event, a signal passed at danger...) expects that result
expect = case.get("expect", "timeout" if duration is not None else "pass")
if expect not in RESULTS:
sys.exit(f"{name}: expect must be one of {', '.join(RESULTS)}")
if watch and duration is None:
sys.exit(f"{name}: --watch needs a case with duration")
result = run_simulator(exe, game_dir, scenery, ini, physicslog,
duration, float(case.get("timeout", 600)), watch, say)
# the logs are ours alone (named after the case scenery), move them out of the game dir
for kind, path in result.logs.items():
shutil.move(path, out / f"{kind}.txt")
if physicslog.exists():
shutil.move(physicslog, out / physicslog.name)
# other trains of the case with a driver, named <something>_<vehicle>, log too
for other in physicslog.parent.glob(f"*_{vehicle}.dat"):
shutil.move(other, out / other.name)
ok = True
sim_t = result.rows[-1][COL_TIME] if result.rows else 0.0
say(f" ran {result.wall_seconds:.0f}s wall, {sim_t:.1f}s of physicslog, results in {out.relative_to(REPO)}")
if watch:
if result.exit_code is not None:
say(f" FAIL simulator exited on its own with code {result.exit_code}")
ok = False
else:
if result.verdict:
say(f" {result.verdict}")
got = EXIT_RESULTS.get(result.exit_code) if result.exit_code is not None else None
if got == "timeout" and "of real time" in result.verdict:
got = WALL_TIMEOUT
passed = got == expect
watchdog = ", from the real time watchdog" if got == WALL_TIMEOUT else ""
say(f" {'PASS' if passed else 'FAIL'} exit code {describe_exit(result.exit_code)}{watchdog}, expected {expect}")
ok &= passed
# a run cut short (the real time watchdog, an error, a crash, or closed by us) could have been writing
# the scenery's geometry cache; the simulator checks only the header of it, so the next runs would read
# a truncated one. it is written again on the next run
if result.failure or (not watch and got in (None, "error", WALL_TIMEOUT)):
scenery_cache(game_dir, name).unlink(missing_ok=True)
if result.failure:
say(f" FAIL {result.failure}")
ok = False
if not result.rows and case.get("checks"):
if not result.failure:
say(f" FAIL no physicslog for '{vehicle}'")
return False
for check in case.get("checks", []):
params = {k: v for k, v in check.items() if k != "type"}
passed, detail = CHECKS[check["type"]](result.rows, **params)
say(f" {'PASS' if passed else 'FAIL'} {check['type']}: {detail}")
ok &= passed
errors = count_lines(out / "errors.txt")
if errors:
say(f" note errors.txt has {errors} line(s)")
if case.get("fail_on_errors") and errors:
ok = False
return ok
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("cases", nargs="*", help="case names from cases/ (default: all)")
ap.add_argument("--game-dir", default=os.environ.get("EU07_GAME_DIR"),
help="game install with scenery/ and dynamic/ (or EU07_GAME_DIR)")
ap.add_argument("--exe", help="simulator executable (default: newest cmake-build-*/bin/Release/eu07*.exe)")
ap.add_argument("--watch", action="store_true",
help="run in a visible window in real time (see --timespeed) instead of the test mode, "
"closed after the case's duration; the exit code isn't checked")
ap.add_argument("--timespeed", type=float, default=1.0,
help="with --watch: simulation time multiplier, the simulator offers 1, 5, 20 and 60 (default 1)")
ap.add_argument("-j", "--jobs", type=int, default=1,
help="cases run at once, each in a simulator of its own (default 1)")
args = ap.parse_args()
if not args.game_dir:
sys.exit("pass --game-dir or set EU07_GAME_DIR")
game_dir = Path(args.game_dir).resolve()
exe = find_exe(args.exe)
print(f"exe: {exe}\ngame: {game_dir}")
names = args.cases or sorted(p.stem for p in CASES_DIR.glob("*.json"))
missing = [n for n in names if not (CASES_DIR / f"{n}.json").exists()]
if missing:
sys.exit("no such case: " + ", ".join(missing))
started = time.monotonic()
results = {}
if args.jobs <= 1:
def say(line, progress=False):
print(line, flush=True)
for n in names:
results[n] = run_case(n, exe, game_dir, args.timespeed, args.watch, say)
else:
# each case prints its block when it is done, so parallel output does not interleave
def job(n):
lines = []
def say(line, progress=False):
if not progress:
lines.append(line)
ok = run_case(n, exe, game_dir, args.timespeed, args.watch, say)
return n, ok, lines
with ThreadPoolExecutor(max_workers=args.jobs) as pool:
for n in names:
print(f"started {n}", flush=True)
for fut in as_completed([pool.submit(job, n) for n in names]):
n, ok, lines = fut.result()
print("\n".join(lines), flush=True)
results[n] = ok
print(f"\n=== summary ({time.monotonic() - started:.0f}s)")
for n in names:
print(f" {'PASS' if results[n] else 'FAIL'} {n}")
return 0 if all(results.values()) else 1
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,268 @@
#include "stdafx.h"
#include "doctest.h"
#include "simulation/simulationtestrules.h"
#include <array>
#include <exception>
#include <new>
#include <stdexcept>
#include <vector>
// the test mode of the simulator (eu07 --test, eu07/maszyna#541): its exit codes, command line and end conditions
namespace {
using simulation::test_result;
simulation::test_command_line parse(std::vector<char const *> Arguments)
{
Arguments.insert(Arguments.begin(), "eu07");
return simulation::parse_test_command_line(Arguments);
}
// a frame of the default 0.03 s
simulation::test_frame frame(bool const Atrest, std::optional<double> const Reported = std::nullopt, std::string_view const Reason = {})
{
return simulation::test_frame{0.03, Reported, Reason, Atrest};
}
// runs frames until the test ends or Seconds of simulation time go by
std::optional<simulation::test_verdict> run(simulation::test_progress &Progress, simulation::test_limits const &Limits, double const Seconds,
simulation::test_frame const &Frame)
{
while (Progress.time < Seconds)
{
if (auto verdict{simulation::evaluate_frame(Progress, Frame, Limits)})
return verdict;
}
return std::nullopt;
}
} // namespace
TEST_CASE("test mode: the exit codes keep clear of the ones of a killed or aborted process")
{
std::array const results{test_result::pass, test_result::fail, test_result::signal_passed_at_danger, test_result::timeout, test_result::standstill, test_result::error};
for (auto const result : results)
{
auto const code{static_cast<int>(result)};
CAPTURE(code);
CHECK(code != 1); // TerminateProcess, most programs' failure
CHECK(code != 3); // abort() on Windows
CHECK(code < 126); // shells, signals on Linux
CHECK(simulation::to_string(result) != "unknown");
}
// the codes are public: scripts and CI read them
CHECK(static_cast<int>(test_result::pass) == 0);
CHECK(static_cast<int>(test_result::fail) == 10);
CHECK(static_cast<int>(test_result::signal_passed_at_danger) == 11);
CHECK(static_cast<int>(test_result::timeout) == 12);
CHECK(static_cast<int>(test_result::standstill) == 13);
CHECK(static_cast<int>(test_result::error) == 14);
}
TEST_CASE("test mode: seconds on the command line are a number, not negative")
{
CHECK(simulation::parse_seconds("240").value_or(-1.0) == doctest::Approx(240.0));
CHECK(simulation::parse_seconds("0.5").value_or(-1.0) == doctest::Approx(0.5));
CHECK(simulation::parse_seconds("0").value_or(-1.0) == doctest::Approx(0.0));
CHECK_FALSE(simulation::parse_seconds("").has_value());
CHECK_FALSE(simulation::parse_seconds("-1").has_value());
CHECK_FALSE(simulation::parse_seconds("60s").has_value());
CHECK_FALSE(simulation::parse_seconds("abc").has_value());
CHECK_FALSE(simulation::parse_seconds("inf").has_value());
CHECK_FALSE(simulation::parse_seconds("nan").has_value());
}
TEST_CASE("test mode: the command line")
{
SUBCASE("without --test there is no test")
{
auto const commandline{parse({"-s", "td.scn", "-v", "ep07-424"})};
CHECK_FALSE(commandline.enabled);
CHECK(commandline.error.empty());
}
SUBCASE("--test and its limits, in any order")
{
auto const commandline{parse({"--test-wall-timeout", "900", "-s", "td.scn", "--test", "--test-timeout", "240"})};
CHECK(commandline.enabled);
CHECK(commandline.timeout.value_or(-1.0) == doctest::Approx(240.0));
CHECK(commandline.wall_timeout.value_or(-1.0) == doctest::Approx(900.0));
CHECK(commandline.error.empty());
}
SUBCASE("a limit not given stays unset")
{
auto const commandline{parse({"--test", "-s", "td.scn"})};
CHECK_FALSE(commandline.timeout.has_value());
CHECK_FALSE(commandline.wall_timeout.has_value());
}
SUBCASE("a limit which isn't a number is an error, not a limit turned off")
{
auto const commandline{parse({"--test", "-s", "td.scn", "--test-timeout", "4m"})};
CHECK(commandline.enabled);
CHECK_FALSE(commandline.timeout.has_value());
CHECK(commandline.error == "--test-timeout needs a number of seconds");
}
SUBCASE("a limit without its value is an error")
{
auto const commandline{parse({"--test", "-s", "td.scn", "--test-wall-timeout"})};
CHECK(commandline.error == "--test-wall-timeout needs a number of seconds");
}
SUBCASE("the first error is the one told")
{
auto const commandline{parse({"--test", "--test-timeout", "x", "--test-wall-timeout", "y"})};
CHECK(commandline.error == "--test-timeout needs a number of seconds");
}
SUBCASE("values of other options aren't taken for options")
{
auto const commandline{parse({"-s", "--test", "-ini", "--test-timeout", "-v", "--test-wall-timeout", "-seed", "--test"})};
CHECK_FALSE(commandline.enabled);
CHECK(commandline.error.empty());
}
SUBCASE("-edit takes no option for its scenery, which is optional")
{
// a scenery after -edit or none: the option after it is read either way
CHECK(parse({"-edit", "--test", "-s", "td.scn"}).enabled);
CHECK(parse({"-edit", "td.scn", "--test"}).enabled);
// -edit leaves -s alone, which takes --test for its scenery, as the simulator reads it
CHECK_FALSE(parse({"-edit", "-s", "--test"}).enabled);
}
}
TEST_CASE("test mode: the real time limit")
{
SUBCASE("given, it is the limit, 0 turns it off")
{
CHECK(simulation::wall_clock_limit(120.0, 3600.0) == doctest::Approx(120.0));
CHECK(simulation::wall_clock_limit(0.0, 3600.0) == doctest::Approx(0.0));
}
SUBCASE("not given, it is the simulation time limit and time for the loading")
{
CHECK(simulation::wall_clock_limit(std::nullopt, 240.0) == doctest::Approx(240.0 + simulation::wall_timeout_loading_allowance));
CHECK(simulation::wall_clock_limit(std::nullopt, 3600.0) == doctest::Approx(3600.0 + simulation::wall_timeout_loading_allowance));
}
SUBCASE("no simulation time limit, no real time limit unless given")
{
CHECK(simulation::wall_clock_limit(std::nullopt, 0.0) == doctest::Approx(0.0));
CHECK(simulation::wall_clock_limit(300.0, 0.0) == doctest::Approx(300.0));
}
}
TEST_CASE("test mode: the end conditions")
{
simulation::test_limits limits;
simulation::test_progress progress;
SUBCASE("nothing happens, nothing ends")
{
CHECK_FALSE(simulation::evaluate_frame(progress, frame(false), limits).has_value());
CHECK(progress.time == doctest::Approx(0.03));
CHECK(progress.resttime == doctest::Approx(0.0));
}
SUBCASE("a cell nobody wrote to says nothing")
{
CHECK_FALSE(simulation::evaluate_frame(progress, frame(false, -1.0), limits).has_value());
CHECK_FALSE(simulation::evaluate_frame(progress, frame(false, 2.0), limits).has_value());
CHECK_FALSE(progress.passrequested);
}
SUBCASE("0 fails at once, with the text of the cell")
{
auto const verdict{simulation::evaluate_frame(progress, frame(false, 0.0, "wrong_track"), limits)};
REQUIRE(verdict.has_value());
CHECK(verdict->result == test_result::fail);
CHECK(verdict->reason == "__test.result set to 0 (wrong_track)");
}
SUBCASE("1 passes once the tested trains have stood for a second")
{
CHECK_FALSE(simulation::evaluate_frame(progress, frame(false, 1.0), limits).has_value());
CHECK(progress.passrequested);
// still moving: the pass waits
CHECK_FALSE(run(progress, limits, 30.0, frame(false, 1.0)).has_value());
auto const stoppedat{progress.time};
auto const verdict{run(progress, limits, 60.0, frame(true, 1.0))};
REQUIRE(verdict.has_value());
CHECK(verdict->result == test_result::pass);
CHECK(progress.time - stoppedat == doctest::Approx(simulation::pass_settle_time).epsilon(0.05));
}
SUBCASE("the pass stays requested when the cell changes back")
{
CHECK_FALSE(simulation::evaluate_frame(progress, frame(false, 1.0), limits).has_value());
auto const verdict{run(progress, limits, 60.0, frame(true, -1.0))};
REQUIRE(verdict.has_value());
CHECK(verdict->result == test_result::pass);
}
SUBCASE("1 with test.no_movement_before_pass no passes at once")
{
limits.no_movement_before_pass = false;
auto const verdict{simulation::evaluate_frame(progress, frame(false, 1.0, "reached_the_stop"), limits)};
REQUIRE(verdict.has_value());
CHECK(verdict->result == test_result::pass);
CHECK(verdict->reason == "__test.result set to 1 (reached_the_stop)");
}
SUBCASE("standing still without a result ends after test.standstill")
{
limits.standstill = 120.0;
auto const verdict{run(progress, limits, 1000.0, frame(true))};
REQUIRE(verdict.has_value());
CHECK(verdict->result == test_result::standstill);
CHECK(progress.time == doctest::Approx(120.0).epsilon(0.001));
}
SUBCASE("a move starts the standstill over")
{
limits.standstill = 120.0;
CHECK_FALSE(run(progress, limits, 100.0, frame(true)).has_value());
CHECK_FALSE(simulation::evaluate_frame(progress, frame(false), limits).has_value());
CHECK(progress.resttime == doctest::Approx(0.0));
CHECK_FALSE(run(progress, limits, 200.0, frame(true)).has_value());
}
SUBCASE("test.pass_on_standstill makes the standstill a pass")
{
limits.standstill = 10.0;
limits.pass_on_standstill = true;
auto const verdict{run(progress, limits, 1000.0, frame(true))};
REQUIRE(verdict.has_value());
CHECK(verdict->result == test_result::pass);
}
SUBCASE("test.standstill 0 never ends the test")
{
limits.standstill = 0.0;
limits.timeout = 0.0;
CHECK_FALSE(run(progress, limits, 5000.0, frame(true)).has_value());
}
SUBCASE("no result by test.timeout")
{
limits.timeout = 240.0;
auto const verdict{run(progress, limits, 1000.0, frame(false))};
REQUIRE(verdict.has_value());
CHECK(verdict->result == test_result::timeout);
CHECK(verdict->reason == "no result after 240 s");
CHECK(progress.time == doctest::Approx(240.0).epsilon(0.001));
}
SUBCASE("a result in the last frame wins over the timeout")
{
limits.timeout = 0.03;
auto const verdict{simulation::evaluate_frame(progress, frame(false, 0.0), limits)};
REQUIRE(verdict.has_value());
CHECK(verdict->result == test_result::fail);
}
}
TEST_CASE("test mode: what ended a run the simulator didn't end")
{
SUBCASE("an exception says what it was")
{
CHECK(simulation::describe_exception(std::make_exception_ptr(std::runtime_error("no scenery"))) == "critical error: no scenery");
// e.g. std::stof on a token of a scenery which isn't a number
CHECK(simulation::describe_exception(std::make_exception_ptr(std::invalid_argument("stof"))) == "critical error: stof");
CHECK(simulation::describe_exception(std::make_exception_ptr(std::bad_alloc())).starts_with("critical error: "));
}
SUBCASE("something thrown which isn't a std::exception")
{
CHECK(simulation::describe_exception(std::make_exception_ptr(42)) == "critical error: an exception of unknown type");
}
SUBCASE("terminated without an exception")
{
CHECK(simulation::describe_exception(nullptr) == "the simulator was terminated");
}
}