18
0
mirror of https://github.com/MaSzyna-EU07/maszyna.git synced 2026-11-19 12:16:02 +01:00

test: scenario tests run in the simulator's test mode and check its exit code

run.py starts eu07 with --test (and --test-timeout <duration> for cases that
record a fixed time) and compares the result from the exit code with the
case's expect key, before the physicslog checks. The 14 cases take about 30 s
with -j 4 instead of minutes. --watch keeps the old windowed run.

New case keys: expect, test_config (lines of a config block), track_events
(lines added to a track of the base scenery, e.g. event0 test_pass). New cases
show each end of the test mode: test_mode_pass, _signal_passed_at_danger,
_timeout and _standstill. The slippery rail cases slide past the red signal,
which now ends a test, so they set test.fail_if_stop_overran no.

README documents the test mode for scenery authors.
This commit is contained in:
jerrrrycho
2026-11-07 13:17:27 +01:00
parent 02e29f6971
commit 6a538d202b
9 changed files with 315 additions and 51 deletions

View File

@@ -1,49 +1,121 @@
# Scenario tests
End-to-end checks that run the real simulator on a scenery and look at what a vehicle
did: did it start, did it keep to the speed limit, did it brake and stop.
End-to-end checks that run the real simulator on a scenery and look at how the run ended
and what a vehicle did: did it start, did it keep to the speed limit, did it brake and stop,
did it pass a signal at stop.
Scenery and vehicle data are not in this repo, so these run against a local game install
only, not in CI.
## The simulator's test mode
The runner drives the simulator's test mode (eu07/maszyna#541), which works without the
runner too and is meant for scenery authors as well:
```
eu07 --test -s <scenery.scn> [--test-timeout <seconds>] [-seed <number|word>] [-ini <settings file>]
```
With `--test` the simulator:
- needs `-s`; `-v` is ignored and every train is driven by AI (ghostview),
- draws nothing: the null renderer loads the 3D models (the physics needs some of their
data) but no textures, and the window stays hidden,
- has no sound, no network, no python, no pause (not even one set by the scenery), and
no crash dialog,
- runs the simulation as fast as it can in fixed steps of 0.03 s of simulation time per
frame (three 10 ms physics steps), whatever the real time, the time speed or the machine,
- seeds its random numbers with a fixed seed unless `-seed` or `randomseed` gives one, so two
runs of a scenery come out the same (the physicslogs of two runs are byte for byte equal),
- quits when the test ends, and says how it ended in the exit code and in a `test: ...`
line on stdout and in the log.
A scenery reports its result to the memory cell `__test.result`, which the simulator creates
(in every run, so a test scenery loads normally too). Write value 1 to it for a pass and 0
for a failure, with an `UpdateValues` event; other values are ignored, and the text, if
given, goes to the log as the reason:
```
event test_pass UpdateValues 0 __test.result reached_the_stop 1 * endevent
event test_fail UpdateValues 0 __test.result wrong_track 0 * endevent
```
(`PutValues` isn't the event for it: it hands a command to a vehicle, it doesn't write a cell.)
A good place for the pass event is `event0` of a track the train should stop on: it runs
while a train stands there, and the length of the track is the margin of the stop.
The test ends with the first of these:
| exit code | result | when |
| --- | --- | --- |
| 0 | pass | `__test.result` was set to 1 and then every vehicle stood still for 1 s, so a train still moving towards a signal at stop has to stop first. Also the standstill end with `test.pass_on_standstill yes` |
| 1 | fail | `__test.result` was set to 0; ends at once |
| 2 | signal passed at danger | a train went past a signal showing stop (`SetVelocity 0 0`; `SetVelocity 0 x` on automatic block, which may be passed after a stop, doesn't count); ends at once, unless `test.fail_if_stop_overran no` |
| 3 | timeout | `test.timeout` seconds of simulation time went by (default 3600) |
| 4 | standstill | every vehicle stood still for `test.standstill` seconds (default 300) and nothing reported a result. Trains waiting at a station or for a signal count as standing, so the limit has to be longer than such waits |
| 5 | error | the run could not start: no `-s`, the scenery failed to load or has no vehicles, or the simulator threw |
A crash ends with the system's code instead (on Windows e.g. `0xC0000005`, -1073741819).
The limits go in a `config` block of the scenery:
```
config
test.timeout 600
test.standstill 120
test.pass_on_standstill no
test.fail_if_stop_overran yes
endconfig
```
`--test-timeout` on the command line wins over `test.timeout` of the scenery.
A signal is passed at danger when the head of a train goes past a signal which its driver
has in its speed table, showing stop. So it's the driver's view that counts: a train without
a driver isn't watched, and neither is a signal the driver never scanned (a train placed at
100 km/h 150 m before a signal at stop got past it before its AI first looked ahead; at
60 km/h it was caught). Outside the test mode the same is written to the log, nothing more.
## Running
```
python tests/scenarios/run.py --game-dir "E:/Program Files (x86)/Steam/steamapps/common/MaSzyna" -j 4
python tests/scenarios/run.py --game-dir "C:/Steam/steamapps/common/MaSzyna" -j 4
```
Python 3.10+, no extra packages. Options:
- `EU07_GAME_DIR` can replace `--game-dir`.
- `--exe path/to/eu07.exe` picks the build to test; the default is the newest
`cmake-build-*/bin/Release/eu07*.exe`. The build needs the `-ini` command line option.
`cmake-build-*/bin/Release/eu07*.exe`. The build needs the `--test` and `-ini` command
line options.
- name one or more cases to run only those, e.g. `run.py ... eu07_td_timetable`.
- `-j N` runs N cases at once, each in its own simulator window. Each case prints its
results when it is done. All 8 cases take about a minute with `-j 4`.
- `--timespeed N` sets the simulation time multiplier, default 20: a 4-minute case runs in
about 20 s. Physics still steps finely (the physicslog keeps its 0.05 s rows) and the speed
profile at 20 and 60 matched a real-time run within a few km/h and seconds. Use
`--timespeed 1` to watch a run.
- `-j N` runs N cases at once. Each case prints its results when it is done. All 14 cases
take about 30 s with `-j 4`, loading included.
- `--watch` runs a case the old way, in a visible window in real time (or faster with
`--timespeed N`), closed after the case's `duration`; the exit code isn't checked then. For
watching what a case does.
Exit code is 0 when every case passed. Logs of each run (`log.txt`, `errors.txt`,
the vehicle's physicslog) are moved to `tests/scenarios/out/<case>/`.
`stdout.txt`, the vehicle's physicslog) are moved to `tests/scenarios/out/<case>/`.
## How it works
For each case the runner:
1. writes `scenery/autotest_<case>.scn` into the game dir: the base scenery with its own
trainsets removed, the start time changed if the case says so, and the case's `.scn`
fragment added: its trainsets after `FirstInit`, everything else (event launchers standing
in for keys the scenery expects a player to press, extra events) before it,
2. starts the simulator with `-s autotest_<case>.scn -v ghostview -ini out/<case>/autotest.ini`;
ghostview leaves every train to the AI, and the extra ini turns on `physicslog`, runs in a
1280x720 window, does not pause when the window loses focus, and writes the logs to
`logs/log (autotest_<case>) <date>.txt` rather than the shared `log.txt` (`multiplelogs`),
3. waits until `physicslog/autotest_<case>.dat` covers `duration` seconds, then closes the
simulator. It gives up early when the log reports `Bad vehicle` / `Bad trainset`, or when
the scenario has loaded and no physicslog appeared,
4. runs the checks on the physicslog.
trainsets removed, the start time changed if the case says so, the case's track events
added, and the case's `.scn` fragment added: its trainsets after `FirstInit`, everything
else (the case's `test_config`, event launchers standing in for keys the scenery expects a
player to press, extra events) before it,
2. starts the simulator with `--test -s autotest_<case>.scn -ini out/<case>/autotest.ini`,
plus `--test-timeout <duration>` when the case gives a `duration`; the extra ini turns on
`physicslog` and writes the logs to `logs/log (autotest_<case>) <date>.txt` rather than the
shared `log.txt` (`multiplelogs`),
3. waits until the simulator quits by itself. It gives up early when the log reports
`Bad vehicle` / `Bad trainset`, when the scenario has loaded and no physicslog appeared, or
after the case's `timeout` of wall-clock time,
4. compares the result from the exit code with the case's `expect`, then runs the checks on
the physicslog.
The lead vehicle is named `autotest_<case>`, so every case has its own physicslog and log
files and cases can run in parallel. The `autotest_*` sceneries (and the simulator's `.sbt`
@@ -60,9 +132,12 @@ scenery is newer.
| `strip_trainsets` | drop the base scenery's trainsets (default `true`) |
| `time` | replace the scenery start time, `HH:MM` |
| `track_friction` | set the friction of every track of the base scenery (default in the sceneries is 0.15), e.g. for slippery rail |
| `track_events` | lines added to tracks of the base scenery, by track name, e.g. `{ "tor1k": "event0 test_pass" }`; only tracks in the scenery file itself, not in its includes |
| `test_config` | lines of a `config` block added to the scenery, e.g. `["test.timeout 90"]` |
| `fragment` | `.scn` file in `cases/` added to the scenery: trainsets after `FirstInit` (before it they derail on unlinked track), the rest before it (launchers read after it never fire) |
| `vars` | values for `{{name}}` in the fragment; a list becomes one line per item |
| `duration` | seconds of simulation to record |
| `expect` | the result the test mode should end with: `pass`, `fail`, `signal_passed_at_danger`, `timeout`, `standstill` or `error`. Default `timeout` for a case with `duration`, else `pass` |
| `duration` | seconds of simulation to record: the test timeout of the run (`--test-timeout`) |
| `timeout` | wall-clock seconds before giving up, loading included |
| `fail_on_errors` | fail when `errors.txt` is not empty (default `false`) |
| `checks` | list of checks, below |
@@ -72,7 +147,14 @@ the one whose physicslog is checked. Other vehicles of the consist take names ma
e.g. `{{vehicle}}_s`. The td.scn cases share two fragments, `td_timetable.scn` (route set,
train runs to Przystanek) and `td_red_signal.scn` (no route, exit signal at stop), and differ
only in `vars.consist`. `td_red_signal_moving.scn` is the red signal one with the train
starting at `vars.velocity` km/h; with `track_friction` 0.03 it locks the wheels when braking.
starting at `vars.velocity` km/h; with `track_friction` 0.03 it locks the wheels when braking
and slides past the signal, so those cases set `test.fail_if_stop_overran no`.
`td_red_signal_pass.scn` adds the `test_pass` event, which `test_mode_pass` puts as `event0`
on `tor1k`, the track in front of the signal.
The `test_mode_*` cases show the ends of the test mode: `test_mode_pass` (exit 0, the pass
event), `test_mode_signal_passed_at_danger` (exit 2), `test_mode_timeout` (exit 3, from
`test.timeout`) and `test_mode_standstill` (exit 4, from `test.standstill`).
Checks (speeds in km/h, times in seconds of physicslog time, distances in metres). Most use
the physicslog Velocity column, which is the wheel speed: it stays at 0 while the wheels are