mirror of
https://github.com/MaSzyna-EU07/maszyna.git
synced 2026-11-19 13:26:08 +01:00
test: scenario tests run in the simulator's test mode and check its exit code
run.py starts eu07 with --test (and --test-timeout <duration> for cases that record a fixed time) and compares the result from the exit code with the case's expect key, before the physicslog checks. The 14 cases take about 30 s with -j 4 instead of minutes. --watch keeps the old windowed run. New case keys: expect, test_config (lines of a config block), track_events (lines added to a track of the base scenery, e.g. event0 test_pass). New cases show each end of the test mode: test_mode_pass, _signal_passed_at_danger, _timeout and _standstill. The slippery rail cases slide past the red signal, which now ends a test, so they set test.fail_if_stop_overran no. README documents the test mode for scenery authors.
This commit is contained in:
@@ -1,49 +1,121 @@
|
|||||||
# Scenario tests
|
# Scenario tests
|
||||||
|
|
||||||
End-to-end checks that run the real simulator on a scenery and look at what a vehicle
|
End-to-end checks that run the real simulator on a scenery and look at how the run ended
|
||||||
did: did it start, did it keep to the speed limit, did it brake and stop.
|
and what a vehicle did: did it start, did it keep to the speed limit, did it brake and stop,
|
||||||
|
did it pass a signal at stop.
|
||||||
|
|
||||||
Scenery and vehicle data are not in this repo, so these run against a local game install
|
Scenery and vehicle data are not in this repo, so these run against a local game install
|
||||||
only, not in CI.
|
only, not in CI.
|
||||||
|
|
||||||
|
## The simulator's test mode
|
||||||
|
|
||||||
|
The runner drives the simulator's test mode (eu07/maszyna#541), which works without the
|
||||||
|
runner too and is meant for scenery authors as well:
|
||||||
|
|
||||||
|
```
|
||||||
|
eu07 --test -s <scenery.scn> [--test-timeout <seconds>] [-seed <number|word>] [-ini <settings file>]
|
||||||
|
```
|
||||||
|
|
||||||
|
With `--test` the simulator:
|
||||||
|
|
||||||
|
- needs `-s`; `-v` is ignored and every train is driven by AI (ghostview),
|
||||||
|
- draws nothing: the null renderer loads the 3D models (the physics needs some of their
|
||||||
|
data) but no textures, and the window stays hidden,
|
||||||
|
- has no sound, no network, no python, no pause (not even one set by the scenery), and
|
||||||
|
no crash dialog,
|
||||||
|
- runs the simulation as fast as it can in fixed steps of 0.03 s of simulation time per
|
||||||
|
frame (three 10 ms physics steps), whatever the real time, the time speed or the machine,
|
||||||
|
- seeds its random numbers with a fixed seed unless `-seed` or `randomseed` gives one, so two
|
||||||
|
runs of a scenery come out the same (the physicslogs of two runs are byte for byte equal),
|
||||||
|
- quits when the test ends, and says how it ended in the exit code and in a `test: ...`
|
||||||
|
line on stdout and in the log.
|
||||||
|
|
||||||
|
A scenery reports its result to the memory cell `__test.result`, which the simulator creates
|
||||||
|
(in every run, so a test scenery loads normally too). Write value 1 to it for a pass and 0
|
||||||
|
for a failure, with an `UpdateValues` event; other values are ignored, and the text, if
|
||||||
|
given, goes to the log as the reason:
|
||||||
|
|
||||||
|
```
|
||||||
|
event test_pass UpdateValues 0 __test.result reached_the_stop 1 * endevent
|
||||||
|
event test_fail UpdateValues 0 __test.result wrong_track 0 * endevent
|
||||||
|
```
|
||||||
|
|
||||||
|
(`PutValues` isn't the event for it: it hands a command to a vehicle, it doesn't write a cell.)
|
||||||
|
A good place for the pass event is `event0` of a track the train should stop on: it runs
|
||||||
|
while a train stands there, and the length of the track is the margin of the stop.
|
||||||
|
|
||||||
|
The test ends with the first of these:
|
||||||
|
|
||||||
|
| exit code | result | when |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 0 | pass | `__test.result` was set to 1 and then every vehicle stood still for 1 s, so a train still moving towards a signal at stop has to stop first. Also the standstill end with `test.pass_on_standstill yes` |
|
||||||
|
| 1 | fail | `__test.result` was set to 0; ends at once |
|
||||||
|
| 2 | signal passed at danger | a train went past a signal showing stop (`SetVelocity 0 0`; `SetVelocity 0 x` on automatic block, which may be passed after a stop, doesn't count); ends at once, unless `test.fail_if_stop_overran no` |
|
||||||
|
| 3 | timeout | `test.timeout` seconds of simulation time went by (default 3600) |
|
||||||
|
| 4 | standstill | every vehicle stood still for `test.standstill` seconds (default 300) and nothing reported a result. Trains waiting at a station or for a signal count as standing, so the limit has to be longer than such waits |
|
||||||
|
| 5 | error | the run could not start: no `-s`, the scenery failed to load or has no vehicles, or the simulator threw |
|
||||||
|
|
||||||
|
A crash ends with the system's code instead (on Windows e.g. `0xC0000005`, -1073741819).
|
||||||
|
|
||||||
|
The limits go in a `config` block of the scenery:
|
||||||
|
|
||||||
|
```
|
||||||
|
config
|
||||||
|
test.timeout 600
|
||||||
|
test.standstill 120
|
||||||
|
test.pass_on_standstill no
|
||||||
|
test.fail_if_stop_overran yes
|
||||||
|
endconfig
|
||||||
|
```
|
||||||
|
|
||||||
|
`--test-timeout` on the command line wins over `test.timeout` of the scenery.
|
||||||
|
|
||||||
|
A signal is passed at danger when the head of a train goes past a signal which its driver
|
||||||
|
has in its speed table, showing stop. So it's the driver's view that counts: a train without
|
||||||
|
a driver isn't watched, and neither is a signal the driver never scanned (a train placed at
|
||||||
|
100 km/h 150 m before a signal at stop got past it before its AI first looked ahead; at
|
||||||
|
60 km/h it was caught). Outside the test mode the same is written to the log, nothing more.
|
||||||
|
|
||||||
## Running
|
## Running
|
||||||
|
|
||||||
```
|
```
|
||||||
python tests/scenarios/run.py --game-dir "E:/Program Files (x86)/Steam/steamapps/common/MaSzyna" -j 4
|
python tests/scenarios/run.py --game-dir "C:/Steam/steamapps/common/MaSzyna" -j 4
|
||||||
```
|
```
|
||||||
|
|
||||||
Python 3.10+, no extra packages. Options:
|
Python 3.10+, no extra packages. Options:
|
||||||
|
|
||||||
- `EU07_GAME_DIR` can replace `--game-dir`.
|
- `EU07_GAME_DIR` can replace `--game-dir`.
|
||||||
- `--exe path/to/eu07.exe` picks the build to test; the default is the newest
|
- `--exe path/to/eu07.exe` picks the build to test; the default is the newest
|
||||||
`cmake-build-*/bin/Release/eu07*.exe`. The build needs the `-ini` command line option.
|
`cmake-build-*/bin/Release/eu07*.exe`. The build needs the `--test` and `-ini` command
|
||||||
|
line options.
|
||||||
- name one or more cases to run only those, e.g. `run.py ... eu07_td_timetable`.
|
- name one or more cases to run only those, e.g. `run.py ... eu07_td_timetable`.
|
||||||
- `-j N` runs N cases at once, each in its own simulator window. Each case prints its
|
- `-j N` runs N cases at once. Each case prints its results when it is done. All 14 cases
|
||||||
results when it is done. All 8 cases take about a minute with `-j 4`.
|
take about 30 s with `-j 4`, loading included.
|
||||||
- `--timespeed N` sets the simulation time multiplier, default 20: a 4-minute case runs in
|
- `--watch` runs a case the old way, in a visible window in real time (or faster with
|
||||||
about 20 s. Physics still steps finely (the physicslog keeps its 0.05 s rows) and the speed
|
`--timespeed N`), closed after the case's `duration`; the exit code isn't checked then. For
|
||||||
profile at 20 and 60 matched a real-time run within a few km/h and seconds. Use
|
watching what a case does.
|
||||||
`--timespeed 1` to watch a run.
|
|
||||||
|
|
||||||
Exit code is 0 when every case passed. Logs of each run (`log.txt`, `errors.txt`,
|
Exit code is 0 when every case passed. Logs of each run (`log.txt`, `errors.txt`,
|
||||||
the vehicle's physicslog) are moved to `tests/scenarios/out/<case>/`.
|
`stdout.txt`, the vehicle's physicslog) are moved to `tests/scenarios/out/<case>/`.
|
||||||
|
|
||||||
## How it works
|
## How it works
|
||||||
|
|
||||||
For each case the runner:
|
For each case the runner:
|
||||||
|
|
||||||
1. writes `scenery/autotest_<case>.scn` into the game dir: the base scenery with its own
|
1. writes `scenery/autotest_<case>.scn` into the game dir: the base scenery with its own
|
||||||
trainsets removed, the start time changed if the case says so, and the case's `.scn`
|
trainsets removed, the start time changed if the case says so, the case's track events
|
||||||
fragment added: its trainsets after `FirstInit`, everything else (event launchers standing
|
added, and the case's `.scn` fragment added: its trainsets after `FirstInit`, everything
|
||||||
in for keys the scenery expects a player to press, extra events) before it,
|
else (the case's `test_config`, event launchers standing in for keys the scenery expects a
|
||||||
2. starts the simulator with `-s autotest_<case>.scn -v ghostview -ini out/<case>/autotest.ini`;
|
player to press, extra events) before it,
|
||||||
ghostview leaves every train to the AI, and the extra ini turns on `physicslog`, runs in a
|
2. starts the simulator with `--test -s autotest_<case>.scn -ini out/<case>/autotest.ini`,
|
||||||
1280x720 window, does not pause when the window loses focus, and writes the logs to
|
plus `--test-timeout <duration>` when the case gives a `duration`; the extra ini turns on
|
||||||
`logs/log (autotest_<case>) <date>.txt` rather than the shared `log.txt` (`multiplelogs`),
|
`physicslog` and writes the logs to `logs/log (autotest_<case>) <date>.txt` rather than the
|
||||||
3. waits until `physicslog/autotest_<case>.dat` covers `duration` seconds, then closes the
|
shared `log.txt` (`multiplelogs`),
|
||||||
simulator. It gives up early when the log reports `Bad vehicle` / `Bad trainset`, or when
|
3. waits until the simulator quits by itself. It gives up early when the log reports
|
||||||
the scenario has loaded and no physicslog appeared,
|
`Bad vehicle` / `Bad trainset`, when the scenario has loaded and no physicslog appeared, or
|
||||||
4. runs the checks on the physicslog.
|
after the case's `timeout` of wall-clock time,
|
||||||
|
4. compares the result from the exit code with the case's `expect`, then runs the checks on
|
||||||
|
the physicslog.
|
||||||
|
|
||||||
The lead vehicle is named `autotest_<case>`, so every case has its own physicslog and log
|
The lead vehicle is named `autotest_<case>`, so every case has its own physicslog and log
|
||||||
files and cases can run in parallel. The `autotest_*` sceneries (and the simulator's `.sbt`
|
files and cases can run in parallel. The `autotest_*` sceneries (and the simulator's `.sbt`
|
||||||
@@ -60,9 +132,12 @@ scenery is newer.
|
|||||||
| `strip_trainsets` | drop the base scenery's trainsets (default `true`) |
|
| `strip_trainsets` | drop the base scenery's trainsets (default `true`) |
|
||||||
| `time` | replace the scenery start time, `HH:MM` |
|
| `time` | replace the scenery start time, `HH:MM` |
|
||||||
| `track_friction` | set the friction of every track of the base scenery (default in the sceneries is 0.15), e.g. for slippery rail |
|
| `track_friction` | set the friction of every track of the base scenery (default in the sceneries is 0.15), e.g. for slippery rail |
|
||||||
|
| `track_events` | lines added to tracks of the base scenery, by track name, e.g. `{ "tor1k": "event0 test_pass" }`; only tracks in the scenery file itself, not in its includes |
|
||||||
|
| `test_config` | lines of a `config` block added to the scenery, e.g. `["test.timeout 90"]` |
|
||||||
| `fragment` | `.scn` file in `cases/` added to the scenery: trainsets after `FirstInit` (before it they derail on unlinked track), the rest before it (launchers read after it never fire) |
|
| `fragment` | `.scn` file in `cases/` added to the scenery: trainsets after `FirstInit` (before it they derail on unlinked track), the rest before it (launchers read after it never fire) |
|
||||||
| `vars` | values for `{{name}}` in the fragment; a list becomes one line per item |
|
| `vars` | values for `{{name}}` in the fragment; a list becomes one line per item |
|
||||||
| `duration` | seconds of simulation to record |
|
| `expect` | the result the test mode should end with: `pass`, `fail`, `signal_passed_at_danger`, `timeout`, `standstill` or `error`. Default `timeout` for a case with `duration`, else `pass` |
|
||||||
|
| `duration` | seconds of simulation to record: the test timeout of the run (`--test-timeout`) |
|
||||||
| `timeout` | wall-clock seconds before giving up, loading included |
|
| `timeout` | wall-clock seconds before giving up, loading included |
|
||||||
| `fail_on_errors` | fail when `errors.txt` is not empty (default `false`) |
|
| `fail_on_errors` | fail when `errors.txt` is not empty (default `false`) |
|
||||||
| `checks` | list of checks, below |
|
| `checks` | list of checks, below |
|
||||||
@@ -72,7 +147,14 @@ the one whose physicslog is checked. Other vehicles of the consist take names ma
|
|||||||
e.g. `{{vehicle}}_s`. The td.scn cases share two fragments, `td_timetable.scn` (route set,
|
e.g. `{{vehicle}}_s`. The td.scn cases share two fragments, `td_timetable.scn` (route set,
|
||||||
train runs to Przystanek) and `td_red_signal.scn` (no route, exit signal at stop), and differ
|
train runs to Przystanek) and `td_red_signal.scn` (no route, exit signal at stop), and differ
|
||||||
only in `vars.consist`. `td_red_signal_moving.scn` is the red signal one with the train
|
only in `vars.consist`. `td_red_signal_moving.scn` is the red signal one with the train
|
||||||
starting at `vars.velocity` km/h; with `track_friction` 0.03 it locks the wheels when braking.
|
starting at `vars.velocity` km/h; with `track_friction` 0.03 it locks the wheels when braking
|
||||||
|
and slides past the signal, so those cases set `test.fail_if_stop_overran no`.
|
||||||
|
`td_red_signal_pass.scn` adds the `test_pass` event, which `test_mode_pass` puts as `event0`
|
||||||
|
on `tor1k`, the track in front of the signal.
|
||||||
|
|
||||||
|
The `test_mode_*` cases show the ends of the test mode: `test_mode_pass` (exit 0, the pass
|
||||||
|
event), `test_mode_signal_passed_at_danger` (exit 2), `test_mode_timeout` (exit 3, from
|
||||||
|
`test.timeout`) and `test_mode_standstill` (exit 4, from `test.standstill`).
|
||||||
|
|
||||||
Checks (speeds in km/h, times in seconds of physicslog time, distances in metres). Most use
|
Checks (speeds in km/h, times in seconds of physicslog time, distances in metres). Most use
|
||||||
the physicslog Velocity column, which is the wheel speed: it stays at 0 while the wheels are
|
the physicslog Velocity column, which is the wheel speed: it stays at 0 while the wheels are
|
||||||
|
|||||||
@@ -4,6 +4,9 @@
|
|||||||
"time": "10:34",
|
"time": "10:34",
|
||||||
"track_friction": 0.03,
|
"track_friction": 0.03,
|
||||||
"fragment": "td_red_signal_moving.scn",
|
"fragment": "td_red_signal_moving.scn",
|
||||||
|
"test_config": [
|
||||||
|
"test.fail_if_stop_overran no"
|
||||||
|
],
|
||||||
"vars": {
|
"vars": {
|
||||||
"velocity": "30",
|
"velocity": "30",
|
||||||
"consist": [
|
"consist": [
|
||||||
|
|||||||
@@ -4,6 +4,9 @@
|
|||||||
"time": "10:34",
|
"time": "10:34",
|
||||||
"track_friction": 0.03,
|
"track_friction": 0.03,
|
||||||
"fragment": "td_red_signal_moving.scn",
|
"fragment": "td_red_signal_moving.scn",
|
||||||
|
"test_config": [
|
||||||
|
"test.fail_if_stop_overran no"
|
||||||
|
],
|
||||||
"vars": {
|
"vars": {
|
||||||
"velocity": "30",
|
"velocity": "30",
|
||||||
"consist": [
|
"consist": [
|
||||||
|
|||||||
8
tests/scenarios/cases/td_red_signal_pass.scn
Normal file
8
tests/scenarios/cases/td_red_signal_pass.scn
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
// added by tests/scenarios/run.py to td.scn, whose own trainsets were removed.
|
||||||
|
// like td_red_signal.scn (no route, the exit signal stacja1_j about 150 m ahead stays at stop),
|
||||||
|
// with the test mode's pass event: the case puts it as event0 on tor1k, the track in front of
|
||||||
|
// the signal, so it runs once the train stands there. the test passes when every vehicle stopped.
|
||||||
|
event test_pass UpdateValues 0 __test.result stopped_before_stacja1_j 1 * endevent
|
||||||
|
trainset td/rozklad tdo_n1734 0.0 0.1
|
||||||
|
{{consist}}
|
||||||
|
endtrainset
|
||||||
31
tests/scenarios/cases/test_mode_pass.json
Normal file
31
tests/scenarios/cases/test_mode_pass.json
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
{
|
||||||
|
"description": "Test mode, pass: EU07 on td.scn with no route set stops before the exit signal stacja1_j; event0 of the track there reports the pass, and the test ends as passed once the train stands.",
|
||||||
|
"base_scenery": "td.scn",
|
||||||
|
"time": "10:34",
|
||||||
|
"fragment": "td_red_signal_pass.scn",
|
||||||
|
"track_events": {
|
||||||
|
"tor1k": "event0 test_pass"
|
||||||
|
},
|
||||||
|
"vars": {
|
||||||
|
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
|
||||||
|
},
|
||||||
|
"expect": "pass",
|
||||||
|
"timeout": 600,
|
||||||
|
"checks": [
|
||||||
|
{
|
||||||
|
"type": "starts",
|
||||||
|
"min_speed": 5,
|
||||||
|
"within": 150
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "stops",
|
||||||
|
"moving_speed": 5,
|
||||||
|
"stop_speed": 0.5,
|
||||||
|
"hold": 0.5
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "distance",
|
||||||
|
"max": 150
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
22
tests/scenarios/cases/test_mode_signal_passed_at_danger.json
Normal file
22
tests/scenarios/cases/test_mode_signal_passed_at_danger.json
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
{
|
||||||
|
"description": "Test mode, signal passed at danger: EU07 on td.scn with no route set starts at 60 km/h about 150 m before the exit signal stacja1_j at stop. Its AI driver is still preparing the engine and doesn't brake, and the run ends when the train passes the signal.",
|
||||||
|
"base_scenery": "td.scn",
|
||||||
|
"time": "10:34",
|
||||||
|
"fragment": "td_red_signal_moving.scn",
|
||||||
|
"vars": {
|
||||||
|
"velocity": "60",
|
||||||
|
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
|
||||||
|
},
|
||||||
|
"expect": "signal_passed_at_danger",
|
||||||
|
"timeout": 600,
|
||||||
|
"checks": [
|
||||||
|
{
|
||||||
|
"type": "max_speed",
|
||||||
|
"limit": 61
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "distance",
|
||||||
|
"min": 140
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
26
tests/scenarios/cases/test_mode_standstill.json
Normal file
26
tests/scenarios/cases/test_mode_standstill.json
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
{
|
||||||
|
"description": "Test mode, standstill: EU07 on td.scn with no route set stops before the exit signal stacja1_j at stop and stays; nothing reports a result, so the test ends once every vehicle stood still for test.standstill 120 seconds (the train waits about 70 s before it pulls off, so less would end it before the start).",
|
||||||
|
"base_scenery": "td.scn",
|
||||||
|
"time": "10:34",
|
||||||
|
"fragment": "td_red_signal.scn",
|
||||||
|
"test_config": [
|
||||||
|
"test.standstill 120"
|
||||||
|
],
|
||||||
|
"vars": {
|
||||||
|
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
|
||||||
|
},
|
||||||
|
"expect": "standstill",
|
||||||
|
"timeout": 600,
|
||||||
|
"checks": [
|
||||||
|
{
|
||||||
|
"type": "stops",
|
||||||
|
"moving_speed": 5,
|
||||||
|
"stop_speed": 0.5,
|
||||||
|
"hold": 5
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "distance",
|
||||||
|
"max": 150
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
21
tests/scenarios/cases/test_mode_timeout.json
Normal file
21
tests/scenarios/cases/test_mode_timeout.json
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
{
|
||||||
|
"description": "Test mode, timeout: EU07 on td.scn with no route set pulls off towards the exit signal stacja1_j at stop; nothing reports a result and test.timeout 90 from the scenery's config block runs out first.",
|
||||||
|
"base_scenery": "td.scn",
|
||||||
|
"time": "10:34",
|
||||||
|
"fragment": "td_red_signal.scn",
|
||||||
|
"test_config": [
|
||||||
|
"test.timeout 90"
|
||||||
|
],
|
||||||
|
"vars": {
|
||||||
|
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
|
||||||
|
},
|
||||||
|
"expect": "timeout",
|
||||||
|
"timeout": 600,
|
||||||
|
"checks": [
|
||||||
|
{
|
||||||
|
"type": "starts",
|
||||||
|
"min_speed": 5,
|
||||||
|
"within": 90
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -2,16 +2,19 @@
|
|||||||
"""Scenario tests: run the real simulator on a scenery and check what a vehicle did.
|
"""Scenario tests: run the real simulator on a scenery and check what a vehicle did.
|
||||||
|
|
||||||
A case (cases/<name>.json) names a base scenery from the game install, a fragment
|
A case (cases/<name>.json) names a base scenery from the game install, a fragment
|
||||||
added to it (usually a trainset with an AI driver), and checks on the vehicle's
|
added to it (usually a trainset with an AI driver), the result the simulator's test
|
||||||
physics log. The runner:
|
mode should end with, and checks on the vehicle's physics log. The runner:
|
||||||
|
|
||||||
1. builds scenery/autotest_<case>.scn in the game dir: the base scenery with its own
|
1. builds scenery/autotest_<case>.scn in the game dir: the base scenery with its own
|
||||||
trainsets removed, optionally a different start time, and the case fragment added,
|
trainsets removed, optionally a different start time, and the case fragment added,
|
||||||
2. starts eu07 with that scenery in ghostview (every train driven by AI) and an extra
|
2. starts eu07 in test mode (--test: every train driven by AI, nothing drawn, fixed
|
||||||
settings file (-ini) that turns on physicslog and runs in a window,
|
time steps as fast as it can) with that scenery and an extra settings file (-ini)
|
||||||
3. waits until the vehicle's physicslog covers `duration` seconds of simulation,
|
that turns on physicslog,
|
||||||
then closes the simulator,
|
3. waits until the simulator quits by itself and reads the test result from its exit
|
||||||
4. evaluates the checks on the log and prints PASS/FAIL per check.
|
code: pass, fail, signal passed at danger, timeout (after `duration` seconds of
|
||||||
|
simulation, if the case gives one), standstill or error,
|
||||||
|
4. compares the result with the one the case expects, evaluates the checks on the
|
||||||
|
physicslog and prints PASS/FAIL per check.
|
||||||
|
|
||||||
Scenery and vehicle data are not part of this repo, so this runs against a local game
|
Scenery and vehicle data are not part of this repo, so this runs against a local game
|
||||||
install only (--game-dir or EU07_GAME_DIR), not in CI. See README.md.
|
install only (--game-dir or EU07_GAME_DIR), not in CI. See README.md.
|
||||||
@@ -49,7 +52,18 @@ COL_COMMAND = 16 # text, read as None
|
|||||||
COL_WHEELSLIP = 20
|
COL_WHEELSLIP = 20
|
||||||
COL_VEHICLE_VELOCITY = 25 # signed vehicle speed, km/h; only in builds which log it
|
COL_VEHICLE_VELOCITY = 25 # signed vehicle speed, km/h; only in builds which log it
|
||||||
|
|
||||||
# settings layered over the user's eu07.ini for a test run
|
# exit codes of the simulator's test mode (eu07 --test), see simulation/simulationtest.h
|
||||||
|
EXIT_RESULTS = {
|
||||||
|
0: "pass",
|
||||||
|
1: "fail",
|
||||||
|
2: "signal_passed_at_danger",
|
||||||
|
3: "timeout",
|
||||||
|
4: "standstill",
|
||||||
|
5: "error",
|
||||||
|
}
|
||||||
|
|
||||||
|
# settings layered over the user's eu07.ini for a test run; the window ones only matter
|
||||||
|
# with --watch, test mode keeps its window hidden and turns the sound off by itself
|
||||||
TEST_INI = """\
|
TEST_INI = """\
|
||||||
fullscreen no
|
fullscreen no
|
||||||
fullscreenwindowed no
|
fullscreenwindowed no
|
||||||
@@ -89,6 +103,17 @@ def set_start_time(text: str, hhmm: str) -> str:
|
|||||||
return re.sub(r"(?im)^(\s*time\s+)\d{1,2}:\d{2}", lambda m: m.group(1) + hhmm, text, count=1)
|
return re.sub(r"(?im)^(\s*time\s+)\d{1,2}:\d{2}", lambda m: m.group(1) + hhmm, text, count=1)
|
||||||
|
|
||||||
|
|
||||||
|
def add_track_events(text: str, events: dict) -> str:
|
||||||
|
"""Adds lines (e.g. "event0 test_pass") to tracks named in events, before their endtrack."""
|
||||||
|
for track, lines in events.items():
|
||||||
|
lines = [lines] if isinstance(lines, str) else lines
|
||||||
|
m = re.search(rf"(?im)^\s*node\s+\S+\s+\S+\s+{re.escape(track)}\s+track\b.*?^(\s*endtrack\b)", text, re.S)
|
||||||
|
if not m:
|
||||||
|
sys.exit(f"track_events: no track {track} in the base scenery itself (not in its includes)")
|
||||||
|
text = text[: m.start(1)] + "".join(line + "\n" for line in lines) + text[m.start(1):]
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
def set_track_friction(text: str, friction: float) -> str:
|
def set_track_friction(text: str, friction: float) -> str:
|
||||||
"""Sets the friction of every track: node <range> <range> <name> track <type> <length> <gauge> <friction> ..."""
|
"""Sets the friction of every track: node <range> <range> <name> track <type> <length> <gauge> <friction> ..."""
|
||||||
return re.sub(r"(?im)^(\s*node\s+\S+\s+\S+\s+\S+\s+track\s+\S+\s+\S+\s+\S+\s+)[-\d.]+",
|
return re.sub(r"(?im)^(\s*node\s+\S+\s+\S+\s+\S+\s+track\s+\S+\s+\S+\s+\S+\s+)[-\d.]+",
|
||||||
@@ -118,6 +143,8 @@ def build_scenery(case: dict, game_dir: Path, case_name: str, vehicle: str) -> s
|
|||||||
text = set_start_time(text, case["time"])
|
text = set_start_time(text, case["time"])
|
||||||
if "track_friction" in case:
|
if "track_friction" in case:
|
||||||
text = set_track_friction(text, case["track_friction"])
|
text = set_track_friction(text, case["track_friction"])
|
||||||
|
if "track_events" in case:
|
||||||
|
text = add_track_events(text, case["track_events"])
|
||||||
fragment = (CASES_DIR / case["fragment"]).read_bytes().decode("latin-1")
|
fragment = (CASES_DIR / case["fragment"]).read_bytes().decode("latin-1")
|
||||||
# vars first: {{vehicle}} inside a var's value gets filled in the second pass
|
# vars first: {{vehicle}} inside a var's value gets filled in the second pass
|
||||||
fragment = fill_vars(fragment, case.get("vars", {}), keep={"vehicle"})
|
fragment = fill_vars(fragment, case.get("vars", {}), keep={"vehicle"})
|
||||||
@@ -126,6 +153,9 @@ def build_scenery(case: dict, game_dir: Path, case_name: str, vehicle: str) -> s
|
|||||||
# events of the case go in before it (read after it they never fire), trainsets after it
|
# events of the case go in before it (read after it they never fire), trainsets after it
|
||||||
# (placed before it they stand on unlinked track and derail at once)
|
# (placed before it they stand on unlinked track and derail at once)
|
||||||
before, trainsets = split_trainsets(fragment)
|
before, trainsets = split_trainsets(fragment)
|
||||||
|
if case.get("test_config"):
|
||||||
|
# settings of the test mode (test.timeout and so on), read like any other config block
|
||||||
|
before = "config\n" + "".join(line + "\n" for line in case["test_config"]) + "endconfig\n" + before
|
||||||
header = f"\n// ---- autotest: {case_name} ----\n"
|
header = f"\n// ---- autotest: {case_name} ----\n"
|
||||||
firstinit = re.search(r"(?im)^\s*firstinit\b", text)
|
firstinit = re.search(r"(?im)^\s*firstinit\b", text)
|
||||||
if firstinit:
|
if firstinit:
|
||||||
@@ -200,20 +230,30 @@ STARTUP_TIMEOUT = 120.0
|
|||||||
@dataclass
|
@dataclass
|
||||||
class RunResult:
|
class RunResult:
|
||||||
rows: list[list[float]] = field(default_factory=list)
|
rows: list[list[float]] = field(default_factory=list)
|
||||||
exit_code: int | None = None # None: still running when we closed it (the normal end)
|
exit_code: int | None = None # None: still running when we closed it (the normal end with --watch)
|
||||||
|
verdict: str = "" # the simulator's own line about the test result
|
||||||
failure: str = "" # why the run was cut short
|
failure: str = "" # why the run was cut short
|
||||||
wall_seconds: float = 0.0
|
wall_seconds: float = 0.0
|
||||||
logs: dict[str, Path] = field(default_factory=dict)
|
logs: dict[str, Path] = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog: Path,
|
def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog: Path,
|
||||||
duration: float, timeout: float, say) -> RunResult:
|
duration: float | None, timeout: float, watch: bool, say) -> RunResult:
|
||||||
|
"""Runs the simulator on the scenery. In test mode it quits by itself and the exit code
|
||||||
|
is the result; with watch it runs in a window and gets closed after `duration`."""
|
||||||
physicslog.unlink(missing_ok=True)
|
physicslog.unlink(missing_ok=True)
|
||||||
|
if watch:
|
||||||
cmd = [str(exe), "-s", scenery, "-v", "ghostview", "-ini", str(ini)]
|
cmd = [str(exe), "-s", scenery, "-v", "ghostview", "-ini", str(ini)]
|
||||||
|
else:
|
||||||
|
cmd = [str(exe), "--test", "-s", scenery, "-ini", str(ini)]
|
||||||
|
if duration is not None:
|
||||||
|
cmd += ["--test-timeout", f"{duration:g}"]
|
||||||
say(" $ " + " ".join(f'"{c}"' if " " in c else c for c in cmd), progress=True)
|
say(" $ " + " ".join(f'"{c}"' if " " in c else c for c in cmd), progress=True)
|
||||||
since = time.time() - 1
|
since = time.time() - 1
|
||||||
started = time.monotonic()
|
started = time.monotonic()
|
||||||
proc = subprocess.Popen(cmd, cwd=game_dir)
|
# the test result line goes to stdout; a file, not a pipe, so a chatty run can't block on it
|
||||||
|
stdout = (ini.parent / "stdout.txt").open("w+b")
|
||||||
|
proc = subprocess.Popen(cmd, cwd=game_dir, stdout=stdout, stderr=subprocess.STDOUT)
|
||||||
result = RunResult()
|
result = RunResult()
|
||||||
last_report = 0.0
|
last_report = 0.0
|
||||||
try:
|
try:
|
||||||
@@ -229,7 +269,7 @@ def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog
|
|||||||
if proc.poll() is not None:
|
if proc.poll() is not None:
|
||||||
result.exit_code = proc.returncode
|
result.exit_code = proc.returncode
|
||||||
break
|
break
|
||||||
if sim_t >= duration:
|
if watch and sim_t >= duration:
|
||||||
break
|
break
|
||||||
if wall > timeout:
|
if wall > timeout:
|
||||||
result.failure = f"timed out after {timeout:.0f}s before {duration}s of simulation"
|
result.failure = f"timed out after {timeout:.0f}s before {duration}s of simulation"
|
||||||
@@ -252,6 +292,10 @@ def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog
|
|||||||
except subprocess.TimeoutExpired:
|
except subprocess.TimeoutExpired:
|
||||||
proc.kill()
|
proc.kill()
|
||||||
result.wall_seconds = time.monotonic() - started
|
result.wall_seconds = time.monotonic() - started
|
||||||
|
stdout.seek(0)
|
||||||
|
result.verdict = next((line.strip() for line in stdout.read().decode(errors="replace").splitlines()
|
||||||
|
if line.startswith("test:")), "")
|
||||||
|
stdout.close()
|
||||||
result.rows = read_physicslog(physicslog)
|
result.rows = read_physicslog(physicslog)
|
||||||
result.logs = find_logs(game_dir, scenery, since)
|
result.logs = find_logs(game_dir, scenery, since)
|
||||||
return result
|
return result
|
||||||
@@ -422,7 +466,14 @@ def count_lines(path: Path) -> int:
|
|||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, say) -> bool:
|
def describe_exit(code: int | None) -> str:
|
||||||
|
if code is None:
|
||||||
|
return "none, still running"
|
||||||
|
name = EXIT_RESULTS.get(code)
|
||||||
|
return f"{code} ({name})" if name else f"{code} (0x{code & 0xFFFFFFFF:08X}, a crash?)"
|
||||||
|
|
||||||
|
|
||||||
|
def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, watch: bool, say) -> bool:
|
||||||
case = json.loads((CASES_DIR / f"{name}.json").read_text(encoding="utf-8"))
|
case = json.loads((CASES_DIR / f"{name}.json").read_text(encoding="utf-8"))
|
||||||
say(f"\n=== {name}: {case.get('description', '')}")
|
say(f"\n=== {name}: {case.get('description', '')}")
|
||||||
out = OUT_DIR / name
|
out = OUT_DIR / name
|
||||||
@@ -436,8 +487,16 @@ def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, say) -> boo
|
|||||||
ini.write_text(TEST_INI.format(timespeed=timespeed), encoding="ascii")
|
ini.write_text(TEST_INI.format(timespeed=timespeed), encoding="ascii")
|
||||||
physicslog = game_dir / "physicslog" / f"{vehicle}.dat"
|
physicslog = game_dir / "physicslog" / f"{vehicle}.dat"
|
||||||
|
|
||||||
|
duration = float(case["duration"]) if "duration" in case else None
|
||||||
|
# a case that only records `duration` seconds ends on the test timeout; one that says
|
||||||
|
# how the test ends (a pass event, a signal passed at danger...) expects that result
|
||||||
|
expect = case.get("expect", "timeout" if duration is not None else "pass")
|
||||||
|
if expect not in EXIT_RESULTS.values():
|
||||||
|
sys.exit(f"{name}: expect must be one of {', '.join(EXIT_RESULTS.values())}")
|
||||||
|
if watch and duration is None:
|
||||||
|
sys.exit(f"{name}: --watch needs a case with duration")
|
||||||
result = run_simulator(exe, game_dir, scenery, ini, physicslog,
|
result = run_simulator(exe, game_dir, scenery, ini, physicslog,
|
||||||
float(case["duration"]), float(case.get("timeout", 600)), say)
|
duration, float(case.get("timeout", 600)), watch, say)
|
||||||
|
|
||||||
# the logs are ours alone (named after the case scenery), move them out of the game dir
|
# the logs are ours alone (named after the case scenery), move them out of the game dir
|
||||||
for kind, path in result.logs.items():
|
for kind, path in result.logs.items():
|
||||||
@@ -448,18 +507,26 @@ def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, say) -> boo
|
|||||||
ok = True
|
ok = True
|
||||||
sim_t = result.rows[-1][COL_TIME] if result.rows else 0.0
|
sim_t = result.rows[-1][COL_TIME] if result.rows else 0.0
|
||||||
say(f" ran {result.wall_seconds:.0f}s wall, {sim_t:.1f}s of physicslog, results in {out.relative_to(REPO)}")
|
say(f" ran {result.wall_seconds:.0f}s wall, {sim_t:.1f}s of physicslog, results in {out.relative_to(REPO)}")
|
||||||
|
if watch:
|
||||||
if result.exit_code is not None:
|
if result.exit_code is not None:
|
||||||
say(f" FAIL simulator exited on its own with code {result.exit_code}")
|
say(f" FAIL simulator exited on its own with code {result.exit_code}")
|
||||||
ok = False
|
ok = False
|
||||||
|
else:
|
||||||
|
if result.verdict:
|
||||||
|
say(f" {result.verdict}")
|
||||||
|
got = EXIT_RESULTS.get(result.exit_code) if result.exit_code is not None else None
|
||||||
|
passed = got == expect
|
||||||
|
say(f" {'PASS' if passed else 'FAIL'} exit code {describe_exit(result.exit_code)}, expected {expect}")
|
||||||
|
ok &= passed
|
||||||
if result.failure:
|
if result.failure:
|
||||||
say(f" FAIL {result.failure}")
|
say(f" FAIL {result.failure}")
|
||||||
ok = False
|
ok = False
|
||||||
if not result.rows:
|
if not result.rows and case.get("checks"):
|
||||||
if not result.failure:
|
if not result.failure:
|
||||||
say(f" FAIL no physicslog for '{vehicle}'")
|
say(f" FAIL no physicslog for '{vehicle}'")
|
||||||
return False
|
return False
|
||||||
|
|
||||||
for check in case["checks"]:
|
for check in case.get("checks", []):
|
||||||
params = {k: v for k, v in check.items() if k != "type"}
|
params = {k: v for k, v in check.items() if k != "type"}
|
||||||
passed, detail = CHECKS[check["type"]](result.rows, **params)
|
passed, detail = CHECKS[check["type"]](result.rows, **params)
|
||||||
say(f" {'PASS' if passed else 'FAIL'} {check['type']}: {detail}")
|
say(f" {'PASS' if passed else 'FAIL'} {check['type']}: {detail}")
|
||||||
@@ -479,12 +546,13 @@ def main() -> int:
|
|||||||
ap.add_argument("--game-dir", default=os.environ.get("EU07_GAME_DIR"),
|
ap.add_argument("--game-dir", default=os.environ.get("EU07_GAME_DIR"),
|
||||||
help="game install with scenery/ and dynamic/ (or EU07_GAME_DIR)")
|
help="game install with scenery/ and dynamic/ (or EU07_GAME_DIR)")
|
||||||
ap.add_argument("--exe", help="simulator executable (default: newest cmake-build-*/bin/Release/eu07*.exe)")
|
ap.add_argument("--exe", help="simulator executable (default: newest cmake-build-*/bin/Release/eu07*.exe)")
|
||||||
# 20 gave the same speed profile as 1 within a few km/h and seconds on eu07_td_timetable,
|
ap.add_argument("--watch", action="store_true",
|
||||||
# in 20 s instead of 4 min; 1 is for watching a run
|
help="run in a visible window in real time (see --timespeed) instead of the test mode, "
|
||||||
ap.add_argument("--timespeed", type=float, default=20.0,
|
"closed after the case's duration; the exit code isn't checked")
|
||||||
help="simulation time multiplier, the simulator offers 1, 5, 20 and 60 (default 20)")
|
ap.add_argument("--timespeed", type=float, default=1.0,
|
||||||
|
help="with --watch: simulation time multiplier, the simulator offers 1, 5, 20 and 60 (default 1)")
|
||||||
ap.add_argument("-j", "--jobs", type=int, default=1,
|
ap.add_argument("-j", "--jobs", type=int, default=1,
|
||||||
help="cases run at once, each in its own simulator window (default 1)")
|
help="cases run at once, each in a simulator of its own (default 1)")
|
||||||
args = ap.parse_args()
|
args = ap.parse_args()
|
||||||
|
|
||||||
if not args.game_dir:
|
if not args.game_dir:
|
||||||
@@ -504,7 +572,7 @@ def main() -> int:
|
|||||||
def say(line, progress=False):
|
def say(line, progress=False):
|
||||||
print(line, flush=True)
|
print(line, flush=True)
|
||||||
for n in names:
|
for n in names:
|
||||||
results[n] = run_case(n, exe, game_dir, args.timespeed, say)
|
results[n] = run_case(n, exe, game_dir, args.timespeed, args.watch, say)
|
||||||
else:
|
else:
|
||||||
# each case prints its block when it is done, so parallel output does not interleave
|
# each case prints its block when it is done, so parallel output does not interleave
|
||||||
def job(n):
|
def job(n):
|
||||||
@@ -512,7 +580,7 @@ def main() -> int:
|
|||||||
def say(line, progress=False):
|
def say(line, progress=False):
|
||||||
if not progress:
|
if not progress:
|
||||||
lines.append(line)
|
lines.append(line)
|
||||||
ok = run_case(n, exe, game_dir, args.timespeed, say)
|
ok = run_case(n, exe, game_dir, args.timespeed, args.watch, say)
|
||||||
return n, ok, lines
|
return n, ok, lines
|
||||||
with ThreadPoolExecutor(max_workers=args.jobs) as pool:
|
with ThreadPoolExecutor(max_workers=args.jobs) as pool:
|
||||||
for n in names:
|
for n in names:
|
||||||
|
|||||||
Reference in New Issue
Block a user