18
0
mirror of https://github.com/MaSzyna-EU07/maszyna.git synced 2026-11-16 14:34:11 +01:00

test: scenario tests run in the simulator's test mode and check its exit code

run.py starts eu07 with --test (and --test-timeout <duration> for cases that
record a fixed time) and compares the result from the exit code with the
case's expect key, before the physicslog checks. The 14 cases take about 30 s
with -j 4 instead of minutes. --watch keeps the old windowed run.

New case keys: expect, test_config (lines of a config block), track_events
(lines added to a track of the base scenery, e.g. event0 test_pass). New cases
show each end of the test mode: test_mode_pass, _signal_passed_at_danger,
_timeout and _standstill. The slippery rail cases slide past the red signal,
which now ends a test, so they set test.fail_if_stop_overran no.

README documents the test mode for scenery authors.
This commit is contained in:
jerrrrycho
2026-11-07 13:17:27 +01:00
parent 02e29f6971
commit 6a538d202b
9 changed files with 315 additions and 51 deletions

View File

@@ -1,49 +1,121 @@
# Scenario tests
End-to-end checks that run the real simulator on a scenery and look at what a vehicle
did: did it start, did it keep to the speed limit, did it brake and stop.
End-to-end checks that run the real simulator on a scenery and look at how the run ended
and what a vehicle did: did it start, did it keep to the speed limit, did it brake and stop,
did it pass a signal at stop.
Scenery and vehicle data are not in this repo, so these run against a local game install
only, not in CI.
## The simulator's test mode
The runner drives the simulator's test mode (eu07/maszyna#541), which works without the
runner too and is meant for scenery authors as well:
```
eu07 --test -s <scenery.scn> [--test-timeout <seconds>] [-seed <number|word>] [-ini <settings file>]
```
With `--test` the simulator:
- needs `-s`; `-v` is ignored and every train is driven by AI (ghostview),
- draws nothing: the null renderer loads the 3D models (the physics needs some of their
data) but no textures, and the window stays hidden,
- has no sound, no network, no python, no pause (not even one set by the scenery), and
no crash dialog,
- runs the simulation as fast as it can in fixed steps of 0.03 s of simulation time per
frame (three 10 ms physics steps), whatever the real time, the time speed or the machine,
- seeds its random numbers with a fixed seed unless `-seed` or `randomseed` gives one, so two
runs of a scenery come out the same (the physicslogs of two runs are byte for byte equal),
- quits when the test ends, and says how it ended in the exit code and in a `test: ...`
line on stdout and in the log.
A scenery reports its result to the memory cell `__test.result`, which the simulator creates
(in every run, so a test scenery loads normally too). Write value 1 to it for a pass and 0
for a failure, with an `UpdateValues` event; other values are ignored, and the text, if
given, goes to the log as the reason:
```
event test_pass UpdateValues 0 __test.result reached_the_stop 1 * endevent
event test_fail UpdateValues 0 __test.result wrong_track 0 * endevent
```
(`PutValues` isn't the event for it: it hands a command to a vehicle, it doesn't write a cell.)
A good place for the pass event is `event0` of a track the train should stop on: it runs
while a train stands there, and the length of the track is the margin of the stop.
The test ends with the first of these:
| exit code | result | when |
| --- | --- | --- |
| 0 | pass | `__test.result` was set to 1 and then every vehicle stood still for 1 s, so a train still moving towards a signal at stop has to stop first. Also the standstill end with `test.pass_on_standstill yes` |
| 1 | fail | `__test.result` was set to 0; ends at once |
| 2 | signal passed at danger | a train went past a signal showing stop (`SetVelocity 0 0`; `SetVelocity 0 x` on automatic block, which may be passed after a stop, doesn't count); ends at once, unless `test.fail_if_stop_overran no` |
| 3 | timeout | `test.timeout` seconds of simulation time went by (default 3600) |
| 4 | standstill | every vehicle stood still for `test.standstill` seconds (default 300) and nothing reported a result. Trains waiting at a station or for a signal count as standing, so the limit has to be longer than such waits |
| 5 | error | the run could not start: no `-s`, the scenery failed to load or has no vehicles, or the simulator threw |
A crash ends with the system's code instead (on Windows e.g. `0xC0000005`, -1073741819).
The limits go in a `config` block of the scenery:
```
config
test.timeout 600
test.standstill 120
test.pass_on_standstill no
test.fail_if_stop_overran yes
endconfig
```
`--test-timeout` on the command line wins over `test.timeout` of the scenery.
A signal is passed at danger when the head of a train goes past a signal which its driver
has in its speed table, showing stop. So it's the driver's view that counts: a train without
a driver isn't watched, and neither is a signal the driver never scanned (a train placed at
100 km/h 150 m before a signal at stop got past it before its AI first looked ahead; at
60 km/h it was caught). Outside the test mode the same is written to the log, nothing more.
## Running
```
python tests/scenarios/run.py --game-dir "E:/Program Files (x86)/Steam/steamapps/common/MaSzyna" -j 4
python tests/scenarios/run.py --game-dir "C:/Steam/steamapps/common/MaSzyna" -j 4
```
Python 3.10+, no extra packages. Options:
- `EU07_GAME_DIR` can replace `--game-dir`.
- `--exe path/to/eu07.exe` picks the build to test; the default is the newest
`cmake-build-*/bin/Release/eu07*.exe`. The build needs the `-ini` command line option.
`cmake-build-*/bin/Release/eu07*.exe`. The build needs the `--test` and `-ini` command
line options.
- name one or more cases to run only those, e.g. `run.py ... eu07_td_timetable`.
- `-j N` runs N cases at once, each in its own simulator window. Each case prints its
results when it is done. All 8 cases take about a minute with `-j 4`.
- `--timespeed N` sets the simulation time multiplier, default 20: a 4-minute case runs in
about 20 s. Physics still steps finely (the physicslog keeps its 0.05 s rows) and the speed
profile at 20 and 60 matched a real-time run within a few km/h and seconds. Use
`--timespeed 1` to watch a run.
- `-j N` runs N cases at once. Each case prints its results when it is done. All 14 cases
take about 30 s with `-j 4`, loading included.
- `--watch` runs a case the old way, in a visible window in real time (or faster with
`--timespeed N`), closed after the case's `duration`; the exit code isn't checked then. For
watching what a case does.
Exit code is 0 when every case passed. Logs of each run (`log.txt`, `errors.txt`,
the vehicle's physicslog) are moved to `tests/scenarios/out/<case>/`.
`stdout.txt`, the vehicle's physicslog) are moved to `tests/scenarios/out/<case>/`.
## How it works
For each case the runner:
1. writes `scenery/autotest_<case>.scn` into the game dir: the base scenery with its own
trainsets removed, the start time changed if the case says so, and the case's `.scn`
fragment added: its trainsets after `FirstInit`, everything else (event launchers standing
in for keys the scenery expects a player to press, extra events) before it,
2. starts the simulator with `-s autotest_<case>.scn -v ghostview -ini out/<case>/autotest.ini`;
ghostview leaves every train to the AI, and the extra ini turns on `physicslog`, runs in a
1280x720 window, does not pause when the window loses focus, and writes the logs to
`logs/log (autotest_<case>) <date>.txt` rather than the shared `log.txt` (`multiplelogs`),
3. waits until `physicslog/autotest_<case>.dat` covers `duration` seconds, then closes the
simulator. It gives up early when the log reports `Bad vehicle` / `Bad trainset`, or when
the scenario has loaded and no physicslog appeared,
4. runs the checks on the physicslog.
trainsets removed, the start time changed if the case says so, the case's track events
added, and the case's `.scn` fragment added: its trainsets after `FirstInit`, everything
else (the case's `test_config`, event launchers standing in for keys the scenery expects a
player to press, extra events) before it,
2. starts the simulator with `--test -s autotest_<case>.scn -ini out/<case>/autotest.ini`,
plus `--test-timeout <duration>` when the case gives a `duration`; the extra ini turns on
`physicslog` and writes the logs to `logs/log (autotest_<case>) <date>.txt` rather than the
shared `log.txt` (`multiplelogs`),
3. waits until the simulator quits by itself. It gives up early when the log reports
`Bad vehicle` / `Bad trainset`, when the scenario has loaded and no physicslog appeared, or
after the case's `timeout` of wall-clock time,
4. compares the result from the exit code with the case's `expect`, then runs the checks on
the physicslog.
The lead vehicle is named `autotest_<case>`, so every case has its own physicslog and log
files and cases can run in parallel. The `autotest_*` sceneries (and the simulator's `.sbt`
@@ -60,9 +132,12 @@ scenery is newer.
| `strip_trainsets` | drop the base scenery's trainsets (default `true`) |
| `time` | replace the scenery start time, `HH:MM` |
| `track_friction` | set the friction of every track of the base scenery (default in the sceneries is 0.15), e.g. for slippery rail |
| `track_events` | lines added to tracks of the base scenery, by track name, e.g. `{ "tor1k": "event0 test_pass" }`; only tracks in the scenery file itself, not in its includes |
| `test_config` | lines of a `config` block added to the scenery, e.g. `["test.timeout 90"]` |
| `fragment` | `.scn` file in `cases/` added to the scenery: trainsets after `FirstInit` (before it they derail on unlinked track), the rest before it (launchers read after it never fire) |
| `vars` | values for `{{name}}` in the fragment; a list becomes one line per item |
| `duration` | seconds of simulation to record |
| `expect` | the result the test mode should end with: `pass`, `fail`, `signal_passed_at_danger`, `timeout`, `standstill` or `error`. Default `timeout` for a case with `duration`, else `pass` |
| `duration` | seconds of simulation to record: the test timeout of the run (`--test-timeout`) |
| `timeout` | wall-clock seconds before giving up, loading included |
| `fail_on_errors` | fail when `errors.txt` is not empty (default `false`) |
| `checks` | list of checks, below |
@@ -72,7 +147,14 @@ the one whose physicslog is checked. Other vehicles of the consist take names ma
e.g. `{{vehicle}}_s`. The td.scn cases share two fragments, `td_timetable.scn` (route set,
train runs to Przystanek) and `td_red_signal.scn` (no route, exit signal at stop), and differ
only in `vars.consist`. `td_red_signal_moving.scn` is the red signal one with the train
starting at `vars.velocity` km/h; with `track_friction` 0.03 it locks the wheels when braking.
starting at `vars.velocity` km/h; with `track_friction` 0.03 it locks the wheels when braking
and slides past the signal, so those cases set `test.fail_if_stop_overran no`.
`td_red_signal_pass.scn` adds the `test_pass` event, which `test_mode_pass` puts as `event0`
on `tor1k`, the track in front of the signal.
The `test_mode_*` cases show the ends of the test mode: `test_mode_pass` (exit 0, the pass
event), `test_mode_signal_passed_at_danger` (exit 2), `test_mode_timeout` (exit 3, from
`test.timeout`) and `test_mode_standstill` (exit 4, from `test.standstill`).
Checks (speeds in km/h, times in seconds of physicslog time, distances in metres). Most use
the physicslog Velocity column, which is the wheel speed: it stays at 0 while the wheels are

View File

@@ -4,6 +4,9 @@
"time": "10:34",
"track_friction": 0.03,
"fragment": "td_red_signal_moving.scn",
"test_config": [
"test.fail_if_stop_overran no"
],
"vars": {
"velocity": "30",
"consist": [

View File

@@ -4,6 +4,9 @@
"time": "10:34",
"track_friction": 0.03,
"fragment": "td_red_signal_moving.scn",
"test_config": [
"test.fail_if_stop_overran no"
],
"vars": {
"velocity": "30",
"consist": [

View File

@@ -0,0 +1,8 @@
// added by tests/scenarios/run.py to td.scn, whose own trainsets were removed.
// like td_red_signal.scn (no route, the exit signal stacja1_j about 150 m ahead stays at stop),
// with the test mode's pass event: the case puts it as event0 on tor1k, the track in front of
// the signal, so it runs once the train stands there. the test passes when every vehicle stopped.
event test_pass UpdateValues 0 __test.result stopped_before_stacja1_j 1 * endevent
trainset td/rozklad tdo_n1734 0.0 0.1
{{consist}}
endtrainset

View File

@@ -0,0 +1,31 @@
{
"description": "Test mode, pass: EU07 on td.scn with no route set stops before the exit signal stacja1_j; event0 of the track there reports the pass, and the test ends as passed once the train stands.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_red_signal_pass.scn",
"track_events": {
"tor1k": "event0 test_pass"
},
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
},
"expect": "pass",
"timeout": 600,
"checks": [
{
"type": "starts",
"min_speed": 5,
"within": 150
},
{
"type": "stops",
"moving_speed": 5,
"stop_speed": 0.5,
"hold": 0.5
},
{
"type": "distance",
"max": 150
}
]
}

View File

@@ -0,0 +1,22 @@
{
"description": "Test mode, signal passed at danger: EU07 on td.scn with no route set starts at 60 km/h about 150 m before the exit signal stacja1_j at stop. Its AI driver is still preparing the engine and doesn't brake, and the run ends when the train passes the signal.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_red_signal_moving.scn",
"vars": {
"velocity": "60",
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
},
"expect": "signal_passed_at_danger",
"timeout": 600,
"checks": [
{
"type": "max_speed",
"limit": 61
},
{
"type": "distance",
"min": 140
}
]
}

View File

@@ -0,0 +1,26 @@
{
"description": "Test mode, standstill: EU07 on td.scn with no route set stops before the exit signal stacja1_j at stop and stays; nothing reports a result, so the test ends once every vehicle stood still for test.standstill 120 seconds (the train waits about 70 s before it pulls off, so less would end it before the start).",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_red_signal.scn",
"test_config": [
"test.standstill 120"
],
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
},
"expect": "standstill",
"timeout": 600,
"checks": [
{
"type": "stops",
"moving_speed": 5,
"stop_speed": 0.5,
"hold": 5
},
{
"type": "distance",
"max": 150
}
]
}

View File

@@ -0,0 +1,21 @@
{
"description": "Test mode, timeout: EU07 on td.scn with no route set pulls off towards the exit signal stacja1_j at stop; nothing reports a result and test.timeout 90 from the scenery's config block runs out first.",
"base_scenery": "td.scn",
"time": "10:34",
"fragment": "td_red_signal.scn",
"test_config": [
"test.timeout 90"
],
"vars": {
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
},
"expect": "timeout",
"timeout": 600,
"checks": [
{
"type": "starts",
"min_speed": 5,
"within": 90
}
]
}

View File

@@ -2,16 +2,19 @@
"""Scenario tests: run the real simulator on a scenery and check what a vehicle did.
A case (cases/<name>.json) names a base scenery from the game install, a fragment
added to it (usually a trainset with an AI driver), and checks on the vehicle's
physics log. The runner:
added to it (usually a trainset with an AI driver), the result the simulator's test
mode should end with, and checks on the vehicle's physics log. The runner:
1. builds scenery/autotest_<case>.scn in the game dir: the base scenery with its own
trainsets removed, optionally a different start time, and the case fragment added,
2. starts eu07 with that scenery in ghostview (every train driven by AI) and an extra
settings file (-ini) that turns on physicslog and runs in a window,
3. waits until the vehicle's physicslog covers `duration` seconds of simulation,
then closes the simulator,
4. evaluates the checks on the log and prints PASS/FAIL per check.
2. starts eu07 in test mode (--test: every train driven by AI, nothing drawn, fixed
time steps as fast as it can) with that scenery and an extra settings file (-ini)
that turns on physicslog,
3. waits until the simulator quits by itself and reads the test result from its exit
code: pass, fail, signal passed at danger, timeout (after `duration` seconds of
simulation, if the case gives one), standstill or error,
4. compares the result with the one the case expects, evaluates the checks on the
physicslog and prints PASS/FAIL per check.
Scenery and vehicle data are not part of this repo, so this runs against a local game
install only (--game-dir or EU07_GAME_DIR), not in CI. See README.md.
@@ -49,7 +52,18 @@ COL_COMMAND = 16 # text, read as None
COL_WHEELSLIP = 20
COL_VEHICLE_VELOCITY = 25 # signed vehicle speed, km/h; only in builds which log it
# settings layered over the user's eu07.ini for a test run
# exit codes of the simulator's test mode (eu07 --test), see simulation/simulationtest.h
EXIT_RESULTS = {
0: "pass",
1: "fail",
2: "signal_passed_at_danger",
3: "timeout",
4: "standstill",
5: "error",
}
# settings layered over the user's eu07.ini for a test run; the window ones only matter
# with --watch, test mode keeps its window hidden and turns the sound off by itself
TEST_INI = """\
fullscreen no
fullscreenwindowed no
@@ -89,6 +103,17 @@ def set_start_time(text: str, hhmm: str) -> str:
return re.sub(r"(?im)^(\s*time\s+)\d{1,2}:\d{2}", lambda m: m.group(1) + hhmm, text, count=1)
def add_track_events(text: str, events: dict) -> str:
"""Adds lines (e.g. "event0 test_pass") to tracks named in events, before their endtrack."""
for track, lines in events.items():
lines = [lines] if isinstance(lines, str) else lines
m = re.search(rf"(?im)^\s*node\s+\S+\s+\S+\s+{re.escape(track)}\s+track\b.*?^(\s*endtrack\b)", text, re.S)
if not m:
sys.exit(f"track_events: no track {track} in the base scenery itself (not in its includes)")
text = text[: m.start(1)] + "".join(line + "\n" for line in lines) + text[m.start(1):]
return text
def set_track_friction(text: str, friction: float) -> str:
"""Sets the friction of every track: node <range> <range> <name> track <type> <length> <gauge> <friction> ..."""
return re.sub(r"(?im)^(\s*node\s+\S+\s+\S+\s+\S+\s+track\s+\S+\s+\S+\s+\S+\s+)[-\d.]+",
@@ -118,6 +143,8 @@ def build_scenery(case: dict, game_dir: Path, case_name: str, vehicle: str) -> s
text = set_start_time(text, case["time"])
if "track_friction" in case:
text = set_track_friction(text, case["track_friction"])
if "track_events" in case:
text = add_track_events(text, case["track_events"])
fragment = (CASES_DIR / case["fragment"]).read_bytes().decode("latin-1")
# vars first: {{vehicle}} inside a var's value gets filled in the second pass
fragment = fill_vars(fragment, case.get("vars", {}), keep={"vehicle"})
@@ -126,6 +153,9 @@ def build_scenery(case: dict, game_dir: Path, case_name: str, vehicle: str) -> s
# events of the case go in before it (read after it they never fire), trainsets after it
# (placed before it they stand on unlinked track and derail at once)
before, trainsets = split_trainsets(fragment)
if case.get("test_config"):
# settings of the test mode (test.timeout and so on), read like any other config block
before = "config\n" + "".join(line + "\n" for line in case["test_config"]) + "endconfig\n" + before
header = f"\n// ---- autotest: {case_name} ----\n"
firstinit = re.search(r"(?im)^\s*firstinit\b", text)
if firstinit:
@@ -200,20 +230,30 @@ STARTUP_TIMEOUT = 120.0
@dataclass
class RunResult:
rows: list[list[float]] = field(default_factory=list)
exit_code: int | None = None # None: still running when we closed it (the normal end)
exit_code: int | None = None # None: still running when we closed it (the normal end with --watch)
verdict: str = "" # the simulator's own line about the test result
failure: str = "" # why the run was cut short
wall_seconds: float = 0.0
logs: dict[str, Path] = field(default_factory=dict)
def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog: Path,
duration: float, timeout: float, say) -> RunResult:
duration: float | None, timeout: float, watch: bool, say) -> RunResult:
"""Runs the simulator on the scenery. In test mode it quits by itself and the exit code
is the result; with watch it runs in a window and gets closed after `duration`."""
physicslog.unlink(missing_ok=True)
cmd = [str(exe), "-s", scenery, "-v", "ghostview", "-ini", str(ini)]
if watch:
cmd = [str(exe), "-s", scenery, "-v", "ghostview", "-ini", str(ini)]
else:
cmd = [str(exe), "--test", "-s", scenery, "-ini", str(ini)]
if duration is not None:
cmd += ["--test-timeout", f"{duration:g}"]
say(" $ " + " ".join(f'"{c}"' if " " in c else c for c in cmd), progress=True)
since = time.time() - 1
started = time.monotonic()
proc = subprocess.Popen(cmd, cwd=game_dir)
# the test result line goes to stdout; a file, not a pipe, so a chatty run can't block on it
stdout = (ini.parent / "stdout.txt").open("w+b")
proc = subprocess.Popen(cmd, cwd=game_dir, stdout=stdout, stderr=subprocess.STDOUT)
result = RunResult()
last_report = 0.0
try:
@@ -229,7 +269,7 @@ def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog
if proc.poll() is not None:
result.exit_code = proc.returncode
break
if sim_t >= duration:
if watch and sim_t >= duration:
break
if wall > timeout:
result.failure = f"timed out after {timeout:.0f}s before {duration}s of simulation"
@@ -252,6 +292,10 @@ def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog
except subprocess.TimeoutExpired:
proc.kill()
result.wall_seconds = time.monotonic() - started
stdout.seek(0)
result.verdict = next((line.strip() for line in stdout.read().decode(errors="replace").splitlines()
if line.startswith("test:")), "")
stdout.close()
result.rows = read_physicslog(physicslog)
result.logs = find_logs(game_dir, scenery, since)
return result
@@ -422,7 +466,14 @@ def count_lines(path: Path) -> int:
return 0
def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, say) -> bool:
def describe_exit(code: int | None) -> str:
if code is None:
return "none, still running"
name = EXIT_RESULTS.get(code)
return f"{code} ({name})" if name else f"{code} (0x{code & 0xFFFFFFFF:08X}, a crash?)"
def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, watch: bool, say) -> bool:
case = json.loads((CASES_DIR / f"{name}.json").read_text(encoding="utf-8"))
say(f"\n=== {name}: {case.get('description', '')}")
out = OUT_DIR / name
@@ -436,8 +487,16 @@ def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, say) -> boo
ini.write_text(TEST_INI.format(timespeed=timespeed), encoding="ascii")
physicslog = game_dir / "physicslog" / f"{vehicle}.dat"
duration = float(case["duration"]) if "duration" in case else None
# a case that only records `duration` seconds ends on the test timeout; one that says
# how the test ends (a pass event, a signal passed at danger...) expects that result
expect = case.get("expect", "timeout" if duration is not None else "pass")
if expect not in EXIT_RESULTS.values():
sys.exit(f"{name}: expect must be one of {', '.join(EXIT_RESULTS.values())}")
if watch and duration is None:
sys.exit(f"{name}: --watch needs a case with duration")
result = run_simulator(exe, game_dir, scenery, ini, physicslog,
float(case["duration"]), float(case.get("timeout", 600)), say)
duration, float(case.get("timeout", 600)), watch, say)
# the logs are ours alone (named after the case scenery), move them out of the game dir
for kind, path in result.logs.items():
@@ -448,18 +507,26 @@ def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, say) -> boo
ok = True
sim_t = result.rows[-1][COL_TIME] if result.rows else 0.0
say(f" ran {result.wall_seconds:.0f}s wall, {sim_t:.1f}s of physicslog, results in {out.relative_to(REPO)}")
if result.exit_code is not None:
say(f" FAIL simulator exited on its own with code {result.exit_code}")
ok = False
if watch:
if result.exit_code is not None:
say(f" FAIL simulator exited on its own with code {result.exit_code}")
ok = False
else:
if result.verdict:
say(f" {result.verdict}")
got = EXIT_RESULTS.get(result.exit_code) if result.exit_code is not None else None
passed = got == expect
say(f" {'PASS' if passed else 'FAIL'} exit code {describe_exit(result.exit_code)}, expected {expect}")
ok &= passed
if result.failure:
say(f" FAIL {result.failure}")
ok = False
if not result.rows:
if not result.rows and case.get("checks"):
if not result.failure:
say(f" FAIL no physicslog for '{vehicle}'")
return False
for check in case["checks"]:
for check in case.get("checks", []):
params = {k: v for k, v in check.items() if k != "type"}
passed, detail = CHECKS[check["type"]](result.rows, **params)
say(f" {'PASS' if passed else 'FAIL'} {check['type']}: {detail}")
@@ -479,12 +546,13 @@ def main() -> int:
ap.add_argument("--game-dir", default=os.environ.get("EU07_GAME_DIR"),
help="game install with scenery/ and dynamic/ (or EU07_GAME_DIR)")
ap.add_argument("--exe", help="simulator executable (default: newest cmake-build-*/bin/Release/eu07*.exe)")
# 20 gave the same speed profile as 1 within a few km/h and seconds on eu07_td_timetable,
# in 20 s instead of 4 min; 1 is for watching a run
ap.add_argument("--timespeed", type=float, default=20.0,
help="simulation time multiplier, the simulator offers 1, 5, 20 and 60 (default 20)")
ap.add_argument("--watch", action="store_true",
help="run in a visible window in real time (see --timespeed) instead of the test mode, "
"closed after the case's duration; the exit code isn't checked")
ap.add_argument("--timespeed", type=float, default=1.0,
help="with --watch: simulation time multiplier, the simulator offers 1, 5, 20 and 60 (default 1)")
ap.add_argument("-j", "--jobs", type=int, default=1,
help="cases run at once, each in its own simulator window (default 1)")
help="cases run at once, each in a simulator of its own (default 1)")
args = ap.parse_args()
if not args.game_dir:
@@ -504,7 +572,7 @@ def main() -> int:
def say(line, progress=False):
print(line, flush=True)
for n in names:
results[n] = run_case(n, exe, game_dir, args.timespeed, say)
results[n] = run_case(n, exe, game_dir, args.timespeed, args.watch, say)
else:
# each case prints its block when it is done, so parallel output does not interleave
def job(n):
@@ -512,7 +580,7 @@ def main() -> int:
def say(line, progress=False):
if not progress:
lines.append(line)
ok = run_case(n, exe, game_dir, args.timespeed, say)
ok = run_case(n, exe, game_dir, args.timespeed, args.watch, say)
return n, ok, lines
with ThreadPoolExecutor(max_workers=args.jobs) as pool:
for n in names: