mirror of
https://github.com/MaSzyna-EU07/maszyna.git
synced 2026-11-19 12:16:02 +01:00
feat(simulation): a real time watchdog ends a test run that hangs
A test run that hung, in the loading or anywhere else, waited until whoever started it killed it. A watchdog thread now ends a run still going test.wall_timeout seconds of real time after the start of the simulator, loading included: it says `test: timeout ... still running after N s of real time` and quits with the timeout code, whatever the main thread is doing, with no dialog. A test that has ended but whose simulator doesn't close ends with its own result. The limit is test.wall_timeout (settings, the scenery's config block) or --test-wall-timeout, which wins. Not given, it is test.timeout + 600 s: ten minutes for the loading, and a simulation that runs at least as fast as real time is never cut short. test.timeout 0 means no watchdog unless the limit is given; test.wall_timeout 0 turns it off. run.py passes the case's timeout as --test-wall-timeout and kills a simulator only 30 s after it. A timeout from the watchdog counts as wall_timeout there, so a run that hung doesn't pass a case expecting the simulation time to run out; test_mode_wall_timeout ends while loading on a limit of 2 s.
This commit is contained in:
@@ -13,7 +13,7 @@ The runner drives the simulator's test mode (eu07/maszyna#541), which works with
|
||||
runner too and is meant for scenery authors as well:
|
||||
|
||||
```
|
||||
eu07 --test -s <scenery.scn> [--test-timeout <seconds>] [-seed <number|word>] [-ini <settings file>]
|
||||
eu07 --test -s <scenery.scn> [--test-timeout <seconds>] [--test-wall-timeout <seconds>] [-seed <number|word>] [-ini <settings file>]
|
||||
```
|
||||
|
||||
With `--test` the simulator:
|
||||
@@ -70,7 +70,7 @@ The test ends with the first of these:
|
||||
| 0 | pass | `__test.result` was set to 1 and then every vehicle stood still for 1 s, so a train still moving towards a signal at stop has to stop first (with `test.no_movement_before_pass no` the pass ends the test at once). Also the standstill end with `test.pass_on_standstill yes` |
|
||||
| 10 | fail | `__test.result` was set to 0; ends at once |
|
||||
| 11 | signal passed at danger | a train went past a signal showing stop (`SetVelocity 0 0`; `SetVelocity 0 x` on automatic block, which may be passed after a stop, doesn't count); ends at once, unless `test.fail_if_stop_overran no` |
|
||||
| 12 | timeout | `test.timeout` seconds of simulation time went by (default 3600) |
|
||||
| 12 | timeout | `test.timeout` seconds of simulation time went by (default 3600), or `test.wall_timeout` seconds of real time since the start (see below) |
|
||||
| 13 | standstill | every vehicle stood still for `test.standstill` seconds (default 300) and nothing reported a result. Trains waiting at a station or for a signal count as standing, so the limit has to be longer than such waits |
|
||||
| 14 | error | the run could not start: no `-s`, the scenery failed to load or has no vehicles, or the simulator threw |
|
||||
|
||||
@@ -84,6 +84,7 @@ The limits go in a `config` block of the scenery:
|
||||
```
|
||||
config
|
||||
test.timeout 600
|
||||
test.wall_timeout 900
|
||||
test.standstill 120
|
||||
test.pass_on_standstill no
|
||||
test.fail_if_stop_overran yes
|
||||
@@ -94,12 +95,22 @@ endconfig
|
||||
| key | default | meaning |
|
||||
| --- | --- | --- |
|
||||
| `test.timeout` | 3600 | seconds of simulation time before the test ends as `timeout`; 0 = never |
|
||||
| `test.wall_timeout` | `test.timeout` + 600 | seconds of real time since the start before the test ends as `timeout`; 0 = never |
|
||||
| `test.standstill` | 300 | seconds with every vehicle still and no result before the test ends as `standstill`; 0 = never |
|
||||
| `test.pass_on_standstill` | no | the standstill end passes the test instead |
|
||||
| `test.fail_if_stop_overran` | yes | a signal passed at danger ends the test; with `no` it's only logged |
|
||||
| `test.no_movement_before_pass` | yes | a pass waits until every vehicle stands still; with `no` it ends the test at once |
|
||||
|
||||
`--test-timeout` on the command line wins over `test.timeout` of the scenery.
|
||||
`--test-timeout` and `--test-wall-timeout` on the command line win over `test.timeout` and
|
||||
`test.wall_timeout` of the scenery and the settings files.
|
||||
|
||||
`test.wall_timeout` is a watchdog in real time, for a run that hangs or crawls: when the
|
||||
simulator still runs that many seconds after its start, loading included, it says `test: timeout`
|
||||
and quits with the timeout code, whatever it is doing at the moment, with no dialog. Not given,
|
||||
it is `test.timeout` + 600 s: ten minutes for the loading, and a simulation that runs at least as
|
||||
fast as real time is never cut short by it. With `test.timeout 0` there is no watchdog unless
|
||||
`test.wall_timeout` is given; `test.wall_timeout 0` turns it off. A test that has ended but whose
|
||||
simulator doesn't close is ended by the watchdog too, with its own result.
|
||||
|
||||
A signal is passed at danger when the head of a train goes past a signal which its driver
|
||||
has in its speed table, showing stop. So it's the driver's view that counts: a train without
|
||||
@@ -120,7 +131,7 @@ Python 3.10+, no extra packages. Options:
|
||||
`cmake-build-*/bin/Release/eu07*.exe`. The build needs the `--test` and `-ini` command
|
||||
line options.
|
||||
- name one or more cases to run only those, e.g. `run.py ... eu07_td_timetable`.
|
||||
- `-j N` runs N cases at once. Each case prints its results when it is done. All 15 cases
|
||||
- `-j N` runs N cases at once. Each case prints its results when it is done. All 16 cases
|
||||
take about 30 s with `-j 4`, loading included.
|
||||
- `--watch` runs a case the old way, in a visible window in real time (or faster with
|
||||
`--timespeed N`), closed after the case's `duration`; the exit code isn't checked then. For
|
||||
@@ -138,13 +149,15 @@ For each case the runner:
|
||||
added, and the case's `.scn` fragment added: its trainsets after `FirstInit`, everything
|
||||
else (the case's `test_config`, event launchers standing in for keys the scenery expects a
|
||||
player to press, extra events) before it,
|
||||
2. starts the simulator with `--test -s autotest_<case>.scn -ini out/<case>/autotest.ini`,
|
||||
plus `--test-timeout <duration>` when the case gives a `duration`; the extra ini turns on
|
||||
2. starts the simulator with `--test -s autotest_<case>.scn -ini out/<case>/autotest.ini
|
||||
--test-wall-timeout <timeout>`, plus `--test-timeout <duration>` when the case gives a
|
||||
`duration`; the extra ini turns on
|
||||
`physicslog` and writes the logs to `logs/log (autotest_<case>) <date>.txt` rather than the
|
||||
shared `log.txt` (`multiplelogs`),
|
||||
3. waits until the simulator quits by itself. It gives up early when the log reports
|
||||
`Bad vehicle` / `Bad trainset`, when the scenario has loaded and no physicslog appeared, or
|
||||
after the case's `timeout` of wall-clock time,
|
||||
`Bad vehicle` / `Bad trainset`, or when the scenario has loaded and no physicslog appeared.
|
||||
After the case's `timeout` of wall-clock time the simulator ends itself with the timeout code;
|
||||
one that doesn't is killed 30 s later,
|
||||
4. compares the result from the exit code with the case's `expect`, then runs the checks on
|
||||
the physicslog.
|
||||
|
||||
@@ -167,9 +180,9 @@ scenery is newer.
|
||||
| `test_config` | lines of a `config` block added to the scenery, e.g. `["test.timeout 90"]` |
|
||||
| `fragment` | `.scn` file in `cases/` added to the scenery: trainsets after `FirstInit` (before it they derail on unlinked track), the rest before it (launchers read after it never fire) |
|
||||
| `vars` | values for `{{name}}` in the fragment; a list becomes one line per item |
|
||||
| `expect` | the result the test mode should end with: `pass`, `fail`, `signal_passed_at_danger`, `timeout`, `standstill` or `error`. Default `timeout` for a case with `duration`, else `pass` |
|
||||
| `expect` | the result the test mode should end with: `pass`, `fail`, `signal_passed_at_danger`, `timeout`, `standstill` or `error`, or `wall_timeout` for the timeout of the real time watchdog (the timeout code with a verdict saying it was the real time; a case expecting `timeout` fails on it). Default `timeout` for a case with `duration`, else `pass` |
|
||||
| `duration` | seconds of simulation to record: the test timeout of the run (`--test-timeout`) |
|
||||
| `timeout` | wall-clock seconds before giving up, loading included |
|
||||
| `timeout` | wall-clock seconds before giving up, loading included: the simulator's `--test-wall-timeout` (default 600) |
|
||||
| `fail_on_errors` | fail when `errors.txt` is not empty (default `false`) |
|
||||
| `checks` | list of checks, below |
|
||||
|
||||
@@ -190,7 +203,8 @@ on `tor1k`, the track in front of the signal.
|
||||
|
||||
The `test_mode_*` cases show the ends of the test mode: `test_mode_pass` (exit 0, the pass
|
||||
event), `test_mode_signal_passed_at_danger` (exit 11), `test_mode_timeout` (exit 12, from
|
||||
`test.timeout`) and `test_mode_standstill` (exit 13, from `test.standstill`).
|
||||
`test.timeout`), `test_mode_standstill` (exit 13, from `test.standstill`) and
|
||||
`test_mode_wall_timeout` (exit 12 from the real time watchdog, which ends it while it loads).
|
||||
`test_mode_pass_while_moving` reports the pass from `event2` of `tor1k` (the direction the train runs), as the train runs onto
|
||||
it, with `test.no_movement_before_pass no`, so the test ends with the train still moving.
|
||||
|
||||
|
||||
11
tests/scenarios/cases/test_mode_wall_timeout.json
Normal file
11
tests/scenarios/cases/test_mode_wall_timeout.json
Normal file
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"description": "Test mode, real time limit: EU07 on td.scn, with a limit of 2 s of real time (the case's timeout, which the runner passes as --test-wall-timeout). The scenery takes longer to load, so the simulator's watchdog ends the run while it loads, with the timeout code and a verdict saying it was the real time.",
|
||||
"base_scenery": "td.scn",
|
||||
"time": "10:34",
|
||||
"fragment": "td_red_signal.scn",
|
||||
"vars": {
|
||||
"consist": "node -1 0 {{vehicle}} dynamic pkp\\4e_v1 4e-004 4e 0 headdriver 0 0 enddynamic"
|
||||
},
|
||||
"expect": "wall_timeout",
|
||||
"timeout": 2
|
||||
}
|
||||
@@ -61,6 +61,10 @@ EXIT_RESULTS = {
|
||||
13: "standstill",
|
||||
14: "error",
|
||||
}
|
||||
# the timeout of the simulator's real time watchdog (test.wall_timeout) has the timeout code too; told
|
||||
# apart by its verdict, so a run that hung doesn't pass a case that expects the simulation time to run out
|
||||
WALL_TIMEOUT = "wall_timeout"
|
||||
RESULTS = (*EXIT_RESULTS.values(), WALL_TIMEOUT)
|
||||
|
||||
# settings layered over the game's eu07.ini for a test run (test mode doesn't read the player's own
|
||||
# in %APPDATA%); the window ones only matter
|
||||
@@ -226,6 +230,8 @@ FATAL_LOG_LINES = ("Bad vehicle:", "Bad trainset:")
|
||||
# time" in the log is not the end of loading (the trainsets after FirstInit still follow),
|
||||
# and with several simulators loading at once the first rows took well over 15 s
|
||||
STARTUP_TIMEOUT = 120.0
|
||||
# wall seconds past a case's timeout before the runner kills a simulator that didn't end itself
|
||||
KILL_GRACE = 30.0
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -246,7 +252,8 @@ def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog
|
||||
if watch:
|
||||
cmd = [str(exe), "-s", scenery, "-v", "ghostview", "-ini", str(ini)]
|
||||
else:
|
||||
cmd = [str(exe), "--test", "-s", scenery, "-ini", str(ini)]
|
||||
# the simulator's own watchdog ends a run that takes too long, with the timeout code and its verdict
|
||||
cmd = [str(exe), "--test", "-s", scenery, "-ini", str(ini), "--test-wall-timeout", f"{timeout:g}"]
|
||||
if duration is not None:
|
||||
cmd += ["--test-timeout", f"{duration:g}"]
|
||||
say(" $ " + " ".join(f'"{c}"' if " " in c else c for c in cmd), progress=True)
|
||||
@@ -272,8 +279,9 @@ def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog
|
||||
break
|
||||
if watch and sim_t >= duration:
|
||||
break
|
||||
if wall > timeout:
|
||||
result.failure = f"timed out after {timeout:.0f}s before {duration}s of simulation"
|
||||
# in test mode a little later than the simulator's own limit, for a run that can't even end itself
|
||||
if wall > timeout + (0 if watch else KILL_GRACE):
|
||||
result.failure = f"timed out after {wall:.0f}s before {duration}s of simulation"
|
||||
break
|
||||
if not result.rows:
|
||||
result.logs = find_logs(game_dir, scenery, since)
|
||||
@@ -506,8 +514,8 @@ def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, watch: bool
|
||||
# a case that only records `duration` seconds ends on the test timeout; one that says
|
||||
# how the test ends (a pass event, a signal passed at danger...) expects that result
|
||||
expect = case.get("expect", "timeout" if duration is not None else "pass")
|
||||
if expect not in EXIT_RESULTS.values():
|
||||
sys.exit(f"{name}: expect must be one of {', '.join(EXIT_RESULTS.values())}")
|
||||
if expect not in RESULTS:
|
||||
sys.exit(f"{name}: expect must be one of {', '.join(RESULTS)}")
|
||||
if watch and duration is None:
|
||||
sys.exit(f"{name}: --watch needs a case with duration")
|
||||
result = run_simulator(exe, game_dir, scenery, ini, physicslog,
|
||||
@@ -530,8 +538,11 @@ def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, watch: bool
|
||||
if result.verdict:
|
||||
say(f" {result.verdict}")
|
||||
got = EXIT_RESULTS.get(result.exit_code) if result.exit_code is not None else None
|
||||
if got == "timeout" and "of real time" in result.verdict:
|
||||
got = WALL_TIMEOUT
|
||||
passed = got == expect
|
||||
say(f" {'PASS' if passed else 'FAIL'} exit code {describe_exit(result.exit_code)}, expected {expect}")
|
||||
watchdog = ", from the real time watchdog" if got == WALL_TIMEOUT else ""
|
||||
say(f" {'PASS' if passed else 'FAIL'} exit code {describe_exit(result.exit_code)}{watchdog}, expected {expect}")
|
||||
ok &= passed
|
||||
if result.failure:
|
||||
say(f" FAIL {result.failure}")
|
||||
|
||||
@@ -80,17 +80,19 @@ TEST_CASE("test mode: the command line")
|
||||
CHECK_FALSE(commandline.enabled);
|
||||
CHECK(commandline.error.empty());
|
||||
}
|
||||
SUBCASE("--test and its limit, in any order")
|
||||
SUBCASE("--test and its limits, in any order")
|
||||
{
|
||||
auto const commandline{parse({"-s", "td.scn", "--test-timeout", "240", "--test"})};
|
||||
auto const commandline{parse({"--test-wall-timeout", "900", "-s", "td.scn", "--test", "--test-timeout", "240"})};
|
||||
CHECK(commandline.enabled);
|
||||
CHECK(commandline.timeout == 240.0);
|
||||
CHECK(commandline.wall_timeout == 900.0);
|
||||
CHECK(commandline.error.empty());
|
||||
}
|
||||
SUBCASE("a limit not given stays unset")
|
||||
{
|
||||
auto const commandline{parse({"--test", "-s", "td.scn"})};
|
||||
CHECK_FALSE(commandline.timeout.has_value());
|
||||
CHECK_FALSE(commandline.wall_timeout.has_value());
|
||||
}
|
||||
SUBCASE("a limit which isn't a number is an error, not a limit turned off")
|
||||
{
|
||||
@@ -101,12 +103,17 @@ TEST_CASE("test mode: the command line")
|
||||
}
|
||||
SUBCASE("a limit without its value is an error")
|
||||
{
|
||||
auto const commandline{parse({"--test", "-s", "td.scn", "--test-timeout"})};
|
||||
auto const commandline{parse({"--test", "-s", "td.scn", "--test-wall-timeout"})};
|
||||
CHECK(commandline.error == "--test-wall-timeout needs a number of seconds");
|
||||
}
|
||||
SUBCASE("the first error is the one told")
|
||||
{
|
||||
auto const commandline{parse({"--test", "--test-timeout", "x", "--test-wall-timeout", "y"})};
|
||||
CHECK(commandline.error == "--test-timeout needs a number of seconds");
|
||||
}
|
||||
SUBCASE("values of other options aren't taken for options")
|
||||
{
|
||||
auto const commandline{parse({"-s", "--test", "-ini", "--test-timeout", "-v", "--test", "-seed", "--test"})};
|
||||
auto const commandline{parse({"-s", "--test", "-ini", "--test-timeout", "-v", "--test-wall-timeout", "-seed", "--test"})};
|
||||
CHECK_FALSE(commandline.enabled);
|
||||
CHECK(commandline.error.empty());
|
||||
}
|
||||
@@ -117,6 +124,25 @@ TEST_CASE("test mode: the command line")
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("test mode: the real time limit")
|
||||
{
|
||||
SUBCASE("given, it is the limit, 0 turns it off")
|
||||
{
|
||||
CHECK(simulation::wall_clock_limit(120.0, 3600.0) == 120.0);
|
||||
CHECK(simulation::wall_clock_limit(0.0, 3600.0) == 0.0);
|
||||
}
|
||||
SUBCASE("not given, it is the simulation time limit and time for the loading")
|
||||
{
|
||||
CHECK(simulation::wall_clock_limit(std::nullopt, 240.0) == 240.0 + simulation::wall_timeout_loading_allowance);
|
||||
CHECK(simulation::wall_clock_limit(std::nullopt, 3600.0) == 3600.0 + simulation::wall_timeout_loading_allowance);
|
||||
}
|
||||
SUBCASE("no simulation time limit, no real time limit unless given")
|
||||
{
|
||||
CHECK(simulation::wall_clock_limit(std::nullopt, 0.0) == 0.0);
|
||||
CHECK(simulation::wall_clock_limit(300.0, 0.0) == 300.0);
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("test mode: the end conditions")
|
||||
{
|
||||
simulation::test_limits limits;
|
||||
|
||||
Reference in New Issue
Block a user