mirror of
https://github.com/MaSzyna-EU07/maszyna.git
synced 2026-11-19 22:44:59 +01:00
Merge pull request #326 from MaSzyna-EU07/test/automatic-coupler-scenarios-master
test(scenarios): automatic coupler cases (re-lands #288)
This commit is contained in:
@@ -135,15 +135,15 @@ Python 3.10+, no extra packages. Options:
|
||||
`cmake-build-*/bin/Release/eu07*.exe`. The build needs the `--test` and `-ini` command
|
||||
line options.
|
||||
- name one or more cases to run only those, e.g. `run.py ... eu07_td_timetable`.
|
||||
- `-j N` runs N cases at once. Each case prints its results when it is done. All 17 cases
|
||||
- `-j N` runs N cases at once. Each case prints its results when it is done. All 19 cases
|
||||
take about 30 s with `-j 4`, loading included.
|
||||
- `--watch` runs a case the old way, in a visible window in real time (or faster with
|
||||
`--timespeed N`), closed after the case's `duration`; the exit code isn't checked then. For
|
||||
watching what a case does.
|
||||
|
||||
Exit code is 0 when every case passed. Logs of each run (`log.txt`, `errors.txt`,
|
||||
`stdout.txt`, the physicslogs of the vehicle and of other trains of the case) are moved to
|
||||
`tests/scenarios/out/<case>/`.
|
||||
`stdout.txt`, the physicslogs of the vehicle and of other trains of the case, the black box as
|
||||
`autotest_<case>.blackbox`) are moved to `tests/scenarios/out/<case>/`.
|
||||
|
||||
## How it works
|
||||
|
||||
@@ -157,14 +157,15 @@ For each case the runner:
|
||||
2. starts the simulator with `--test -s autotest_<case>.scn -ini out/<case>/autotest.ini
|
||||
--test-wall-timeout <timeout>`, plus `--test-timeout <duration>` when the case gives a
|
||||
`duration`; the extra ini turns on
|
||||
`physicslog` and writes the logs to `logs/log (autotest_<case>) <date>.txt` rather than the
|
||||
`physicslog` (and with it the black box of the run, `physicslog/autotest_<case>_<date>.blackbox`)
|
||||
and writes the logs to `logs/log (autotest_<case>) <date>.txt` rather than the
|
||||
shared `log.txt` (`multiplelogs`),
|
||||
3. waits until the simulator quits by itself. It gives up early when the log reports
|
||||
`Bad vehicle` / `Bad trainset`, or when the scenario has loaded and no physicslog appeared.
|
||||
After the case's `timeout` of wall-clock time the simulator ends itself with the timeout code;
|
||||
one that doesn't is killed 30 s later,
|
||||
4. compares the result from the exit code with the case's `expect`, then runs the checks on
|
||||
the physicslog.
|
||||
the physicslog, the log and the black box.
|
||||
|
||||
The lead vehicle is named `autotest_<case>`, so every case has its own physicslog and log
|
||||
files and cases can run in parallel. The `autotest_*` sceneries (and the simulator's `.sbt`
|
||||
@@ -184,6 +185,7 @@ which could have been writing it.
|
||||
| `track_friction` | set the friction of every track of the base scenery (default in the sceneries is 0.15), e.g. for slippery rail |
|
||||
| `track_events` | lines added to tracks of the base scenery, by track name, e.g. `{ "tor1k": "event0 test_pass" }`; only tracks in the scenery file itself, not in its includes |
|
||||
| `test_config` | lines of a `config` block added to the scenery, e.g. `["test.timeout 90"]`; the block always has `test.vehicle` with the case's lead vehicle, so only its train counts for the pass and the standstill |
|
||||
| `ini` | extra `eu07.ini` lines for the run, after the common ones of the runner's `-ini` file, e.g. `["crashdamage yes"]` |
|
||||
| `fragment` | `.scn` file in `cases/` added to the scenery: trainsets after `FirstInit` (before it they derail on unlinked track), the rest before it (launchers read after it never fire) |
|
||||
| `vars` | values for `{{name}}` in the fragment; a list becomes one line per item |
|
||||
| `expect` | the result the test mode should end with: `pass`, `fail`, `signal_passed_at_danger`, `timeout`, `standstill` or `error`, or `wall_timeout` for the timeout of the real time watchdog (the timeout code with a verdict saying it was the real time; a case expecting `timeout` fails on it). Default `timeout` for a case with `duration`, else `pass` |
|
||||
@@ -219,6 +221,15 @@ into the test timeout).
|
||||
`test_mode_pass_while_moving` reports the pass from `event2` of `tor1k` (the direction the train runs), as the train runs onto
|
||||
it, with `test.no_movement_before_pass no`, so the test ends with the train still moving.
|
||||
|
||||
The `en57_2000_automatic_coupler_*` cases (issue #86) share `td_automatic_coupler_hit.scn`: an
|
||||
EN57-2000 unit without a driver runs at `vars.velocity` km/h into a standing one, from
|
||||
`35.6 - vars.offset` m behind it, with `crashdamage yes` in `ini`. At 50 km/h the automatic
|
||||
couplers mustn't catch and the log has to show the collision and the derailment; at 4 km/h they
|
||||
couple and neither shows up: the black box has to have the last car of the standing unit,
|
||||
`{{vehicle}}_rb`, and the first of the rolling one, `{{vehicle}}_hit_ra`, coupled (the simulator
|
||||
writes nothing to the log when couplers catch). Nothing reports a result, so both end on the 60 s
|
||||
test timeout.
|
||||
|
||||
Checks (speeds in km/h, times in seconds of physicslog time, distances in metres). Most use
|
||||
the physicslog Velocity column, which is the wheel speed: it stays at 0 while the wheels are
|
||||
locked in a skid. The ones marked *vehicle speed* use the vehicle speed column at the end of
|
||||
@@ -244,6 +255,19 @@ the row instead, logged by builds from this branch on.
|
||||
after moving off, moves no more than `max_distance` metres, e.g. no creeping on locked wheels.
|
||||
- `ends_moving` `min_speed`: the last row of the physicslog is at `min_speed` or faster, e.g.
|
||||
the test ended without waiting for the train to stop.
|
||||
- `log_contains` `pattern`: a line of the run's `log.txt` matches the regex, e.g. the
|
||||
`Bad driving: ... collided` message of a collision.
|
||||
- `log_lacks` `pattern`: no line of the run's `log.txt` matches the regex.
|
||||
- `couples` `vehicle`, `other`: the black box of the run has the two vehicles coupled to each
|
||||
other (the coupler itself, whatever else of the hoses and the cables) at some time after the
|
||||
start, and still coupled at the end. The vehicles go by their node names, with `{{vehicle}}`
|
||||
filled in as in the fragment, e.g. `{{vehicle}}_rb`. Fails when the two were coupled from the
|
||||
start, came apart again, or the run left no black box. The black box is looked at once a
|
||||
second, so the time given is that of the first look after the couplers caught.
|
||||
|
||||
New check types go in `CHECKS` in `run.py`; each gets the physicslog rows and the check's
|
||||
parameters and returns `(passed, detail)`.
|
||||
parameters and returns `(passed, detail)`. Checks on the log go in `LOG_CHECKS` and get the
|
||||
text of `log.txt` instead of the rows. Checks on the black box (`blackbox_doc.md` in the root of
|
||||
the repo; the simulator writes it along with the physicslogs) go in `BLACKBOX_CHECKS` and get
|
||||
what `read_blackbox()` read of it: the names of the vehicles and the coupling records. A check
|
||||
on another kind of record has to add it there.
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"description": "Issue #86, the control case: an EN57-2000 unit without a driver rolls at 4 km/h into a standing one on td.scn. Their automatic couplers catch: the black box of the run has the last car of the standing unit and the first of the rolling one coupled. Nothing collides nor derails; nothing reports a result, so the run ends on the 60 s test timeout.",
|
||||
"base_scenery": "td.scn",
|
||||
"ini": ["crashdamage yes"],
|
||||
"fragment": "td_automatic_coupler_hit.scn",
|
||||
"vars": {
|
||||
"velocity": "4",
|
||||
"offset": "30"
|
||||
},
|
||||
"expect": "timeout",
|
||||
"duration": 60,
|
||||
"timeout": 300,
|
||||
"checks": [
|
||||
{ "type": "couples", "vehicle": "{{vehicle}}_rb", "other": "{{vehicle}}_hit_ra" },
|
||||
{ "type": "log_lacks", "pattern": "collided" },
|
||||
{ "type": "log_lacks", "pattern": "derailed" }
|
||||
]
|
||||
}
|
||||
17
tests/scenarios/cases/en57_2000_automatic_coupler_crash.json
Normal file
17
tests/scenarios/cases/en57_2000_automatic_coupler_crash.json
Normal file
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"description": "Issue #86: an EN57-2000 unit without a driver runs at 50 km/h into a standing one on td.scn. Their automatic couplers mustn't catch at that speed: the units collide and derail; the run ends on the 60 s test timeout.",
|
||||
"base_scenery": "td.scn",
|
||||
"ini": ["crashdamage yes"],
|
||||
"fragment": "td_automatic_coupler_hit.scn",
|
||||
"vars": {
|
||||
"velocity": "50",
|
||||
"offset": "0"
|
||||
},
|
||||
"expect": "timeout",
|
||||
"duration": 60,
|
||||
"timeout": 300,
|
||||
"checks": [
|
||||
{ "type": "log_contains", "pattern": "Bad driving: autotest_\\S+ and autotest_\\S+ collided" },
|
||||
{ "type": "log_contains", "pattern": "Bad driving: autotest_\\S+ derailed" }
|
||||
]
|
||||
}
|
||||
19
tests/scenarios/cases/td_automatic_coupler_hit.scn
Normal file
19
tests/scenarios/cases/td_automatic_coupler_hit.scn
Normal file
@@ -0,0 +1,19 @@
|
||||
// added by tests/scenarios/run.py to td.scn, whose own trainsets were removed.
|
||||
// two EN57-2000 units, which have automatic couplers at their cab ends, on the straight track 1
|
||||
// of Stacja1: one stands where td.scn keeps its timetable train, with an AI driver and no
|
||||
// timetable, so it stays put and logs its physics; the other has no driver and rolls into it at
|
||||
// {{velocity}} km/h, its front coupler hitting the rear one of the first.
|
||||
// a trainset's first vehicle stands {{offset}} m past point 1 of its track, the rest towards point 1,
|
||||
// and a positive velocity runs towards point 2. tdo_n1734 is the 100 m track after tdo_n1725, a unit
|
||||
// is 64.4 m long: the first one reaches back to 35.6 m into tdo_n1725, so the gap is 35.6 - offset m.
|
||||
trainset none tdo_n1734 0.0 0.0
|
||||
node -1 0 {{vehicle}} dynamic pkp\en57-2000_v1 en57-2040_ra 6ba 0 headdriver 255 0 enddynamic
|
||||
node -1 0 {{vehicle}}_s dynamic pkp\en57-2000_v1 en57-2040_s 6bs 0 nobody 255 0 enddynamic
|
||||
node -1 0 {{vehicle}}_rb dynamic pkp\en57-2000_v1 en57-2040_rb 6bb 0 nobody 0 0 enddynamic
|
||||
endtrainset
|
||||
|
||||
trainset none tdo_n1725 {{offset}} {{velocity}}
|
||||
node -1 0 {{vehicle}}_hit_ra dynamic pkp\en57-2000_v1 en57-2040_ra 6ba 0 nobody 255 0 enddynamic
|
||||
node -1 0 {{vehicle}}_hit_s dynamic pkp\en57-2000_v1 en57-2040_s 6bs 0 nobody 255 0 enddynamic
|
||||
node -1 0 {{vehicle}}_hit_rb dynamic pkp\en57-2000_v1 en57-2040_rb 6bb 0 nobody 0 0 enddynamic
|
||||
endtrainset
|
||||
@@ -31,6 +31,7 @@ import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
@@ -230,6 +231,81 @@ def read_text(path: Path | None) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
# the black box of a run, see blackbox_doc.md: written along with the physicslogs, to
|
||||
# physicslog/<scenery>_<date>.blackbox. A header, then records: a byte of the kind, the length, the content
|
||||
BLACKBOX_MAGIC = b"EU07BBX\0"
|
||||
BLACKBOX_HEADER = 12 # the magic, the version of the format, two bytes reserved
|
||||
BLACKBOX_VEHICLE = 0x12 # a vehicle: its number and its name
|
||||
BLACKBOX_COUPLING = 0x37 # a coupler of a vehicle coupled, uncoupled, or its hoses and cables changed
|
||||
COUPLING_COUPLER = 0x01 # of the bits telling what is coupled: the coupler itself
|
||||
|
||||
|
||||
@dataclass
|
||||
class Coupling:
|
||||
"""A coupling record: what a coupler of a vehicle is coupled to from `time` on."""
|
||||
time: float
|
||||
vehicle: int # vehicles by their numbers, 0 for none
|
||||
side: int # 0 the front coupler, 1 the rear one
|
||||
other: int
|
||||
other_side: int
|
||||
flags: int # what is coupled, 0 nothing
|
||||
previous_flags: int
|
||||
previous_other: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class Blackbox:
|
||||
vehicles: dict[int, str] = field(default_factory=dict) # names by number
|
||||
couplings: list[Coupling] = field(default_factory=list) # in the order written
|
||||
|
||||
|
||||
def find_blackbox(game_dir: Path, scenery: str, since: float) -> Path | None:
|
||||
"""The black box of one run: the simulator names it after the scenery and the time it started."""
|
||||
stem = re.sub(r"[^0-9A-Za-z_-]", "_", scenery.rsplit(".", 1)[0])
|
||||
# the whole name, so a case whose name another one starts with doesn't take that one's file
|
||||
name = re.compile(re.escape(stem) + r"_\d{4}-\d\d-\d\d_\d\d-\d\d-\d\d\.blackbox")
|
||||
files = [p for p in (game_dir / "physicslog").glob(f"{stem}_*.blackbox")
|
||||
if name.fullmatch(p.name) and p.stat().st_mtime >= since]
|
||||
return max(files, key=lambda p: p.stat().st_mtime, default=None)
|
||||
|
||||
|
||||
def read_blackbox(path: Path | None) -> Blackbox | None:
|
||||
"""The vehicles and the coupling records of a black box; None when there is none or it can't be read."""
|
||||
try:
|
||||
data = path.read_bytes() if path else b""
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
if data[:len(BLACKBOX_MAGIC)] != BLACKBOX_MAGIC:
|
||||
return None
|
||||
box = Blackbox()
|
||||
at = BLACKBOX_HEADER
|
||||
try:
|
||||
while at + 5 <= len(data):
|
||||
kind, length = struct.unpack_from("<BI", data, at)
|
||||
content = data[at + 5: at + 5 + length]
|
||||
if len(content) < length:
|
||||
break # a record still being written when the run ended
|
||||
at += 5 + length
|
||||
if kind == BLACKBOX_VEHICLE:
|
||||
number, = struct.unpack_from("<I", content)
|
||||
# a text: its length in 7 bit groups, low first, the high bit set on all but the last
|
||||
size, shift, start = 0, 0, 4
|
||||
while True:
|
||||
byte = content[start]
|
||||
start += 1
|
||||
size |= (byte & 0x7F) << shift
|
||||
shift += 7
|
||||
if byte < 0x80:
|
||||
break
|
||||
box.vehicles[number] = content[start: start + size].decode("utf-8", errors="replace")
|
||||
elif kind == BLACKBOX_COUPLING:
|
||||
# the position of the coupler follows
|
||||
box.couplings.append(Coupling(*struct.unpack_from("<dIBIBBBI", content)))
|
||||
except (struct.error, IndexError):
|
||||
return None # a record shorter than its kind is
|
||||
return box
|
||||
|
||||
|
||||
# load problems that leave the case without its vehicle; no point waiting for the timeout
|
||||
FATAL_LOG_LINES = ("Bad vehicle:", "Bad trainset:")
|
||||
# wall seconds from start to the first physicslog row before giving up. "Scenario loading
|
||||
@@ -248,6 +324,7 @@ class RunResult:
|
||||
failure: str = "" # why the run was cut short
|
||||
wall_seconds: float = 0.0
|
||||
logs: dict[str, Path] = field(default_factory=dict)
|
||||
blackbox: Path | None = None
|
||||
|
||||
|
||||
def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog: Path,
|
||||
@@ -313,6 +390,7 @@ def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog
|
||||
stdout.close()
|
||||
result.rows = read_physicslog(physicslog)
|
||||
result.logs = find_logs(game_dir, scenery, since)
|
||||
result.blackbox = find_blackbox(game_dir, scenery, since)
|
||||
return result
|
||||
|
||||
|
||||
@@ -476,6 +554,63 @@ CHECKS = {
|
||||
}
|
||||
|
||||
|
||||
def check_log_contains(log, pattern):
|
||||
"""A line of the run's log.txt matches the regex, e.g. the "Bad driving" message of a collision."""
|
||||
found = next((line.strip() for line in log.splitlines() if re.search(pattern, line)), None)
|
||||
if found is None:
|
||||
return False, f"no log line matches /{pattern}/"
|
||||
return True, f"log: {found}"
|
||||
|
||||
|
||||
def check_log_lacks(log, pattern):
|
||||
"""No line of the run's log.txt matches the regex."""
|
||||
found = [line.strip() for line in log.splitlines() if re.search(pattern, line)]
|
||||
if found:
|
||||
return False, f"{len(found)} log line(s) match /{pattern}/, first: {found[0]}"
|
||||
return True, f"no log line matches /{pattern}/"
|
||||
|
||||
|
||||
# checks on the text of the run's log.txt rather than on the physicslog rows
|
||||
LOG_CHECKS = {
|
||||
"log_contains": check_log_contains,
|
||||
"log_lacks": check_log_lacks,
|
||||
}
|
||||
|
||||
|
||||
def check_couples(box, vehicle, other):
|
||||
"""The two vehicles couple in the run and stay coupled to its end; coupled from the start doesn't count."""
|
||||
numbers = {name: number for number, name in box.vehicles.items()}
|
||||
unknown = [name for name in (vehicle, other) if name not in numbers]
|
||||
if unknown:
|
||||
return False, f"the black box has no vehicle {' nor '.join(unknown)}"
|
||||
pair = {numbers[vehicle], numbers[other]}
|
||||
# of the two vehicles of a coupling the one with the lower number tells it, each change of that coupler
|
||||
# under the same vehicle and side: the record which coupled the two, for as long as they stay coupled
|
||||
made, parted = {}, None
|
||||
for c in box.couplings:
|
||||
coupler = (c.vehicle, c.side)
|
||||
if {c.vehicle, c.other} == pair and c.flags & COUPLING_COUPLER:
|
||||
made.setdefault(coupler, c)
|
||||
elif coupler in made:
|
||||
parted = (made.pop(coupler), c)
|
||||
if not made:
|
||||
if parted:
|
||||
return False, f"{vehicle} and {other} coupled at t={parted[0].time:.1f}s but came apart at t={parted[1].time:.1f}s"
|
||||
return False, f"none of the {len(box.couplings)} coupling record(s) of the black box couples {vehicle} and {other}"
|
||||
c = min(made.values(), key=lambda c: c.time)
|
||||
if c.time <= 0:
|
||||
return False, f"{vehicle} and {other} were coupled when the recording started"
|
||||
ends = ("front", "rear")
|
||||
return True, (f"at t={c.time:.1f}s the {ends[c.side]} coupler of {box.vehicles[c.vehicle]} coupled to the "
|
||||
f"{ends[c.other_side]} one of {box.vehicles[c.other]}, coupling 0x{c.flags:02X}")
|
||||
|
||||
|
||||
# checks on the black box of the run rather than on the physicslog rows
|
||||
BLACKBOX_CHECKS = {
|
||||
"couples": check_couples,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- main
|
||||
|
||||
|
||||
@@ -513,7 +648,9 @@ def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, watch: bool
|
||||
vehicle = f"autotest_{name}"
|
||||
scenery = build_scenery(case, game_dir, name, vehicle)
|
||||
ini = out / "autotest.ini"
|
||||
ini.write_text(TEST_INI.format(timespeed=timespeed), encoding="ascii")
|
||||
# the case's own settings go last, over the common ones
|
||||
ini.write_text(TEST_INI.format(timespeed=timespeed) + "".join(line + "\n" for line in case.get("ini", [])),
|
||||
encoding="ascii")
|
||||
physicslog = game_dir / "physicslog" / f"{vehicle}.dat"
|
||||
|
||||
duration = float(case["duration"]) if "duration" in case else None
|
||||
@@ -535,6 +672,9 @@ def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, watch: bool
|
||||
# other trains of the case with a driver, named <something>_<vehicle>, log too
|
||||
for other in physicslog.parent.glob(f"*_{vehicle}.dat"):
|
||||
shutil.move(other, out / other.name)
|
||||
# the black box is named after the case scenery too
|
||||
if result.blackbox:
|
||||
result.blackbox = Path(shutil.move(result.blackbox, out / f"{vehicle}.blackbox"))
|
||||
|
||||
ok = True
|
||||
sim_t = result.rows[-1][COL_TIME] if result.rows else 0.0
|
||||
@@ -566,9 +706,27 @@ def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, watch: bool
|
||||
say(f" FAIL no physicslog for '{vehicle}'")
|
||||
return False
|
||||
|
||||
log = read_text(out / "log.txt")
|
||||
box = read_blackbox(result.blackbox)
|
||||
for check in case.get("checks", []):
|
||||
params = {k: v for k, v in check.items() if k != "type"}
|
||||
passed, detail = CHECKS[check["type"]](result.rows, **params)
|
||||
if check["type"] in LOG_CHECKS:
|
||||
# with no log text a log check proves nothing, log_lacks would pass vacuously
|
||||
if log.strip():
|
||||
passed, detail = LOG_CHECKS[check["type"]](log, **params)
|
||||
else:
|
||||
passed, detail = False, "log.txt is missing or empty"
|
||||
elif check["type"] in BLACKBOX_CHECKS:
|
||||
if box is not None:
|
||||
# vehicles are named as in the fragment
|
||||
params = {k: v.replace("{{vehicle}}", vehicle) if isinstance(v, str) else v for k, v in params.items()}
|
||||
passed, detail = BLACKBOX_CHECKS[check["type"]](box, **params)
|
||||
elif result.blackbox:
|
||||
passed, detail = False, f"{result.blackbox.name} is not a black box, or is damaged"
|
||||
else:
|
||||
passed, detail = False, "the run left no black box (physicslog/*.blackbox); the build is too old, or has blackbox no"
|
||||
else:
|
||||
passed, detail = CHECKS[check["type"]](result.rows, **params)
|
||||
say(f" {'PASS' if passed else 'FAIL'} {check['type']}: {detail}")
|
||||
ok &= passed
|
||||
|
||||
|
||||
Reference in New Issue
Block a user