18
0
mirror of https://github.com/MaSzyna-EU07/maszyna.git synced 2026-11-19 22:44:59 +01:00

Merge pull request #326 from MaSzyna-EU07/test/automatic-coupler-scenarios-master

test(scenarios): automatic coupler cases (re-lands #288)
This commit is contained in:
maj00r
2026-11-11 22:41:42 +01:00
committed by GitHub
5 changed files with 244 additions and 8 deletions

View File

@@ -135,15 +135,15 @@ Python 3.10+, no extra packages. Options:
`cmake-build-*/bin/Release/eu07*.exe`. The build needs the `--test` and `-ini` command
line options.
- name one or more cases to run only those, e.g. `run.py ... eu07_td_timetable`.
- `-j N` runs N cases at once. Each case prints its results when it is done. All 17 cases
- `-j N` runs N cases at once. Each case prints its results when it is done. All 19 cases
take about 30 s with `-j 4`, loading included.
- `--watch` runs a case the old way, in a visible window in real time (or faster with
`--timespeed N`), closed after the case's `duration`; the exit code isn't checked then. For
watching what a case does.
Exit code is 0 when every case passed. Logs of each run (`log.txt`, `errors.txt`,
`stdout.txt`, the physicslogs of the vehicle and of other trains of the case) are moved to
`tests/scenarios/out/<case>/`.
`stdout.txt`, the physicslogs of the vehicle and of other trains of the case, the black box as
`autotest_<case>.blackbox`) are moved to `tests/scenarios/out/<case>/`.
## How it works
@@ -157,14 +157,15 @@ For each case the runner:
2. starts the simulator with `--test -s autotest_<case>.scn -ini out/<case>/autotest.ini
--test-wall-timeout <timeout>`, plus `--test-timeout <duration>` when the case gives a
`duration`; the extra ini turns on
`physicslog` and writes the logs to `logs/log (autotest_<case>) <date>.txt` rather than the
`physicslog` (and with it the black box of the run, `physicslog/autotest_<case>_<date>.blackbox`)
and writes the logs to `logs/log (autotest_<case>) <date>.txt` rather than the
shared `log.txt` (`multiplelogs`),
3. waits until the simulator quits by itself. It gives up early when the log reports
`Bad vehicle` / `Bad trainset`, or when the scenario has loaded and no physicslog appeared.
After the case's `timeout` of wall-clock time the simulator ends itself with the timeout code;
one that doesn't is killed 30 s later,
4. compares the result from the exit code with the case's `expect`, then runs the checks on
the physicslog.
the physicslog, the log and the black box.
The lead vehicle is named `autotest_<case>`, so every case has its own physicslog and log
files and cases can run in parallel. The `autotest_*` sceneries (and the simulator's `.sbt`
@@ -184,6 +185,7 @@ which could have been writing it.
| `track_friction` | set the friction of every track of the base scenery (default in the sceneries is 0.15), e.g. for slippery rail |
| `track_events` | lines added to tracks of the base scenery, by track name, e.g. `{ "tor1k": "event0 test_pass" }`; only tracks in the scenery file itself, not in its includes |
| `test_config` | lines of a `config` block added to the scenery, e.g. `["test.timeout 90"]`; the block always has `test.vehicle` with the case's lead vehicle, so only its train counts for the pass and the standstill |
| `ini` | extra `eu07.ini` lines for the run, after the common ones of the runner's `-ini` file, e.g. `["crashdamage yes"]` |
| `fragment` | `.scn` file in `cases/` added to the scenery: trainsets after `FirstInit` (before it they derail on unlinked track), the rest before it (launchers read after it never fire) |
| `vars` | values for `{{name}}` in the fragment; a list becomes one line per item |
| `expect` | the result the test mode should end with: `pass`, `fail`, `signal_passed_at_danger`, `timeout`, `standstill` or `error`, or `wall_timeout` for the timeout of the real time watchdog (the timeout code with a verdict saying it was the real time; a case expecting `timeout` fails on it). Default `timeout` for a case with `duration`, else `pass` |
@@ -219,6 +221,15 @@ into the test timeout).
`test_mode_pass_while_moving` reports the pass from `event2` of `tor1k` (the direction the train runs), as the train runs onto
it, with `test.no_movement_before_pass no`, so the test ends with the train still moving.
The `en57_2000_automatic_coupler_*` cases (issue #86) share `td_automatic_coupler_hit.scn`: an
EN57-2000 unit without a driver runs at `vars.velocity` km/h into a standing one, from
`35.6 - vars.offset` m behind it, with `crashdamage yes` in `ini`. At 50 km/h the automatic
couplers mustn't catch and the log has to show the collision and the derailment; at 4 km/h they
couple and neither shows up: the black box has to have the last car of the standing unit,
`{{vehicle}}_rb`, and the first of the rolling one, `{{vehicle}}_hit_ra`, coupled (the simulator
writes nothing to the log when couplers catch). Nothing reports a result, so both end on the 60 s
test timeout.
Checks (speeds in km/h, times in seconds of physicslog time, distances in metres). Most use
the physicslog Velocity column, which is the wheel speed: it stays at 0 while the wheels are
locked in a skid. The ones marked *vehicle speed* use the vehicle speed column at the end of
@@ -244,6 +255,19 @@ the row instead, logged by builds from this branch on.
after moving off, moves no more than `max_distance` metres, e.g. no creeping on locked wheels.
- `ends_moving` `min_speed`: the last row of the physicslog is at `min_speed` or faster, e.g.
the test ended without waiting for the train to stop.
- `log_contains` `pattern`: a line of the run's `log.txt` matches the regex, e.g. the
`Bad driving: ... collided` message of a collision.
- `log_lacks` `pattern`: no line of the run's `log.txt` matches the regex.
- `couples` `vehicle`, `other`: the black box of the run has the two vehicles coupled to each
other (the coupler itself, whatever else of the hoses and the cables) at some time after the
start, and still coupled at the end. The vehicles go by their node names, with `{{vehicle}}`
filled in as in the fragment, e.g. `{{vehicle}}_rb`. Fails when the two were coupled from the
start, came apart again, or the run left no black box. The black box is looked at once a
second, so the time given is that of the first look after the couplers caught.
New check types go in `CHECKS` in `run.py`; each gets the physicslog rows and the check's
parameters and returns `(passed, detail)`.
parameters and returns `(passed, detail)`. Checks on the log go in `LOG_CHECKS` and get the
text of `log.txt` instead of the rows. Checks on the black box (`blackbox_doc.md` in the root of
the repo; the simulator writes it along with the physicslogs) go in `BLACKBOX_CHECKS` and get
what `read_blackbox()` read of it: the names of the vehicles and the coupling records. A check
on another kind of record has to add it there.

View File

@@ -0,0 +1,18 @@
{
"description": "Issue #86, the control case: an EN57-2000 unit without a driver rolls at 4 km/h into a standing one on td.scn. Their automatic couplers catch: the black box of the run has the last car of the standing unit and the first of the rolling one coupled. Nothing collides nor derails; nothing reports a result, so the run ends on the 60 s test timeout.",
"base_scenery": "td.scn",
"ini": ["crashdamage yes"],
"fragment": "td_automatic_coupler_hit.scn",
"vars": {
"velocity": "4",
"offset": "30"
},
"expect": "timeout",
"duration": 60,
"timeout": 300,
"checks": [
{ "type": "couples", "vehicle": "{{vehicle}}_rb", "other": "{{vehicle}}_hit_ra" },
{ "type": "log_lacks", "pattern": "collided" },
{ "type": "log_lacks", "pattern": "derailed" }
]
}

View File

@@ -0,0 +1,17 @@
{
"description": "Issue #86: an EN57-2000 unit without a driver runs at 50 km/h into a standing one on td.scn. Their automatic couplers mustn't catch at that speed: the units collide and derail; the run ends on the 60 s test timeout.",
"base_scenery": "td.scn",
"ini": ["crashdamage yes"],
"fragment": "td_automatic_coupler_hit.scn",
"vars": {
"velocity": "50",
"offset": "0"
},
"expect": "timeout",
"duration": 60,
"timeout": 300,
"checks": [
{ "type": "log_contains", "pattern": "Bad driving: autotest_\\S+ and autotest_\\S+ collided" },
{ "type": "log_contains", "pattern": "Bad driving: autotest_\\S+ derailed" }
]
}

View File

@@ -0,0 +1,19 @@
// added by tests/scenarios/run.py to td.scn, whose own trainsets were removed.
// two EN57-2000 units, which have automatic couplers at their cab ends, on the straight track 1
// of Stacja1: one stands where td.scn keeps its timetable train, with an AI driver and no
// timetable, so it stays put and logs its physics; the other has no driver and rolls into it at
// {{velocity}} km/h, its front coupler hitting the rear one of the first.
// a trainset's first vehicle stands {{offset}} m past point 1 of its track, the rest towards point 1,
// and a positive velocity runs towards point 2. tdo_n1734 is the 100 m track after tdo_n1725, a unit
// is 64.4 m long: the first one reaches back to 35.6 m into tdo_n1725, so the gap is 35.6 - offset m.
trainset none tdo_n1734 0.0 0.0
node -1 0 {{vehicle}} dynamic pkp\en57-2000_v1 en57-2040_ra 6ba 0 headdriver 255 0 enddynamic
node -1 0 {{vehicle}}_s dynamic pkp\en57-2000_v1 en57-2040_s 6bs 0 nobody 255 0 enddynamic
node -1 0 {{vehicle}}_rb dynamic pkp\en57-2000_v1 en57-2040_rb 6bb 0 nobody 0 0 enddynamic
endtrainset
trainset none tdo_n1725 {{offset}} {{velocity}}
node -1 0 {{vehicle}}_hit_ra dynamic pkp\en57-2000_v1 en57-2040_ra 6ba 0 nobody 255 0 enddynamic
node -1 0 {{vehicle}}_hit_s dynamic pkp\en57-2000_v1 en57-2040_s 6bs 0 nobody 255 0 enddynamic
node -1 0 {{vehicle}}_hit_rb dynamic pkp\en57-2000_v1 en57-2040_rb 6bb 0 nobody 0 0 enddynamic
endtrainset

View File

@@ -31,6 +31,7 @@ import json
import os
import re
import shutil
import struct
import subprocess
import sys
import time
@@ -230,6 +231,81 @@ def read_text(path: Path | None) -> str:
return ""
# the black box of a run, see blackbox_doc.md: written along with the physicslogs, to
# physicslog/<scenery>_<date>.blackbox. A header, then records: a byte of the kind, the length, the content
BLACKBOX_MAGIC = b"EU07BBX\0"
BLACKBOX_HEADER = 12 # the magic, the version of the format, two bytes reserved
BLACKBOX_VEHICLE = 0x12 # a vehicle: its number and its name
BLACKBOX_COUPLING = 0x37 # a coupler of a vehicle coupled, uncoupled, or its hoses and cables changed
COUPLING_COUPLER = 0x01 # of the bits telling what is coupled: the coupler itself
@dataclass
class Coupling:
"""A coupling record: what a coupler of a vehicle is coupled to from `time` on."""
time: float
vehicle: int # vehicles by their numbers, 0 for none
side: int # 0 the front coupler, 1 the rear one
other: int
other_side: int
flags: int # what is coupled, 0 nothing
previous_flags: int
previous_other: int
@dataclass
class Blackbox:
vehicles: dict[int, str] = field(default_factory=dict) # names by number
couplings: list[Coupling] = field(default_factory=list) # in the order written
def find_blackbox(game_dir: Path, scenery: str, since: float) -> Path | None:
"""The black box of one run: the simulator names it after the scenery and the time it started."""
stem = re.sub(r"[^0-9A-Za-z_-]", "_", scenery.rsplit(".", 1)[0])
# the whole name, so a case whose name another one starts with doesn't take that one's file
name = re.compile(re.escape(stem) + r"_\d{4}-\d\d-\d\d_\d\d-\d\d-\d\d\.blackbox")
files = [p for p in (game_dir / "physicslog").glob(f"{stem}_*.blackbox")
if name.fullmatch(p.name) and p.stat().st_mtime >= since]
return max(files, key=lambda p: p.stat().st_mtime, default=None)
def read_blackbox(path: Path | None) -> Blackbox | None:
"""The vehicles and the coupling records of a black box; None when there is none or it can't be read."""
try:
data = path.read_bytes() if path else b""
except FileNotFoundError:
return None
if data[:len(BLACKBOX_MAGIC)] != BLACKBOX_MAGIC:
return None
box = Blackbox()
at = BLACKBOX_HEADER
try:
while at + 5 <= len(data):
kind, length = struct.unpack_from("<BI", data, at)
content = data[at + 5: at + 5 + length]
if len(content) < length:
break # a record still being written when the run ended
at += 5 + length
if kind == BLACKBOX_VEHICLE:
number, = struct.unpack_from("<I", content)
# a text: its length in 7 bit groups, low first, the high bit set on all but the last
size, shift, start = 0, 0, 4
while True:
byte = content[start]
start += 1
size |= (byte & 0x7F) << shift
shift += 7
if byte < 0x80:
break
box.vehicles[number] = content[start: start + size].decode("utf-8", errors="replace")
elif kind == BLACKBOX_COUPLING:
# the position of the coupler follows
box.couplings.append(Coupling(*struct.unpack_from("<dIBIBBBI", content)))
except (struct.error, IndexError):
return None # a record shorter than its kind is
return box
# load problems that leave the case without its vehicle; no point waiting for the timeout
FATAL_LOG_LINES = ("Bad vehicle:", "Bad trainset:")
# wall seconds from start to the first physicslog row before giving up. "Scenario loading
@@ -248,6 +324,7 @@ class RunResult:
failure: str = "" # why the run was cut short
wall_seconds: float = 0.0
logs: dict[str, Path] = field(default_factory=dict)
blackbox: Path | None = None
def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog: Path,
@@ -313,6 +390,7 @@ def run_simulator(exe: Path, game_dir: Path, scenery: str, ini: Path, physicslog
stdout.close()
result.rows = read_physicslog(physicslog)
result.logs = find_logs(game_dir, scenery, since)
result.blackbox = find_blackbox(game_dir, scenery, since)
return result
@@ -476,6 +554,63 @@ CHECKS = {
}
def check_log_contains(log, pattern):
"""A line of the run's log.txt matches the regex, e.g. the "Bad driving" message of a collision."""
found = next((line.strip() for line in log.splitlines() if re.search(pattern, line)), None)
if found is None:
return False, f"no log line matches /{pattern}/"
return True, f"log: {found}"
def check_log_lacks(log, pattern):
"""No line of the run's log.txt matches the regex."""
found = [line.strip() for line in log.splitlines() if re.search(pattern, line)]
if found:
return False, f"{len(found)} log line(s) match /{pattern}/, first: {found[0]}"
return True, f"no log line matches /{pattern}/"
# checks on the text of the run's log.txt rather than on the physicslog rows
LOG_CHECKS = {
"log_contains": check_log_contains,
"log_lacks": check_log_lacks,
}
def check_couples(box, vehicle, other):
"""The two vehicles couple in the run and stay coupled to its end; coupled from the start doesn't count."""
numbers = {name: number for number, name in box.vehicles.items()}
unknown = [name for name in (vehicle, other) if name not in numbers]
if unknown:
return False, f"the black box has no vehicle {' nor '.join(unknown)}"
pair = {numbers[vehicle], numbers[other]}
# of the two vehicles of a coupling the one with the lower number tells it, each change of that coupler
# under the same vehicle and side: the record which coupled the two, for as long as they stay coupled
made, parted = {}, None
for c in box.couplings:
coupler = (c.vehicle, c.side)
if {c.vehicle, c.other} == pair and c.flags & COUPLING_COUPLER:
made.setdefault(coupler, c)
elif coupler in made:
parted = (made.pop(coupler), c)
if not made:
if parted:
return False, f"{vehicle} and {other} coupled at t={parted[0].time:.1f}s but came apart at t={parted[1].time:.1f}s"
return False, f"none of the {len(box.couplings)} coupling record(s) of the black box couples {vehicle} and {other}"
c = min(made.values(), key=lambda c: c.time)
if c.time <= 0:
return False, f"{vehicle} and {other} were coupled when the recording started"
ends = ("front", "rear")
return True, (f"at t={c.time:.1f}s the {ends[c.side]} coupler of {box.vehicles[c.vehicle]} coupled to the "
f"{ends[c.other_side]} one of {box.vehicles[c.other]}, coupling 0x{c.flags:02X}")
# checks on the black box of the run rather than on the physicslog rows
BLACKBOX_CHECKS = {
"couples": check_couples,
}
# ---------------------------------------------------------------- main
@@ -513,7 +648,9 @@ def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, watch: bool
vehicle = f"autotest_{name}"
scenery = build_scenery(case, game_dir, name, vehicle)
ini = out / "autotest.ini"
ini.write_text(TEST_INI.format(timespeed=timespeed), encoding="ascii")
# the case's own settings go last, over the common ones
ini.write_text(TEST_INI.format(timespeed=timespeed) + "".join(line + "\n" for line in case.get("ini", [])),
encoding="ascii")
physicslog = game_dir / "physicslog" / f"{vehicle}.dat"
duration = float(case["duration"]) if "duration" in case else None
@@ -535,6 +672,9 @@ def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, watch: bool
# other trains of the case with a driver, named <something>_<vehicle>, log too
for other in physicslog.parent.glob(f"*_{vehicle}.dat"):
shutil.move(other, out / other.name)
# the black box is named after the case scenery too
if result.blackbox:
result.blackbox = Path(shutil.move(result.blackbox, out / f"{vehicle}.blackbox"))
ok = True
sim_t = result.rows[-1][COL_TIME] if result.rows else 0.0
@@ -566,9 +706,27 @@ def run_case(name: str, exe: Path, game_dir: Path, timespeed: float, watch: bool
say(f" FAIL no physicslog for '{vehicle}'")
return False
log = read_text(out / "log.txt")
box = read_blackbox(result.blackbox)
for check in case.get("checks", []):
params = {k: v for k, v in check.items() if k != "type"}
passed, detail = CHECKS[check["type"]](result.rows, **params)
if check["type"] in LOG_CHECKS:
# with no log text a log check proves nothing, log_lacks would pass vacuously
if log.strip():
passed, detail = LOG_CHECKS[check["type"]](log, **params)
else:
passed, detail = False, "log.txt is missing or empty"
elif check["type"] in BLACKBOX_CHECKS:
if box is not None:
# vehicles are named as in the fragment
params = {k: v.replace("{{vehicle}}", vehicle) if isinstance(v, str) else v for k, v in params.items()}
passed, detail = BLACKBOX_CHECKS[check["type"]](box, **params)
elif result.blackbox:
passed, detail = False, f"{result.blackbox.name} is not a black box, or is damaged"
else:
passed, detail = False, "the run left no black box (physicslog/*.blackbox); the build is too old, or has blackbox no"
else:
passed, detail = CHECKS[check["type"]](result.rows, **params)
say(f" {'PASS' if passed else 'FAIL'} {check['type']}: {detail}")
ok &= passed