Frametests: variant suffix/compare-suffix for shared references, output dir 'out'

Variants can now declare a compare-suffix (defaulting to the variant's own
suffix) to compare against another variant's reference images, enabling
shared references across e.g. opengl/vulkan when their output matches.
Reference images are only generated for variants whose suffix equals their
compare-suffix; other variants fail clearly if the shared reference is
missing. Rename the default output directory from frametest-out to out.

.gitignore

@@ -61,7 +61,7 @@ imgui.ini
 PPSSPPControls.dat
 
 # Frametest output (see frametests.py)
-frametest-out/
+out/
 
 # Gradle/Android Studio
 .gradle

docs/frametest.md

@@ -26,7 +26,7 @@ python3 frametests.py frametests/frametests.json
 The first run generates reference images for any dumps that don't have them
 yet (status `NEW`) and compares the rest (status `PASS`/`FAIL`). A summary is
 printed and a self-contained HTML report is written to
-`frametests/frametest-out/report.html` (default output dir; gitignored).
+`frametests/out/report.html` (default output dir; gitignored).
 
 Exit code is `0` if all tests passed, `1` if anything failed, `2` on
 configuration/usage errors.
@@ -35,18 +35,22 @@ configuration/usage errors.
 
 The configuration is a single JSON file. All relative paths are resolved
 against the config file's directory, so the same script works with any test
-set by just pointing `--config` at a different file.
+set by just pointing `--config` at a different file. The config lives with
+the test set it describes (e.g. `frametests/frametests.json`) and is not
+committed to the repository.
 
 ```json
 {
 	"testRoot": "dumps",
 	"refRoot": "ref",
-	"outputRoot": "frametest-out",
+	"outputRoot": "out",
 	"headlessPath": "",
 	"timeout": 60,
 	"maxMse": 0.0,
 	"variants": {
-		"soft": "--graphics=software"
+		"soft": {
+			"args": "--graphics=software"
+		}
 	}
 }
 ```
@@ -55,42 +59,50 @@ set by just pointing `--config` at a different file.
 |-----------------|-----------------------------------------------------------------------------|
 | `testRoot`      | Directory tree containing frame dumps (`.ppdmp` files, possibly wrapped in `.zip`). |
 | `refRoot`       | Where reference images live; mirrors the `testRoot` tree.                   |
-| `outputRoot`    | Where logs, diff images, the report, and generated references go.           |
+| `outputRoot`    | Where logs, diff images, the report, and generated references go (default `out`). |
 | `headlessPath`  | Path to the `PPSSPPHeadless` binary. Empty = auto-detect (see below).        |
 | `timeout`       | Per-test timeout in seconds.                                                 |
 | `maxMse`        | Maximum allowed MSE for screenshot comparison (0 = exact match).            |
-| `variants`      | Map of variant name to command line arguments for the headless binary.       |
+| `variants`      | Map of variant name to variant configuration (see below).                   |
 
 ### Variants
 
-Each test runs once per variant. The variant name is appended to the
-reference image filename, so each variant gets its own reference set:
+Each test runs once per variant. A variant has a **suffix** and a
+**compare-suffix**:
 
-```
-dumps/Depth/11578 Virtua Tennis pause menu ULES00126_0002.zip
-  → ref/Depth/11578 Virtua Tennis pause menu ULES00126_0002-soft.png
-```
-
-The variant value is an arbitrary command line argument string passed to
-`PPSSPPHeadless`, so new rendering configurations are just new entries (or a
-different config file on a machine with the right GPU):
+- The **suffix** (default: the variant name) is appended to the variant's own
+  output image filenames (`actuals`, `diffs`, `logs`).
+- The **compare-suffix** (default: the suffix) names the reference image the
+  variant compares against: `<name>-<compare-suffix>.png`.
+- **Reference images are only generated for variants whose suffix matches
+  their compare-suffix.** This lets several variants share a single reference
+  set - e.g. OpenGL and Vulkan outputs should be nearly bitwise identical, so
+  a `gl` variant can compare against the `soft` references:
 
 ```json
 "variants": {
-	"soft":  "--graphics=software",
-	"gl":    "--graphics=opengl",
-	"gl-4x": "--graphics=opengl --resolution-scale=4"
+	"soft": { "args": "--graphics=software" },
+	"gl":   { "args": "--graphics=opengl", "compare-suffix": "soft" },
+	"vul":  { "args": "--graphics=vulkan", "compare-suffix": "soft" }
 }
 ```
 
+A plain string value (`"soft": "--graphics=software"`) is shorthand for
+`{ "args": "...", "suffix": "soft", "compare-suffix": "soft" }`.
+
 ## How a test runs
 
 For each dump (recursively under `testRoot`) and each variant:
 
-- If the reference image `<name>-<variant>.png` is **missing**: the dump is
-  rendered and the output saved as the new reference. Status: `NEW`. This is
-  how references are created - run locally, then commit the generated images
-  (also copied to `<outputRoot>/generated/` for convenience).
+- If the variant's reference image `<name>-<compare-suffix>.png` is
+  **missing** and the variant generates references (suffix ==
+  compare-suffix): the dump is rendered and the output saved as the new
+  reference. Status: `NEW`. This is how references are created - run locally,
+  then commit the generated images (also copied to `<outputRoot>/generated/`
+  for convenience).
+- If the reference is **missing** for a variant that doesn't generate
+  references (suffix != compare-suffix): Status: `ERROR` - the shared
+  reference needs to be generated by the matching variant first.
 - If the reference **exists**: the dump is rendered, the output saved to
   `<outputRoot>/actuals/`, and compared against the reference using MSE
   (mean squared error over R, G, B per pixel, alpha ignored). A visual
@@ -161,12 +173,12 @@ show up inline in the GitHub Actions log. A typical CI job:
   uses: actions/upload-artifact@v4
   with:
     name: frametest-report
-    path: frametests/frametest-out/
+    path: frametests/out/
 ```
 
 A missing reference image in CI means the test set is incomplete - the run
 fails and the generated reference is available in
-`frametests/frametest-out/generated/` (part of the uploaded artifact) so it
+`frametests/out/generated/` (part of the uploaded artifact) so it
 can be committed.
 
 ## Custom machines

frametests.py

@@ -268,14 +268,31 @@ def main():
 
     test_root = (config_dir / config["testRoot"]).resolve()
     ref_root = (config_dir / config["refRoot"]).resolve()
-    output_root = (config_dir / config.get("outputRoot", "frametest-out")).resolve()
+    output_root = (config_dir / config.get("outputRoot", "out")).resolve()
     timeout = float(config.get("timeout", 60))
     max_mse = float(config.get("maxMse", 0.0))
-    variants = config.get("variants", {})
-    if not variants:
+    raw_variants = config.get("variants", {})
+    if not raw_variants:
         print("ERROR: no variants defined in %s" % config_path, file=sys.stderr)
         return 2
 
+    # Each variant has a suffix (used for its output images) and a compare-suffix
+    # (the reference image it compares against).  Reference images are only
+    # generated for variants whose suffix matches their compare-suffix, so that
+    # other variants can share a reference (e.g. gl comparing against the soft
+    # reference).  A plain string entry means args only, with both suffixes
+    # defaulting to the variant name.
+    variants = {}
+    for key, entry in raw_variants.items():
+        if isinstance(entry, str):
+            args_str = entry
+            suffix = key
+        else:
+            args_str = entry.get("args", "")
+            suffix = entry.get("suffix", key)
+        compare_suffix = entry.get("compare-suffix", suffix) if not isinstance(entry, str) else suffix
+        variants[key] = {"args": shlex.split(args_str), "suffix": suffix, "compare_suffix": compare_suffix}
+
     headless = find_headless(config_dir, config.get("headlessPath", ""))
     if headless is None:
         print("ERROR: PPSSPPHeadless binary not found. Set 'headlessPath' in %s or PPSSPP_HEADLESS, or run from the repo root." % config_path, file=sys.stderr)
@@ -303,7 +320,13 @@ def main():
 
     print("Headless: %s" % headless)
     print("Test root: %s" % test_root)
-    print("Variants: %s" % ", ".join(variants.keys()))
+    variant_desc = []
+    for key, v in variants.items():
+        if v["compare_suffix"] == v["suffix"]:
+            variant_desc.append(key)
+        else:
+            variant_desc.append("%s (ref: %s)" % (key, v["compare_suffix"]))
+    print("Variants: %s" % ", ".join(variant_desc))
     print("Running %d dumps..." % len(dumps))
 
     results = []
@@ -316,21 +339,30 @@ def main():
         rel = dump.relative_to(test_root)
         base = strip_dump_extensions(dump.name)
         rel_dir = rel.parent
-        for variant, variant_args_str in variants.items():
-            variant_args = shlex.split(variant_args_str)
-            ref_path = ref_root / rel_dir / ("%s-%s.png" % (base, variant))
+        for variant, vinfo in variants.items():
+            variant_args = vinfo["args"]
+            generate_refs = vinfo["suffix"] == vinfo["compare_suffix"]
+            ref_path = ref_root / rel_dir / ("%s-%s.png" % (base, vinfo["compare_suffix"]))
             actual_path = actual_dir / rel_dir / ("%s-%s.png" % (base, variant))
             diff_path = diff_dir / rel_dir / ("%s-%s.png" % (base, variant))
             log_path = log_dir / rel_dir / ("%s-%s.log" % (base, variant))
             for p in (ref_path, actual_path, diff_path, log_path):
                 p.parent.mkdir(parents=True, exist_ok=True)
 
-            status, mse, output, timed_out = run_test(
-                headless, dump, variant_args, ref_path, actual_path, diff_path,
-                max_mse, timeout, output_root)
+            if not ref_path.exists() and not generate_refs:
+                status = STATUS_ERROR
+                mse = None
+                output = ""
+                timed_out = False
+            else:
+                status, mse, output, timed_out = run_test(
+                    headless, dump, variant_args, ref_path, actual_path, diff_path,
+                    max_mse, timeout, output_root)
 
             detail = ""
-            if status == STATUS_FAIL:
+            if status == STATUS_ERROR and not ref_path.exists() and not generate_refs:
+                detail = "no reference image '%s' available for this variant (references are only generated for variants whose suffix matches their compare-suffix)" % vinfo["compare_suffix"]
+            elif status == STATUS_FAIL:
                 if timed_out:
                     detail = "timed out"
                 elif mse is None:
@@ -357,7 +389,7 @@ def main():
                     print("::error file=%s::%s errored (%s)" % (rel, variant, detail))
             elif status == STATUS_NEW:
                 new_refs += 1
-                gen_path = generated_dir / rel_dir / ("%s-%s.png" % (base, variant))
+                gen_path = generated_dir / rel_dir / ("%s-%s.png" % (base, vinfo["compare_suffix"]))
                 gen_path.parent.mkdir(parents=True, exist_ok=True)
                 shutil.copyfile(ref_path, gen_path)
                 if strict: