From d40953d09bb684d580cf106520853c29fbc2a9b1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20Rydg=C3=A5rd?= Date: Sat, 19 Sep 2026 10:20:58 -0600 Subject: [PATCH] Document the headless traps that produce convincing but bogus measurements Running a commercial game through headless to measure something has four ways to report a clean pass for a run that proved nothing, and they compound: --log is needed before anything is printed (and it goes to stderr, so grepping forces the streams together), the memory stick defaults to one beside the exe rather than the app's, which means no installed firmware and a silent LLE-to-HLE fallback, an unrecognised parameter then hides in the merged output, and a count of zero errors reads identically whether the code ran or never got there. All four of these cost a round of wrong results while investigating ME memory. Also records that the line-ending check has to be done on the bytes, since Git Bash's grep normalises them: the obvious `grep -c $'\r$'` for a stray LF reports every file clean, which is how a handful of LF lines got into two CRLF files here without the usual whole-file-rewrite tell in git diff --stat. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 17 ++++++++++++++++- docs/debugging.md | 25 +++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 812e758400..e146b6d48a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,7 +12,7 @@ for it: | Doc | When you need it | |---|---| | [docs/building.md](docs/building.md) | Build commands for every target (VS/MSBuild, CMake, UWP, legacy Android NDK, libretro), unit tests, pspautotests | -| [docs/debugging.md](docs/debugging.md) | Driving the WebSocket debugger and PPSSPPHeadless from a script, breakpoint reliability per CPU backend, debugging a game that works on hardware | +| [docs/debugging.md](docs/debugging.md) | Driving the WebSocket debugger and PPSSPPHeadless from a script, measuring a commercial game with headless, breakpoint reliability per CPU backend, debugging a game that works on hardware | | [docs/DebuggerThreading.md](docs/DebuggerThreading.md) | `Core_RunOnCPUThread` / `g_frameMutex` / shutdown-lock rules - required reading before touching debugger code | | [docs/HLEModules.md](docs/HLEModules.md) | Adding an HLE module or function, and the seven build files a new source file goes in | | [docs/translations.md](docs/translations.md) | Translating UI strings with Tools/langtool | @@ -48,6 +48,14 @@ for it: `newline=''` silently converts the whole file, turning a two-line addition into a 5000-line diff. Check `git diff --stat` before committing - a whole-file rewrite is obvious there and invisible in the editor. Prefer the Edit tool, which does exact string replacement and can't do this. + + **Don't verify this with `grep -c $'\r$'`** - Git Bash's grep normalises line endings, so it counts + a bare-LF line as having a CR and reports any file as clean. The opposite mistake to a whole-file + rewrite is inserting a handful of LF lines into a CRLF file (a Python `"""..."""` block written with + `newline=''` does exactly this), and that shows in `--stat` only as the lines you meant to add. Check + the bytes instead, e.g. `python -c "d=open(F,'rb').read(); print(d.count(b'\r\n'), d.count(b'\n'))"`, + and treat git's "LF will be replaced by CRLF the next time Git touches it" as the warning it is + rather than autocrlf noise. 6. **Don't feed Python to `bash -c` via a heredoc when the code needs a literal backslash in its *output*.** Git Bash strips one level of escaping on the way in even with a quoted delimiter (`<<'PY'`), so `"\\n"` reaches Python as `"\n"` and writes a real newline into the file - no error, @@ -111,6 +119,13 @@ python test.py -g --graphics=software around a hundred failures that mean nothing is wrong. The only meaningful result is `0 tests failed`. (The debug-CRT "Detected memory leaks!" dump after the summary line is normal, not a failure.) +When the thing under test is a commercial game rather than a suite, headless needs `--log` before it prints +anything (to stderr), and defaults its memory stick to `/memstick` rather than the app's - so the +firmware you installed in the app isn't there, and LLE modules silently fall back to HLE. A run configured +differently from what you asked for, or one that never reached the code, produces the same all-zero counts as +a clean one, so assert the exit code and a positive "we got here" counter before believing any error count. +The traps in full: [docs/debugging.md](docs/debugging.md). + New unit tests are added to `availableTests`; large ones go in their own file in `unittest/`, which has to be listed in **three** build files, not two: `CMakeLists.txt`, `unittest/UnitTests.vcxproj` (and its `.filters`), and `android/jni/Android.mk`, which builds a unit test executable of its own. Miss the last diff --git a/docs/debugging.md b/docs/debugging.md index 04e57cc5e8..0d286ded64 100644 --- a/docs/debugging.md +++ b/docs/debugging.md @@ -146,6 +146,31 @@ A working invocation, and the traps around it: a lot of emulation speed (a 25-second run of a game playing video wrote 450k lines and ran several times slower than real time, which on its own looks like the stall you're hunting). +## Measuring a commercial game with headless + +Headless will happily run a game and print nothing, or run *a different configuration than the one you asked +for*, and both look like a clean pass if you are counting log lines. Four traps, each of which silently +produced a round of bogus results here: + +- **`--log` is required for any log output at all**, and the log goes to **stderr**. Without it you get only + headless's own lines (`Loaded State`, `TIMEOUT`); `-d`/`-v` set the level but don't turn the printf logger + on. So folding stderr in isn't optional if you're grepping, which is what makes the third trap bite. +- **Headless's memory stick is not the app's.** It defaults to `/memstick` (`headless/Headless.cpp`), + so `Windows///memstick`, while the app derives its own from `installed.txt` or Documents + (`InitMemstickDirectory` in `Windows/main.cpp`, which carries a TODO about sharing the derivation). Headless + *creates* that directory, empty `PSP/NAND/flash0` and all, so firmware installed through the app is invisible + and every LLE module quietly falls back to HLE - at which point you are measuring the HLE you were trying to + compare against. Pass `--memstick=` explicitly; it resolves relative to the CWD, not the exe. +- **Check the exit code.** An unrecognised parameter prints `Error: ...` to stderr and exits 1, which is + correct and easy to throw away: fold stderr into stdout (which the first trap forces), count grep hits, and a + run that never started reports the same all-zero line as a clean one. +- **Zero is not a pass.** Measuring by log-grepping needs a positive precondition asserted separately ("did + this run reach the code at all"), or "0 errors" also means "0 anything" - which is equally what a failed + boot, a savestate that never reaches the cutscene, and a silent HLE fallback produce. + +The last three compound: the fix is to treat the run's exit code and a positive "we got here" counter as +preconditions, and only then believe the error counts. + ## Debugging and breakpoint considerations It might be worth trying the interpreter - all types of breakpoints are the most reliable with this CPU backend.