Document the headless traps that produce convincing but bogus measurements

Running a commercial game through headless to measure something has four ways to
report a clean pass for a run that proved nothing, and they compound: --log is
needed before anything is printed (and it goes to stderr, so grepping forces the
streams together), the memory stick defaults to one beside the exe rather than
the app's, which means no installed firmware and a silent LLE-to-HLE fallback,
an unrecognised parameter then hides in the merged output, and a count of zero
errors reads identically whether the code ran or never got there.

All four of these cost a round of wrong results while investigating ME memory.

Also records that the line-ending check has to be done on the bytes, since Git
Bash's grep normalises them: the obvious `grep -c $'\r$'` for a stray LF reports
every file clean, which is how a handful of LF lines got into two CRLF files
here without the usual whole-file-rewrite tell in git diff --stat.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Henrik RydgårdandClaude Opus 5 committed 2026-09-20 16:39:35 -06:00
1 parent 1a08f3c88c
commit d40953d09b
2 files changed
+41 -1

No files matched your search

+16 -1
View File
@@ -12,7 +12,7 @@ for it:
| Doc | When you need it |
|---|---|
| [docs/building.md](docs/building.md) | Build commands for every target (VS/MSBuild, CMake, UWP, legacy Android NDK, libretro), unit tests, pspautotests |
| [docs/debugging.md](docs/debugging.md) | Driving the WebSocket debugger and PPSSPPHeadless from a script, breakpoint reliability per CPU backend, debugging a game that works on hardware |
| [docs/debugging.md](docs/debugging.md) | Driving the WebSocket debugger and PPSSPPHeadless from a script, measuring a commercial game with headless, breakpoint reliability per CPU backend, debugging a game that works on hardware |
| [docs/DebuggerThreading.md](docs/DebuggerThreading.md) | `Core_RunOnCPUThread` / `g_frameMutex` / shutdown-lock rules - required reading before touching debugger code |
| [docs/HLEModules.md](docs/HLEModules.md) | Adding an HLE module or function, and the seven build files a new source file goes in |
| [docs/translations.md](docs/translations.md) | Translating UI strings with Tools/langtool |
@@ -48,6 +48,14 @@ for it:
`newline=''` silently converts the whole file, turning a two-line addition into a 5000-line diff.
Check `git diff --stat` before committing - a whole-file rewrite is obvious there and invisible in
the editor. Prefer the Edit tool, which does exact string replacement and can't do this.
**Don't verify this with `grep -c $'\r$'`** - Git Bash's grep normalises line endings, so it counts
a bare-LF line as having a CR and reports any file as clean. The opposite mistake to a whole-file
rewrite is inserting a handful of LF lines into a CRLF file (a Python `"""..."""` block written with
`newline=''` does exactly this), and that shows in `--stat` only as the lines you meant to add. Check
the bytes instead, e.g. `python -c "d=open(F,'rb').read(); print(d.count(b'\r\n'), d.count(b'\n'))"`,
and treat git's "LF will be replaced by CRLF the next time Git touches it" as the warning it is
rather than autocrlf noise.
6. **Don't feed Python to `bash -c` via a heredoc when the code needs a literal backslash in its
*output*.** Git Bash strips one level of escaping on the way in even with a quoted delimiter
(`<<'PY'`), so `"\\n"` reaches Python as `"\n"` and writes a real newline into the file - no error,
@@ -111,6 +119,13 @@ python test.py -g --graphics=software
around a hundred failures that mean nothing is wrong. The only meaningful result is `0 tests failed`.
(The debug-CRT "Detected memory leaks!" dump after the summary line is normal, not a failure.)
When the thing under test is a commercial game rather than a suite, headless needs `--log` before it prints
anything (to stderr), and defaults its memory stick to `<exe dir>/memstick` rather than the app's - so the
firmware you installed in the app isn't there, and LLE modules silently fall back to HLE. A run configured
differently from what you asked for, or one that never reached the code, produces the same all-zero counts as
a clean one, so assert the exit code and a positive "we got here" counter before believing any error count.
The traps in full: [docs/debugging.md](docs/debugging.md).
New unit tests are added to `availableTests`; large ones go in their own file in `unittest/`, which has
to be listed in **three** build files, not two: `CMakeLists.txt`, `unittest/UnitTests.vcxproj` (and its
`.filters`), and `android/jni/Android.mk`, which builds a unit test executable of its own. Miss the last