Merge pull request #22355 from hrydgard/headless-gl-offscreen

Make headless work off-screen with OpenGL on the Mac

docs/debugging.md

@@ -141,8 +141,8 @@ A working invocation, and the traps around it:
   calls `gpu->PerformWriteFormattedFromMemory()`. The software renderer reads that memory directly and so renders
   the frame whether or not anything was notified. A missing notification therefore looks perfect under
   `--graphics=software` and shows up as a frozen screen on every real backend, while the decode logs keep scrolling
-  past as if all were well. Reproduce display bugs on `--graphics=d3d11` or `--graphics=vulkan` (`directx9` is not
-  a valid value) and compare `--screenshot-save=` output, not the log.
+  past as if all were well. Reproduce display bugs on a hardware backend (see "Choosing a GPU backend" below;
+  `directx9` is not a valid value) and compare `--screenshot-save=` output, not the log.
 - **`wsdbg --launch` only works with the headless build.** The app build is a GUI-subsystem exe with no stdout, so
   the `Listening on port N` line never reaches the launcher and it gives up. Start it yourself with an explicit
   `--debugger=PORT` and point wsdbg at that port. Note also that `--debugger-run` is `CmdLineMode::Headless`; the
@@ -178,39 +178,7 @@ produced a round of bogus results here:
 The last three compound: the fix is to treat the run's exit code and a positive "we got here" counter as
 preconditions, and only then believe the error counts.
 
-**If headless goes silent early in a game's boot, check which GPU backend it's actually using.** From
-551e4cd0ab (2026-08-04) until 2026-09-22, headless without `--graphics` quietly ran the OpenGL backend instead of
-the software renderer the README promises (`bSoftwareRendering` defaulted to false). Under Mesa llvmpipe on
-Linux/WSL, OpenGL hung every commercial game tried early in boot, e.g. AI Go right after its
-`sceKernelCreateCallback`, with no CPU use. Neither `--timeout-wall` nor `--timeout-emulated` fired, because both
-are checked only when the emulation loop comes back around, so a blocked host thread defeats them. `test.py`
-passes no `--graphics` either, so it stalled the same way. The default is software again; the OpenGL hang itself
-isn't fixed.
-
-**For game runs, prefer `--graphics=vulkan`.** Headless renders it offscreen, into images of its own with no
-window, surface or swapchain, so it needs no display, and works on macOS through MoltenVK. It's far faster than
-the software renderer, which runs display lists synchronously inside `sceGeListEnQueue` and so dominates any
-profile of the emulator thread: 30 emulated seconds of God of War take about 4 seconds instead of a minute. OpenGL
-headless deadlocks at startup on macOS too, main thread in `GLRenderManager::ThreadFrame` with no CPU use. The
-pspautotests pass on Vulkan apart from 17 GPU tests, whose references are hardware screenshots that the hardware
-backends don't match exactly (edge pixels, dithering, filtering, Metal's always-on primitive restart), so keep
-`--graphics=software` for those.
-
-That cost a lot of time because the first theories were confounded. Runs "worked in the background and hung in
-the foreground" only because the background ones happened to have `--graphics=software` added. Change one variable
-at a time, and diff the full command lines of a good and a bad run before theorising about the environment. If a
-run is still silent a few seconds in, it isn't going to recover, so don't wait out the full timeout. Other things
-that made it worse:
-
-- `docs/debugging.md` wasn't read first, so the `timeout`-wrapper and full-`--log` warnings above were missed.
-- The runs were wrapped in `timeout`, and a tool-call limit shorter than that sent them to the background. The
-  follow-up `pkill -f` then killed its own shell rather than the emulators, so hung instances piled up across runs.
-- `--memstick` was blamed first. It matters for LLE firmware modules, but a game that needs none (AI Go) stalled
-  just the same without it.
-- Pass `--graphics=software` whenever rendering doesn't matter, even though it's the default, so a copied
-  command line doesn't depend on the default.
-
-For the silent-fallback half of this, headless refuses the run rather than substituting: an explicit
+Rather than silently falling back to HLE, headless refuses the run: an explicit
 `--disable-hle=` whose firmware module isn't there names the module, prints the `flash0:/kd` and memory stick
 it looked in (usually enough to spot that it's the one beside the exe), and exits 1. Only an *explicit*
 `--disable-hle` binds - sceMpeg and sceMp4 are LLE by default and still fall back quietly, or every run on a
@@ -224,6 +192,25 @@ sorted - three appeared identically with `--force-hle=16`, which made them the g
 fourth only under the real module, which made it ours. Neither `--nand=` pointing somewhere empty
 nor `--appendconfig` does this job: the firmware gets found anyway and the setting is per-game.
 
+### Choosing a GPU backend
+
+- **`--graphics=software`** (the default) needs no GPU at all, and matches the GPU pspautotests' reference
+  screenshots (taken on a PSP) best. It's slow for games, though: it runs display lists synchronously inside
+  `sceGeListEnQueue`, which then dominates any profile of the emulator thread.
+- **`--graphics=vulkan`** renders offscreen, into images of its own with no window or swapchain, so it needs no
+  display. On macOS it goes through MoltenVK.
+- **`--graphics=opengl`** renders offscreen on macOS (a CGL context with a framebuffer object of its own), and
+  through a hidden SDL window elsewhere.
+- **`--graphics=d3d11`** (Windows) renders through a hidden window.
+
+For game runs, prefer a hardware backend: 30 emulated seconds of God of War take 3-4 seconds instead of a minute.
+The pspautotests pass on Vulkan and OpenGL except for 17 GPU tests, whose references are hardware screenshots
+that the hardware backends don't match exactly.
+
+Pass `--graphics` explicitly even when you want the default, so a copied command line doesn't depend on it. If a
+run goes silent with no CPU use, a host thread is blocked, and neither `--timeout-wall` nor `--timeout-emulated`
+will end it, as both are only checked when the emulation loop comes around.
+
 ## Debugging and breakpoint considerations
 
 It might be worth trying the interpreter - all types of breakpoints are the most reliable with this CPU backend.

headless/Headless.cpp

@@ -330,12 +330,14 @@ struct AutoTestOptions {
 	int requiredDisableHLE;
 };
 
-// Ends a frame of the draw context the way the app does, presenting it. With Vulkan, presenting is
-// also what returns a frame's image, and a frame that isn't presented would wait forever for its
-// next one. The other backends present to a hidden window, which could wait for vsync.
+// Ends a frame of the draw context the way the app does, presenting it. Unpresented frames never
+// finish: with Vulkan, presenting is what returns a frame's image, and OpenGL's render thread only
+// finishes a frame when it's presented, so either would eventually wait forever. Neither waits for
+// vsync here (Vulkan and, on macOS, OpenGL render offscreen, and the hidden-window OpenGL context
+// swaps with interval 0). D3D11 presents to a hidden window, which could.
 static void EndDrawFrame(Draw::DrawContext *draw) {
 	draw->EndFrame();
-	if (GetGPUBackend() == GPUBackend::VULKAN) {
+	if (GetGPUBackend() == GPUBackend::VULKAN || GetGPUBackend() == GPUBackend::OPENGL) {
 		draw->Present(Draw::PresentMode::FIFO);
 	}
 }
@@ -964,6 +966,11 @@ int main(int argc, const char* argv[]) {
 			vulkanContext->SetOffscreen(480, 272);
 			graphicsContext = vulkanContext;
 			deviceSetting = &g_Config.sVulkanDevice;
+#if PPSSPP_PLATFORM(MAC) && defined(SDL)
+		} else if (gpuCore == GPUCORE_GLES) {
+			// So does OpenGL, into a framebuffer object of its own.
+			graphicsContext = new CGLHeadlessGraphicsContext(480, 272);
+#endif
 		} else {
 			// TODO: Will we need a larger window for higher resolutions? Well, not if we use buffered rendering.
 			window = CreateHiddenWindow(480, 272, cmdLineOptions.gpuBackend.value_or(GPUBackend::OPENGL), &windowDesc);

headless/SDLHeadlessGLGraphicsContext.cpp

@@ -25,7 +25,12 @@
 #include "headless/SDLHeadlessGLGraphicsContext.h"
 #include "Common/GPU/OpenGL/GLCommon.h"
 #include "Common/GPU/OpenGL/GLFeatures.h"
+#if PPSSPP_PLATFORM(MAC)
+// After glew, which has its own definitions of what this would include.
+#include <OpenGL/OpenGL.h>
+#endif
 #include "Common/GPU/thin3d_create.h"
+#include "Common/StringUtils.h"
 #include "Common/File/VFS/VFS.h"
 #include "Common/File/VFS/DirectoryReader.h"
 #include "Common/GPU/GraphicsContext.h"
@@ -92,6 +97,92 @@ bool SDLHeadlessGLGraphicsContext::InitSurface(WindowSystem winsys, void *data1,
 	return true;
 }
 
+#if PPSSPP_PLATFORM(MAC)
+
+// Bound by the GL backend wherever it would bind the default framebuffer.
+extern GLuint g_defaultFBO;
+
+bool CGLHeadlessGraphicsContext::InitAPI(void *wnd, std::string *deviceName, std::string *errorMessage) {
+	const CGLPixelFormatAttribute attributes[] = {
+		kCGLPFAAccelerated,
+		kCGLPFAOpenGLProfile, (CGLPixelFormatAttribute)kCGLOGLPVersion_GL4_Core,
+		kCGLPFAColorSize, (CGLPixelFormatAttribute)24,
+		kCGLPFAAlphaSize, (CGLPixelFormatAttribute)8,
+		(CGLPixelFormatAttribute)0,
+	};
+	CGLPixelFormatObj pixelFormat = nullptr;
+	GLint formatCount = 0;
+	if (CGLChoosePixelFormat(attributes, &pixelFormat, &formatCount) != kCGLNoError || !pixelFormat) {
+		*errorMessage = "CGLChoosePixelFormat failed";
+		return false;
+	}
+	CGLContextObj context = nullptr;
+	CGLError err = CGLCreateContext(pixelFormat, nullptr, &context);
+	CGLDestroyPixelFormat(pixelFormat);
+	if (err != kCGLNoError) {
+		*errorMessage = StringFromFormat("CGLCreateContext failed: %s", CGLErrorString(err));
+		return false;
+	}
+	context_ = context;
+	CGLSetCurrentContext(context);
+
+	// Core profile drivers leave some extensions out of the list, so glew has to look for them anyway.
+	SetGLCoreContext(true);
+	glewExperimental = true;
+	if (glewInit() != GLEW_OK) {
+		*errorMessage = "Failed to initialize glew";
+		return false;
+	}
+	// glew causes an invalid enum error with core profiles, ignore it.
+	glGetError();
+
+	// There's no drawable, so the backbuffer is a framebuffer object of our own.
+	glGenRenderbuffers(1, &colorBuffer_);
+	glBindRenderbuffer(GL_RENDERBUFFER, colorBuffer_);
+	glRenderbufferStorage(GL_RENDERBUFFER, GL_RGBA8, width_, height_);
+	glGenRenderbuffers(1, &depthStencilBuffer_);
+	glBindRenderbuffer(GL_RENDERBUFFER, depthStencilBuffer_);
+	glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH24_STENCIL8, width_, height_);
+	glGenFramebuffers(1, &fbo_);
+	glBindFramebuffer(GL_FRAMEBUFFER, fbo_);
+	glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, colorBuffer_);
+	glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_STENCIL_ATTACHMENT, GL_RENDERBUFFER, depthStencilBuffer_);
+	if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE) {
+		*errorMessage = "The offscreen framebuffer is incomplete";
+		return false;
+	}
+	g_defaultFBO = fbo_;
+
+	CheckGLExtensions();
+	SetGPUBackend(GPUBackend::OPENGL);
+	draw_ = Draw::T3DCreateGLContext(false);
+	renderManager_ = (GLRenderManager *)draw_->GetNativeObject(Draw::NativeObject::RENDER_MANAGER);
+	renderManager_->SetInflightFrames(g_Config.iInflightFrames);
+	bool success = draw_->CreatePresets();
+	_assert_(success);
+	// Nothing to swap. Flushing keeps the frames moving like a swap would.
+	renderManager_->SetSwapFunction([]() {
+		glFlush();
+	});
+	return success;
+}
+
+void CGLHeadlessGraphicsContext::ShutdownSurface() {
+	delete draw_;
+	draw_ = nullptr;
+
+	g_defaultFBO = 0;
+	glBindFramebuffer(GL_FRAMEBUFFER, 0);
+	glDeleteFramebuffers(1, &fbo_);
+	glDeleteRenderbuffers(1, &colorBuffer_);
+	glDeleteRenderbuffers(1, &depthStencilBuffer_);
+	CGLSetCurrentContext(nullptr);
+	CGLDestroyContext((CGLContextObj)context_);
+	context_ = nullptr;
+}
+
+#endif
+
 bool SDLHeadlessGLGraphicsContext::InitAPI(void *wnd, std::string *deviceName, std::string *errorMessage) {
 	SDL_Init(SDL_INIT_VIDEO);
 

headless/SDLHeadlessGLGraphicsContext.h

@@ -19,6 +19,7 @@
 
 #ifdef SDL
 
+#include "ppsspp_config.h"
 #include <SDL3/SDL.h>
 
 #include "Common/GPU/GraphicsContext.h"
@@ -70,4 +71,58 @@ private:
 void *CreateHiddenWindow(int w, int h, GPUBackend backend, WindowDesc *desc);
 void DestroyHiddenWindow(void *window, WindowDesc desc);
 
+#if PPSSPP_PLATFORM(MAC)
+
+// An OpenGL context with no window: a CGL context without a drawable, rendering into a framebuffer
+// object of its own that stands in for the backbuffer (through g_defaultFBO). Core profile, like
+// the SDL app uses on macOS.
+class CGLHeadlessGraphicsContext : public GraphicsContext {
+public:
+	CGLHeadlessGraphicsContext(int width, int height) : width_(width), height_(height) {}
+	~CGLHeadlessGraphicsContext() { delete draw_; }
+
+	bool InitAPI(void *wnd, std::string *deviceNameSetting, std::string *errorMessage) override;
+	bool InitSurface(WindowSystem winsys, void *data1, void *data2, std::string *errorMessage) override {
+		return true;
+	}
+	void ShutdownSurface() override;
+
+	bool NeedsSeparateEmuThread() const override { return true; }
+
+	Draw::DrawContext *GetDrawContext() override {
+		return draw_;
+	}
+
+	void ThreadStart() override {
+		renderManager_->ThreadStart(draw_);
+	}
+
+	bool ThreadFrame() override {
+		return renderManager_->ThreadFrame();
+	}
+
+	void ThreadEnd() override {
+		renderManager_->ThreadEnd();
+	}
+
+	void Resize() override {}
+
+	// Call from emu thread
+	void NotifyEmuThreadExit() override {
+		renderManager_->NotifyEmuThreadExit();
+	}
+
+private:
+	Draw::DrawContext *draw_ = nullptr;
+	GLRenderManager *renderManager_ = nullptr;
+	void *context_ = nullptr;  // CGLContextObj
+	unsigned int fbo_ = 0;
+	unsigned int colorBuffer_ = 0;
+	unsigned int depthStencilBuffer_ = 0;
+	int width_;
+	int height_;
+};
+
+#endif
+
 #endif