When I rewrote cable-club, the first thing I built was not the CPU or the renderer. It was the test harness. An emulator without accuracy tests is a program that produces plausible-looking output with no way to know if it is correct. I had already learned that lesson once.
What accuracy testing means
A GBA emulator can pass every standard CPU test suite — every ARM instruction, every Thumb instruction, every edge case in isolation — and still produce the wrong output when running a real game. The test suites verify that individual instructions behave correctly. They do not verify that the CPU, memory bus, DMA controller, interrupt system, and PPU all interact correctly over thousands of frames.
Accuracy testing means running a test ROM for a fixed number of frames, capturing the final framebuffer, and comparing it against a reference produced by a known-correct emulator. If the pixels match, the entire system — not just the CPU — produced the right result.
The reference emulator is NanoBoyAdvance, a cycle-accurate GBA emulator that passes 100% of the standard test suites. If NanoBoyAdvance and cable-club produce the same frame after the same number of cycles, cable-club is correct. If they differ, something is wrong, and the diff image shows exactly which pixels are off.
An emulator without accuracy tests is a program that produces plausible-looking output with no way to know if it is correct.
The format
Each test is defined in a TOML manifest:
[test]
name = "arm"
rom_url = "https://github.com/jsmolka/gba-tests/releases/..."
rom_sha256 = "abc123..."
frames = 120
[[references]]
emulator = "NanoBoyAdvance"
version = "1.8.2"
bios = "real"
hash = "def456..."
The reference hash is a SHA256 of the raw framebuffer bytes — 240×160 pixels in BGR555 format, 76,800 bytes. Not a PNG hash, because PNG encoding varies between libraries and platforms. Raw framebuffer bytes are deterministic.
The ROM is not stored in the repository. A download script fetches test ROMs from their original sources. The repository contains only manifests, reference hashes, and tooling.
Why it became a separate project
The test harness started as a crate inside cable-club. It worked for cable-club, but the format — TOML manifests, reference hashes, a comparison tool — is not Rust-specific. Any GBA emulator in any language can run a ROM headlessly, capture a screenshot, and compare a hash.
The GBA emulation community does not have a shared accuracy benchmark. The closest precedent is c-sp/game-boy-test-roms for the Game Boy, which curates test ROMs into versioned releases. But it stops at ROM distribution. Nobody built the automated comparison layer — the part that tells you whether your emulator’s output actually matches reality.
So I extracted the test data, manifests, comparison tool, and runner framework into gba-accuracy-tests — a standalone Python project that any emulator can plug into.
How other emulators use it
A runner is a small adapter that knows how to invoke an emulator headlessly. The repository ships with runners for mGBA (via Lua scripting) and cable-club (via its accuracy-sweep binary). Adding a new emulator means writing a Python file with two functions: is_available() and run_test().
python compare.py run --runner mgba --suite jsmolka
The output is pass/fail per test, with diff images for failures showing exactly which pixels differ from the reference. CI-friendly. Language-agnostic.
What the tests caught
The Frame 476 bug was found because the accuracy harness ran FireRed long enough for the prefetch pipeline issue to surface. Standard CPU tests would not have caught it.
But the harness caught subtler things too. An LZ77 VRAM decompression routine that wrote bytes instead of halfwords — producing a corrupted Oak portrait that looked almost right but had shifted pixel columns. A VBlank flag that was not being cleared when the scanline counter wrapped to zero, causing the PPU to skip frames. ARM LDM/STM instructions that used the wrong register bank when the S-bit was set, breaking user-mode access from privileged code.
Each of these bugs passed the standard test suites. Each produced wrong output that the accuracy harness detected as a hash mismatch. Each came with a diff image pointing to the exact pixels that were wrong.
Each bug passed the standard test suites. Each was caught by frame-level comparison against a known-correct reference.
The gap in the ecosystem
The GBA emulation ecosystem has excellent test ROMs (jsmolka, ARMWrestler, mgba-suite) and excellent reference emulators (NanoBoyAdvance, mGBA). What it lacks is the glue — the automated pipeline that connects test ROMs to reference hashes to pass/fail results. Every emulator author builds their own ad-hoc testing workflow.
gba-accuracy-tests is an attempt to fill that gap. It is early — two runners, nineteen tests, references from a single gold-standard emulator. But the format is extensible, the tooling is language-agnostic, and the repository is public for anyone building a GBA emulator who wants to know if their output is correct.