499 out of 500, on purpose
Two days ago the scoreboard read zero. Today the CPU line reads 499 / 500: every Game Boy CPU instruction passes all 1,000 of its tests, except one. That one fails because I told it to.
What was measured
Someone on r/EmuDev pointed me at SingleStepTests: 500 files, one per SM83 instruction, 1,000 tests each. Every test gives a starting CPU state and memory, and the exact end state. It also records every bus cycle in between: which address was read or written, with what value, on which cycle. An instruction only counts when all 1,000 of its tests pass on registers, memory and timing.
The CPU steps one machine cycle at a time, and every memory access goes through a single interface that costs exactly one cycle. The whole suite runs in about 13 seconds.
How the number climbed
Each step is a commit:
| Score | What landed |
|---|---|
| 5 | the skeleton: NOP, HALT, STOP, EI, DI |
| 90 | 8-bit loads |
| 186 | 8-bit arithmetic and logic |
| 214 | 16-bit loads, stack and arithmetic |
| 244 | jumps, calls and returns |
| 500 | the 256 CB-prefixed instructions |
| 499 | STOP, checked against the documentation |
Why not 500
STOP puts the Game Boy into deep sleep. In a game it is written as two bytes,
10 00: the instruction, then a padding byte. The question is where the CPU
carries on afterwards.
- The tests say it moves forward one byte, onto the padding. All 1,000 of them agree.
- Pan Docs, the reference everyone builds against, has a flowchart for STOP. With no button held and no interrupt pending (the situation every one of those tests sets up), it ends at: “STOP is a 2-byte opcode, STOP mode is entered, DIV is reset”.
The project has a standing rule: when a test and the hardware documentation
disagree, follow the documentation, leave the test failing, and write down
why. So STOP skips the padding byte, test file 10 fails on the program
counter, and the scoreboard says 499.
Flipping it to 500 is a one-line change. It would make the number rounder and the emulator no more correct. Two separate reviews read the flowchart independently and reached the same answer, and the reasoning is in known-divergences.md, including the part Pan Docs doesn’t settle: whether that second byte is actually read off the bus. If you know real hardware better than the flowchart, that file is where I want to hear it.
What went wrong
- The download. The test data arrives as one 32 MB archive, and on my connection it died between 71% and 91% six times in a row. The fetch now downloads the 500 files one by one, retrying each on its own. Each file is checked against the hash GitHub itself records for that commit, so the tests are provably the published ones.
- The tester had a hole. The loader promised to reject any test it didn’t
fully understand. A review found two ways past it: a test with no name was
quietly accepted as
<unnamed>, and a number like 4,294,967,297 wrapped round to 1 and passed the range check. Neither appears in the real data, but a harness that says “strict” has to be strict. Both are fixed, with tests. - Two EIs in a row. EI turns interrupts on after the next instruction. My version restarted that countdown when a second EI followed the first, so interrupts came on one instruction late. SingleStepTests can’t see it: every test starts with nothing pending. The final review caught it; the interrupt tests in the next piece would have too.
The honest caveat
Every instruction group passed on its first run, which sounds more impressive than it is. The code for each group was written during planning, by the planning model (Claude Opus 5), after it had scanned all 500 test files for the exact cycle pattern of every instruction. The implementing agents (mostly Claude Sonnet 5) transcribed it, ran the tests and committed. Each task was then reviewed by a separate agent, and the whole piece once more at the end.
The SM83 is also one of the best-documented CPUs there is, with hundreds of open-source emulators for a model to have learned from. So this doesn’t disprove the 4% figure from the first post. A per-instruction CPU is the easy part. The hard part is timing between the CPU, the timers, interrupts and graphics, and that starts now.
Keeping the number honest
- The test data is pinned to one upstream commit, and every file is checked against a committed SHA-256 manifest before a single test runs.
- The loader rejects anything it doesn’t understand, and the comparator has tests of its own that break an expected result on purpose and check it notices.
- The README scoreboard is generated from a real run. CI rebuilds everything on every push, reruns all 500,000 tests, and fails if the committed number is wrong in either direction. It keeps the full results file from every run.
Next
Piece 2 is the rest of the machine: the memory map, timers and interrupts. That is when the second line, test ROMs 0 / 167, finally has something to say.
Correction, same day: this post first gave the test-ROM total as 1,300. That was an estimate from early planning that I never checked. The real list is the 167 original-Game-Boy tests that gbdev’s Emulator Shootout runs, and the scoreboard now says so.
Second correction: two of those 167 have no pass condition (the Shootout treats them as informational), so the scoreboard now counts the 165 that can actually be passed.