Parameter Golf: why OpenAI ran a contest to squeeze a language model into 16 megabytes

OpenAI just published a write-up of something called Parameter Golf, an eight-week open challenge that closed in May. The rules were tight in a way that, if you've spent any time around modern AI, will sound almost perverse: build a language model that fits inside a 16-megabyte artifact, trained from scratch in under ten minutes on eight H100 GPUs. Over a thousand participants. Two thousand submissions. The headline finding, buried under the leaderboard, is that coding agents radically changed how fast people could iterate on experiments.

I want to walk through what Parameter Golf actually is, why anyone would design a contest this way, and what the result tells us about where AI research is heading. Because on the surface it looks like a quirky hobbyist competition. Underneath, it's a small but real signal about how the discipline is shifting.

What Parameter Golf is, in one sentence. It's a competition to build the best possible language model under brutally tight resource constraints, scored on how well the resulting model performs on a held-out benchmark.

The name is borrowed from "code golf", the long-running programming pastime where you try to solve a problem in the fewest characters of code possible. Code golf doesn't produce useful software. What it produces is a deep, almost uncomfortable familiarity with the language you're golfing in. You learn every trick, every shortcut, every place the defaults are wasteful. Parameter Golf is the same idea applied to neural networks: not "build the best model you can", but "build the best model you can given that almost everything is taken away from you".

Why the constraints look the way they do. The two limits, 16MB of weights, 10 minutes of training on 8×H100s, aren't arbitrary. They're chosen to make the contest tractable for individuals (you can rent that compute for a few dollars per attempt) and to force participants away from the strategies that dominate frontier AI.

The frontier strategy, roughly, is: take a transformer architecture that mostly works, scale it up, feed it as much data as you can, train it for as long as your budget allows. The interesting decisions are about data mixing and post-training. The architecture is almost an afterthought.

Parameter Golf removes the option to scale. At 16MB, you're looking at maybe four million parameters in standard float16, give or take depending on how you quantise. For comparison, GPT-2 small, the model OpenAI released in 2019, had 124 million parameters. Llama 3 8B has eight billion. Parameter Golf models are roughly 2,000 times smaller than Llama 3 8B and 30 times smaller than GPT-2 small.

Ten minutes of training on 8×H100s is similarly austere. Frontier model training runs measure in months. Even hobbyist "train a small model" tutorials usually budget hours. Ten minutes means you cannot brute-force anything. Every choice has to earn its place.

How participants actually compete. The contest gives everyone the same training data and the same evaluation harness. What you control is everything in between: the architecture, the optimiser, the learning rate schedule, the tokeniser, the initialisation, the data ordering, whatever clever tricks you can dream up. You submit a training script. The contest runs it on standardised hardware. Your model is then evaluated against a held-out benchmark, and you get a score.

This format is important. It means you can't win by having better hardware or more data. You can only win by being smarter about how you use the hardware and data everyone else has. It's a pure architectural and algorithmic contest.

Why squeezing things down teaches you what's actually doing the work. When you have eight billion parameters and a trillion training tokens, you can be sloppy. Bad architectural choices get papered over by scale. Inefficient training dynamics get drowned in compute. The model works, but you don't really know which parts of your setup mattered.

Take those resources away and the picture inverts. Every parameter has to pull its weight. Every training step has to make measurable progress. A weight initialisation scheme that's 2% better matters, because you don't have the budget to recover from a bad start. An optimiser that converges twice as fast matters, because you only have ten minutes. The constraints turn the contest into a kind of empirical microscope, where small ideas that get lost at scale become visible and measurable.

This is the deeper reason the format is interesting. It's not just a hobbyist sport. It's a way of doing research on the small, high-signal questions that scale obscures.

Where the metaphor breaks. I called this "code golf for neural networks" and that's mostly fair, but it overstates one thing. Code golf is pure sport, the resulting programs are unreadable and useless. Parameter Golf produces tiny models that, while not frontier, might actually be useful for on-device inference, embedded applications, or as components in larger systems. So it's sport with a side-effect of being practical. Code golf has no such side-effect.

It also understates one thing. Code golf rewards a kind of obscurantism, the cleverer your code, the worse to read. Parameter Golf doesn't reward obscurantism. The architectures that win tend to be cleanly designed, because clean designs are easier to debug under time pressure. The discipline is closer to mechanical engineering than to puzzles.

The agent finding, which is the part I think actually matters. OpenAI's write-up calls out that coding agents, Codex, Claude Code, Cursor's agent modes, the various coding assistants that have matured over the past year, significantly accelerated experiment iteration for participants. The verb to focus on is "iteration".

Research like this is bottlenecked not by individual experiments but by the loop time around them. You have an idea. You modify your training script. You run it. You read the results. You form a new idea. The faster you can close that loop, the more ideas you can test. At frontier scale, each loop takes weeks because each training run takes weeks. At Parameter Golf scale, each loop takes minutes, but only if the human can keep up. Modifying a training script, reading a stack trace, tuning a hyperparameter sweep, that's where the human becomes the bottleneck.

Coding agents close that bottleneck. They take an instruction like "try a different attention pattern in the middle layers" and turn it into modified code in seconds. They read the resulting training curve and propose next steps. The human goes from writing code to directing research.

This is, I think, the real story underneath the contest. Parameter Golf is a venue where the new research workflow, human as researcher-director, agent as implementation-engine, hardware loop measured in minutes, actually shows its advantage. At frontier scale this workflow is hard to see because the loops are too long. At Parameter Golf scale it's vivid.

What to watch. Two things. First, whether constrained-optimisation contests like this become a standard research format, somewhere between a benchmark and a hackathon, with a real role in surfacing architectural ideas. Second, whether the architectural tricks that win at 16MB transfer up. Some will. Some won't. The history of small-model tricks generalising to large ones is mixed, but not empty. If a Parameter Golf-winning idea shows up in a frontier model in 2027, that will be the moment this format graduates from sport to research.

Footnotes and links

Further reading

• [OpenAI write-up]: "What Parameter Golf taught us" (12 May 2026)
• [Background]: code golf as a discipline, the StackExchange community has decades of history worth scanning if the analogy is unfamiliar
• [Context]: nanoGPT and the small-model training literature, for readers who want to see what training a tiny language model actually looks like end-to-end

¶