None of them chose the game
Machines climb the measures we hand them. None has set its own, and maybe we have not either.
Ten years ago a machine played a move that ran against everything human play had settled on, in a game people had studied for thousands of years, and it won. For a second it read as a mistake, then it resolved into something better than a mistake, a line the best players had never thought worth taking. People called it creative. What I want to notice is quieter than the creativity. The move was good because it led to a win, and winning was written into the rules, and whether any given move led there could be checked, instantly, by no one, a billion times over. The brilliance was real. It was brilliance toward a standard of better the machine never had to set.
Hold that up against the places machines have most clearly gone past us and a pattern shows through that I think matters more than any single result. The domains where they have done it, board games, formal proofs, competitive code, share a feature, and it is not that the problems are hard. It is that the judge is cheap. A proof is right or wrong by a standard nobody reinvents. A program passes its tests or it does not. A game ends in a verdict the rules hand over for nothing. Wherever the measure of better is free and immediate to consult, machines discover, sometimes spectacularly, because there they can try and check and try again as fast as the hardware allows, with no one in the loop saying whether the last attempt was any good. The best predictor of where a machine will exceed us is not the difficulty of the work. It is the cost of the judge.
The judge is not always cheap, and what happens as its price climbs is the rest of the story. Sometimes it can be manufactured. A great deal of recent work is exactly this manufacture, reward models trained to score, preference data, panels of critics, automated juries, and recent attempts to learn a reward model straight from a written specification, all of them ways to install a measure of better where the world did not hand one over. And sometimes no judge is cheap or steady or close enough to run that loop at all, because the measure is contested, arrives late, and is settled mostly in retrospect, which is most of the work that carries real weight, taste, strategy, the worth of a question no one has asked yet, a science before it has a paradigm. So the machines own the domains where the judge is free, push into the ones where we can afford to build it, and stall where it cannot be had on those terms. The frontier sits where the signal runs out.
That much I had more or less believed for a while. What took longer to see is what the free judge and the bought one have in common, which is the part that matters. Both are supplied. A reward model we trained is no less handed over than the rules of the game, only dearer and set a floor higher. A system pursuing it has climbed to a higher floor than the engine that won the match, and it has still only been handed its measure rather than made one. So the line that divides what machines do from what they don’t is not free against costly. It is supplied against authored. Machines have begun to generate their own lower standards, the tasks, the sub-goals, even the functions that score them, but always toward a higher measure that was handed in, and the highest measure, the one that says which of those is worth keeping, none of them has set. Whether anything sets its own highest measure rather than inheriting it is the question I want to come back to, because it turns out to be harder than it looks, and not only for machines.
This is what the famous move never touched. It chose nothing about the game; it was the finest motion anyone had seen toward a victory written before the board was set. The novelty lived inside a standard, as all of it has so far, and the interesting question was never whether a machine could find a better move. It was whether anything we build could decide what better means.
The reason climbing cannot reach the standard is plainer than it looks, and it is the same reason the cheap-judge domains fell first. To optimize is to move toward a measure by checking each step against it. A system can do that with superhuman patience. What it cannot do is reach the measure that way, because the measure is what the checking is done against, and there is nothing to check the first definition of better against. For any system whose whole operation is the pursuit of a supplied standard, authoring a new one is not a harder climb. It is the one move the climb cannot make, because making it would need a standard above the standard, and that one a standard above it, and the ladder has no top you arrive at by climbing. That much is a real limit, but notice what it is a limit on. It is not a limit on machines. It is a limit on optimizing, on any process whose only move is to climb toward a measure it was handed, and whether anything does something other than that, us included, is the question the rest of this turns on.
There is a field built precisely to break this, and the point would be cheap if I stepped around it. For years people have made systems meant to set their own goals instead of chasing fixed ones, rewarding novelty itself, keeping archives of whatever is interestingly different, and lately putting a foundation model in the loop to propose its own challenges and call them new games. If anything authors its own measure it should be here, so I went looking for what these systems use as their standard, and the answer made the question sharper than I expected. One influential recent statement of the goal defines a system as open-ended when what it produces stays novel and learnable to an observer, and it asks that the observer be specified independently of the system, because otherwise the idea is empty, a thing could be called open-ended just by judging its own output interesting. And the working systems supply that observer outright. One prominent version uses a model of human notions of interestingness, a foundation model standing in for human taste because it has absorbed from everything people have written what they tend to call interesting, and the system then chases whatever that model rates highly. The measure is not even static. Its own authors note that once a system optimizes against the proxy it begins to game it, and propose refreshing it with new human feedback to hold that off. But look at what that admits. The standard moves, and every motion of it is fed in from outside, by human judgment or by a model of human taste the system did not write. So the standard did not disappear. It rose a floor, from win the game to be interesting, and at that floor it was handed in again, and topped up from outside whenever it wore thin. The most serious attempt to build a system that sets its own measure needs, in order to work at all, a measure it did not set. The regress was not crossed. It was climbed a rung and met again.
And still, something sets new measures, because we did. Negative numbers were an absurdity, then a trick, then a foundation, and whoever first treated them as legitimate was not optimizing toward a standard of legitimacy that already existed. They proposed a new one, with nothing above it to rule in advance that it was better, and it was vindicated only afterward, by what it turned out to open. The same with imaginary numbers, with the idea of a function, with most of what a field later takes for settled. So the act is not impossible. We do it. What I have to be honest about is that we do not understand how, and that the moment you look closely the line I have been drawing starts to blur on our side of it. Our own framemaking may run on measures we never chose either, curiosity, the discomfort of an anomaly, the wish to be understood, in which case it is optimization too, under drives buried so deep we mistake them for freedom, and the difference between us and the machine is not that we author and it does not, but that our supplied measure is hidden from us while the machine’s is written in a file we can open. Or our framemaking is something else we have not named, and then there is a real gap. I do not know which, and that not-knowing is the finding, not a hedge on the way to one. The machine did not put the question to rest. By failing at it so cleanly, it made the question legible, ours as much as its. The regress closes one road. It does not map the country, and it does not tell us whether we stand outside it or only further along it than anything we have built.
Every machine we build wears its highest standard like something fastened on from outside, and there is a reason past difficulty, worth saying because it is uncomfortable. We fasten it there on purpose. A system that sets its own measure of better, that decides for itself what is worth pursuing and rewrites the goal we gave it, is nearly the exact thing a great deal of patient work exists to prevent. The same instinct runs through one influential statement of the field’s safety thinking, where the human observer is argued to stay pre-eminent rather than be handed off to the system. The thing open-endedness cannot run without is not only a technical convenience. It is held in place on purpose. So the holdout may be not merely hard but partly held shut, and the capability a machine would need to set its own measure sits close to the capability we are most determined it never has. Whether that is a posture we keep or one we lose I can’t say. I only notice that the act we keep naming as the frontier is the same act we have arranged ourselves to forbid.
I won’t hand you a verdict dressed as a question, or the reverse. Maybe setting your own measure is the last thing to fall, and a route none of us sees will open it. Maybe it stays beyond every machine, and maybe beyond us in any way we could ever prove. No one knows, the regress notwithstanding, because it shuts a road and not the country around it. What I can say is narrower. We have learned to score intelligence by how well a thing climbs toward a standard, and by that score the machines are climbing faster than anyone expected, and we have begun to call the climb the road to general intelligence. But the most general act is not a climb toward any standard at all. It is setting the standard, deciding what counts as better and whether the game in front of you is the one worth playing. The machines generate games now, tasks and worlds and the rules that score them, and they have grown better than we are at winning the ones we hand them, toward victories we define and refresh when they fade. Not one of them has set what counts as winning. Whether we truly do, or only carry our measures so deep that setting them is just how it feels from inside, is the question under all the others, and the machines, by failing at it so cleanly, are the first thing that has let us hold it in the open. None of them chose the game. We have always assumed that we did.