For OpenAI, o1 represents a step toward its broader goal of human-like artificial intelligence. More practically, it does a better job at writing code and solving multistep problems than previous models. But it’s also more expensive and slower to use than GPT-4o. OpenAI is calling this release of o1 a “preview” to emphasize how nascent it is.
The training behind o1 is fundamentally different from its predecessors, OpenAI’s research lead, Jerry Tworek, tells me, though the company is being vague about the exact details. He says o1 “has been trained using a completely new optimization algorithm and a new training dataset specifically tailored for it.”
OpenAI taught previous GPT models to mimic patterns from its training data. With o1, it trained the model to solve problems on its own using a technique known as reinforcement learning, which teaches the system through rewards and penalties. It then uses a “chain of thought” to process queries, similarly to how humans process problems by going through them step-by-step.
At the same time, o1 is not as capable as GPT-4o in a lot of areas. It doesn’t do as well on factual knowledge about the world. It also doesn’t have the ability to browse the web or process files and images. Still, the company believes it represents a brand-new class of capabilities. It was named o1 to indicate “resetting the counter back to 1.”
I think this is the most important part (emphasis mine):
As a result of this new training methodology, OpenAI says the model should be more accurate. “We have noticed that this model hallucinates less,” Tworek says. But the problem still persists. “We can’t say we solved hallucinations.”
So for those not familar with machine learning, which was the practical business use case for “AI” before LLMs took the world by storm, that is what they are describing as reinforcement learning. Both are valid terms for it.
It’s how you can make an AI that plays Mario Kart. You establish goals that grant points, stuff to avoid that loses points, and what actions it can take each “step”. Then you give it the first frame of a Mario Kart race, have it try literally every input it can put in that frame, then evaluate the change in points that results. You branch out from that collection of “frame 2s” and do the same thing again and again, checking more and more possible future states.
At some point you use certain rules to eliminate certain branches on this tree of potential future states, like discarding branches where it’s driving backwards. That way you can start opptimizing towards the options at any given time that get the most points im the end. Keep the amount of options being evaluated to an amount you can push through your hardware.
Eventually you try enough things enough times that you can pretty consistently use the data you gathered to make the best choice on any given frame.
The jank comes from how the points are configured. Like AI for a delivery robot could prioritize jumping off balconies if it prioritizes speed over self preservation.
Some of these pitfalls are easy to create rules around for training. Others are far more subtle and difficult to work around.
Some people in the video game TAS community (custom building a frame by frame list of the inputs needed to beat a game as fast as possible, human limits be damned) are already using this in limited capacities to automate testing approaches to particularly challenging sections of gameplay.
So it ends up coming down to complexity. Making an AI to play Pacman is relatively simple. There are only 4 options every step, the direction the joystick is held. So you have 4n states to keep track of, where n is the number of steps forward you want to look.
Trying to do that with language, and arguing that you can get reliable results with any kind of consistency, is blowing smoke. They can’t even clearly state what outcomes they are optimizing for with their “reward” function. God only knows what edge cases they’ve overlooked.
My complete out of my ass guess is that they did some analysis on response to previous gpt output, tried to distinguish between positive and negative responses (or at least distinguish against responses indicating that it was incorrect). They then used that as some sort of positive/negative points heuristic.
People have been speculating for a while that you could do that, crank up the “randomness”, have it generate multiple responses behind the scenes and then pit those “pre-responses” against each other and use that criteria to choose the best option of the “pre-responses”. They could even A/B test the responses over multiple users, and use the user responses as further “positive/negative points” reinforcement to feed back into it in a giant loop.
Again, completely pulled from my ass. Take with a boulder of salt.
You’re under arrest. That’s ass-salt.
Fuck you, that made me smile. And I haven’t even had my coffee yet.
Sorry for hitting you at a vulnerable time.
To be a little nitpicky most of the AI that can play Mario kart are trained not with a reinforcement learning algorithm, but woth a genetic algorithm, which is a sort of different thing.
Reinforcement learning is rather like how you teach a child. Show them a bunch of good stuff, and show them a bunch of bad stuff, and tell them which is the good stuff and which is the bad stuff.
Genetic algorithms are where you just leave it alone, simulate the evolutionary process on an accelerated time scale, and let normal evolutionary processes take over. Much easier, and less processor intensive, plus you don’t need huge corpuses of data. But it takes ages, and it also sometimes results in weird behaviors because evolution finds a solution you never thought of, or it finds a solution to a different problem to the one you were trying to get it to find a solution to.
Those outcomes seem especially beneficial.
Is this process something that distributed computing could be leveraged for, akin to SETI@home?
I work in computer science but not really anything to do with AI so I’m only adjacently knowledgeable about it. But my understanding is unfortunately, no not really. The problem would be that if you run a bunch of evolutions in parallel you just get a bunch of independent AIs, all with slightly different parameters but they’re incapable of working together because they weren’t evolved to work together, they were evolved independently.
In theory you could come up with some kind of file format that allowed for the transfer of AI between each cluster, but you’d probably spend as much time transferring AI as you saved by having multiple iterations run at the same time. It’s n^n problem, where n is the number of AIs you have.
Genetic algorithms is a sort of broad category and there’s certainly ways you could federate and parallelize. I think autoML basically applies this within the ML space (multiple trainings explore a solution topology and convergence progress is compared between epochs, with low performers dropping out). Keep in mind, you can also use a genetic algorithm to learn how to explore an old fashioned state tree.