OpenAI o1: Reasoning Breakthrough
OpenAI releases o1 model, demonstrating powerful reasoning and problem-solving capabilities.
Overview
In September 2024, OpenAI released the o1 model series (formerly codenamed "Strawberry"). o1 models use "chain-of-thought" technology, performing deep reasoning before answering, able to solve complex math, science, and coding problems. o1 achieved 83% accuracy on AMC math competitions and surpassed average human expert level on PhD-level science questions. o1's release marked the evolution of LLMs from "fast intuition" to "careful deliberation," opening a new direction for reasoning enhancement.
Study Notes
Background, mechanics, impact, and key takeaways in one read
Background & Context
OpenAI o1 was released on September 12, 2024 in the blog "Learning to reason with LLMs." It marked a shift from "answer in one forward pass" to "think, then answer." The backdrop: models like GPT-4o still struggled on competition math, coding, and PhD-level science. o1 uses large-scale reinforcement learning to teach the model to "use its chain of thought productively"——generating a long internal reasoning before answering, and improving as it thinks longer. o1-preview and o1-mini opened the same day; the full o1 shipped on December 5, 2024. It far surpassed GPT-4o on AIME, Codeforces, and GPQA, opening a new axis of "test-time compute" scaling.
Key Points
- 1It is the first model explicitly positioned as a "reasoning model": it generates a long internal chain of thought before answering.
- 2Large-scale RL teaches the model to use its chain of thought well; the reward is answer correctness, not human preference.
- 3Its performance keeps rising with both train-time and test-time compute, opening a new axis of reasoning-time scaling.
- 4On AIME 2024 it scored 74% single-sample and 83% by 64-sample consensus, far above GPT-4o's ~12%.
- 5On Codeforces it reached the 89th percentile; on GPQA diamond it first surpassed human PhD experts.
- 6Its chain of thought is hidden from users (only a summary shows), for safety, compliance, and anti-distillation.
How It Works
o1's core is not a new architecture (still a Transformer decoder) but a new training paradigm. In RL it is encouraged to produce a long chain of thought, then rewarded by whether the final answer is correct——unlike traditional RLHF, which uses human preference as the reward; o1's reward is the task outcome itself. Training lets the model learn to hesitate, reflect, correct, and switch approaches: it breaks hard problems into steps, backtracks when stuck, and retries differently. The key phenomenon is test-time scaling: giving the model more thinking tokens (test-time compute) smoothly raises accuracy, unlike pretraining scaling bounded by data and compute. The cost is higher latency and expense, and for safety and anti-distillation OpenAI hides the full chain of thought, showing only a brief summary.
Impact & Legacy
o1 proved that "letting the model think longer" yields gains rivaling or beating bigger parameters, establishing test-time compute as a new scaling axis continued by o3 and o4-mini. It redrew baselines in math, coding, and science reasoning and spawned ideas like "thinking budgets" and agentic reasoning (thinking while calling tools). The controversy: hidden chains of thought weaken interpretability and accountability, and long reasoning raises latency, cost, and energy. The field rethought the old "intelligence equals parameters" assumption toward co-optimizing training and inference.
Key Takeaways
- Reasoning can be elicited by RL that makes the model "think by itself," rewarded by answer correctness, not preference.
- Test-time compute is a second scaling axis after pretraining and deserves its own investment.
- The self-evolved chain of thought (reflection, backtracking) is key to hard problem solving.
- There is tension between capability and interpretability: hiding the chain aids safety but costs transparency.
Glossary
Allocating more "thinking tokens" to a model at inference to boost performance, a new scaling dimension alongside training-time compute.
The model's step-by-step intermediate reasoning written before the final answer, helping solve hard problems.
A large model trained with RL to explicitly generate a long internal reasoning before answering, such as OpenAI o1.