catching up on large language models: from conventional LLMs to reasoning models
9 min read
·…
tl;dr:Notes on how reinforcement learning and verifiable rewards reshaped LLM reasoning.
Development on Brevia has recently reached a pause. Wanting something worthwhile to work on at home, I first looked into sparse autoencoders and realized how out of date parts of my understanding of large language models had become. So I am refreshing that knowledge system through a small series of posts on important developments in LLMs since 2024.
My own mental model of LLMs was still rooted in 2024, when Mixtral 8×7B and Llama 3 had just arrived. I had done some SFT and RAG work; mixture-of-experts architectures were not yet widespread in LLMs. The dominant training pipeline was pre-training, then SFT, then RLHF, yielding a conversational model. The underlying idea was still the scaling law: vendors were in an arms race to increase parameter counts. Scaling laws no longer command quite the same attention, while reasoning and agents have become everyday concepts. Where I once had to wire together LangChain, rules, and hand-written tools for tool use and web search, an LLM’s agentic capabilities can now handle that work with ease.
This post looks at what I consider the most consequential breakthrough of the past two years: the reasoning model. I also recommend this excellent overview on Zhihu and this thoughtful analysis.
From Llama 3 to OpenAI o1
Back in 2024, the typical LLM training process began by pre-training a next-token-prediction model, then fine-tuning it on instruction-and-response pairs. RLHF had taken off, and reinforcement-learning post-training methods such as DPO and PPO were being used to steer models toward preferred outputs. Mainstream training paradigms held that a model’s capabilities were fixed when training ended.
For straightforward problems, those language models could still produce answers directly. Fine-tuning and post-training data covered simple arithmetic and logical deductions reasonably well. But as problems grew more complex and required multiple dependent steps, chain-of-thought emerged: after receiving an input, a model emits several rounds of reasoning tokens before using the final round to produce its answer.
Why does chain-of-thought work? The early research path went roughly like this. Researchers first noticed the value of intermediate steps: OpenAI’s 2021 paper introducing GSM8K created a math-reasoning dataset in a “question–steps–answer” format for fine-tuning GPT-3. The idea of intermediate steps had therefore appeared even earlier.
Researchers then found that irrelevant context or subjective details in a prompt can easily distract an LLM.
In 2022, Chain-of-Thought Prompting showed that adding an instruction such as “think step by step” to a prompt could substantially improve a model’s reasoning ability.
In 2023, OpenAI introduced self-consistency: sampling multiple reasoning paths and voting on the final answer can improve reasoning performance further.
Chain-of-thought delivered a clear improvement at the time. But because this apparent thinking was elicited by a prompt, a model might simply have learned from training data to generate text that resembles reasoning rather than genuinely following a process of noticing an error, backtracking, trying another method, and validating the answer.
A useful way to understand the limitation is that a language model is trained to maximize token likelihood, not the logical correctness of its reasoning process. A trajectory that is linguistically likely and resembles a sound derivation therefore does not guarantee a correct final answer.
In September 2024, OpenAI released o1 and argued publicly that large-scale reinforcement learning could teach models to use chain-of-thought more effectively. Performance improved with both training-time and test-time compute. In practice, this meant longer reasoning-token sequences, reasoning , search strategies, and verification and reranking strategies.
Previously, suppose the training set contained a question with correct answer . Under SFT, the target output for might be: “Because and , therefore .” The model directly optimizes that reasoning process, learning the path supplied by the teacher. It is plainly not learning to reason for itself; it is learning to imitate the form of reasoning.
Reinforcement learning is different. It gives the model no reasoning target—only a problem and an answer verifier. The model must explore reasoning paths on its own until it finds a correct answer. Different paths receive different rewards according to the verifier’s result, and the model learns better paths by optimizing that reward. Math and code problems have objective, verifiable answers, so they do not require human feedback or a reward model. This makes their reinforcement-learning paths the easiest to optimize. The approach is called RLVR (Reinforcement Learning with Verifiable Rewards). Unlike RLHF, which relies on human feedback and a reward model, RLVR only needs to determine whether the verifier’s result is correct.
o1’s most important breakthrough was demonstrating that large-scale RL can substantially improve how models use chain-of-thought, opening test-time compute as a new scaling dimension.
From DeepSeek-R1-Zero to DeepSeek-R1
DeepSeek-R1 is widely considered one of the most important papers of the past two years because it disclosed the full reasoning-RL route that OpenAI had kept under wraps for so long. R1-Zero even skipped SFT, applying large-scale reinforcement learning directly to a base model. It developed substantial reasoning ability, along with longer reasoning and reflection-like behavior, but had poor readability, mixed languages, and other presentation issues.
The emergence of this standard paradigm led to a wave of reasoning models. Previously, to teach a model to navigate a maze with SFT, one had to give it the canonical path. With RL, it only needs a reward for reaching the exit and zero reward for hitting a wall. In that process, it may learn better strategies—possibly strategies people would not have imagined and that would never appear in SFT data.
R1-Zero used two rule-based rewards. Accuracy rewards determine whether a final answer is correct—for example, with deterministic rules for mathematics and compilers plus test cases for code. Format rewards check that the model places its reasoning inside <think>...</think> tags as required.
Although the training process did not use data containing self-correction formats, researchers found that the model began producing reasoning traces in its answers. It developed ways to check itself, often with thought paths such as “wait…”. This was considered an important breakthrough. As training progressed, its thinking time gradually increased and more complex reasoning behavior began to emerge.
Subsequent work has suggested that the base model may already possess some reasoning ability, and that RL amplifies this ability, making self-checking behavior more likely to appear in its reasoning paths.
R1-Zero was a major breakthrough, but its reasoning was difficult for people to read: languages were mixed, formatting was irregular, and the expression was hard to follow. The subsequent DeepSeek-R1 therefore added SFT back on top of R1-Zero.
The full training pipeline has four stages: Stage 1, cold-start SFT; Stage 2, reasoning-oriented reinforcement learning; Stage 3, rejection sampling and supervised fine-tuning; and Stage 4, reinforcement learning for all scenarios.
Cold-start SFT teaches the model to present reasoning paths in a human-readable form. Reasoning RL teaches it better reasoning strategies. The final SFT and RL stages further teach it to present those paths readably.
In simple terms: pre-training gives a model capabilities; SFT specifies behavioral patterns, output formats, and style; RL lets it explore high-value strategies.
GRPO
The central RL method used in DeepSeek-R1 is GRPO (Group Relative Policy Optimization), which later drew enormous attention. Its advantage is a substantial reduction in training cost. PPO normally requires a policy model, a reference model, and a reward model, which is already expensive. GRPO instead groups a set of answers to a given problem by their quality, then updates the policy so that high-quality paths become more likely.
Reading the paper from beginning to end, I was struck by how little it relies on flashy terminology or formulas. It is full of strong engineering thinking: the problems are clearly defined, the reasoning flows naturally, and the implementation is solid. It is enjoyable to read because it explores and validates from observations, rather than drawing a target around an arrow already fired.
What a reasoning model really is
At its core, a reasoning model is still a next-token-prediction model. Its post-training changes the distribution of reasoning paths across problems, making high-value paths easier for the model to produce.
Computationally, what a model calls thinking is simply the generation of more intermediate tokens. As the context grows, reasoning tokens function as a scratchpad: they give the model more contextual information from which to produce a better result.
My understanding is that the breakthrough of reasoning models is chiefly enabled by math and code problems having unambiguous answers, which makes large-scale, inexpensive reasoning-RL training with RLVR possible. Subjective reasoning problems that are more closely tied to human values still need further research.
August 24, 2026, Suzhou