Survey of RL for LRMs

1. Preliminary Definitions of RL Modeling for LRMs
- 1.1. Reinforcement Learning
- 1.2. Frontier Models
2. Foundational Components of RL for LRMs
3. Problems in RL for LRMs
4. Training Resources for RL
5. Application of RL
6. Future Directions in RL for Language Models

1. Preliminary Definitions of RL Modeling for LRMs

We quickly go through the basic components of RL, and describe how language models can be adapted into the RL framework.

1.1. Reinforcement Learning

RL is a general framework for sequential decision making, in which an agent interacts with an environment by taking actions to maximize cumulative reward. The main components include state space \(\mathcal{S}\), action space \(\mathcal{A}\), transition function \( \mathcal{P}:\mathcal{S}\times\mathcal{A}\mapsto\mathcal{S} \), reward function \( R:\mathcal{S}\times\mathcal{A}\mapsto \mathbb{R} \), and a discounting factor \( \gamma \in [0,1] \).

At each step, the agent observes a state \( s_{t} \), selects an action \( a_{t} \) based on policy \( \pi_{\theta} \) parameterized by \( \theta \), transits to a new state \( s_{t+1} \) and receives a reward \( r_{t} \).

When mapping to the context of language models:

Prompt \(x\) can be mapped to the initial world state or env context, drawn from a data distribution and corresponding to the dataset \( \mathcal{D} \).
Policy \( \pi_{\theta} \) represents the language model itself, which generates a sequence of length \(T\) denoting as \( y=(y_{1}, y_{2}, \dots, y_{T}) \).
State \( s_{t} \) defined as prompt + tokens generated so far, i.e., \( s_{t}=(x,a_{1:t-1}) \)
Action \( a_{t} \) can be a single token (token-level), a segment of tokens (step-level), or the entire generated sequence (sequence-level), dependent on granularity.
Transition Dynamics \(\mathcal{P}\) is usually deterministic in the context of LLMs since \( s_{t+1} = [s_{t}, a_{t}] \)
Reward \( R(x,y),r_{t} \) is assigned based on action granularity.
Return \(G\) is the cumulative reward of the whole trajectory \(y\) for prompt \(x\)

In this setting, the learning objective is to maximize the expected cumulative reward over that data distribution \(\mathcal{D}\), i.e.,

\[ \theta^{\ast}=\max_{\theta} \mathcal{J}(\theta):=\mathbb{E}_{x\sim\mathcal{D}, y\sim \pi_{\theta}(x)}[G] \]

In practice, we usually have a reference policy \(\pi_{ref}\) for model regularization. Often implemented as KL-divergence, to stablize modle policy training and maintain language quality (by avoiding model parameters from changing too much).

1.2. Frontier Models

There are 3 major directions: LRMs, agentic LRMs and multimodal LRMs.

2. Foundational Components of RL for LRMs

2.1. Reward Design

2.1.1. Verifiable Rewards

Only tasks comparatively easy to verify benefit from outcome-based RL algorithms. Tasks lacking fast or objective verification remain challenging for outcome-based RL.

Rule-based Rewards
In rule-based rewards method, we usually have a programmatic verifier. They provide binary and checkable signals in domains with objectively verifiable outcomes, for example, coding and mathematics, thus improving models’ performance in these 2 fields through accuracy and format checks.
- Accuracy rewards. For tasks with deterministic outcomes, like math problems, we compare generated answer with reference answer. For coding tests, we may use unit tests or compiler signals.
- Format rewards impose structural constraint requiring the model to follow, like <think></think> chain of thought format.
Rule-based Verifiers

Usually rely on a large collection of predefined equivalent rules to check if the generated answer matches the ground truth.

2.1.2. Generative Rewards

Generative reward models (GenRMs) extend RL to subjective, non-verifiable domains by providing nuanced, text-based feedback. A dominant trend is training RMs to reason before judging, often using structured rubrics to guide evaluation or co-evolving them with the policy model in a unified RL loop.

Model-based Verifiers for Verifiable Tasks

Use model to evaluate model’s free-form output with reference answer. This technique is adopted in augmenting existing rule-based systems, as well as multi-domain veirifiers that handles diverse data types and reasoning tasks.
Generative Rewards for Non-Verifiable Tasks
AKA assessment-based GenRMs, which can be roughly categorized into:

“Reasoning reward models” (learning to think). Train the reward models to explicitly reason before judging. It could be done through prompting or train reward models with RL using simple and verifiable meta-rewards based on correctness of their final verdict.

“Rubric-based rewards” (structuring subjectivity). While rule-based methods rely on hardcoded logic for objective and verifiable tasks, rubric-based methods leverage natural language descriptions to capture criteria for subjective, non-verifiable domains where binary rules are insufficient. A common way is to use LLM to generate or follow a checklist of principles to guide its assessment. Another approach is to decompose high-level tasks into a set of verifiable proxy questions or generate domain specific principles. Moreover, rubric itself can be served as instructional scaffolds to guide policy exploration, and as criteria for the final reward, e.g. RLMT¹ uses model-rewarded thinking to improve reasoning and chat capabilities.

“Co-Evolving Systems” (Unifying Policy and Reward). Generator and verifier improve together.
- Self-Rewarding that a single model generates its own training signals. Either alternatively play roles as policy and verifier, or performs self-correction based on its own critique, or internalizes the reward function via post-completion learning.
- Co-Optimization where the policy and a separate reward model are trained concurrently.

2.1.3. Dense Rewards

Dense rewards provide fine-grained credit assignment and improve training efficiency and optimization stability in RL. However, scaling remains challenging for open-domain tasks due to difficulty in defining dense rewards or using verifiers.

2.1.4. Unsupervised Rewards

Unsupervised rewards eliminate human annotation bottleneck, making reward signal generation scalable. Main approaches either from model’s own processes (output consistency, internal confidence, self-generated knowledge), or from automated external sources (heuristics, data corpora).

2.1.5. Reward Shaping

2.2. Policy Optimization

2.2.1. Policy Gradient Objective

2.2.2. Critic-based Algorithms

2.2.3. Critic-Free Algorithms

2.2.4. Off-Policy Optimization

2.2.5. Regularization Objectives

2.3. Sampling Strategies

2.3.1. Dynamic and Structured Sampling

2.3.2. Sampling Hyper Parameters

3. Problems in RL for LRMs

3.1. Role of RL

3.2. RL vs. SFT

3.3. Model Priors

3.4. Training Recipes

3.5. Reward Definitions

4. Training Resources for RL

4.1. Static Corpus

4.2. Dynamic Environments

4.3. Training Infrastructure

5. Application of RL

6. Future Directions in RL for Language Models

Footnotes:

Yang Zhou, Sunzhu Li, Shunyu Liu, Wenkai Fang, Jiale Zhao, Jingwen Yang, Jianwei Lv, Kongcheng Zhang, Yihe Zhou, Hengtong Lu, et al. Breaking the exploration bottleneck: Rubric-scaffolded reinforcement learning for general llm reasoning. arXiv preprint arXiv:2508.16949, 2025f.