OraRL
arXiv 2026 · Submitted

Annotations as Rollouts

Efficient and scalable reinforcement learning for unified video MLLMs. OraRL turns every annotation into a reliable positive rollout while preserving on-policy exploration.

Yunheng Li1 Guohong Mu1 Hao Li2 Shengsheng Qian3 Dingwen Zhang2 Qibin Hou1,4† Ming-Ming Cheng1,4
1Nankai University 2Northwestern Polytechnical University 3Institute of Automation, CAS 4NKIARI
arXiv submitted Code Data Models
Unified video perception 7 task families
Video segmentation example with a highlighted mask from the paper Segmentation
Temporal grounding example with a highlighted interval from the paper Temporal grounding
Spatial grounding example with a highlighted bounding box from the paper Spatial grounding
Visual tracking example with target boxes from the paper Visual tracking
Spatial intelligence example with a highlighted metric relation from the paper Spatial intelligence
Video-ORA-9B One model and one RL recipe across localization, tracking, segmentation, QA, and spatial reasoning.
7task families
100ktraining prompts
1.48×faster updates
36.8×less decoding time
00 · Video

From annotations to rollouts.

A visual overview of the motivation, method, results, and efficiency.

Project overview 01:38 · Silent · English captions
01 · Core idea

Use the annotation twice.

Existing video RL uses annotations only to score sampled responses. OraRL also serializes each annotation as an oracle rollout—a direct, task-independent positive target.

Paper figure comparing SFT, GRPO, mixed-policy training, and OraRL
Figure 2 · Training paradigms OraRL excludes ground truth from the on-policy baseline while retaining it as an optimization target.
01

Reliable positive anchor

Every training group receives one precise, task-valid positive rollout.

02

Preserved policy comparison

On-policy rewards alone define the baseline, preventing advantage inversion.

03

Exploration remains intact

The annotation supplements sampled behavior rather than replacing it.

02 · Method

Decouple comparison from guidance.

The policy group sets a clean reference. The oracle–policy gap then strengthens promising rollouts and supplies a separate detached anchor.

Overview of the OraRL training framework
Figure 3 · OraRL framework Annotation rollout, oracle-free baseline, gap-aware guidance, and sign-balanced selection in one update.
01

Annotation rollout

Serialize intervals, boxes, masks, trajectories, or answers into the model’s native response format.

02

Oracle-free baseline

Normalize only on-policy rewards so rollouts keep their correct policy-relative signs.

03

Gap-aware guidance

Use the oracle–policy gap for a bounded directional gain and a detached oracle advantage.

04

Balanced pruning

Retain strong positive and negative signals with the oracle, then restore update moments.

03 · Results

One model, seven task families.

Video-ORA-9B improves over its backbone and every displayed baseline across unified video perception—without chain-of-thought decoding.

Video-ORA results across seven video perception task families
Figure 1 · Unified performance Representative examples and family-level scores over common benchmark coverage.
Temporal grounding 66.0

mIoU, up from the prior best of 62.5.

Visual tracking 78.2

Average overlap on GOT-10k.

Segmentation 70.4

Aggregate cIoU / J&F family score.

VSI-Bench 73.1

Best reported overall average.

04 · Efficiency

Less compute in training and inference.

Selective back-propagation shortens updates, while answer-only generation removes costly reasoning traces at inference time.

Paper figure comparing OraRL training cost and inference latency
Figure 7 · Efficiency OraRL gives the strongest training trade-off and avoids the decoding cost of chain-of-thought.
Training update 93.9 → 62.4 s / step

Higher temporal mIoU with substantially less update time than GRPO.

Answer decoding 4,780 → 130 ms

Answer-only decoding on ten-minute videos, without generated reasoning traces.

Citation

Annotations as Rollouts

The public arXiv identifier will be added after announcement.

@article{li2026orarl,
  title   = {Annotations as Rollouts: Efficient and Scalable
             Reinforcement Learning for Video MLLMs},
  author  = {Li, Yunheng and Mu, Guohong and Li, Hao and
             Qian, Shengsheng and Zhang, Dingwen and Hou, Qibin
             and Cheng, Ming-Ming},
  journal = {arXiv preprint},
  year    = {2026}
}