How AI Researchers Are Making LLM Training More Efficient With Difficulty-Targeted Data Selection

Reinforcement learning (RL) has become a go-to method for fine-tuning large language models (LLMs), especially when it comes to boosting their reasoning skills. But there’s a catch: RL fine-tuning is notoriously resource-intensive, and until now, researchers have largely ignored the problem of data efficiency. A new paper from researchers at UIUC, NYU, UT Austin, and Microsoft introduces two techniques that could change that—difficulty-targeted online data selection and rollout replay—cutting training time by up to 65% while maintaining performance.

The Problem: RL Fine-Tuning Is Expensive

Training LLMs with RL isn’t just slow; it’s expensive. For example, fine-tuning a relatively small 1.5B-parameter model on just 40,000 samples can cost over 3,800 A100 GPU hours—roughly $4,500 in compute costs—before even scaling to larger models. The issue? Traditional RL fine-tuning methods waste time on uninformative data—questions that are either too easy (where the model always gets it right) or too hard (where it always fails).

The Solution: Smarter Data Selection and Replay

The team’s approach tackles inefficiency in two ways:

1. Difficulty-Targeted Online Data Selection (DOTS)

• Instead of randomly sampling training questions, DOTS prioritizes questions of moderate difficulty—those where the model has a roughly 50% chance of success.
• To avoid the computational cost of evaluating every question, the researchers developed an attention-based framework that predicts difficulty by comparing new questions to a small reference set.

1. Rollout Replay (RR)

• Rather than generating fresh rollouts (model responses) for every training step, RR reuses recent rollouts stored in a buffer, reducing per-step computation by 11–13%.
• A modified GRPO loss ensures stability, preventing performance degradation despite the off-policy updates.

Results: Faster Training, Same Performance

Across six LLM-dataset combinations, the method reduced total training time by 25–65% while matching the performance of standard GRPO training. Key findings:

• Convergence Speed: DOTS alone reduced the number of training steps needed by 13–60%.
• Per-Step Cost: RR slashed wall-clock time per step by reusing rollouts.
• Effective Questions: DOTS selected 25% more “effective” questions (those providing non-zero gradients) than random sampling.

Why It Matters

This work shifts the focus from algorithmic improvements to data-centric optimizations in RL fine-tuning. By making better use of training data, it could lower the barrier to deploying RL-tuned LLMs in business applications—where efficiency directly translates to cost savings.

For AI teams fine-tuning models for reasoning tasks (e.g., math, code generation, or structured QA), these techniques offer a way to train faster without sacrificing quality. And with rollout generation often consuming half of total step time, the savings could be even greater for longer-context models.

The Bottom Line

RL fine-tuning doesn’t have to be a compute black hole. By focusing on data efficiency, this research opens the door to more scalable—and affordable—LLM training pipelines.