Oral at the Meta-Agents Workshop, NeurIPS 2026

RLVR from Within Learning to Ask to Elicit Intrinsic Rewards for Test-Time Reasoning

Kuang-Da Wang1,†, Kai-Jie Lin1,†, Guo-Xun Ko1, Yu-Heng Lin1, Yu Heng Hung1, Travis M. Bartley2, Shuoyang Ding2, Yu-Chiang Frank Wang2,3, Chao-Han Huck Yang2, Guan-Ting Liu2, Wen-Chih Peng1, Ping-Chun Hsieh1

1National Yang Ming Chiao Tung University   2NVIDIA   3National Taiwan University   †Equal contribution

The answer's in the question. AskR learns to ask the right one.

A process reward model has to learn, from final outcomes alone, how much each intermediate thought is worth. AskR sidesteps that credit-assignment problem. It learns which sub-questions to ask so that a frozen reasoner's own yes/no checks on each answer become predictive of final correctness, and then lets a test-time optimizer use those checks as its reward.

Bring RLVR's signal inside the reasoner

RLVR improves reasoning by rewarding verifiable final answers, but only during training, and only by updating the backbone. At test time there is no verifier. AskR keeps the backbone frozen and turns its self-verification into a dense, internal reward over sub-questions and sub-answers (SubQAs).

Left: RLVR trains the backbone with an external, sparse reward on full trajectories. Right: AskR keeps the backbone frozen and scores multiple SubQAs with an internal, dense reward that a test-time optimizer maximizes.
From external, sparse rewards to internal, dense ones. Training-time RLVR needs a trainable backbone and an external verifier. AskR needs neither at test time: the frozen reasoner verifies its own SubQAs, and the optimizer maximizes that reward.
Left: a process reward model assigns values to thoughts and suffers from credit assignment. Right: AskR steers the reasoner toward SubQA trajectories whose self-verification scores are high when the answer is correct.
Learning values versus learning what to ask. A PRM must assign credit to each thought from the final answer alone. AskR instead learns which states to expose, steering toward SubQA trajectories where high self-verification goes with a correct answer.

Watch AskR ask

AskR writes the sub-questions. The frozen reasoner answers them in one pass and checks its own answers, and a test-time optimizer keeps the trajectory with the highest intrinsic reward.

A white truck behind trees and a guardrail; a small tissue box sits on its dashboard.
Close-up of the dashboard: a small blue tissue box.

What is the color of the tissue box?

  • (a) gray
  • (b) white
  • (c) black
  • (d) blue

Test-time optimizer: candidate trajectories (illustrative scores)

  1. #1(b) white0.25
  2. #2(d) blue0.75
  3. #3(a) gray0.50
  4. #4(d) blue1.00
  5. #5(c) black0.25

Press play, or step through with the arrows, to watch AskR plan the sub-questions and the reasoner answer them.

AskR asks
Reasoner answers every sub-question and the problem in one pass
  1. Q1What objects/features are visible in the image? ✓ verified
    A white truck, a tree, some vegetation, a guardrail, a tissue box.
  2. Q2Can you identify any objects that resemble a tissue box? ✓ verified
    Yes, there is a small, rectangular object on the dashboard of the truck that resembles a tissue box.
  3. Q3If a tissue box is present, what is its color based on the visible features? ✓ verified
    The tissue box appears to be a light purple or lavender color.
  4. Q4Considering the options provided, which color best matches the tissue box's appearance? ✓ verified
    The color that best matches the tissue box's appearance is blue.

Final answer: (d) blue

Intrinsic reward \(r_\phi = (1+1+1+1)/4 = 1.00\)

Highest reward wins: option D (blue). Correct.

How AskR works

Two models with separate jobs. The asker \(g_\phi\) only writes sub-questions and is the only thing trained. The reasoner \(\pi_\theta\) answers, verifies and stays frozen throughout.

Training: AskR learns what to ask Test time: the intrinsic reward picks the reasoning

Problem Answer key No answer key
AskRwrites sub-questions
trained
Reasoneranswers and self-verifies
frozen

One SubQA trajectory

  1. q1a1✓
  2. q2a2✓
  3. q3a3✗
  4. q4a4✓

Final answer B

Sampled trajectories

  1. τ1A0.50
  2. τ2B0.75
  3. τ3B1.00
  4. τ4C0.25

Intrinsic reward

3 of 4 checks pass: r = 0.75

Signed by the final answer

B matches the answer key

−10+1

Training reward +0.75. A wrong answer would get −0.75.

Test-time optimizer

Ranks trajectories by intrinsic reward and keeps the best, for example with Best-of-N, TPO or TMPC.

Answer B, from τ3

The training reward updates AskR only. The reasoner stays frozen.

Training
Test time

  1. Ask

    The asker writes up to ten sub-questions. The frozen reasoner answers each one in turn.

    \[ Q=(q_1,\dots,q_K)\sim g_\phi(\cdot\mid x) \]
  2. Verify

    The same reasoner judges each SubQA as correct or not. The intrinsic reward is the fraction judged correct.

    \[ r_\phi(x,\tau)=(v_1+\dots+v_K)/K \]
  3. Learn what to ask

    GRPO trains the asker on the intrinsic reward, signed by final-answer correctness. No sub-question labels.

    \[ R_\phi=(2V(\hat y,y)-1)\,r_\phi(x,\tau) \]

Why sign the reward?

Maximizing self-verification alone would teach the asker to pose easy questions. Signing it by correctness rewards high scores only when they come with a right answer, so a high score becomes evidence of correctness. Try it.

−10+1

Quantitative results

Same reasoner, same optimizer, same search budget: swapping a trained PRM for AskR's intrinsic reward helps under every optimizer, on text and on vision.

Accuracy of PRM and AskR rewards under BoN, TPO and TMPC
PRM reward AskR reward RLVR (trained backbone) CoT

One asker, many reasoners

We train one AskR from Qwen3-4B and reuse it, unchanged, to steer Qwen3 reasoners from 1.7B to 32B (average of MMLU-Pro, LAB-Bench and SciEval, with TMPC).

It beats chain-of-thought at every scale: +9.96 points at 1.7B and still +4.70 at 32B, a reasoner eight times its size. The untrained asker gains at most +1.69, so the improvement comes from what AskR learned to ask.

Average accuracy versus Qwen3 reasoner scale from 1.7B to 32B. Trained AskR is highest at every scale; untrained AskR tracks chain-of-thought.

Across model families

Trained from Qwen3-4B or Gemma3-4B, AskR with TMPC beats both chain-of-thought and RLVR on average, on text and on vision.

Accuracy (%) with Qwen3-4B and Gemma3-4B backbones.
Backbone / methodMMLU-ProLAB-BenchSciEvalAvg.V* BenchMMStarCV-BenchAvg.

Training makes the reward predictive

A reward is only useful if it tracks correctness. Before training, the reasoner's self-verification barely predicts whether the final answer is right (AUROC 0.55 and 0.60, against 0.5 for chance). After training, the same frozen reasoner's checks become clearly predictive, on both backbones.

Training on correctness alone does not get there: replacing the signed reward with answer correctness drops the TMPC average from 57.17 to 53.69 on Qwen2.5-7B and from 66.57 to 61.85 on Qwen3-4B.

Pearson, Spearman and AUROC between intrinsic reward and correctness, before and after AskR training, for Qwen2.5-7B and Qwen3-4B. All rise after training.

What the reward buys

Grid comparing AskR Pass@1 within intrinsic-reward buckets against CoT Pass@N for N from 1 to 8.
One high-reward answer versus many samples. On MMLU-Pro, AskR's single answer in the top reward bucket (63.5) beats CoT given eight tries (61.6).
Density of intrinsic reward for direct sampling, BoN, TPO and TMPC. TMPC concentrates most mass near a reward of 1.
Optimizers climb the reward. Distribution of intrinsic reward on MMLU-Pro for direct sampling and three optimizers. TMPC moves the most mass toward a reward of 1.