Outcome rewards · RLVR
Check only the final answer. A correct answer reached from a wrong world still earns full reward.
TL;DR Language models often fail physics because they reason inside the wrong physical world. RLAWM trains a model to build the right world and to reason inside it, by probing its hidden belief from two sides during reinforcement learning. Nothing extra runs at inference.
+12.2points on PhysReason over the strongest baseline
+5.2points on PHYSICS over the strongest baseline
2.45×on olympiad-level physics, zero-shot
+10.0points on AIME 2026: physics training transfers to math
1 · Motivation
The failure is in the world, not the algebra
Research-level physics benchmarks show a recurring pattern. Models silently change a given, invent a law, or add an assumption that turns a hard problem into a familiar one, and then reason flawlessly inside that corrupted world. We call this modeling drift.
-
1
Modeling error
Wrong geometry: the model assumes \(h_B - h_A = R\), although the problem says \(AB\) is horizontal.
-
2
Reasoning in the wrong world
It writes the energy equation from that geometry and obtains the impossible \(v_O^2 < 0\).
-
3
Drift for self-consistency
Instead of fixing the geometry, it rewrites the problem and declares \(E \leftarrow 0\).
Modeling drift is the divergence between a model’s inferred axiomatic world model (AWM) and the true physical world specified in the problem. Unlike factual hallucination, it happens at the level of premises: the model reasons coherently inside a wrong physics world.
Paper Figure 1 · click to enlarge
Why current rewards miss it
Step-level rewards · PRMs, LLM judges, rubrics
Score whether each step looks plausible or locally correct, relative to the premises the model has already adopted.
What none of them checks
Whether the model is reasoning in the right world at all. RLAWM rewards that world directly.
The opening
In physics, verifying a claim is easier than deriving it: checking a conservation law or a scaling relation is straightforward. But a reasoning trace is static text. It can be read, not probed.
Our idea
Turn the model’s belief into something that can be compared with the true world and executed inside it. Every probe becomes a training signal on the belief itself.
2 · Method
Reaching a belief you cannot see
The axiomatic world model is the policy’s belief about a problem’s physical setup. It lives in weights and hidden states, so no reward can read or edit it directly. RLAWM reaches it from two sides, and trains on what it finds: first learn the right world, then learn to reason inside it.
Stage I · passive probe
Make the belief speak
The policy writes its world down as a structured [modeling] block. We read it and compare it, motif by motif, with the problem’s true axiom set.
→ modeling reward \(R_M\): is this the right world?
Stage II · active probe
Make the belief act
The policy’s reasoning is compiled into a program and dropped into an executable model of the true world, built in code for each problem. We run it, perturb it, and check it against physical law.
→ solving reward \(R_S\): does it reason correctly in that world?
I
Stage I · modeling reward \(R_M\)
Learn the right world
Make the belief speak
Before solving, the policy writes the world it believes in as a [modeling] block with five fields. Hover a field to see what it pins down. Switch to the drifted belief of Figure 1: the error is now visible in writing, where a reward can reach it.
Blocks are illustrative, written in the paper’s schema (Appendix I) for the problem above. The drifted block is reconstructed from Figure 1.
Compare the believed world with the true one · PMF
Physics Modeling Fidelity never asks an LLM to judge physics. A frozen LLM only extracts tuples from both blocks under one schema; a deterministic rule matches them and computes F1. Three motif types catch different failures, and \(R_M = \tfrac13\,(\mathrm{F1}_{\text{QEL}} + \mathrm{F1}_{\text{VCL}} + \mathrm{F1}_{\text{AEB}})\).
[modeling] text→frozen extractorschema-bounded→motif tuples→rule match+ soft matcher→F1 per motif
Inject an error:
\(R_M\)—
Motif sets are illustrative, in the PMF schema of Appendices J–K. The expected effect of each error type follows Table 13 of the paper.
II
Stage II · solving reward \(R_S\)
Reason correctly in that world
Build the true world in code
For every training problem, a checker is synthesized once, before training, and then frozen. It binds the problem’s entities to a shared library of 147 reusable physics-law templates. The templates hold general laws and their applicability conditions, not answers.
- 1Extract entities
- 2Bind law templates
- 3Emit assertions
- 4Validate in a sandbox
Problem entities
Template library 147 laws extracted from PhysReason solutions · examples
PhysicsChecker generated for this problem
1,080 checkers for the PhysReason training set
$5.35 total API cost, about half a cent each
0 manual authoring
Drop the belief into that world and probe it
The policy writes its derivation as pseudocode. A frozen translator (Qwen2.5-Coder-1.5B) compiles it into a program; if the program does not run, \(R_S = 0\). The checker then runs it and fires three kinds of probes. Pick a rollout from Appendix F and watch.
PhysicsChecker
Executable world for this problem
true world’s response
rollout’s program
law templates bound to this problem
Both rollouts state the same premises; only one carries them through.
Invariants
Do the core physical relations hold?
work done by the world—
rollout’s ΔKE—
Trace consistency
Do its own quantities agree with each other?
Response under perturbation
Change an input, re-run the same program: does it respond like the true world?
Extra probe for illustration: the checker of Appendix M perturbs only \(R\), so this one is not scored.
invariants–
trace–
perturbation–
solving reward only\(R_S = \tfrac{\#\,\text{passed tests}}{\#\,\text{tests}}\)—enters the total reward \(R = 0.4R_M + 0.4R_S + 0.2R_A\)
Scores follow the illustrative checker of Appendix M, which has one test per category. Real checkers run 7.34 assertions per rollout on average.
↻
Training · \(R = 0.4R_M + 0.4R_S + 0.2R_A\)
Close the loop
Update the belief
Training is an SFT cold start that teaches the [modeling] / [solving] format, followed by GRPO on the joint reward. GRPO learns only from reward differences inside a group of 8 rollouts. Take a hard problem where every rollout gets the final answer wrong.
The eight rollouts are illustrative. The measured numbers are from Table 4 and Appendix N.
Internalize the laws
The extractor, translator and checkers exist only during training. Because each checker is built from general law templates, optimizing \(R_S\) does not fit the policy to one checker; it pushes law-abiding execution into the policy itself. At inference the model answers alone, at the cost of ordinary chain-of-thought.
policywith its AWM
motif extractorgpt-4o-mini · PMF
translatorQwen2.5-Coder-1.5B
checkersone per problem
answerno extra cost
If the laws were internalized rather than the checkers memorized, the gains should persist where no checker ever existed: at large \(k\), on harder problems, on new sources, and even in math. The results test exactly that.
Glossary · how this page maps to the paper
In this pageIn the paper
The model’s belief worldThe policy’s internal AWM, written out as the five-field
[modeling] block \(\hat m\)The true worldThe problem’s axiom set, represented by the reference structure \(m^*\)
Make the belief speakPMF compares \(\hat m\) with \(m^*\) through QEL, VCL and AEB motifs, giving \(R_M\)
A world built in codeA per-problem PhysicsChecker \(g\), bound from 147 reusable physics-law templates
Make the belief actPseudocode \(c\), translated into an executable program \(\hat w = f(c)\), run in a sandbox, perturbed and checked, giving \(R_S\)
Update the beliefGRPO on \(R = 0.4R_M + 0.4R_S + 0.2R_A\) after an SFT cold start
Internalize the lawsNo checker at inference; the gains transfer across difficulty, source and domain
Paper Figure 2 · click to enlarge
3 · Results
Consistent gains that transfer
All methods share the same SFT cold start and GRPO setup; only the reward differs. Baselines: RLVR (final-answer reward), Direct-Likert and Reference-Likert (LLM judges, without and with the reference solution), and Rubrics as Rewards. SubPass@k is the share of sub-questions answered correctly, keeping the best of k samples.
Main results · SubPass@1 (%)
RLAWM has the best Overall score on both datasets for all three backbones.
- Up to +12.2 points on PhysReason (Qwen3-VL-4B) and +5.2 on PHYSICS (Qwen3-VL-8B) over the strongest baseline.
- The margin is largest on Easy and Medium problems and narrows on Difficult ones: when the model can already reason, modeling drift is the bottleneck.
- Gains vary by subdomain. The explicit coordinate frames of E&M and optics are easier to capture than the operator algebra of quantum and statistical mechanics.
Expanding the reasoning boundary · SubPass@k
Best of \(k\) samples on PhysReason.
- As \(k\) grows, RLVR and Reference-Likert fall below the untrained base model, the diversity collapse reported by Yue et al. (2025).
- RLAWM stays on top at every \(k\): rewarding the world, not one reasoning template, keeps exploration alive.
Transfer without a checker · zero-shot
Trained only on PhysReason. SubPass@1 (%).
- 2.45× on olympiad-level physics (Qwen3-VL-8B: 15.2 vs 6.2), and the gain holds at 4B (11.3 vs 4.5).
- +10.0 on AIME 2026 under the same Zero-RL protocol: the axiomatic core of physics carries over to math.
4 · Case studies
What each reward sees
Three rollouts from the paper. The first shows why both levels are needed; the other two mark where the checker’s judgment ends.
Appendix F · arc-and-field problem
Case 1 · timeline
[representation] type: timeline
S2: B → O normal force does no work;
gravity and electric force both do work
[sub2] W_g + W_E = ½ m v_O² − ½ m v_B²
W_g = −mgR ; W_E = 2mgR
⇒ v_O = √(3gR)
[sub3] a_x = g, a_y = 0 ⇒ y² = 6Rx
Case 2 · interaction_graph
[representation] type: interaction_graph
edges: field → ball: qE = √2 mg at 45°
rail → ball: normal force does no work
[sub2] F_Ey = mg cancels gravity
⇒ kinetic energy from B to O unchanged
⇒ v_O = v_B = √(gR)
[sub3] a_x = g, a_y = 0 ⇒ y² = 2Rx
\(R_M\) · modeling
Both identify gravity, the electric force and the non-working normal force. The declared worlds are nearly the same.
\(R_S\) · solving
Case 2 drops the work of the horizontal electric force. The work–energy invariant fails: \(\Delta KE = 0 \ne mgR\). Run it in M4 ↑
\(R_A\) · answer
Case 1: 3/3 sub-answers. Case 2: 1/3 (only \(E_p\)).
Takeaway. Valid premises do not guarantee valid execution, and a self-consistent trajectory (\(y^2 = 2Rx\) follows from its wrong \(v_O\)) is not a correct one. This is why RLAWM rewards both levels.
Appendix N · false negative · three-step gas cycle
Rollout all 3 sub-answers correct
[constraints] first law as dU = Q + W,
W = work done ON the gas
[sub1] W_1 = dU_1 = P_0V_0 (r^(γ−1) − 1)/(γ−1)
[sub2] W_2 = −P_0V_0 (r^γ − r^(γ−1))
Q_2 = dU_2 − W_2
[sub3] η = W_by / Q_2
Checker tests \(R_S = 0.56\)
first law · step 1
first law · step 2
isobaric work · trace
net work · perturbation
other tests ×5 ✓
The checker writes \(\Delta U = Q - W\) with \(W\) the work done by the gas. Every test that reads a work term sees the opposite sign.
Takeaway. The law is the same on both sides; only its written form differs. Tests that fix a convention the law leaves open under-reward valid physics. This motivates convention-invariant tests and adaptive probes.
Appendix N · false positive · series RLC resonance
Rollout sub-answer 3 wrong
[sub3] half power: X = ±R
w_plus = +R/(2L) + √(R²/(4L²) + 1/(LC))
w_minus = −R/(2L) + √(R²/(4L²) + 1/(LC))
dw = w_plus − w_minus → emit dw ✓ (= R/L)
[subans 3] Δω = R/(2L) ✗
\(R_S\) · solving
Every emitted quantity is correct, including dw. The checker audits emitted values, not the written answer: \(R_S = 1.0\).
\(R_A\) · answer
The written closed form \(R/(2L)\) is wrong (it should be \(R/L\)). The answer reward catches it.
Takeaway. What a rollout writes after its last emit is invisible to the checker. This is the most common false-positive pattern, and the reason the joint reward keeps \(R_A\).
5 · Conclusion
Reward the world, not just the answer
RLAWM turns the verification–derivation asymmetry of axiomatic domains into a training signal. It rewards the world a model believes in and how it reasons inside that world. The gains hold across backbones, seeds and three kinds of distribution shift, with no checker at inference.
Cite
BibTeX
@inproceedings{yang2026axiomatic,
title = {Axiomatic World Modeling for Physics Reasoning},
author = {Yang, Xinye and Liu, Zhenyang and Wang, Yuxuan and Lei, Yuanyuan},
booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
year = {2026}
}