BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning

Does pure RL with verifiable rewards teach models physics, or just clever pattern-matching?

We explored this question in our new paper by training a compact 1.5B reasoning model on beam statics (a classic structural engineering problem). We used parameter-efficient RLVR with strict, binary correctness rewards from symbolic solvers—meaning zero teacher-generated reasoning traces were used.

Here is what we found:

The good: The best BeamPERL checkpoint achieved a massive 66.7% improvement in Pass@1 over the base model.

The catch: The learned reasoning is highly anisotropic. The model generalizes beautifully to compositional changes (like adding more loads to the beam), but completely breaks down under topological shifts (like moving the locations of the supports) - even though the underlying equilibrium equations remain exactly the same.

Insights into training dynamics: Intermediate checkpoints actually yielded the strongest reasoning. Continued optimization degraded robustness while maintaining the reward, leaning into reward-hacking.

Key takeaway: Outcome-level alignment with exact physics rewards seems to induce procedural solution templates, not a true internalization of the governing equations. The precision of a reward signal - even when analytically perfect - does not by itself guarantee transferable physical reasoning.

To move beyond template-matching toward robust scientific comprehension, verifiable rewards likely need to be paired with structured reasoning scaffolding.

We are looking forward to discussing with the community.