Model Summary

This model enhances the reasoning capabilities of the small 1.5B parameter DeepSeek-R1-Distill-Qwen-1.5B LLM using reinforcement learning (RL). Trained efficiently on 4 A40 GPUs in under 24 hours, it achieves significant gains in mathematical reasoning benchmarks (e.g., 80% accuracy on AMC23, 46.7% on AIME24, surpassing o1-preview). This cost-effective approach demonstrates the potential of RL for boosting reasoning in resource-constrained settings.

Evaluation

Performance Highlights

Open-RS1: 53.0% avg. score
Open-RS2: 55.7% avg. score, 80.0% on AMC23
Open-RS3: 56.3% avg. score, 46.7% on AIME24 (outperforms o1-preview at 44.6%)
Competitive MATH-500 scores; Minerva lags behind 7B models.

Cost Efficiency

Our approach uses 7,000 samples (42,000 total outputs) and costs ~$42 on 4x A40 GPUs in 24 hours, compared to thousands of dollars for baseline models.

Citation

If this project aids your work, please cite it as:

@inproceedings{
    dang2026reinforcement,
    title={Reinforcement Learning for Reasoning in Small {LLM}s: What Works and What Doesn{\textquoteright}t},
    author={Quy-Anh Dang and Chris Ngo},
    booktitle={Logical and Symbolic Reasoning in Language Models @ AAAI 2026},
    year={2026},
    url={https://openreview.net/forum?id=3pWL6Zxc4A}
}

For more details, including usage instructions and further evaluation results, please refer to our GitHub repository.