README.md

---
language: en
license: apache-2.0
tags:
- reward-model
- preference-learning
- agentic-reasoning
- outcome-reward-model
- pairwise-preference
datasets:
- LossFunctionLover/orm-pairwise-preference-pairs
metrics:
- accuracy
pipeline_tag: text-classification
model-index:
- name: pairwise-orm-model
  results:
  - task:
      type: preference-learning
      name: Pairwise Preference Ranking
    dataset:
      name: ORM Pairwise Preference Pairs
      type: LossFunctionLover/orm-pairwise-preference-pairs
    metrics:
    - type: accuracy
      value: 96.3
      name: Pairwise Accuracy
    - type: confidence_interval
      value: "[95.3%, 97.1%]"
      name: Bootstrap 90% CI
---

# Pairwise Outcome Reward Model (ORM)

<div align="center">

**A Robust Preference Learning Model for Agentic Reasoning Systems**

[![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-yellow)](https://huggingface.co/datasets/LossFunctionLover/orm-pairwise-preference-pairs)

</div>

## 📋 Model Description

This is a **Pairwise Outcome Reward Model (ORM)** designed for agentic reasoning systems. The model learns to rank reasoning traces through relative preference judgments rather than absolute quality scores, achieving superior stability and reproducibility compared to traditional pointwise approaches.

**Key Achievements:**
- ✅ **96.3% pairwise accuracy** with tight confidence intervals [95.3%, 97.1%]
- ✅ **Stable training** in just 800 optimization steps (~10 minutes on single GPU)
- ✅ **Strong anti-symmetry** (swapped accuracy: 3.75%, correlation: -0.998)
- ✅ **Calibrated uncertainty** on near-tie cases
- ✅ **Length-robust** performance (95.5% - 99.7% across token ranges)
- ✅ **Frozen base model** architecture for reproducibility

## 🎯 Intended Use

This model is designed for:
- **Best-of-N sampling** in reasoning tasks
- **Candidate ranking** in agentic search and tree-based reasoning
- **Outcome-level feedback** in multi-step reasoning systems
- **Integration with Process Reward Models (PRMs)** for comprehensive evaluation
- **Agentic frameworks** like MagiCore-Agentic for robust decision-making

## 🏗️ Architecture

```
Input Text (Reasoning Trace)
    ↓
[Frozen Base LM Encoder]  ← Pre-trained, frozen during training
    ↓
[Final Non-Padding Token Pooling (attention-mask aware)]
    ↓
[Lightweight Linear Head]    ← Only these parameters are trained
    ↓
Scalar Reward Score
```

**Design Philosophy:**
- **Frozen encoder**: Leverages pre-trained representations, reduces overfitting
- **Lightweight head**: <1M trainable parameters for stability
- **Minimal architecture**: Prioritizes reproducibility over complexity

## 📊 Training Details

### Dataset Construction

The model was trained on a carefully curated pairwise preference dataset derived from high-quality reasoning traces:

**Original Pointwise Dataset:**
- Train: 9,482 examples
- Validation: 524 examples  
- Test: 547 examples
- Labels: Binary (correct=1, incorrect=0)

**Quality Validation (Base Model Log-Probability Analysis):**
- Pearson correlation: **r = 0.87** (p < 1e-162)
- Spearman correlation: **ρ = 0.83** (p < 1e-134)
- Base model pairwise accuracy: **98.2%**
- Mean log-prob (positive): -2.17
- Mean log-prob (negative): -3.64

These metrics confirm strong signal separation in the base model, validating dataset quality before pairwise transformation.

**Pairwise Dataset Construction:**

The pointwise data was transformed into pairwise preferences using a global sampling strategy:

```python
# For each positive example, sample N negative examples
# Creates (chosen, rejected) pairs where chosen=correct, rejected=incorrect
```

**Dataset Statistics:**
- **Training pairs**: 41,656 (8 negatives per positive)
- **Validation pairs**: 1,144 (4 negatives per positive)
- **Test pairs**: 1,232 (4 negatives per positive)

Each pair contains:
- `chosen`: Correct reasoning trace (label=1)
- `rejected`: Incorrect reasoning trace (label=0)
- `meta`: Chain IDs and labels for traceability

**Curation Process:**
- ✅ **Weeks of manual quality control** on original dataset
- ✅ **Rigorous filtering** for correctness and reasoning quality
- ✅ **Balanced sampling** across reasoning patterns and lengths
- ✅ **Verified anti-symmetry** through base model analysis

### Training Configuration

**Hyperparameters:**
- **Base Model**: facebook/opt-1.3b
- **Trainable Parameters**: Scoring head only (~500K-1M params)
- **Optimizer**: AdamW
  - Learning rate: 2e-5
  - Betas: (0.9, 0.999)
  - Weight decay: 0.01
- **Learning Rate Schedule**: Linear warmup (50 steps) + constant
- **Batch Size**: 8 pairs
- **Gradient Clipping**: Max norm 1.0
- **Training Steps**: 800
- **Mixed Precision**: FP16
- **Hardware**: Single GPU (A100/V100)
- **Training Time**: ~10 minutes

**Loss Function:**
```python
# Logistic pairwise ranking loss
L = -log(sigmoid(f(x_chosen) - f(x_rejected)))
```

## 🔬 Evaluation Results

### Main Performance (Test Set: 1,232 pairs)

| Metric | Value |
|--------|-------|
| **Pairwise Accuracy** | **96.3%** |
| Bootstrap 90% CI | [95.3%, 97.1%] |
| Mean Margin | 1.40 |
| Median Margin | 1.52 |
| Std Deviation | 1.12 |
| Incorrect/Tied Pairs | 3.7% |

### Length-Based Robustness

| Token Range | Accuracy | Sample Size |
|-------------|----------|-------------|
| 0-128 tokens | 95.5% | 442 pairs |
| 128-256 tokens | **99.7%** | 332 pairs |
| 256+ tokens | 96.1% | 458 pairs |

**Key Insight**: Model does not exploit length heuristics; benefits from additional context in medium-length range.

### Anti-Symmetry Validation (Label-Swap Test)

| Metric | Value | Expected |
|--------|-------|----------|
| Swapped Accuracy | 3.75% | ~3.7% ✅ |
| Mean Swapped Margin | -1.40 | -1.40 ✅ |
| Correlation (Original vs Swapped) | -0.998 | ~-1.0 ✅ |

**Conclusion**: Model learns true preference ordering, not positional artifacts.

### Near-Tie Uncertainty Calibration

| Margin Threshold | Accuracy | Interpretation |
|------------------|----------|----------------|
| \|Δ\| < 0.05 | 43% | Low confidence → near chance |
| \|Δ\| < 0.10 | 48% | Uncertain region |
| \|Δ\| < 0.20 | 60% | Moderate confidence |
| \|Δ\| < 0.50 | 71% | Higher confidence |

**Key Insight**: Smooth calibration curve indicates well-calibrated uncertainty—critical for agentic systems that need to defer when uncertain.

## 💻 Usage

### Installation

```bash
pip install transformers torch huggingface_hub
```

### Basic Usage

```python
import torch
from transformers import AutoModel, AutoTokenizer
from huggingface_hub import hf_hub_download

# Download the trained model weights
model_path = hf_hub_download(
    repo_id="LossFunctionLover/pairwise-orm-model",
    filename="pairwise_orm.pt"
)

# Load the base encoder (frozen during training)
base_model = AutoModel.from_pretrained("facebook/opt-1.3b")
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")

# Load the trained scoring head weights
ckpt = torch.load(model_path, map_location="cpu")
state = ckpt["model_state"] if "model_state" in ckpt else ckpt

head_state = {
    k.replace("score.", ""): v
    for k, v in state.items()
    if k.startswith("score.")
}

assert set(head_state.keys()) == {"weight", "bias"}

# Initialize scoring head (single linear layer)
hidden_size = base_model.config.hidden_size
scoring_head = torch.nn.Linear(hidden_size, 1)
scoring_head.load_state_dict(state)

# Move to device
device = "cuda" if torch.cuda.is_available() else "cpu"
base_model.eval().to(device)
scoring_head.eval().to(device)

# Score a single reasoning trace
def score_trace(trace_text: str) -> float:
    """
    Compute scalar reward for a reasoning trace.
    Higher scores indicate better reasoning quality.
    """
    inputs = tokenizer(
        trace_text,
        return_tensors="pt",
        truncation=True,
        max_length=512,
        padding=True
    )
    inputs = {k: v.to(device) for k, v in inputs.items()}
    
    with torch.no_grad():
        # Get base model embeddings
        encoder_outputs = base_model(**inputs)
        # Pool at actual sequence end (accounts for padding)
        seq_lengths = inputs["attention_mask"].sum(dim=1) - 1
        pooled = encoder_outputs.last_hidden_state[torch.arange(seq_lengths.size(0)), seq_lengths]
        # Get reward score
        score = scoring_head(pooled).squeeze(-1).cpu().item()
    
    return score

# Example: Compare two reasoning traces
trace_1 = """
1. Calculate the cost per item: $20 / 4 = $5
2. Calculate total for 10 items: $5 × 10 = $50
3. Apply 10% discount: $50 × 0.9 = $45

Final Answer: $45
"""

trace_2 = """
1. Assume linear growth incorrectly
2. Multiply by unrelated constant
3. Round result arbitrarily

Final Answer: $38
"""

score_1 = score_trace(trace_1)
score_2 = score_trace(trace_2)

print(f"Trace 1 score: {score_1:.3f}")
print(f"Trace 2 score: {score_2:.3f}")
print(f"Preferred trace: {'Trace 1' if score_1 > score_2 else 'Trace 2'}")
print(f"Confidence (margin): {abs(score_1 - score_2):.3f}")
```

### Batch Scoring for Best-of-N Sampling

```python
def rank_candidates(candidates: list[str], return_scores: bool = False):
    """
    Rank multiple candidate reasoning traces.
    
    Args:
        candidates: List of reasoning trace strings
        return_scores: If True, return (ranked_candidates, scores)
    
    Returns:
        Ranked list of candidates (best first)
    """
    scores = [score_trace(cand) for cand in candidates]
    
    # Sort by score descending
    ranked_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)
    ranked_candidates = [candidates[i] for i in ranked_indices]
    
    if return_scores:
        ranked_scores = [scores[i] for i in ranked_indices]
        return ranked_candidates, ranked_scores
    
    return ranked_candidates

# Example usage
candidates = [trace_1, trace_2, ...]  # Multiple traces for same problem
best_trace = rank_candidates(candidates)[0]
```

### Integration with Agentic Systems

```python
# Example: Use ORM for tree search pruning
def should_expand_node(reasoning_trace: str, threshold: float = 0.0) -> bool:
    """
    Decide whether to expand a reasoning node based on ORM score.
    """
    score = score_trace(reasoning_trace)
    return score > threshold

# Example: Combine with PRM for comprehensive evaluation
def hybrid_evaluation(trace: str, orm_model, prm_model):
    """
    Combine outcome-level (ORM) and process-level (PRM) rewards.
    """
    orm_score = score_trace(trace)  # Outcome quality
    prm_scores = prm_model.score_steps(trace)  # Step-level correctness
    
    # Weighted combination
    final_score = 0.5 * orm_score + 0.5 * prm_scores.mean()
    return final_score
```

## 🔗 Related Work & Citation

This work builds upon and complements:

- **MagiCore-Agentic** ([Liu et al., 2024](https://arxiv.org/abs/2409.12147)): Robust multi-step reasoning through agentic orchestration
- **Training Verifiers** ([Cobbe et al., 2021](https://arxiv.org/abs/2110.14168)): Math word problem verification
- **Process & Outcome Feedback** ([Uesato et al., 2022](https://arxiv.org/abs/2211.14275)): Combining reward signals

### Citation

If you use this model in your research, please cite:

```bibtex
@article{mishra2026orm,
  title={Stable Outcome Reward Modeling via Pairwise Preference Learning},
  author={Mishra, Aklesh},
  journal={arXiv preprint},
  year={2026},
  note={Under review}
}
```

## 🔗 Resources

- 📄 **Paper**: Submitted to arXiv (under review)
- 💾 **Dataset**: [HuggingFace](https://huggingface.co/datasets/LossFunctionLover/orm-pairwise-preference-pairs)

## 📧 Contact

**Aklesh Mishra**
- Email: akleshmishra7@gmail.com
- Independent Researcher

## 📝 License

This model is released under the **Apache 2.0 License**.

## 🙏 Acknowledgments

This research builds upon months of dedicated work in preference learning and agentic reasoning systems. Special thanks to:

- The **MagiCore-Agentic** team for their inspiring work on multi-step agentic reasoning
- The broader ML community for foundational research in reward modeling and RLHF
- Contributors to open-source tools (Transformers, PyTorch) that made this work possible

## 📊 Model Performance Summary

```
╔════════════════════════════════════════════════════════════╗
║  Pairwise ORM - Key Metrics                                ║
╠════════════════════════════════════════════════════════════╣
║  Pairwise Accuracy:        96.3% [95.3%, 97.1%]            ║
║  Training Steps:           800 (~10 min on single GPU)     ║
║  Dataset Quality (r):      0.87 (Pearson)                  ║
║  Anti-symmetry:            -0.998 correlation              ║
║  Length Robustness:        95.5% - 99.7% across ranges     ║
║  Uncertainty Calibration:  Smooth degradation near ties    ║
╚════════════════════════════════════════════════════════════╝
```

---

**Last Updated**: January 22, 2026