| <h1 align="center">🚀 Remedy-R: Generative Reasoning Models for MT Evaluation</h1> |
| <p align="center"><b>Reasoning-driven, reinforcement-trained metrics for machine translation evaluation</b></p> |
|
|
| --- |
|
|
| ## ✨ What is Remedy-R? |
|
|
| **Remedy-R** is a family of **reasoning-based MT evaluation models** trained with **reinforcement learning via verifiable rewards (RLVR)** on **pairwise human translation preferences**. |
|
|
| Instead of directly regressing a scalar score, Remedy-R: |
|
|
| - Generates **step-by-step analyses** of *accuracy*, *fluency*, and *completeness*. |
| - Outputs a **final numeric score in [0, 100]** that can be parsed and used like a standard metric. |
| - Is trained with **PPO + rule-based rewards** that check whether predicted preferences match human rankings and calibrate scores toward human ratings. |
| - Supports both **reference-based** and **reference-free (QE)** evaluation. |
|
|
| On WMT22–24 and MSLC24-style OOD stress tests, Remedy-R: |
|
|
| - **Surpasses** strong LLM-as-judge methods. |
| - Matches top-performing scalar SOTA metrics. |
| - Remains **robust under OOD conditions** such as source copy, empty translations, wrong language, and mixed-language outputs. |
| - Enables **Test-Time Scaling (TTS)** via multiple reasoning passes, improving segment-level meta-evaluation. |
| - Powers **Remedy-R Agent**, an evaluate–revise pipeline that improves translations for diverse base systems. |
|
|
| --- |
|
|
| ## 📚 Contents |
|
|
| - [✨ What is Remedy-R?](#-what-is-remedy-r) |
| - [📚 Contents](#-contents) |
| - [📦 Installation](#-installation) |
| - [From PyPI (recommended)](#from-pypi-recommended) |
| - [From source](#from-source) |
| - [⚙️ Requirements](#️-requirements) |
| - [🧠 Model Zoo](#-model-zoo) |
| - [🚀 Quickstart](#-quickstart) |
| - [CLI: Local vLLM Inference](#cli-local-vllm-inference) |
| - [Reference-Free / QE Mode](#reference-free--qe-mode) |
| - [Test-Time Scaling (TTS)](#test-time-scaling-tts) |
| - [🌐 Optional: vLLM Online Serving](#-optional-vllm-online-serving) |
| - [📄 Outputs](#-outputs) |
| - [📚 Citation](#-citation) |
|
|
| --- |
|
|
| ## 📦 Installation |
|
|
| ### From PyPI (unavailable for now) |
|
|
| ```bash |
| pip install --upgrade pip |
| pip install remedy-r-mt-eval |
| ```` |
|
|
| This installs the `remedy_r` package and the CLI entrypoint `remedy-r-score` (plus related tools). |
|
|
| ### From source |
|
|
| ```bash |
| git clone https://github.com/Smu-Tan/Remedy-R.git |
| cd Remedy-R |
| pip install -e . |
| ``` |
|
|
| --- |
|
|
| ## ⚙️ Requirements |
|
|
| Core runtime dependencies (see `pyproject.toml` for exact versions): |
|
|
| * Python ≥ 3.10 (tested mostly with 3.12) |
| * [PyTorch](https://pytorch.org/) with GPU support |
| * [vLLM](https://github.com/vllm-project/vllm) for efficient batched inference |
| * `transformers`, `numpy`, `pandas`, `tqdm` |
|
|
| You also need: |
|
|
| * At least **1 GPU (16–24 GB)** for 7B models |
| * More memory/GPUs for 14B/32B models or large batch sizes |
|
|
| --- |
|
|
| ## 🧠 Model Zoo |
|
|
| Remedy-R models are hosted on HuggingFace under `ShaomuTan/`: |
|
|
| | Model | Size | Base model | Mode | Link | |
| | ------------ | ---- | ----------- | -------- | --------------------------- | |
| | Remedy-R-7B | 7B | Qwen2.5-7B | Ref + QE | [🤗 HuggingFace](https://huggingface.co/ShaomuTan/Remedy-R-7B) | |
| | Remedy-R-14B | 14B | Qwen2.5-14B | Ref + QE | [🤗 HuggingFace](https://huggingface.co/ShaomuTan/Remedy-R-14B) | |
| | Remedy-R-32B | 32B | Qwen2.5-32B | Ref + QE | [🤗 HuggingFace](https://huggingface.co/ShaomuTan/Remedy-R-32B) | |
|
|
| You can cache them locally: |
|
|
| ```bash |
| HF_HUB_ENABLE_HF_TRANSFER=1 \ |
| huggingface-cli download ShaomuTan/Remedy-R-14B \ |
| --local-dir Models/Remedy-R-14B |
| ``` |
|
|
| Then point `--model` to either the **HF ID** or the **local path**. |
|
|
| --- |
|
|
| ## 🚀 Quickstart |
|
|
| ### CLI: Local vLLM Inference |
|
|
| The main entrypoint is: |
|
|
| ```bash |
| remedy-r-score \ |
| --model "$MODEL_CHECKPOINT" \ |
| --save_metric_name "$METRIC_NAME" \ |
| --output_dir "$DATA_DIR" \ |
| --max-tokens "$MAX_TOKENS" \ |
| --tp_size "$TP_SIZE" \ |
| --dp_size "$DP_SIZE" \ |
| --temperature "$DEC_TEMPERATURE" \ |
| --repetition_penalty "$REPETITION_PENALTY" \ |
| --gpu-memory-utilization "$GPU_MEM_UTIL" \ |
| --max-model-len "$MAX_MODEL_LEN" \ |
| --seed "$SEED" \ |
| --src-file "$SRC_FILE" \ |
| --mt-file "$MT_FILE" \ |
| --lp "$LP" \ |
| ``` |
|
|
| **Key arguments** |
|
|
| * `--model` : HF repo ID or local checkpoint |
| * `--src-file` : Source sentences (one per line) |
| * `--mt-file` : MT outputs (one per line) |
| * `--ref-file` : Reference translations (optional; enables ref-based mode) |
| * `--lp` : Language-pair codes (e.g., `en-de`) |
| * `--output_dir` : Output folder |
| * `--temperature` : Generation temperature |
| * `--tp_size` : Tensor parallel size |
| * `--dp_size` : Data parallel size |
| * `--num-seqs` : Max parallel sequences per step |
| * `--max-tokens` : Max generation token numebrs |
| * `--gpu-memory-utilization` : vLLM memory ratio (e.g. 0.9) |
|
|
|
|
| You can also call the CLI via Python: |
|
|
| ```bash |
| python -m remedy_r.cli.score \ |
| --model ShaomuTan/Remedy-R-7B \ |
| ... |
| ``` |
|
|
| --- |
|
|
| ### Reference-Free / QE Mode |
|
|
| If you don’t have references, just drop `--ref-file` and add `--no-ref`: |
|
|
| ```bash |
| remedy-r-score \ |
| --model ShaomuTan/Remedy-R-7B \ |
| --src-file ./testcase/en.src \ |
| --mt-file ./testcase/en-de.hyp \ |
| --no-ref \ |
| --src-lang en \ |
| --tgt-lang de \ |
| --save-dir ./testcase \ |
| --cache-dir ./Models |
| ``` |
|
|
| The prompt automatically switches to **reference-free quality estimation** while keeping the same [0, 100] score scale. |
|
|
| --- |
|
|
| ### Test-Time Scaling (TTS) |
|
|
| Remedy-R supports **Test-Time Scaling** by averaging multiple independent evaluation passes with different seeds: |
|
|
| ```bash |
| remedy-r-score \ |
| --model ShaomuTan/Remedy-R-14B \ |
| --src-file ./testcase/en.src \ |
| --mt-file ./testcase/en-de.hyp \ |
| --ref-file ./testcase/de.ref \ |
| --src-lang en --tgt-lang de \ |
| --save-dir ./testcase_tts \ |
| --TTS \ |
| --best-of-n 4 \ |
| --seed 42 |
| ``` |
|
|
| * `--TTS` : Enable multi-pass evaluation |
| * `--best-of-n` : Number of independent passes (e.g., 2–6) |
| * Scores are averaged; the detailed per-pass scores can be optionally logged. |
|
|
| TTS typically improves **segment-level pairwise accuracy** and stabilizes scores for difficult segments. |
|
|
| --- |
|
|
| ## 🌐 Optional: vLLM Online Serving |
|
|
| To avoid re-loading the model for every scoring run, you can: |
|
|
| 1. **Start a local vLLM server** (OpenAI-compatible): |
|
|
| ```bash |
| remedy-r-serve \ |
| --model ShaomuTan/Remedy-R-14B \ |
| --port 8000 \ |
| --max-model-len 4096 \ |
| --gpu-memory-utilization 0.9 |
| ``` |
|
|
| 2. **Score via the server**: |
|
|
| ```bash |
| remedy-r-score \ |
| --src-file ./testcase/en.src \ |
| --mt-file ./testcase/en-de.hyp \ |
| --ref-file ./testcase/de.ref \ |
| --lp en-de \ |
| --save_metric_name Remedy-R-14B \ |
| --save-dir ./testcase_server \ |
| --server-url http://localhost:8000/v1 |
| ``` |
|
|
| Internally this reuses the same Remedy-R prompting and scoring logic, but routes generation requests through the running vLLM server instead of instantiating `LLM()` in every process. |
|
|
| --- |
|
|
| ## 📄 Outputs |
|
|
| For each language pair `SRC-TGT`, Remedy-R writes: |
|
|
| * `results.jsonl` |
| * `segment_scores.tsv` |
| * `system_score.txt` |
|
|
|
|
| ## 📚 Citation |
|
|
| If you use Remedy-R or this codebase, please cite: |
|
|
| Arxiv coming soon... |
|
|
