docs/adrs/ADR-004-replaysim-normalization.md

# ADR-004 — Replaysim normalization layer for the trace-replay channel

**Status**: Accepted
**Date**: 2026-05-26
**Wave**: 13 (deep work loop, expansion phase)

## Context

The brief's V5 clause says:

> use traces from an llm-application usage then replay the traces with
> different models to see at each llm-step what the llm would do. by doing
> this we get distillation data from any number of models that could be
> used to train the target model further

The user added 2026-05-26: *"see if we can leverage [a normalization
library] to normalize the data while also making the replaysim dataset
generation."*

Currently the framework has `composer_replication.teacher_replay`:
- `replay_trace()` — N-teacher OpenRouter replay, returns
  `list[TeacherCallResult]`
- `extract_dpo_pairs()` — converts teacher disagreement to `list[DPOPair]`

This produces preference-pair training data, but with **zero normalization**:
no dedup, no length filtering, no language detection, no quality
filtering, no chat-template validation. The output is closer to "raw
LLM API responses" than "training-ready dataset."

For the replaysim to power downstream RL training (V6), the dataset needs
to be production-quality. Hand-rolling that pipeline is a tax we'd rather
not pay.

## Options considered

Audited five candidates in `docs/research/REPLAYSIM_NORMALIZATION_RECONNAISSANCE.md`:

| Library | License | Multi-turn? | DPO pairs? | Streaming? | GPU? | Verdict |
|---|---|---|---|---|---|---|
| HuggingFace `datatrove` | MIT | ❌ flat-text only | ❌ | ✅ | ❌ | Deal-breaker on multi-turn |
| Alibaba `data-juicer` | Apache-2 | ✅ native `messages` ops | ✅ `pair_preference_mapper` | ✅ | ❌ for ops we need | **Chosen** |
| NVIDIA `nemo-curator` | Apache-2 | partial | ❌ | ✅ | ✅ mandatory for differentiating ops | Reject — GPU-bound for the ops we need |
| Argilla `distilabel` | Apache-2 | ✅ native chat | ✅ formatters | ✅ | ❌ | Reject — would replace teacher orchestration, not just normalize |
| Databricks `lilac` | — | n/a | n/a | n/a | n/a | Reject — archived 2024-03 |

## Decision

**Adopt `data-juicer` (Alibaba/modelscope, Apache-2.0, last push 2026-05-25, 6.4k★).**

Reasons:

1. **It's the only candidate with native multi-turn + DPO support in the
   *normalization* op-graph.** Has `pair_preference_mapper`,
   `dialog_intent_detection_mapper`, `dialog_topic_detection_mapper`,
   etc. that operate on chat-format messages directly.

2. **CPU-runnable for our op set.** The differentiating ops we need
   (length filter, language ID, chat-template validation, dedup) all
   work on CPU. We avoid the NeMo-Curator GPU dependency entirely.

3. **Streaming-friendly.** Op graph is a DAG; we can pipe `replay_trace`
   output into the graph during generation, not as a post-hoc pass. This
   matters for cost discipline — bad teacher outputs get filtered before
   contributing to OpenRouter spend on subsequent steps.

4. **YAML-recipe driven.** Recipes live in `recipes/replaysim/` and can
   be version-controlled. A user can swap normalization recipes without
   touching framework code.

## Consequences

### Accepted

- New module `composer_replication.replaysim` lifts the existing
  `teacher_replay` logic out of the package's flat namespace and adds:
  - `composer_replication.replaysim.normalize` — `DJNormalizer` adapter
    that wraps `data-juicer` op graphs around `replay_trace` output
  - `recipes/replaysim/default.yaml` — base normalization recipe (length
    filter + chat-template validation + per-turn dedup)
  - Optional `recipes/replaysim/with_disagreement_filter.yaml` — adds a
    semantic-similarity filter that drops "false disagreements" where
    teachers used different wording for the same answer
- New optional dependency `[replaysim]` extra in `pyproject.toml`:
  `pip install -e .[replaysim]` pulls `data-juicer`. Core install
  doesn't require it.
- The existing `replay_trace` and `extract_dpo_pairs` keep their
  signatures. The normalizer is opt-in via a `normalizer=` kwarg on a
  new `replay_and_normalize_trace` convenience function.

### One-day spike before merge

`pair_preference_mapper` in data-juicer might unconditionally re-synthesize
the `rejected` text via an LLM call. We already have `rejected` from
teacher disagreement and don't want to pay another API call. The recon
flagged this — verify by reading the mapper's source, and if it's LLM-bound,
substitute a plain validator that checks the field exists + isn't empty.

If the spike fails (the mapper IS LLM-bound and isn't easily replaceable),
fall back to writing a custom `DJOp` subclass that validates pre-existing
DPO pairs without re-synthesis. ~50 LOC.

### Rejected paths

- **`datatrove`**: would have required hand-rolling all chat-template logic
  on top of flat-text ops. Bigger ongoing maintenance cost than
  data-juicer's native multi-turn support.
- **`nemo-curator`**: GPU-mandatory ops mean we'd need to pay for GPU during
  dataset generation (separate from the replay phase, which is already
  GPU-free). Net cost increase for no quality win.
- **`distilabel`**: too broad — its pipeline abstraction would replace our
  `replay_trace` entirely. We'd lose direct OpenRouter cost control + the
  audit trail. Possible v0.3 migration if data-juicer becomes a bottleneck.

### Future work

- v0.2: add a `recipes/replaysim/altered_minds.yaml` for the user's
  `altered-minds` workstream tie-in (per Wave 13 expansion)
- v0.3: revisit if `distilabel` becomes more mature and the migration
  cost vs ongoing-maintenance balance shifts

## Source

`docs/research/REPLAYSIM_NORMALIZATION_RECONNAISSANCE.md` (2026-05-26
subagent recon, primary-sourced from each repo's GitHub + DeepWiki).