README.md

---
license: apache-2.0
library_name: pytorch
tags:
- code
- classification
- code-filtering
- algorithmic-relevance
- ultradata
language:
- en
- zh
---

# UltraData-Code-L2-Classifier

<p align="center">
<a href="https://huggingface.co/collections/openbmb/ultradata">📦 UltraData Collection</a> |
<a href="https://ultradata.openbmb.cn/">🌐 UltraData</a> |
<a href="https://huggingface.co/collections/openbmb/minicpm5">🤗 MiniCPM5 Series</a> |
📖 Tech Report (Coming Soon) |
<a href="https://huggingface.co/datasets/openbmb/UltraData-Code">🤗 UltraData-Code Dataset</a>
</p>

<p align="center">English | <a href="https://huggingface.co/openbmb/UltraData-Code-L2-Classifier/blob/main/README_ZH.md">中文</a></p>

## 📚 Introduction

***UltraData-Code-L2-Classifier*** is a suite of language-specific, file-level scorers designed for the 11 programming languages represented in [UltraData-Code-L1](https://huggingface.co/datasets/openbmb/UltraData-Code).

For each language, we train and apply a dedicated L2 scorer to the corresponding files in UltraData-Code-L1. The selected files form [UltraData-Code-L2](https://huggingface.co/datasets/openbmb/UltraData-Code), a corpus designed to concentrate interpretable computation and problem-solving logic. The resulting L2 corpus contains approximately **400B tokens** and retains approximately **12.23% of L1 files** overall.

## 💡 Highlights

> **Abstract:** Code generation has become a core capability of large language models, and code data is a central part of the pre-training process that develops it. As code corpora continue to grow, their scale, diversity, and quality increasingly shape the capabilities learned during pre-training. Following a tiered data management perspective, we present **UltraData-Code** as a family of four connected data states, from repository archival at L0 through standardized natural code at L1, algorithmic selection at L2, and task-oriented synthesis at L3. L0 archives the latest revision on the default branch of each public GitHub repository with its file structure, relationships, and provenance. L1 applies scalable filtering, normalization, and near deduplication to obtain standardized natural code. L2 then selects algorithmically relevant files from L1 using language-adaptive signals from file roles and heuristics, together with code quality constraints, yielding approximately 400B tokens of **UltraData-Code-L2**, spanning 11 programming languages. L3 applies task-oriented synthesis to algorithmic files from L2, turning each implementation into a structured programming exercise, generating approximately 150B tokens of **UltraData-Code-L3** in the same 11 languages. Under controlled 10B-token continual pre-training of a 1B model, training on L2 instead of L1 raises pass@1 on EvalPlus by **7.80 points** and on MultiPL-E by **5.13 points**, while exceeding Stack-Edu by 4.37 and 3.05 points, respectively. Replacing half of the L2 training tokens with L3 yields a further gain of **8.42 points** on EvalPlus and **8.07 points** on MultiPL-E over L2-only training, while exceeding the strongest synthetic data baseline by 5.57 and 7.80 points, respectively. When the training budget increases to 100B tokens, gains from L2 selection and L3 synthesis further widen on both benchmarks.

- **A connected code data construction pipeline.** UltraData-Code links repository archival and standardized natural code with selection and structured synthesis, yielding approximately 400B tokens at L2 and 150B tokens at L3 across 11 programming languages.
- **Fine-grained curation of algorithmically relevant code.** We develop a language-adaptive selection framework that combines file role supervision with language-specific heuristic cues to learn algorithmic relevance beyond explicit ALGO files. The framework reuses precomputed semantic embeddings across role, relevance, and quality models, and applies role and quality constraints during selection.
- **Implementation-grounded, task-oriented synthesis.** We introduce a structured synthesis protocol that converts each algorithmically relevant implementation selected by L2 into a programming exercise, jointly generating a standalone task, analysis, solution, and test candidates from the same source implementation. This transformation preserves the source code's computational intent while adding explicit task and solution supervision for code generation.

<div align="center">
  <img src="assets/ultradata-code-l0-l3-overview.png" width="760" alt="UltraData-Code L0-L3 construction pipeline"/>
</div>

## 🔎 L2 Selection Framework

The L2 framework is applied independently to each of the 11 languages in UltraData-Code-L1:

1. **Encode files.** Each file is represented with a 1024-dimensional embedding from [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B).
2. **Predict file roles.** The role classifier assigns one of `ALGO`, `WEB`, `TOOL`, `DATA`, `TEST`, `CONFIG`, or `EXCLUDE`.
3. **Construct dual-cue supervision.** ALGO role labels are combined with language-specific heuristic evidence to identify algorithmically relevant files beyond the ALGO role alone.
4. **Score relevance and quality.** The relevance model outputs `algo_rel_score`; the quality model outputs `quality_score`.
5. **Calibrate and select.** A language-specific relevance threshold is calibrated on held-out data. A file is retained when it passes the relevance and quality thresholds and is not predicted as `EXCLUDE`.

<div align="center">
  <img src="assets/ultradata-code-l2.png" width="760" alt="UltraData-Code L2 selection framework"/>
</div>

## 📈 Evaluation Results

The 10B-token comparisons use the same 1B foundation model, training settings, decontamination, and evaluation protocol.

**Python results: Algorithmic selection yields the strongest natural code result.** UltraData-Code-L2-py improves the average EvalPlus pass@1 by **17.91 points** over UltraData-Code-L1-py and by **8.46 points** over Stack-Edu-py.

<div align="center">
  <img src="assets/python-training-dynamics.png" alt="Python training dynamics" width="760"/>
</div>

<div align="center">
  <img src="assets/python_results.png" alt="Python main results table" width="900"/>
</div>

**Multilingual results: L2 selection improves over L1 baselines.** UltraData-Code-L2 exceeds UltraData-Code-L1 by **7.80 points** on EvalPlus and **5.13 points** on MultiPL-E, and exceeds Stack-Edu by **4.37** and **3.05 points**, respectively.

<div align="center">
  <img src="assets/multilingual-training-dynamics.png" alt="Multilingual training dynamics" width="760"/>
</div>

<div align="center">
  <img src="assets/multilingual_results.png" alt="Multilingual main results table" width="900"/>
</div>

## 🚀 Usage of UltraData-Code-L2 Classifier

Install the runtime dependencies:

```bash
pip install numpy torch pyarrow
```

### Score embeddings in Python

```python
import sys
sys.path.insert(0, "scripts")
from scorers import LanguageScorers

scorers = LanguageScorers.load("classifiers/cpp")

# float32 [N, 1024]; paths are repository-relative and aligned with embeddings
out = scorers.score(embeddings, paths)
```

Use the matching language directory for each input language. Real repository-relative paths are required by the relevance model.

### Score Parquet files

[`scripts/score_parquet.py`](scripts/score_parquet.py) accepts one Parquet file or recursively walks a directory. Inputs must contain an `embedding` column and either `relative_path` or `meta.file_path`.

```bash
SCORERS=classifiers/cpp
DATA_PATH=data/input
SAVE_PATH=data/output

python scripts/score_parquet.py \
  --scorers ${SCORERS} \
  --input ${DATA_PATH} \
  --output-root ${SAVE_PATH} \
  --batch-size 1024 \
  --device cpu \
  --apply-policy
```

The output mirrors the input directory structure and preserves all existing columns. Use `--describe` to inspect a scorer set, `--limit N` to process at most `N` Parquet files, or `--algo-rel-min`, `--quality-min`, and `--exclude-categories` to override policy values.

### Output fields

The classifier appends the following results to each input file:

```text
category: {Predicted file role}
cls_confidence: {File role confidence}
algo_rel_score: {Algorithmic-relevance score}
quality_score: {Code quality score}
selected: {Whether the file passes the selection policy}
```

- `category` is one of `ALGO`, `WEB`, `TOOL`, `DATA`, `TEST`, `CONFIG`, or `EXCLUDE`.
- `cls_confidence` and `algo_rel_score` range from `0` to `1`; `quality_score` ranges from `0` to `10`.
- `selected` is added only when `--apply-policy` is enabled. It marks rows that pass the relevance threshold, the `EXCLUDE` veto, and the quality threshold; rows are not deleted.

### Notes

- Inputs must use 1024-dimensional embeddings produced by [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B).
- The relevance scorer also uses the repository-relative path. Use real paths aligned with the embeddings.
- [`config.json`](config.json) provides defaults. Command-line options can override these defaults.
- The scoring script is single-process. For corpus-scale inference, schedule multiple shards or subdirectories externally.

## ❤️ Acknowledgements

This release builds on [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B), [PyTorch](https://pytorch.org/), [PyArrow](https://arrow.apache.org/docs/python/). Thanks for their awesome work! Open-source contributions make UltraData-Code possible! 🙌

## 💳 License

This project is released under the [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license.

## 📖 Citation

If you find **UltraData-Code** or this classifier useful, please consider citing:

```bibtex
@misc{ultradata_code,
  title        = {{UltraData-Code}: From Raw Repositories to Algorithmically Dense and Task-Oriented Code Data},
  author       = {Chengying Tu and Hengyu Zhao and Shuaikang Xue and Zhongming Qu and Jihao Zhou and Xinle Lin and Junshao Guo and Zixuan Fu and Qiang Ma and Jie Zhou and Chaojun Xiao and Hongfei Yan and Yudong Wang and Xu Han and Zhiyuan Liu and Maosong Sun},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/openbmb/UltraData-Code}}
}
```