update README
Browse files- README.md +27 -36
- README_ZH.md +36 -45
README.md
CHANGED
|
@@ -22,27 +22,21 @@ language:
|
|
| 22 |
<a href="https://huggingface.co/datasets/openbmb/UltraData-Code">🤗 UltraData-Code Dataset</a>
|
| 23 |
</p>
|
| 24 |
|
| 25 |
-
<p align="center">English | <a href="https://huggingface.co/
|
| 26 |
|
| 27 |
## 📚 Introduction
|
| 28 |
|
| 29 |
-
***UltraData-Code-L2-Classifier*** is a suite of language-specific, file-level scorers for the 11 programming languages
|
| 30 |
|
| 31 |
-
For each language,
|
| 32 |
-
|
| 33 |
-
## 📢 What's New
|
| 34 |
-
|
| 35 |
-
- **[2026.09.07]** ***UltraData-Code-L2-Classifier*** is released, together with [***UltraData-Code***](https://huggingface.co/datasets/openbmb/UltraData-Code) (**L2** \~400B tokens / **L3** \~150B tokens across 11 languages). 🚀🚀🚀
|
| 36 |
-
- **[2026.09.07]** ***[MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B) is released!***, the second model in the MiniCPM5 series after MiniCPM5-1B. 🚀🚀🚀
|
| 37 |
-
- **[2026.02.08]** The [***UltraData***](https://ultradata.openbmb.cn/) platform is now live, introducing the [L0-L4 tiered data management framework](https://arxiv.org/pdf/2602.09003). 🔍🔍🔍
|
| 38 |
|
| 39 |
## 💡 Highlights
|
| 40 |
|
| 41 |
-
> **Abstract:** Code generation
|
| 42 |
|
| 43 |
-
- **A connected code data construction pipeline
|
| 44 |
-
- **Fine-grained curation of algorithmically relevant code
|
| 45 |
-
- **
|
| 46 |
|
| 47 |
<div align="center">
|
| 48 |
<img src="assets/ultradata-code-l0-l3-overview.png" width="760" alt="UltraData-Code L0-L3 construction pipeline"/>
|
|
@@ -64,12 +58,9 @@ The L2 framework is applied independently to each of the 11 languages in UltraDa
|
|
| 64 |
|
| 65 |
## 📈 Evaluation Results
|
| 66 |
|
| 67 |
-
The 10B-token comparisons use the same 1B
|
| 68 |
-
|
| 69 |
-
### Python Results
|
| 70 |
|
| 71 |
-
|
| 72 |
-
- **Structured synthesis improves the matched mixture.** The equal-token UltraData-Code-L2-py + UltraData-Code-L3-py mixture reaches a **46.43** average pass@1, **3.30 points** above training on L2 alone and **4.46 points** above the matched L2 + SwallowCode-v2 mixture.
|
| 73 |
|
| 74 |
<div align="center">
|
| 75 |
<img src="assets/python-training-dynamics.png" alt="Python training dynamics" width="760"/>
|
|
@@ -79,11 +70,7 @@ The 10B-token comparisons use the same 1B base model, architecture, optimization
|
|
| 79 |
<img src="assets/python_results.png" alt="Python main results table" width="900"/>
|
| 80 |
</div>
|
| 81 |
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
- **L2 selection improves over L1 baselines.** UltraData-Code-L2 exceeds UltraData-Code-L1 by **7.80 points** on EvalPlus and **5.13 points** on MultiPL-E, and exceeds Stack-Edu by **4.37** and **3.05 points**, respectively.
|
| 85 |
-
- **L3 synthesis provides a further gain.** Replacing half of the L2 training tokens with L3 yields **8.42** and **8.07 points** over training on L2 alone on EvalPlus and MultiPL-E. Under the same 1:1 token mix, UltraData-Code-L2-L3 exceeds the strongest reported synthetic baseline by **5.57** and **7.80 points**, respectively.
|
| 86 |
-
- **The advantage persists during scaling.** At 100B tokens, the L2-L3 mixture reaches 57.06 on EvalPlus and 39.54 on MultiPL-E, exceeding training on L2 alone by 10.11 and 12.39 points and the matched CodeAlchemy-code-qa mixture by 10.86 and 9.60 points.
|
| 87 |
|
| 88 |
<div align="center">
|
| 89 |
<img src="assets/multilingual-training-dynamics.png" alt="Multilingual training dynamics" width="760"/>
|
|
@@ -93,7 +80,9 @@ The 10B-token comparisons use the same 1B base model, architecture, optimization
|
|
| 93 |
<img src="assets/multilingual_results.png" alt="Multilingual main results table" width="900"/>
|
| 94 |
</div>
|
| 95 |
|
| 96 |
-
## 🚀 Usage
|
|
|
|
|
|
|
| 97 |
|
| 98 |
```bash
|
| 99 |
pip install numpy torch pyarrow
|
|
@@ -112,7 +101,7 @@ scorers = LanguageScorers.load("classifiers/cpp")
|
|
| 112 |
out = scorers.score(embeddings, paths)
|
| 113 |
```
|
| 114 |
|
| 115 |
-
Use the matching language directory
|
| 116 |
|
| 117 |
### Score Parquet files
|
| 118 |
|
|
@@ -132,32 +121,34 @@ python scripts/score_parquet.py \
|
|
| 132 |
--apply-policy
|
| 133 |
```
|
| 134 |
|
| 135 |
-
The output mirrors the input directory structure and
|
| 136 |
|
| 137 |
### Output fields
|
| 138 |
|
|
|
|
|
|
|
| 139 |
```text
|
| 140 |
category: {Predicted file role}
|
| 141 |
-
cls_confidence: {File
|
| 142 |
algo_rel_score: {Algorithmic-relevance score}
|
| 143 |
-
quality_score: {Code
|
| 144 |
selected: {Whether the file passes the selection policy}
|
| 145 |
```
|
| 146 |
|
| 147 |
-
- `category`
|
| 148 |
-
- `cls_confidence`
|
| 149 |
-
- `selected` is
|
| 150 |
|
| 151 |
### Notes
|
| 152 |
|
| 153 |
-
-
|
| 154 |
-
- The relevance scorer also uses the repository-relative path
|
| 155 |
-
- [`config.json`](config.json)
|
| 156 |
-
-
|
| 157 |
|
| 158 |
## ❤️ Acknowledgements
|
| 159 |
|
| 160 |
-
This release builds on [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B), [PyTorch](https://pytorch.org/),
|
| 161 |
|
| 162 |
## 💳 License
|
| 163 |
|
|
|
|
| 22 |
<a href="https://huggingface.co/datasets/openbmb/UltraData-Code">🤗 UltraData-Code Dataset</a>
|
| 23 |
</p>
|
| 24 |
|
| 25 |
+
<p align="center">English | <a href="https://huggingface.co/openbmb/UltraData-Code-L2-Classifier/blob/main/README_ZH.md">中文</a></p>
|
| 26 |
|
| 27 |
## 📚 Introduction
|
| 28 |
|
| 29 |
+
***UltraData-Code-L2-Classifier*** is a suite of language-specific, file-level scorers designed for the 11 programming languages represented in [UltraData-Code-L1](https://huggingface.co/datasets/openbmb/UltraData-Code).
|
| 30 |
|
| 31 |
+
For each language, we train and apply a dedicated L2 scorer to the corresponding files in UltraData-Code-L1. The selected files form [UltraData-Code-L2](https://huggingface.co/datasets/openbmb/UltraData-Code), a corpus designed to concentrate interpretable computation and problem-solving logic. The resulting L2 corpus contains approximately **400B tokens** and retains approximately **12.23% of L1 files** overall.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
## 💡 Highlights
|
| 34 |
|
| 35 |
+
> **Abstract:** Code generation has become a core capability of large language models, and code data is a central part of the pre-training process that develops it. As code corpora continue to grow, their scale, diversity, and quality increasingly shape the capabilities learned during pre-training. Following a tiered data management perspective, we present **UltraData-Code** as a family of four connected data states, from repository archival at L0 through standardized natural code at L1, algorithmic selection at L2, and task-oriented synthesis at L3. L0 archives the latest revision on the default branch of each public GitHub repository with its file structure, relationships, and provenance. L1 applies scalable filtering, normalization, and near deduplication to obtain standardized natural code. L2 then selects algorithmically relevant files from L1 using language-adaptive signals from file roles and heuristics, together with code quality constraints, yielding approximately 400B tokens of **UltraData-Code-L2**, spanning 11 programming languages. L3 applies task-oriented synthesis to algorithmic files from L2, turning each implementation into a structured programming exercise, generating approximately 150B tokens of **UltraData-Code-L3** in the same 11 languages. Under controlled 10B-token continual pre-training of a 1B model, training on L2 instead of L1 raises pass@1 on EvalPlus by **7.80 points** and on MultiPL-E by **5.13 points**, while exceeding Stack-Edu by 4.37 and 3.05 points, respectively. Replacing half of the L2 training tokens with L3 yields a further gain of **8.42 points** on EvalPlus and **8.07 points** on MultiPL-E over L2-only training, while exceeding the strongest synthetic data baseline by 5.57 and 7.80 points, respectively. When the training budget increases to 100B tokens, gains from L2 selection and L3 synthesis further widen on both benchmarks.
|
| 36 |
|
| 37 |
+
- **A connected code data construction pipeline.** UltraData-Code links repository archival and standardized natural code with selection and structured synthesis, yielding approximately 400B tokens at L2 and 150B tokens at L3 across 11 programming languages.
|
| 38 |
+
- **Fine-grained curation of algorithmically relevant code.** We develop a language-adaptive selection framework that combines file role supervision with language-specific heuristic cues to learn algorithmic relevance beyond explicit ALGO files. The framework reuses precomputed semantic embeddings across role, relevance, and quality models, and applies role and quality constraints during selection.
|
| 39 |
+
- **Implementation-grounded, task-oriented synthesis.** We introduce a structured synthesis protocol that converts each algorithmically relevant implementation selected by L2 into a programming exercise, jointly generating a standalone task, analysis, solution, and test candidates from the same source implementation. This transformation preserves the source code's computational intent while adding explicit task and solution supervision for code generation.
|
| 40 |
|
| 41 |
<div align="center">
|
| 42 |
<img src="assets/ultradata-code-l0-l3-overview.png" width="760" alt="UltraData-Code L0-L3 construction pipeline"/>
|
|
|
|
| 58 |
|
| 59 |
## 📈 Evaluation Results
|
| 60 |
|
| 61 |
+
The 10B-token comparisons use the same 1B foundation model, training settings, decontamination, and evaluation protocol.
|
|
|
|
|
|
|
| 62 |
|
| 63 |
+
**Python results: Algorithmic selection yields the strongest natural code result.** UltraData-Code-L2-py improves the average EvalPlus pass@1 by **17.91 points** over UltraData-Code-L1-py and by **8.46 points** over Stack-Edu-py.
|
|
|
|
| 64 |
|
| 65 |
<div align="center">
|
| 66 |
<img src="assets/python-training-dynamics.png" alt="Python training dynamics" width="760"/>
|
|
|
|
| 70 |
<img src="assets/python_results.png" alt="Python main results table" width="900"/>
|
| 71 |
</div>
|
| 72 |
|
| 73 |
+
**Multilingual results: L2 selection improves over L1 baselines.** UltraData-Code-L2 exceeds UltraData-Code-L1 by **7.80 points** on EvalPlus and **5.13 points** on MultiPL-E, and exceeds Stack-Edu by **4.37** and **3.05 points**, respectively.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
|
| 75 |
<div align="center">
|
| 76 |
<img src="assets/multilingual-training-dynamics.png" alt="Multilingual training dynamics" width="760"/>
|
|
|
|
| 80 |
<img src="assets/multilingual_results.png" alt="Multilingual main results table" width="900"/>
|
| 81 |
</div>
|
| 82 |
|
| 83 |
+
## 🚀 Usage of UltraData-Code-L2 Classifier
|
| 84 |
+
|
| 85 |
+
Install the runtime dependencies:
|
| 86 |
|
| 87 |
```bash
|
| 88 |
pip install numpy torch pyarrow
|
|
|
|
| 101 |
out = scorers.score(embeddings, paths)
|
| 102 |
```
|
| 103 |
|
| 104 |
+
Use the matching language directory for each input language. Real repository-relative paths are required by the relevance model.
|
| 105 |
|
| 106 |
### Score Parquet files
|
| 107 |
|
|
|
|
| 121 |
--apply-policy
|
| 122 |
```
|
| 123 |
|
| 124 |
+
The output mirrors the input directory structure and preserves all existing columns. Use `--describe` to inspect a scorer set, `--limit N` to process at most `N` Parquet files, or `--algo-rel-min`, `--quality-min`, and `--exclude-categories` to override policy values.
|
| 125 |
|
| 126 |
### Output fields
|
| 127 |
|
| 128 |
+
The classifier appends the following results to each input file:
|
| 129 |
+
|
| 130 |
```text
|
| 131 |
category: {Predicted file role}
|
| 132 |
+
cls_confidence: {File role confidence}
|
| 133 |
algo_rel_score: {Algorithmic-relevance score}
|
| 134 |
+
quality_score: {Code quality score}
|
| 135 |
selected: {Whether the file passes the selection policy}
|
| 136 |
```
|
| 137 |
|
| 138 |
+
- `category` is one of `ALGO`, `WEB`, `TOOL`, `DATA`, `TEST`, `CONFIG`, or `EXCLUDE`.
|
| 139 |
+
- `cls_confidence` and `algo_rel_score` range from `0` to `1`; `quality_score` ranges from `0` to `10`.
|
| 140 |
+
- `selected` is added only when `--apply-policy` is enabled. It marks rows that pass the relevance threshold, the `EXCLUDE` veto, and the quality threshold; rows are not deleted.
|
| 141 |
|
| 142 |
### Notes
|
| 143 |
|
| 144 |
+
- Inputs must use 1024-dimensional embeddings produced by [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B).
|
| 145 |
+
- The relevance scorer also uses the repository-relative path. Use real paths aligned with the embeddings.
|
| 146 |
+
- [`config.json`](config.json) provides defaults. Command-line options can override these defaults.
|
| 147 |
+
- The scoring script is single-process. For corpus-scale inference, schedule multiple shards or subdirectories externally.
|
| 148 |
|
| 149 |
## ❤️ Acknowledgements
|
| 150 |
|
| 151 |
+
This release builds on [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B), [PyTorch](https://pytorch.org/), [PyArrow](https://arrow.apache.org/docs/python/). Thanks for their awesome work! Open-source contributions make UltraData-Code possible! 🙌
|
| 152 |
|
| 153 |
## 💳 License
|
| 154 |
|
README_ZH.md
CHANGED
|
@@ -8,54 +8,45 @@
|
|
| 8 |
<a href="https://huggingface.co/datasets/openbmb/UltraData-Code">🤗 UltraData-Code 数据集</a>
|
| 9 |
</p>
|
| 10 |
|
| 11 |
-
<p align="center"><a href="https://huggingface.co/
|
| 12 |
|
| 13 |
## 📚 简介
|
| 14 |
|
| 15 |
-
***UltraData-Code-L2-Classifier*** 是
|
| 16 |
|
| 17 |
-
对每种语言
|
| 18 |
-
|
| 19 |
-
## 📢 最新动态
|
| 20 |
-
|
| 21 |
-
- **[2026.09.07]** ***UltraData-Code-L2-Classifier*** 正式发布,同步开源 [***UltraData-Code***](https://huggingface.co/datasets/openbmb/UltraData-Code)(**L2** 约 400B tokens / **L3** 约 150B tokens,覆盖 11 种语言)。🚀🚀🚀
|
| 22 |
-
- **[2026.09.07]** ***[MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B) 正式发布!*** 作为 MiniCPM5 系列在 MiniCPM5-1B 之后的第二款模型。🚀🚀🚀
|
| 23 |
-
- **[2026.02.08]** [***UltraData***](https://ultradata.openbmb.cn/) 平台正式上线,并发布 [L0-L4 分级数据管理框架](https://arxiv.org/pdf/2602.09003)。🔍🔍🔍
|
| 24 |
|
| 25 |
## 💡 亮点
|
| 26 |
|
| 27 |
-
> **摘要:** 代码生成
|
| 28 |
|
| 29 |
-
- **完整的代码数据构建流水线:** 将仓库归档和规范化自然代码与细粒度精
|
| 30 |
-
- **细粒度算法相关代码精
|
| 31 |
-
- **
|
| 32 |
|
| 33 |
<div align="center">
|
| 34 |
<img src="assets/ultradata-code-l0-l3-overview.png" width="760" alt="UltraData-Code L0-L3 构建流程"/>
|
| 35 |
</div>
|
| 36 |
|
| 37 |
-
## 🔎 L2 筛选
|
| 38 |
|
| 39 |
-
L2
|
| 40 |
|
| 41 |
-
1. **编码文件
|
| 42 |
-
2. **预测文件角色
|
| 43 |
-
3. **构造双线索监督
|
| 44 |
-
4. **
|
| 45 |
-
5. **校准并筛选
|
| 46 |
|
| 47 |
<div align="center">
|
| 48 |
-
<img src="assets/ultradata-code-l2.png" width="760" alt="UltraData-Code L2 筛选
|
| 49 |
</div>
|
| 50 |
|
| 51 |
## 📈 评测结果
|
| 52 |
|
| 53 |
-
10B-token 对比均使用相同的 1B 基座模型、
|
| 54 |
-
|
| 55 |
-
### Python 结果
|
| 56 |
|
| 57 |
-
|
| 58 |
-
- **结构化合成提升等 token 混合训练效果。** UltraData-Code-L2-py 与 UltraData-Code-L3-py 的等 token 混合训练平均 pass@1 达到 **46.43**,比仅使用 L2 高 **3.30 个百分点**,比对应的 L2 + SwallowCode-v2 混合训练高 **4.46 个百分点**。
|
| 59 |
|
| 60 |
<div align="center">
|
| 61 |
<img src="assets/python-training-dynamics.png" alt="Python 训练动态" width="760"/>
|
|
@@ -65,11 +56,7 @@ L2 框架分别独立应用于 UltraData-Code-L1 中的 11 种语言:
|
|
| 65 |
<img src="assets/python_results.png" alt="Python 主结果表" width="900"/>
|
| 66 |
</div>
|
| 67 |
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
- **L2 精选优于自然代码基线。** UltraData-Code-L2 相比 UltraData-Code-L1 在 EvalPlus 和 MultiPL-E 上分别提升 **7.80** 和 **5.13 个百分点**,相比 Stack-Edu 分别提升 **4.37** 和 **3.05 个百分点**。
|
| 71 |
-
- **L3 合成带来进一步提升。** 将一半 L2 训练 token 替换为 L3 后,相比仅使用 L2 的训练,EvalPlus 和 MultiPL-E 分别提升 **8.42** 和 **8.07 个百分点**。在相同的 1:1 token 配比下,UltraData-Code-L2-L3 混合训练相比最强的已报告合成数据基线,两个基准分别提升 **5.57** 和 **7.80 个百分点**。
|
| 72 |
-
- **优势在规模扩展中持续存在。** 在 100B tokens 训练规模下,L2-L3 混合训练在 EvalPlus 和 MultiPL-E 上分别达到 57.06 和 39.54,相比仅使用 L2 的训练分别高 10.11 和 12.39 个百分点,相比对应的 CodeAlchemy-code-qa 混合训练分别高 10.86 和 9.60 个百分点。
|
| 73 |
|
| 74 |
<div align="center">
|
| 75 |
<img src="assets/multilingual-training-dynamics.png" alt="多语言训练动态" width="760"/>
|
|
@@ -79,7 +66,9 @@ L2 框架分别独立应用于 UltraData-Code-L1 中的 11 种语言:
|
|
| 79 |
<img src="assets/multilingual_results.png" alt="多语言主结果表" width="900"/>
|
| 80 |
</div>
|
| 81 |
|
| 82 |
-
## 🚀 使用方法
|
|
|
|
|
|
|
| 83 |
|
| 84 |
```bash
|
| 85 |
pip install numpy torch pyarrow
|
|
@@ -94,15 +83,15 @@ from scorers import LanguageScorers
|
|
| 94 |
|
| 95 |
scorers = LanguageScorers.load("classifiers/cpp")
|
| 96 |
|
| 97 |
-
# float32 [N, 1024];paths
|
| 98 |
out = scorers.score(embeddings, paths)
|
| 99 |
```
|
| 100 |
|
| 101 |
-
请使用
|
| 102 |
|
| 103 |
-
###
|
| 104 |
|
| 105 |
-
[`scripts/score_parquet.py`](scripts/score_parquet.py) 可处理单个 Parquet 文件
|
| 106 |
|
| 107 |
```bash
|
| 108 |
SCORERS=classifiers/cpp
|
|
@@ -118,10 +107,12 @@ python scripts/score_parquet.py \
|
|
| 118 |
--apply-policy
|
| 119 |
```
|
| 120 |
|
| 121 |
-
输出会镜像输入目录结构,并保留原有列。使用 `--describe` 查看打分器配置,使用 `--limit N`
|
| 122 |
|
| 123 |
### 输出字段
|
| 124 |
|
|
|
|
|
|
|
| 125 |
```text
|
| 126 |
category: {预测的文件角色}
|
| 127 |
cls_confidence: {文件角色置信度}
|
|
@@ -130,20 +121,20 @@ quality_score: {代码质量分数}
|
|
| 130 |
selected: {是否通过筛选策略}
|
| 131 |
```
|
| 132 |
|
| 133 |
-
- `category`
|
| 134 |
-
- `cls_confidence`
|
| 135 |
-
-
|
| 136 |
|
| 137 |
### 注意事项
|
| 138 |
|
| 139 |
-
-
|
| 140 |
-
- 相关性打分器还会使用仓库相对路径
|
| 141 |
-
- [`config.json`](config.json) 默认值
|
| 142 |
-
-
|
| 143 |
|
| 144 |
## ❤️ 致谢
|
| 145 |
|
| 146 |
-
本
|
| 147 |
|
| 148 |
## 💳 许可证
|
| 149 |
|
|
|
|
| 8 |
<a href="https://huggingface.co/datasets/openbmb/UltraData-Code">🤗 UltraData-Code 数据集</a>
|
| 9 |
</p>
|
| 10 |
|
| 11 |
+
<p align="center"><a href="https://huggingface.co/openbmb/UltraData-Code-L2-Classifier/blob/main/README.md">English</a> | 中文</p>
|
| 12 |
|
| 13 |
## 📚 简介
|
| 14 |
|
| 15 |
+
***UltraData-Code-L2-Classifier*** 是一组编程语言特定的文件级打分器,针对 [UltraData-Code-L1](https://huggingface.co/datasets/openbmb/UltraData-Code) 中的 11 种编程语言分别设计。
|
| 16 |
|
| 17 |
+
我们针对每种语言训练并应用对应的 L2 打分器,对 UltraData-Code-L1 中相应语言的文件进行筛选。筛选后的文件共同构成 [UltraData-Code-L2](https://huggingface.co/datasets/openbmb/UltraData-Code),其目标是集中可解释的计算与问题求解逻辑。最终 L2 数据约包含 **400B tokens**,总体保留 L1 约 **12.23% 的文件**。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
|
| 19 |
## 💡 亮点
|
| 20 |
|
| 21 |
+
> **摘要:** 代码生成逐渐成为大语言模型的核心能力,而代码数据则是培养这一能力的预训练过程中的关键组成部分。随着代码语料规模不断增长,其规模、多样性与质量日益影响模型在预训练阶段习得的能力。遵循分级数据治理视角,我们提出 **UltraData-Code**,将其构建为由四个相互衔接的数据状态组成的数据家族:从 L0 的代码仓库归档,到 L1 的标准化自然代码、L2 的算法导向筛选,再到 L3 的任务导向合成。L0 归档每个公开 GitHub 仓库默认分支上的最新版本,并保留其文件结构、关联关系与来源信息。L1 通过可扩展的过滤、规范化与近重复去重,获得标准化自然代码。随后,L2 基于由文件角色与启发式规则构成的语言自适应信号,并结合代码质量约束,从 L1 中筛选与算法相关的文件,最终得到覆盖 11 种编程语言、规模约为 400B tokens 的 **UltraData-Code-L2**。L3 对 L2 中的算法文件实施任务导向合成,将每个实现转化为结构化编程练习,生成覆盖相同 11 种语言、规模约为 150B tokens 的 **UltraData-Code-L3**。在相同的 10B-token 持续预训练设置下,对一个 1B 模型使用 L2 而非 L1 进行训练,可使其在 EvalPlus 和 MultiPL-E 上的 pass@1 分别提高 **7.80 个百分点** 和 **5.13 个百分点**,同时分别超过 Stack-Edu 4.37 个百分点和 3.05 个百分点。将一半的 L2 训练 tokens 替换为 L3 后,相较于仅使用 L2 训练,模型在 EvalPlus 和 MultiPL-E 上可进一步提升 **8.42 个百分点** 和 **8.07 个百分点**,同时分别超过最强的合成数据基线 5.57 个百分点和 7.80 个百分点。当训练预算增加至 100B tokens 时,L2 筛选与 L3 合成在两个基准上的增益均进一步扩大。
|
| 22 |
|
| 23 |
+
- **完整的代码数据构建流水线:** 将仓库归档和规范化自然代码与细粒度精筛、结构化合成连接起来,在 11 种编程语言上分别形成约 400B tokens 的 L2 和约 150B tokens 的 L3。
|
| 24 |
+
- **细粒度算法相关代码精筛:** 我们提出一种语言自适应筛选框架,将文件角色监督与语言特定启发式线索结合,学习超越显式 ALGO 文件的算法相关性。该框架在文件角色、算法相关性和代码质量模型之间复用预计算语义表征,最终结合角色与质量约束完成筛选。
|
| 25 |
+
- **任务导向合成:** 我们提出一种结构化合成方案,将 L2 筛选出的每个算法相关实现转化为一道编程练习,并基于同一源实现联合生成独立完整的任务、分析、解答与候选测试用例。该转换既保留了源代码的计算意图,又为代码生成引入了显式的任务监督与解答监督。
|
| 26 |
|
| 27 |
<div align="center">
|
| 28 |
<img src="assets/ultradata-code-l0-l3-overview.png" width="760" alt="UltraData-Code L0-L3 构建流程"/>
|
| 29 |
</div>
|
| 30 |
|
| 31 |
+
## 🔎 L2 筛选流程
|
| 32 |
|
| 33 |
+
L2 流程分别应用于 UltraData-Code-L1 中的 11 种语言:
|
| 34 |
|
| 35 |
+
1. **编码文件:** 使用 [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) 为每个文件生成 1024 维 embedding。
|
| 36 |
+
2. **预测文件角色:** 将文件分为 `ALGO`、`WEB`、`TOOL`、`DATA`、`TEST`、`CONFIG` 或 `EXCLUDE`。
|
| 37 |
+
3. **构造双线索监督:** 将 ALGO 角色标注与语言特定启发式证据结合,覆盖 ALGO 之外的算法相关文件。
|
| 38 |
+
4. **预测相关性和质量:** 相关性模型输出 `algo_rel_score`,质量模型输出 `quality_score`。
|
| 39 |
+
5. **校准并筛选:** 在留出数据上为每种语言校准相关性阈值。文件需通过相关性阈值和质量阈值且不被预测为 `EXCLUDE`,才会保留。
|
| 40 |
|
| 41 |
<div align="center">
|
| 42 |
+
<img src="assets/ultradata-code-l2.png" width="760" alt="UltraData-Code L2 筛选流程"/>
|
| 43 |
</div>
|
| 44 |
|
| 45 |
## 📈 评测结果
|
| 46 |
|
| 47 |
+
10B-token 对比均使用相同的 1B 基座模型、训练设置、去污流程和评测协议。
|
|
|
|
|
|
|
| 48 |
|
| 49 |
+
**Python 结果:算法相关性精筛带来最强的自然代码结果。** UltraData-Code-L2-py 的 EvalPlus 平均 pass@1 比 UltraData-Code-L1-py 提升 **17.91 个百分点**,比 Stack-Edu-py 提升 **8.46 个百分点**。
|
|
|
|
| 50 |
|
| 51 |
<div align="center">
|
| 52 |
<img src="assets/python-training-dynamics.png" alt="Python 训练动态" width="760"/>
|
|
|
|
| 56 |
<img src="assets/python_results.png" alt="Python 主结果表" width="900"/>
|
| 57 |
</div>
|
| 58 |
|
| 59 |
+
**多语言结果:L2 精筛优于自然代码基线。** UltraData-Code-L2 相比 UltraData-Code-L1 在 EvalPlus 和 MultiPL-E 上分别提升 **7.80** 和 **5.13 个百分点**,相比 Stack-Edu 分别提升 **4.37** 和 **3.05 个百分点**。
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
|
| 61 |
<div align="center">
|
| 62 |
<img src="assets/multilingual-training-dynamics.png" alt="多语言训练动态" width="760"/>
|
|
|
|
| 66 |
<img src="assets/multilingual_results.png" alt="多语言主结果表" width="900"/>
|
| 67 |
</div>
|
| 68 |
|
| 69 |
+
## 🚀 UltraData-Code-L2 打分器使用方法
|
| 70 |
+
|
| 71 |
+
安装依赖:
|
| 72 |
|
| 73 |
```bash
|
| 74 |
pip install numpy torch pyarrow
|
|
|
|
| 83 |
|
| 84 |
scorers = LanguageScorers.load("classifiers/cpp")
|
| 85 |
|
| 86 |
+
# float32 [N, 1024];paths 与 embeddings 顺序一致,使用仓库相对路径
|
| 87 |
out = scorers.score(embeddings, paths)
|
| 88 |
```
|
| 89 |
|
| 90 |
+
请为每种输入语言使用对应的语言目录。相关性模型需要真实的仓库相对路径。
|
| 91 |
|
| 92 |
+
### Parquet 文件打分
|
| 93 |
|
| 94 |
+
[`scripts/score_parquet.py`](scripts/score_parquet.py) 可处理单个 Parquet 文件或递归处理目录。输入必须包含 `embedding` 列,以及 `relative_path` 列或包含 `file_path` 的 `meta` 结构。
|
| 95 |
|
| 96 |
```bash
|
| 97 |
SCORERS=classifiers/cpp
|
|
|
|
| 107 |
--apply-policy
|
| 108 |
```
|
| 109 |
|
| 110 |
+
输出会镜像输入目录结构,并保留所有原有列。使用 `--describe` 查看打分器配置,使用 `--limit N` 限制处理的 Parquet 文件数,或使用 `--algo-rel-min`、`--quality-min` 和 `--exclude-categories` 覆盖策略值。
|
| 111 |
|
| 112 |
### 输出字段
|
| 113 |
|
| 114 |
+
打分器会在输入文件中追加以下结果:
|
| 115 |
+
|
| 116 |
```text
|
| 117 |
category: {预测的文件角色}
|
| 118 |
cls_confidence: {文件角色置信度}
|
|
|
|
| 121 |
selected: {是否通过筛选策略}
|
| 122 |
```
|
| 123 |
|
| 124 |
+
- `category` 为 `ALGO`、`WEB`、`TOOL`、`DATA`、`TEST`、`CONFIG` 或 `EXCLUDE` 之一。
|
| 125 |
+
- `cls_confidence` 和 `algo_rel_score` 的范围为 `0` 到 `1`,`quality_score` 的范围为 `0` 到 `10`。
|
| 126 |
+
- 只有启用 `--apply-policy` 时才会添加 `selected`。该字段标记同时通过相关性阈值、`EXCLUDE` veto 和质量阈值的行,不会删除未通过的行。
|
| 127 |
|
| 128 |
### 注意事项
|
| 129 |
|
| 130 |
+
- 输入必须使用 [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) 生成的 1024 维 embedding。
|
| 131 |
+
- 算法相关性打分器还会使用仓库相对路径,请传入与 embedding 顺序一致的真实路径。
|
| 132 |
+
- [`config.json`](config.json) 提供通用默认值。命令行参数可以覆盖这些默认值。
|
| 133 |
+
- 当前打分脚本为单进程实现。大规模语料推理时,建议在外部按数据分片或子目录进行调度。
|
| 134 |
|
| 135 |
## ❤️ 致谢
|
| 136 |
|
| 137 |
+
本项目基于 [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B)、[PyTorch](https://pytorch.org/)、[PyArrow](https://arrow.apache.org/docs/python/) 。感谢这些优秀开源工作,正是开源社区的贡献让 UltraData-Code 成为可能!🙌
|
| 138 |
|
| 139 |
## 💳 许可证
|
| 140 |
|