BigDong commited on
Commit
f18137a
·
1 Parent(s): 2c36e1e

update README

Browse files
Files changed (2) hide show
  1. README.md +27 -36
  2. README_ZH.md +36 -45
README.md CHANGED
@@ -22,27 +22,21 @@ language:
22
  <a href="https://huggingface.co/datasets/openbmb/UltraData-Code">🤗 UltraData-Code Dataset</a>
23
  </p>
24
 
25
- <p align="center">English | <a href="https://huggingface.co/datasets/openbmb/UltraData-Code-L2-Classifier/blob/main/README_ZH.md">中文</a></p>
26
 
27
  ## 📚 Introduction
28
 
29
- ***UltraData-Code-L2-Classifier*** is a suite of language-specific, file-level scorers for the 11 programming languages covered by [UltraData-Code](https://huggingface.co/datasets/openbmb/UltraData-Code). It is the L2 selection model used to build **UltraData-Code-L2** from UltraData-Code-L1.
30
 
31
- For each language, a dedicated scorer ranks L1 files by algorithmic relevance and code quality. Selected files form UltraData-Code-L2 (~**400B tokens**), retaining about **12.23%** of L1 files overall. The goal is to concentrate interpretable computation and problem-solving logic for code pretraining.
32
-
33
- ## 📢 What's New
34
-
35
- - **[2026.09.07]** ***UltraData-Code-L2-Classifier*** is released, together with [***UltraData-Code***](https://huggingface.co/datasets/openbmb/UltraData-Code) (**L2** \~400B tokens / **L3** \~150B tokens across 11 languages). 🚀🚀🚀
36
- - **[2026.09.07]** ***[MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B) is released!***, the second model in the MiniCPM5 series after MiniCPM5-1B. 🚀🚀🚀
37
- - **[2026.02.08]** The [***UltraData***](https://ultradata.openbmb.cn/) platform is now live, introducing the [L0-L4 tiered data management framework](https://arxiv.org/pdf/2602.09003). 🔍🔍🔍
38
 
39
  ## 💡 Highlights
40
 
41
- > **Abstract:** Code generation is a core capability of large language models, and code data is central to the pretraining process that develops it. As code corpora grow, their scale, diversity, and quality increasingly shape the capabilities learned during pretraining. Following a tiered data management perspective, we present **UltraData-Code** as a family of four connected data states: repository archival at L0, standardized natural code at L1, algorithmic selection at L2, and task-oriented synthesis at L3. L0 archives the latest revision on the default branch of each public GitHub repository with its file structure, relationships, and provenance. L1 applies scalable filtering, normalization, and near deduplication to obtain standardized natural code. L2 selects algorithmically relevant files from L1 using language-adaptive signals from file roles and heuristics, together with code-quality constraints, yielding approximately 400B tokens across 11 programming languages. L3 applies task-oriented synthesis to algorithmic files from L2, turning each implementation into a structured programming exercise with aligned task, analysis, solution, and test fields, and generating approximately 150B tokens in the same 11 languages. Under controlled 10B-token continual pretraining of a 1B model, training on L2 instead of L1 raises pass@1 on EvalPlus by **7.80 points** and on MultiPL-E by **5.13 points**, while exceeding Stack-Edu by 4.37 and 3.05 points, respectively. Replacing half of the L2 training tokens with L3 yields a further gain of **8.42 points** on EvalPlus and **8.07 points** on MultiPL-E over training on L2 alone, while exceeding the strongest synthetic data baseline by 5.57 and 7.80 points, respectively.
42
 
43
- - **A connected code data construction pipeline:** UltraData-Code links repository archival and standardized natural code with fine-grained selection and structured synthesis, yielding approximately 400B tokens at L2 and 150B tokens at L3 across 11 programming languages.
44
- - **Fine-grained curation of algorithmically relevant code:** We introduce file-level algorithmic relevance as a selection objective and combine file-role cues with language-specific heuristic evidence to identify computational content beyond files explicitly labeled as algorithm implementations.
45
- - **Task-oriented synthesis grounded in implementations:** Each selected implementation serves as the common source for a task, analysis, solution, and test record, preserving computational intent while adding task semantics and structured supervision.
46
 
47
  <div align="center">
48
  <img src="assets/ultradata-code-l0-l3-overview.png" width="760" alt="UltraData-Code L0-L3 construction pipeline"/>
@@ -64,12 +58,9 @@ The L2 framework is applied independently to each of the 11 languages in UltraDa
64
 
65
  ## 📈 Evaluation Results
66
 
67
- The 10B-token comparisons use the same 1B base model, architecture, optimization settings, preprocessing, decontamination, and evaluation protocol.
68
-
69
- ### Python Results
70
 
71
- - **Algorithmic selection yields the strongest natural code result.** UltraData-Code-L2-py improves the average EvalPlus pass@1 by **17.91 points** over UltraData-Code-L1-py and by **8.46 points** over Stack-Edu-py. Ultra-Stack-v3-py reaches the second-highest natural code average, while UltraData-Code-L2-py records the strongest average overall among the natural code settings.
72
- - **Structured synthesis improves the matched mixture.** The equal-token UltraData-Code-L2-py + UltraData-Code-L3-py mixture reaches a **46.43** average pass@1, **3.30 points** above training on L2 alone and **4.46 points** above the matched L2 + SwallowCode-v2 mixture.
73
 
74
  <div align="center">
75
  <img src="assets/python-training-dynamics.png" alt="Python training dynamics" width="760"/>
@@ -79,11 +70,7 @@ The 10B-token comparisons use the same 1B base model, architecture, optimization
79
  <img src="assets/python_results.png" alt="Python main results table" width="900"/>
80
  </div>
81
 
82
- ### Multilingual Results
83
-
84
- - **L2 selection improves over L1 baselines.** UltraData-Code-L2 exceeds UltraData-Code-L1 by **7.80 points** on EvalPlus and **5.13 points** on MultiPL-E, and exceeds Stack-Edu by **4.37** and **3.05 points**, respectively.
85
- - **L3 synthesis provides a further gain.** Replacing half of the L2 training tokens with L3 yields **8.42** and **8.07 points** over training on L2 alone on EvalPlus and MultiPL-E. Under the same 1:1 token mix, UltraData-Code-L2-L3 exceeds the strongest reported synthetic baseline by **5.57** and **7.80 points**, respectively.
86
- - **The advantage persists during scaling.** At 100B tokens, the L2-L3 mixture reaches 57.06 on EvalPlus and 39.54 on MultiPL-E, exceeding training on L2 alone by 10.11 and 12.39 points and the matched CodeAlchemy-code-qa mixture by 10.86 and 9.60 points.
87
 
88
  <div align="center">
89
  <img src="assets/multilingual-training-dynamics.png" alt="Multilingual training dynamics" width="760"/>
@@ -93,7 +80,9 @@ The 10B-token comparisons use the same 1B base model, architecture, optimization
93
  <img src="assets/multilingual_results.png" alt="Multilingual main results table" width="900"/>
94
  </div>
95
 
96
- ## 🚀 Usage
 
 
97
 
98
  ```bash
99
  pip install numpy torch pyarrow
@@ -112,7 +101,7 @@ scorers = LanguageScorers.load("classifiers/cpp")
112
  out = scorers.score(embeddings, paths)
113
  ```
114
 
115
- Use the matching language directory under `classifiers/` (`cpp`, `csharp`, `go`, `java`, `javascript`, `php`, `python`, `r`, `ruby`, `rust`, `shell`). Real repository-relative paths are required by the relevance model.
116
 
117
  ### Score Parquet files
118
 
@@ -132,32 +121,34 @@ python scripts/score_parquet.py \
132
  --apply-policy
133
  ```
134
 
135
- The output mirrors the input directory structure and keeps existing columns. Use `--describe` to inspect a scorer set, `--limit N` to process at most `N` Parquet files, or `--algo-rel-min`, `--quality-min`, and `--exclude-categories` to override policy values.
136
 
137
  ### Output fields
138
 
 
 
139
  ```text
140
  category: {Predicted file role}
141
- cls_confidence: {File-role confidence}
142
  algo_rel_score: {Algorithmic-relevance score}
143
- quality_score: {Code-quality score}
144
  selected: {Whether the file passes the selection policy}
145
  ```
146
 
147
- - `category` {`ALGO`, `WEB`, `TOOL`, `DATA`, `TEST`, `CONFIG`, `EXCLUDE`}
148
- - `cls_confidence` / `algo_rel_score` `[0, 1]`; `quality_score` `[0, 10]`
149
- - `selected` is written only with `--apply-policy`; it marks policy passes and does not delete rows
150
 
151
  ### Notes
152
 
153
- - Embeddings must be 1024-d from [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B).
154
- - The relevance scorer also uses the repository-relative path; keep paths aligned with embeddings.
155
- - [`config.json`](config.json) defaults: `algo_rel_min = 0.8`, `quality_min = 3.0`, veto category `EXCLUDE`. CLI flags override these.
156
- - `score_parquet.py` is single-process; shard externally for large corpora.
157
 
158
  ## ❤️ Acknowledgements
159
 
160
- This release builds on [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B), [PyTorch](https://pytorch.org/), and [PyArrow](https://arrow.apache.org/docs/python/). Thanks for their awesome work! 🙌
161
 
162
  ## 💳 License
163
 
 
22
  <a href="https://huggingface.co/datasets/openbmb/UltraData-Code">🤗 UltraData-Code Dataset</a>
23
  </p>
24
 
25
+ <p align="center">English | <a href="https://huggingface.co/openbmb/UltraData-Code-L2-Classifier/blob/main/README_ZH.md">中文</a></p>
26
 
27
  ## 📚 Introduction
28
 
29
+ ***UltraData-Code-L2-Classifier*** is a suite of language-specific, file-level scorers designed for the 11 programming languages represented in [UltraData-Code-L1](https://huggingface.co/datasets/openbmb/UltraData-Code).
30
 
31
+ For each language, we train and apply a dedicated L2 scorer to the corresponding files in UltraData-Code-L1. The selected files form [UltraData-Code-L2](https://huggingface.co/datasets/openbmb/UltraData-Code), a corpus designed to concentrate interpretable computation and problem-solving logic. The resulting L2 corpus contains approximately **400B tokens** and retains approximately **12.23% of L1 files** overall.
 
 
 
 
 
 
32
 
33
  ## 💡 Highlights
34
 
35
+ > **Abstract:** Code generation has become a core capability of large language models, and code data is a central part of the pre-training process that develops it. As code corpora continue to grow, their scale, diversity, and quality increasingly shape the capabilities learned during pre-training. Following a tiered data management perspective, we present **UltraData-Code** as a family of four connected data states, from repository archival at L0 through standardized natural code at L1, algorithmic selection at L2, and task-oriented synthesis at L3. L0 archives the latest revision on the default branch of each public GitHub repository with its file structure, relationships, and provenance. L1 applies scalable filtering, normalization, and near deduplication to obtain standardized natural code. L2 then selects algorithmically relevant files from L1 using language-adaptive signals from file roles and heuristics, together with code quality constraints, yielding approximately 400B tokens of **UltraData-Code-L2**, spanning 11 programming languages. L3 applies task-oriented synthesis to algorithmic files from L2, turning each implementation into a structured programming exercise, generating approximately 150B tokens of **UltraData-Code-L3** in the same 11 languages. Under controlled 10B-token continual pre-training of a 1B model, training on L2 instead of L1 raises pass@1 on EvalPlus by **7.80 points** and on MultiPL-E by **5.13 points**, while exceeding Stack-Edu by 4.37 and 3.05 points, respectively. Replacing half of the L2 training tokens with L3 yields a further gain of **8.42 points** on EvalPlus and **8.07 points** on MultiPL-E over L2-only training, while exceeding the strongest synthetic data baseline by 5.57 and 7.80 points, respectively. When the training budget increases to 100B tokens, gains from L2 selection and L3 synthesis further widen on both benchmarks.
36
 
37
+ - **A connected code data construction pipeline.** UltraData-Code links repository archival and standardized natural code with selection and structured synthesis, yielding approximately 400B tokens at L2 and 150B tokens at L3 across 11 programming languages.
38
+ - **Fine-grained curation of algorithmically relevant code.** We develop a language-adaptive selection framework that combines file role supervision with language-specific heuristic cues to learn algorithmic relevance beyond explicit ALGO files. The framework reuses precomputed semantic embeddings across role, relevance, and quality models, and applies role and quality constraints during selection.
39
+ - **Implementation-grounded, task-oriented synthesis.** We introduce a structured synthesis protocol that converts each algorithmically relevant implementation selected by L2 into a programming exercise, jointly generating a standalone task, analysis, solution, and test candidates from the same source implementation. This transformation preserves the source code's computational intent while adding explicit task and solution supervision for code generation.
40
 
41
  <div align="center">
42
  <img src="assets/ultradata-code-l0-l3-overview.png" width="760" alt="UltraData-Code L0-L3 construction pipeline"/>
 
58
 
59
  ## 📈 Evaluation Results
60
 
61
+ The 10B-token comparisons use the same 1B foundation model, training settings, decontamination, and evaluation protocol.
 
 
62
 
63
+ **Python results: Algorithmic selection yields the strongest natural code result.** UltraData-Code-L2-py improves the average EvalPlus pass@1 by **17.91 points** over UltraData-Code-L1-py and by **8.46 points** over Stack-Edu-py.
 
64
 
65
  <div align="center">
66
  <img src="assets/python-training-dynamics.png" alt="Python training dynamics" width="760"/>
 
70
  <img src="assets/python_results.png" alt="Python main results table" width="900"/>
71
  </div>
72
 
73
+ **Multilingual results: L2 selection improves over L1 baselines.** UltraData-Code-L2 exceeds UltraData-Code-L1 by **7.80 points** on EvalPlus and **5.13 points** on MultiPL-E, and exceeds Stack-Edu by **4.37** and **3.05 points**, respectively.
 
 
 
 
74
 
75
  <div align="center">
76
  <img src="assets/multilingual-training-dynamics.png" alt="Multilingual training dynamics" width="760"/>
 
80
  <img src="assets/multilingual_results.png" alt="Multilingual main results table" width="900"/>
81
  </div>
82
 
83
+ ## 🚀 Usage of UltraData-Code-L2 Classifier
84
+
85
+ Install the runtime dependencies:
86
 
87
  ```bash
88
  pip install numpy torch pyarrow
 
101
  out = scorers.score(embeddings, paths)
102
  ```
103
 
104
+ Use the matching language directory for each input language. Real repository-relative paths are required by the relevance model.
105
 
106
  ### Score Parquet files
107
 
 
121
  --apply-policy
122
  ```
123
 
124
+ The output mirrors the input directory structure and preserves all existing columns. Use `--describe` to inspect a scorer set, `--limit N` to process at most `N` Parquet files, or `--algo-rel-min`, `--quality-min`, and `--exclude-categories` to override policy values.
125
 
126
  ### Output fields
127
 
128
+ The classifier appends the following results to each input file:
129
+
130
  ```text
131
  category: {Predicted file role}
132
+ cls_confidence: {File role confidence}
133
  algo_rel_score: {Algorithmic-relevance score}
134
+ quality_score: {Code quality score}
135
  selected: {Whether the file passes the selection policy}
136
  ```
137
 
138
+ - `category` is one of `ALGO`, `WEB`, `TOOL`, `DATA`, `TEST`, `CONFIG`, or `EXCLUDE`.
139
+ - `cls_confidence` and `algo_rel_score` range from `0` to `1`; `quality_score` ranges from `0` to `10`.
140
+ - `selected` is added only when `--apply-policy` is enabled. It marks rows that pass the relevance threshold, the `EXCLUDE` veto, and the quality threshold; rows are not deleted.
141
 
142
  ### Notes
143
 
144
+ - Inputs must use 1024-dimensional embeddings produced by [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B).
145
+ - The relevance scorer also uses the repository-relative path. Use real paths aligned with the embeddings.
146
+ - [`config.json`](config.json) provides defaults. Command-line options can override these defaults.
147
+ - The scoring script is single-process. For corpus-scale inference, schedule multiple shards or subdirectories externally.
148
 
149
  ## ❤️ Acknowledgements
150
 
151
+ This release builds on [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B), [PyTorch](https://pytorch.org/), [PyArrow](https://arrow.apache.org/docs/python/). Thanks for their awesome work! Open-source contributions make UltraData-Code possible! 🙌
152
 
153
  ## 💳 License
154
 
README_ZH.md CHANGED
@@ -8,54 +8,45 @@
8
  <a href="https://huggingface.co/datasets/openbmb/UltraData-Code">🤗 UltraData-Code 数据集</a>
9
  </p>
10
 
11
- <p align="center"><a href="https://huggingface.co/datasets/openbmb/UltraData-Code-L2-Classifier/blob/main/README.md">English</a> | 中文</p>
12
 
13
  ## 📚 简介
14
 
15
- ***UltraData-Code-L2-Classifier*** 是面向 [UltraData-Code](https://huggingface.co/datasets/openbmb/UltraData-Code) 覆盖的 11 种编程语言的一组语言特定、文件级打它是从 UltraData-Code-L1 构建 **UltraData-Code-L2** 所使用的 L2 筛选模型。
16
 
17
- 对每种语言,专用打分器按算法相关性和代码质量对 L1 文件排序。筛选后的文件构成 UltraData-Code-L2约 **400B tokens**,总体保留 L1 约 **12.23%** 的文件。目标是为代码预训练集中可解释的计算与问题求解逻辑。
18
-
19
- ## 📢 最新动态
20
-
21
- - **[2026.09.07]** ***UltraData-Code-L2-Classifier*** 正式发布,同步开源 [***UltraData-Code***](https://huggingface.co/datasets/openbmb/UltraData-Code)(**L2** 约 400B tokens / **L3** 约 150B tokens,覆盖 11 种语言)。🚀🚀🚀
22
- - **[2026.09.07]** ***[MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B) 正式发布!*** 作为 MiniCPM5 系列在 MiniCPM5-1B 之后的第二款模型。🚀🚀🚀
23
- - **[2026.02.08]** [***UltraData***](https://ultradata.openbmb.cn/) 平台正式上线,并发布 [L0-L4 分级数据管理框架](https://arxiv.org/pdf/2602.09003)。🔍🔍🔍
24
 
25
  ## 💡 亮点
26
 
27
- > **摘要:** 代码生成大语言模型的核心能力,代码数据是培养能力的预训练过程的重要组成部分。随着代码语料规模不断增长,其规模、多样性质量日益决定模型在预训练中学的能力。遵循分级数据治理视角,我们 **UltraData-Code** 构建为四个相互接的数据状态:L0 仓库归档L1 规范化自然代码、L2 算法相关性精 L3 面向任务合成。L0 保留每个公开 GitHub 仓库默认分支的最新版本其文件结构、文件关系来源信息。L1 通过规模化过滤、规范化近重复去重得标准化自然代码。L2 基于文件角色语言自适应启发式信号代码质量约束,从 L1 中筛选算法相关文件,覆盖 11 种编程语言、约 400B tokens。L3 L2 中的算法相关文件为基础,将每个实现转任务、分析、解答和测试字段对齐的结构化编程练习,覆盖相同 11 种语言、约 150B tokens。在相同的 1B 模型和 10B-token 持续预训练设置下,使用 L2 替代 L1 ,EvalPlus 的 pass@1 提 **7.80 个百分点**,MultiPL-E 提升 **5.13 个百分点**,相比 Stack-Edu 分别提升 4.37 和 3.05 个百分点。将一半 L2 训练 token 替换为 L3 后,相仅使用 L2 训练,EvalPlus 进一步提升 **8.42 个百分点**,MultiPL-E 进一步提升 **8.07 个百分点**,相比最强的合成数据基线分别提升 5.57 和 7.80 个百分点。
28
 
29
- - **完整的代码数据构建流水线:** 将仓库归档和规范化自然代码与细粒度精、结构化合成连接起来,在 11 种编程语言上分别形成约 400B tokens 的 L2 和约 150B tokens 的 L3。
30
- - **细粒度算法相关代码精:** 将文件级算法相关性定义为择目标结合文件角色线索与语言特定启发式证据识别显式算法实现标签范围的计算内容
31
- - **面向任务且以实现为基础的合成:** 每个精选实现为任务、分析、解答测试记录的共同来源,在保留计算意图的同时补充任务语义和结构化监督。
32
 
33
  <div align="center">
34
  <img src="assets/ultradata-code-l0-l3-overview.png" width="760" alt="UltraData-Code L0-L3 构建流程"/>
35
  </div>
36
 
37
- ## 🔎 L2 筛选框架
38
 
39
- L2 框架分别独立应用于 UltraData-Code-L1 中的 11 种语言:
40
 
41
- 1. **编码文件** 使用 [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) 为每个文件生成 1024 维 embedding。
42
- 2. **预测文件角色** 角色分类器将文件分为 `ALGO`、`WEB`、`TOOL`、`DATA`、`TEST`、`CONFIG` 或 `EXCLUDE`。
43
- 3. **构造双线索监督** 将 ALGO 角色标注与语言特定启发式证据结合,识别 ALGO 角色之外的算法相关文件。
44
- 4. **打分相关性质量** 相关性模型输出 `algo_rel_score`,质量模型输出 `quality_score`。
45
- 5. **校准并筛选** 在留出数据上为每种语言校准相关性阈值。文件需通过相关性和质量阈值被预测为 `EXCLUDE`,才会保留。
46
 
47
  <div align="center">
48
- <img src="assets/ultradata-code-l2.png" width="760" alt="UltraData-Code L2 筛选框架"/>
49
  </div>
50
 
51
  ## 📈 评测结果
52
 
53
- 10B-token 对比均使用相同的 1B 基座模型、模型架构、优化设置、预处理、去污流程和评测协议。
54
-
55
- ### Python 结果
56
 
57
- - **算法相关性精带来最强的自然代码结果。** UltraData-Code-L2-py 的 EvalPlus 平均 pass@1 比 UltraData-Code-L1-py 提升 **17.91 个百分点**,比 Stack-Edu-py 提升 **8.46 个百分点**。Ultra-Stack-v3-py 在自然代码设置中取得第二高的平均分,而 UltraData-Code-L2-py 在自然代码设置中取得最高平均分。
58
- - **结构化合成提升等 token 混合训练效果。** UltraData-Code-L2-py 与 UltraData-Code-L3-py 的等 token 混合训练平均 pass@1 达到 **46.43**,比仅使用 L2 高 **3.30 个百分点**,比对应的 L2 + SwallowCode-v2 混合训练高 **4.46 个百分点**。
59
 
60
  <div align="center">
61
  <img src="assets/python-training-dynamics.png" alt="Python 训练动态" width="760"/>
@@ -65,11 +56,7 @@ L2 框架分别独立应用于 UltraData-Code-L1 中的 11 种语言:
65
  <img src="assets/python_results.png" alt="Python 主结果表" width="900"/>
66
  </div>
67
 
68
- ### 多语言结果
69
-
70
- - **L2 精选优于自然代码基线。** UltraData-Code-L2 相比 UltraData-Code-L1 在 EvalPlus 和 MultiPL-E 上分别提升 **7.80** 和 **5.13 个百分点**,相比 Stack-Edu 分别提升 **4.37** 和 **3.05 个百分点**。
71
- - **L3 合成带来进一步提升。** 将一半 L2 训练 token 替换为 L3 后,相比仅使用 L2 的训练,EvalPlus 和 MultiPL-E 分别提升 **8.42** 和 **8.07 个百分点**。在相同的 1:1 token 配比下,UltraData-Code-L2-L3 混合训练相比最强的已报告合成数据基线,两个基准分别提升 **5.57** 和 **7.80 个百分点**。
72
- - **优势在规模扩展中持续存在。** 在 100B tokens 训练规模下,L2-L3 混合训练在 EvalPlus 和 MultiPL-E 上分别达到 57.06 和 39.54,相比仅使用 L2 的训练分别高 10.11 和 12.39 个百分点,相比对应的 CodeAlchemy-code-qa 混合训练分别高 10.86 和 9.60 个百分点。
73
 
74
  <div align="center">
75
  <img src="assets/multilingual-training-dynamics.png" alt="多语言训练动态" width="760"/>
@@ -79,7 +66,9 @@ L2 框架分别独立应用于 UltraData-Code-L1 中的 11 种语言:
79
  <img src="assets/multilingual_results.png" alt="多语言主结果表" width="900"/>
80
  </div>
81
 
82
- ## 🚀 使用方法
 
 
83
 
84
  ```bash
85
  pip install numpy torch pyarrow
@@ -94,15 +83,15 @@ from scorers import LanguageScorers
94
 
95
  scorers = LanguageScorers.load("classifiers/cpp")
96
 
97
- # float32 [N, 1024];paths 仓库相对路径,并与 embeddings 对齐
98
  out = scorers.score(embeddings, paths)
99
  ```
100
 
101
- 请使用 `classifiers/` 下对应语言目录(`cpp`、`csharp`、`go`、`java`、`javascript`、`php`、`python`、`r`、`ruby`、`rust`、`shell`)。相关性模型需要真实的仓库相对路径。
102
 
103
- ### Parquet 文件打分
104
 
105
- [`scripts/score_parquet.py`](scripts/score_parquet.py) 可处理单个 Parquet 文件或递归处理目录。输入必须包含 `embedding` 列,以及 `relative_path` 或 `meta.file_path`。
106
 
107
  ```bash
108
  SCORERS=classifiers/cpp
@@ -118,10 +107,12 @@ python scripts/score_parquet.py \
118
  --apply-policy
119
  ```
120
 
121
- 输出会镜像输入目录结构,并保留原有列。使用 `--describe` 查看打分器配置,使用 `--limit N` 最多处理 `N` 个 Parquet 文件,或使用 `--algo-rel-min`、`--quality-min` 和 `--exclude-categories` 覆盖策略值。
122
 
123
  ### 输出字段
124
 
 
 
125
  ```text
126
  category: {预测的文件角色}
127
  cls_confidence: {文件角色置信度}
@@ -130,20 +121,20 @@ quality_score: {代码质量分数}
130
  selected: {是否通过筛选策略}
131
  ```
132
 
133
- - `category` {`ALGO`, `WEB`, `TOOL`, `DATA`, `TEST`, `CONFIG`, `EXCLUDE`}
134
- - `cls_confidence` / `algo_rel_score` `[0, 1]``quality_score` `[0, 10]`
135
- - 仅在启用 `--apply-policy` 时写入 `selected`;它标记通过策略的行,不会删除未通过的行
136
 
137
  ### 注意事项
138
 
139
- - embedding 必须来自 [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) 的 1024 维表示
140
- - 相关性打分器还会使用仓库相对路径保持路径与 embedding 对齐
141
- - [`config.json`](config.json) 默认值:`algo_rel_min = 0.8`、`quality_min = 3.0`,veto 类别为 `EXCLUDE`。命令行参数优先
142
- - `score_parquet.py` 为单进程实现大规模语料在外部分片���度。
143
 
144
  ## ❤️ 致谢
145
 
146
- 发布基于 [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B)、[PyTorch](https://pytorch.org/)[PyArrow](https://arrow.apache.org/docs/python/)。感谢这些优秀开源工作!🙌
147
 
148
  ## 💳 许可证
149
 
 
8
  <a href="https://huggingface.co/datasets/openbmb/UltraData-Code">🤗 UltraData-Code 数据集</a>
9
  </p>
10
 
11
+ <p align="center"><a href="https://huggingface.co/openbmb/UltraData-Code-L2-Classifier/blob/main/README.md">English</a> | 中文</p>
12
 
13
  ## 📚 简介
14
 
15
+ ***UltraData-Code-L2-Classifier*** 是一组编程语言特定的文件级打分器,针对 [UltraData-Code-L1](https://huggingface.co/datasets/openbmb/UltraData-Code) 的 11 种编程语言分别设计
16
 
17
+ 我们针对每种语言训练并应对应的 L2 打分器UltraData-Code-L1 中相应语言的文件进行筛选。筛选后的文件共同构成 [UltraData-Code-L2](https://huggingface.co/datasets/openbmb/UltraData-Code),其目标是集中可解释的计算与问题求解逻辑。最终 L2 数据包含 **400B tokens**,总体保留 L1 约 **12.23% 的文件**
 
 
 
 
 
 
18
 
19
  ## 💡 亮点
20
 
21
+ > **摘要:** 代码生成逐渐成为大语言模型的核心能力,代码数据是培养这一能力的预训练过程关键组成部分。随着代码语料规模不断增长,其规模、多样性质量日益影响模型在预训练阶段的能力。遵循分级数据治理视角,我们提出 **UltraData-Code**,将其构建为四个相互接的数据状态组成的数据家族L0 的代码仓库归档,到 L1 的标准化自然代码、L2 算法导向筛,再到 L3 任务导向合成。L0 归档每个公开 GitHub 仓库默认分支的最新版本,并保留其文件结构、关联关来源信息。L1 通过可扩展的过滤、规范化近重复去重,获得标准化自然代码。随后,L2 基于文件角色与启发式规则构成的语言自适应信号,并结合代码质量约束,从 L1 中筛选算法相关文件,最终得到覆盖 11 种编程语言、规模 400B tokens 的 **UltraData-Code-L2**。L3 L2 中的算法文件实施任务导向合成,将每个实现转为结构化编程练习,生成覆盖相同 11 种语言、规模 150B tokens 的 **UltraData-Code-L3**。在相同的 10B-token 持续预训练设置下,对一个 1B 模型使用 L2 而非 L1 进行训练可使其在 EvalPlus 和 MultiPL-E 上的 pass@1 分别 **7.80 个百分点** **5.13 个百分点**,同时分别超过 Stack-Edu 4.37 个百分点和 3.05 个百分点。将一半 L2 训练 tokens 替换为 L3 后,相较于仅使用 L2 训练,模型在 EvalPlus 和 MultiPL-E 上可进一步提升 **8.42 个百分点** **8.07 个百分点**,同时分别超过最强的合成数据基线 5.57 个百分点和 7.80 个百分点。当训练预算增加至 100B tokens 时,L2 筛选与 L3 合成在两个基准上的增益均进一步扩大。
22
 
23
+ - **完整的代码数据构建流水线:** 将仓库归档和规范化自然代码与细粒度精、结构化合成连接起来,在 11 种编程语言上分别形成约 400B tokens 的 L2 和约 150B tokens 的 L3。
24
+ - **细粒度算法相关代码精:** 我们提出一种语言自适应筛框架文件角色监督与语言特定启发式线索结合学习显式 ALGO 文件的算法相关性。该框架在文件角色、算法相关性和代码质量模型之间复用预计算语义表征,最终结合角色与质量约束完成筛选
25
+ - **任务导向合成:** 我们提出一种结构化合成方案,将 L2 筛选出的每个算法相关实现转化一道编程练习,并基于同一源实现联合生成独立完整的任务、分析、解答与候选测试用例。该转换既保留了源代码的计算意图,又为代码生成引入了显式的任务监督与解答监督
26
 
27
  <div align="center">
28
  <img src="assets/ultradata-code-l0-l3-overview.png" width="760" alt="UltraData-Code L0-L3 构建流程"/>
29
  </div>
30
 
31
+ ## 🔎 L2 筛选流程
32
 
33
+ L2 流程分别应用于 UltraData-Code-L1 中的 11 种语言:
34
 
35
+ 1. **编码文件** 使用 [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) 为每个文件生成 1024 维 embedding。
36
+ 2. **预测文件角色** 将文件分为 `ALGO`、`WEB`、`TOOL`、`DATA`、`TEST`、`CONFIG` 或 `EXCLUDE`。
37
+ 3. **构造双线索监督** 将 ALGO 角色标注与语言特定启发式证据结合,覆盖 ALGO 之外的算法相关文件。
38
+ 4. **预测相关性质量** 相关性模型输出 `algo_rel_score`,质量模型输出 `quality_score`。
39
+ 5. **校准并筛选** 在留出数据上为每种语言校准相关性阈值。文件需通过相关性阈值和质量阈值且被预测为 `EXCLUDE`,才会保留。
40
 
41
  <div align="center">
42
+ <img src="assets/ultradata-code-l2.png" width="760" alt="UltraData-Code L2 筛选流程"/>
43
  </div>
44
 
45
  ## 📈 评测结果
46
 
47
+ 10B-token 对比均使用相同的 1B 基座模型、训练设置、去污流程和评测协议。
 
 
48
 
49
+ **Python 结果:算法相关性精带来最强的自然代码结果。** UltraData-Code-L2-py 的 EvalPlus 平均 pass@1 比 UltraData-Code-L1-py 提升 **17.91 个百分点**,比 Stack-Edu-py 提升 **8.46 个百分点**。
 
50
 
51
  <div align="center">
52
  <img src="assets/python-training-dynamics.png" alt="Python 训练动态" width="760"/>
 
56
  <img src="assets/python_results.png" alt="Python 主结果表" width="900"/>
57
  </div>
58
 
59
+ **多语言结果:L2 精筛优于自然代码基线。** UltraData-Code-L2 相比 UltraData-Code-L1 在 EvalPlus 和 MultiPL-E 上分别提升 **7.80** 和 **5.13 个百分点**,相比 Stack-Edu 分别提升 **4.37** 和 **3.05 个百分点**。
 
 
 
 
60
 
61
  <div align="center">
62
  <img src="assets/multilingual-training-dynamics.png" alt="多语言训练动态" width="760"/>
 
66
  <img src="assets/multilingual_results.png" alt="多语言主结果表" width="900"/>
67
  </div>
68
 
69
+ ## 🚀 UltraData-Code-L2 打分器使用方法
70
+
71
+ 安装依赖:
72
 
73
  ```bash
74
  pip install numpy torch pyarrow
 
83
 
84
  scorers = LanguageScorers.load("classifiers/cpp")
85
 
86
+ # float32 [N, 1024];paths 与 embeddings 顺序一致,使用仓库相对路径
87
  out = scorers.score(embeddings, paths)
88
  ```
89
 
90
+ 为每种输入语言使用对应语言目录。相关性模型需要真实的仓库相对路径。
91
 
92
+ ### Parquet 文件打分
93
 
94
+ [`scripts/score_parquet.py`](scripts/score_parquet.py) 可处理单个 Parquet 文件或递归处理目录。输入必须包含 `embedding` 列,以及 `relative_path` 包含 `file_path` 的 `meta` 结构
95
 
96
  ```bash
97
  SCORERS=classifiers/cpp
 
107
  --apply-policy
108
  ```
109
 
110
+ 输出会镜像输入目录结构,并保留所有原有列。使用 `--describe` 查看打分器配置,使用 `--limit N` 限制处理 Parquet 文件,或使用 `--algo-rel-min`、`--quality-min` 和 `--exclude-categories` 覆盖策略值。
111
 
112
  ### 输出字段
113
 
114
+ 打分器会在输入文件中追加以下结果:
115
+
116
  ```text
117
  category: {预测的文件角色}
118
  cls_confidence: {文件角色置信度}
 
121
  selected: {是否通过筛选策略}
122
  ```
123
 
124
+ - `category` `ALGO``WEB``TOOL``DATA``TEST``CONFIG` `EXCLUDE` 之一。
125
+ - `cls_confidence` `algo_rel_score` 的范围为 `0` 到 `1``quality_score` 的范围为 `0` 到 `10`
126
+ - 只有启用 `--apply-policy` 时才会添加 `selected`。该字段标记同时通过相关性阈值、`EXCLUDE` veto 和质量阈值的行,不会删除未通过的行
127
 
128
  ### 注意事项
129
 
130
+ - 输入必须使用 [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) 生成的 1024 维 embedding
131
+ - 算法相关性打分器还会使用仓库相对路径传入与 embedding 顺序一致的真实路径
132
+ - [`config.json`](config.json) 提供通用默认值。命令行参数可以覆盖这些默认值
133
+ - 当前打分脚本为单进程实现大规模语料推理时,建议在外部按数据分片或子目录进行调度。
134
 
135
  ## ❤️ 致谢
136
 
137
+ 项目基于 [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B)、[PyTorch](https://pytorch.org/)[PyArrow](https://arrow.apache.org/docs/python/) 。感谢这些优秀开源工作,正是开源社区的贡献让 UltraData-Code 成为可能!🙌
138
 
139
  ## 💳 许可证
140