add README
Browse files
README.md
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
tags:
|
| 4 |
+
- code
|
| 5 |
+
- latent-reasoning
|
| 6 |
+
- codi
|
| 7 |
+
- experiment-archive
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
# codi-trace — checkpoint archive
|
| 11 |
+
|
| 12 |
+
CODI latent-reasoning distillation 实验的 **checkpoint 备份**(非成果发布)。学生模型在每个执行帧把
|
| 13 |
+
`$LOCALS` 状态换成一个递归 latent block(`<|latent_start|>` + latent steps + `<|latent_end|>`),
|
| 14 |
+
从 Stage-1 SFT 模型蒸馏而来。基座为 Qwen2.5-Coder-1.5B/3B。
|
| 15 |
+
|
| 16 |
+
> ⚠️ 这些 **不是** 标准 `from_pretrained` 权重。`pytorch_model.bin` 是训练用 `CodiModel` 的
|
| 17 |
+
> `state_dict`(键带 `model.` / `prj.` 前缀),需用随仓附带的 `code/` 加载。
|
| 18 |
+
|
| 19 |
+
## 目录结构
|
| 20 |
+
|
| 21 |
+
```
|
| 22 |
+
README.md
|
| 23 |
+
code/ # 冻结的加载代码快照(只读),与这批权重对应
|
| 24 |
+
checkpoints/<run>/checkpoint-<step>/
|
| 25 |
+
pytorch_model.bin # CodiModel state_dict(含 model.* 与 prj.*)
|
| 26 |
+
thought_projector.pt # 投影器权重(亦含于 pytorch_model.bin)
|
| 27 |
+
config.json # 基座 arch 配置
|
| 28 |
+
tokenizer.json, merges.txt, vocab.json, tokenizer_config.json,
|
| 29 |
+
special_tokens_map.json, added_tokens.json, chat_template.jinja
|
| 30 |
+
trainer_state.json # 训练 loss 历史
|
| 31 |
+
```
|
| 32 |
+
|
| 33 |
+
训练恢复用文件(`optimizer.pt` / `scheduler.pt` / `rng_state_*.pth` / `training_args.bin`)**未包含**,
|
| 34 |
+
故无法精确 resume,仅供加载推理 / 评测。
|
| 35 |
+
|
| 36 |
+
## run 命名图例
|
| 37 |
+
|
| 38 |
+
| 片段 | 含义 |
|
| 39 |
+
|---|---|
|
| 40 |
+
| `sft…` | Stage-1 SFT(显式 trace,teacher) |
|
| 41 |
+
| `codi<size>_a<α>_b<β>_g<γ>_ls<n>` | 多帧 CODI:α/β/γ = teacher/student/KD 损失权重,ls = latent_steps |
|
| 42 |
+
| `…_ss<p>` | scheduled sampling 概率 p |
|
| 43 |
+
| `codi_frozen_<size>_<hidden\|logit>` | 冻结 SFT teacher;KD 对齐最后层 hidden 或 logit |
|
| 44 |
+
| `codi-single-<size>` | 单 block CODI(arXiv 2502.21074 忠实版) |
|
| 45 |
+
|
| 46 |
+
## 加载
|
| 47 |
+
|
| 48 |
+
```python
|
| 49 |
+
import sys; sys.path.insert(0, "code")
|
| 50 |
+
from eval.eval_cruxeval_codi import load_codi
|
| 51 |
+
|
| 52 |
+
# latent_steps 要与该 run 名里的 ls<n> 一致(默认 1)
|
| 53 |
+
tok, ids, codi = load_codi(
|
| 54 |
+
"checkpoints/codi_frozen_3b_hidden/checkpoint-1000", latent_steps=1, dev=0)
|
| 55 |
+
```
|
| 56 |
+
|
| 57 |
+
依赖:`torch`、`transformers`(`wandb` 仅训练时惰性需要,加载无需)。CRUXEval-O 评测见
|
| 58 |
+
`code/eval/eval_cruxeval_codi.py`。
|