ljsysfurry commited on
Commit
4348ae3
·
verified ·
1 Parent(s): 0e30e77

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +22 -19
README.md CHANGED
@@ -5,6 +5,8 @@ tags:
5
  - llm-inference
6
  - optimization
7
  - compression
 
 
8
  language:
9
  - zh
10
  - en
@@ -12,33 +14,34 @@ language:
12
 
13
  # 📄 KV Cache Compression for Text-Only LLMs
14
 
15
- **纯文本大语言模型 KV 缓存压缩 — 完整技术报告**
16
 
17
- ## 内容
18
 
19
- | 文件 | 说明 |
20
- |------|------|
21
- | `kv_cache_compression_report.md` | 📊 完整技术报告架构/选型/实现/评测) |
22
- | `kv_compress_plan.md` | 🗺 详细实施计划(四层方案/场景矩阵/路线图) |
 
 
23
 
24
- ## 核心方案
25
 
26
- 四层正交叠加的 KV 缓存压缩
27
 
28
- ```
29
- L1: INT8 per-channel 量化 → 显存 -50%, 几乎无损
30
- L2: StreamingLLM 滑窗 → 长对话稳定, 零成本
31
- L3: H2O 驱逐 → 再省 40%, 按场景启用
32
- L4: K/V 分离存储 → 显存换回溯能力
33
- ```
34
 
35
- **场景压缩率**:客服 4-6x / 长文 6-8x / 代码 6-8x / RAG 3-4x / 推理 3-4x
36
 
37
- ## 关键洞察
38
 
39
- > 纯文本模型可激进压缩(量化+驱逐),推理模型(R1 类)必须保守(量化+滑窗+分级)。
40
- > **场景判定比算法本身更重要。**
41
 
42
  ---
43
 
44
- *Cloud LTE Studio · 2026-08-06 · MIT License*
 
5
  - llm-inference
6
  - optimization
7
  - compression
8
+ - mla
9
+ - deepseek
10
  language:
11
  - zh
12
  - en
 
14
 
15
  # 📄 KV Cache Compression for Text-Only LLMs
16
 
17
+ **纯文本大语言模型 KV 缓存压缩 — 完整技术报告** | **Full Technical Reports**
18
 
19
+ ## Contents
20
 
21
+ | File | Description |
22
+ |------|-------------|
23
+ | `mla_absorbed_cache_report.md` | 🚀 MLA 吸收式缓存优化中文) |
24
+ | `mla_absorbed_cache_report_en.md` | 🚀 **Absorbed MLA Cache Optimization (English)** |
25
+ | `kv_cache_compression_report.md` | 📊 KV 缓存压缩完整方案(中文) |
26
+ | `kv_compress_plan.md` | 🗺 详细实施计划(中文) |
27
 
28
+ ## 🚀 Highlight: Absorbed MLA Cache (270KB → 8.4KB/token, 32×)
29
 
30
+ DeepSeek-V2-Lite MLA 优化,在 L40S 实测
31
 
32
+ | Approach | KV/token | Compression | Error |
33
+ |----------|----------|-------------|-------|
34
+ | Standard MHA | 270 KB | 1x | — |
35
+ | Absorbed MLA | 30.4 KB | 8.9x | 0 |
36
+ | + per-channel INT8 | 15.2 KB | 17.8x | 0.011 |
37
+ | + INT4 (extreme) | **8.4 KB** | **32x** | 0.112 |
38
 
39
+ **Single L40S: 1.24M tokens context.**
40
 
41
+ ## Key Insight
42
 
43
+ > Text-only models can be aggressively compressed (quantization + eviction), while reasoning models (R1-class) must be conservative (quantization + sliding window + tiered storage). **Scenario classification matters more than the algorithm itself.**
 
44
 
45
  ---
46
 
47
+ *Cloud LTE Studio · 2026-08-08 · MIT License*