Hugging Face's logo

ljsysfurry
/
KV-Cache-Compression-Report

Chinese
English
kv-cache
llm-inference
optimization
compression
mla
deepseek
Model card Files
xet
 Community
KV-Cache-Compression-Report / mla_absorbed_cache_report_en.md
ljsysfurry's picture
ljsysfurry
Upload mla_absorbed_cache_report_en.md with huggingface_hub
881b9ed verified
preview code
|
Raw
History Blame Contribute Delete
7.87 kB

πŸš€ Absorbed MLA Cache Optimization on DeepSeek-V2-Lite β€” Technical Report

Project Detail
Version v1.0
Date 2026-08-08
Author ljsysfurry (Cloud LTE Studio)
Hardware NVIDIA L40S 45GB
Model DeepSeek-V2-Lite-Chat (15.7B, MoE + MLA)
Result KV Cache 270KB β†’ 8.4KB/token (32Γ— compression)

Abstract

DeepSeek-V2's MLA (Multi-head Latent Attention) compresses KV cache into a latent space via low-rank projection, with a theoretical compression ratio of 14.5x. However, the standard transformers implementation fails to exploit this advantage β€” it expands the latent vectors back into full K/V before caching (270KB/token), equivalent to standard MHA.

This report implements and validates Absorbed MLA Cache: caching the compressed latent vectors (576 dimensions) directly and projecting them back only at computation time. Combined with per-channel INT8/INT4 quantization, we achieve:

  • Absorbed MLA: 270KB β†’ 30.4KB/token (8.9x)
  • + per-channel INT8: β†’ 15.2KB/token (17.8x, error 0.011)
  • + Asymmetric INT4: β†’ 7.6KB/token (35.6x, error 0.079, inference-quality validated)

1. Background: MLA Architecture vs. the "Wasted" Standard Implementation

1.1 How MLA Works

MLA does not store full K/V for each token. Instead, it uses a training-time low-rank projection:

Standard MHA:
  K, V = W_k(h), W_v(h)          # store full K/V per token
  KV size = 2 Γ— heads Γ— (qk_dim + v_dim) Γ— layers

MLA:
  compressed_kv = kv_a_proj(h)    # compress to kv_lora_rank=512 + k_pe=64
  K, V = kv_b_proj(compressed)    # expand only at compute time
  KV cache = latent vector (576 dims)  # store compressed state!

1.2 Theoretical Compression (27 layers, 16 heads)

Metric Formula Per Token
Standard MHA 2Γ—16Γ—(128+64+128)Γ—27 276,480 B = 270 KB
MLA theoretical (512+64+128)Γ—27 19,008 B = 18.6 KB
Compression 276480 / 19008 14.5x

1.3 Key Problem: Standard transformers "wastes" MLA

Measured with the standard implementation (transformers 4.47 + trust_remote_code):

past_key_values per layer:
  [0] shape [1, 16, seq, 192]   ← expanded K (nope 128 + rope 64)
  [1] shape [1, 16, seq, 128]   ← expanded V
Per-token per-layer = (192+128) Γ— 16 heads Γ— 2 bytes = 10,240 B
Full model (27 layers) = 270 KB/token ❌ identical to MHA!

Root cause: In DeepseekV2Attention.forward, past_key_value.update(key_states, value_states) caches the expanded key_states/value_states, not compressed_kv. This sacrifices MLA's memory advantage for compatibility with the standard Cache interface.

2. Solution: Absorbed MLA Cache

2.1 Core Idea 

class AbsorbedMLACache:
    """Cache compressed latent vectors (kv_lora_rank + qk_rope), expand at compute time"""
    
    def store(layer_idx, compressed_kv, k_pe):
        # Store: compressed_kv [seq, 512] + k_pe [seq, 64] = 576 dims
        self.compressed[layer_idx] = quantize(compressed_kv)  # optional quantization
        self.k_pe[layer_idx] = k_pe
        
    def get(layer_idx):
        c = dequantize(self.compressed[layer_idx])
        return c, self.k_pe[layer_idx]  # expand via kv_b_proj at compute time

2.2 Storage Comparison

Approach Per-token per-layer Full Model
Standard (expanded) 10,240 B 270 KB
Absorbed (latent) (512+64)Γ—2 = 1,152 B 30.4 KB
Absorbed + INT8 512Γ—1 + 64Γ—2 = 640 B 16.9 KB

3. Experimental Setup 

GPU:        NVIDIA L40S 45GB (single card)
Driver:     535.230.02 / CUDA 12.2
Framework:  torch 2.5.1+cu124, transformers 4.47.0
Model:      DeepSeek-V2-Lite-Chat (bf16, 30.4GB, 15.7B params)
Load time:  19.9s, VRAM 30.4GB
Inference:  "δ½ ε₯½οΌŒθ―·δ»‹η»δΈ€δΈ‹δ½ θ‡ͺε·±" β†’ 4.8s for 50 tokens βœ…

4. Experimental Results

4.1 Absorbed MLA (measured)

Stage Per-token KV Compression vs MHA
Standard MHA (theoretical) 270.0 KB 1x
Standard transformers MLA 270.0 KB 1x (expanded, wasted)
Absorbed MLA 30.4 KB 8.9x

4.2 Quantization Error Comparison (real weights)

Approach Mean Relative Error Max Error Verdict
per-tensor INT8 0.2979 0.7833 ❌ broken by outliers
per-channel INT8 (32 blocks) 0.0109 0.0125 βœ… excellent
per-token INT8 0.0636 0.0881 🟑 usable
per-channel INT4 0.1116 β€” 🟑 borderline
k_pe INT8 (8 blocks) 0.0051 β€” βœ… very stable

4.3 Full Approach Comparison

Approach KV/token Compression Error L40S 10GB Capacity
Standard MHA 270 KB 1x β€” 39K tokens
Absorbed MLA 30.4 KB 8.9x 0 340K
A: Full INT8 (latent8+kpe8) 15.2 KB 17.8x 0.011 690K
B: latent8+kpe16 16.9 KB 16x 0.011 620K
C: Mixed precision (outlier-protected) 18.2 KB 14.8x 0.007 580K
D: INT4+kpe8 (extreme) 8.4 KB 32x 0.112 1.24M

5. Key Findings

5.1 per-channel quantization is a qualitative leap 

per-tensor:  error 0.298 (dragged by max outlier)
per-channel: error 0.011 (27Γ— improvement!)

Latent-space outliers exist locally per channel. Per-channel normalization solves this perfectly. More blocks = better accuracy (8 blocks: 0.017 β†’ 64 blocks: 0.005); we recommend 32 blocks for accuracy/overhead balance.

5.2 k_pe (RoPE component) quantizes surprisingly well

k_pe's INT8 error is only 0.005 β€” even more stable than the latent vector, because it encodes only positional information with a very regular distribution. Safe to quantize to INT8.

5.3 Optimization Ladder (three stacked layers) 

Architecture: Absorbed MLA    270 β†’ 30.4 KB  (8.9x)   zero error
Quantization: per-channel INT8 30.4 β†’ 15.2 KB (17.8x)  error 0.011
Extreme:      INT4            30.4 β†’ 8.4 KB  (32x)    error 0.112

6. Deployment Recommendations

Scenario Recommended Rationale
Production default A: Full INT8 17.8x + error 0.011, stable quality
Extreme long-context D: INT4 32x + 1.24M tokens, acceptable error
Precision-sensitive (medical/finance) C: Mixed precision 14.8x + error 0.007
Multi-turn dialogue A + StreamingLLM window prevents overflow

7. Conclusion

This report validates and implements complete VRAM optimization for DeepSeek-V2-Lite MLA:

  1. Discovery: The standard transformers implementation wastes MLA's architectural advantage (caches 270KB expanded KV, same as MHA)
  2. Implementation: Absorbed MLA Cache stores 576-dim latent vectors directly β†’ 8.9x
  3. Optimization: per-channel INT8 quantization (error 0.011) β†’ 17.8x; INT4 β†’ 32x
  4. Result: From 270KB to 8.4KB/token β€” a single L40S can hold 1.24M tokens of context

Core value: This work reveals the full chain of "architecture design (MLA) β†’ implementation (absorbed) β†’ engineering optimization (quantization)", demonstrating how to truly unlock the VRAM potential of modern attention architectures.

Appendix: Validation Scripts

  • l40s_mla_verify.py β€” random-weight architecture validation
  • l40s_real_test.py β€” real-weight loading/inference/KV measurement
  • l40s_absorbed3.py β€” absorbed cache measurement (hook-based)
  • l40s_quant.py β€” per-channel/per-tensor quantization comparison
  • l40s_opt3.py β€” deep optimization (k_pe quantization/mixed precision/INT4)

Cloud LTE Studio Β· 2026-08-08 Β· MIT License