Speculative Speculative Decoding

Here are the main results from "Speculative Speculative Decoding" (SSD), which introduces a framework to parallelize drafting and verification in LLM inference:

1. Core Performance Results

SSD achieves up to 2× speedup over optimized speculative decoding (SD) and up to 5× over autoregressive (AR) decoding (Figure 7, Table B.3).

Figure 7 (End-to-End Evaluation) shows that SSD strictly improves the throughput-latency Pareto frontier across batch sizes, achieving lower latency without sacrificing throughput despite using additional compute (the draft model runs on separate hardware—1× H100 for drafting vs 4× H100 for the target).

2. Key Innovation: Parallelizing Speculation and Verification

Figure 1 illustrates the architectural shift:

• Left (Standard SD): The verifier waits idly while the draft model speculates, creating a sequential dependency.
• Center (SSD): The draft model runs asynchronously on separate hardware (1× H100) and predicts likely verification outcomes preemptively, preparing a "speculation cache." When verification completes, if the outcome was predicted, tokens are returned immediately with zero drafting overhead.

3. The Saguaro Algorithm: Three Key Optimizations

A. Saguaro Cache: Geometric Fan-Out Strategy (Section 4.1)

To predict which verification outcomes to prepare for, Saguaro uses a geometric fan-out strategy (Theorem 12) rather than uniform allocation.

• Figure 3 shows cache hit rates follow a power law: rejection rates scale as $1/F^r$ where $F$ is fan-out.
• Figure 4 demonstrates that geometric fan-out (allocating more guesses to earlier positions in the sequence) improves cache hit rates by up to 90% and end-to-end speed, especially at higher temperatures where uniform strategies fail.

B. Saguaro Sampling: Balancing Acceptance vs. Cache Hits (Section 4.2)

There is a fundamental tension between:

• High acceptance rate (draft tokens match target distribution)
• High cache hit rate (predicting the bonus token correctly)

Figure 5 shows Saguaro sampling introduces a tunable hyperparameter $C \in [0,1]$ that downweights probabilities on cached tokens during drafting. This:

• Increases residual probability mass on cached tokens (making the bonus token easier to predict)
• Creates a trade-off curve where lower $C$ → higher cache hits but lower acceptance rate
• Theorem 15 proves cache hit rate increases monotonically as $C \to 0$

C. Saguaro Fallback: Optimal Cache Miss Handling (Section 4.3)

When the cache misses (verification outcome not predicted), the optimal strategy depends on batch size:

Figure 6 shows:

• Small batches: Use the slow, high-quality primary speculator as backup
• Large batches: Switch to a fast backup speculator (e.g., random tokens or n-grams) at critical batch size $b^*$ (Theorem 17)
• At batch size 16, scaling draft compute (more GPUs → larger fan-out) continues to improve speedups

4. Theoretical Guarantees

• Corollary 8: SSD is strictly faster than SD whenever cache hit rate $p_{\text{hit}} > 0$
• Corollary 9 (Speedup Sandwich): Bounds the speedup of SSD over SD as proportional to $(1 + T_{\text{SD}}) \cdot \frac{E_{\text{hit}}}{E_{\text{SD}}} \cdot p_{\text{hit}}$, where $T_{\text{SD}}$ is draft latency and $E$ represents expected tokens generated

5. Implementation Architecture

The system uses a custom PyTorch inference engine with:

• PagedAttention and continuous batching
• Custom sparse attention masks (Figure 8