\documentclass[10pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amssymb}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{xcolor}
\usepackage{caption}
\usepackage[hidelinks]{hyperref}
\usepackage{microtype}

\input{numbers}

\captionsetup{font=small,labelfont=bf}
\setlength{\parskip}{1pt}

\title{\bf Quantization as Cache Amplification:\\[2pt]
Trillion-Parameter Mixture-of-Experts Inference on a Commodity Laptop}

\author{Kavin Kumar\\ \small Neural Metrics\\ \small\texttt{kavin.kum016@neuralmetrics.ai}}
\date{\today}

\begin{document}
\maketitle

\begin{abstract}
Weight quantization is usually justified as footprint reduction. We argue that
for offloaded mixture-of-experts (MoE) inference this framing misses where the
leverage actually is. In an offloaded MoE engine the binding resource is not
storage capacity but the fraction of expert slots resident in DRAM, and storage
traffic depends on that fraction through a cache hit rate that is both concave
and, for recency-based policies, \emph{discontinuous}. We show on real routing
traces that a least-recently-used expert cache collapses to a hit rate of
\emph{exactly zero} whenever its capacity falls below the $kL$ expert slots a
single token touches --- a cyclic-reference pathology we measure at precisely
the predicted threshold. Quantization is what moves a system across that
threshold, so bits buy throughput super-proportionally to their compression
ratio.

We make this quantitative on one commodity laptop (\nGPUName, 4\,GB VRAM;
32\,GB DRAM; consumer NVMe). We contribute (i) a sub-2-bit post-training codec
combining randomized Hadamard incoherence processing, multi-stage residual
vector quantization, and block-LDL error feedback; (ii) \emph{frequency-conditioned
bit allocation}, which spends bits on experts in proportion to measured
activation frequency and so improves distortion and cache residency together;
and (iii) an end-to-end roofline in which every hardware term is measured on the
host and the cache term is modelled analytically and validated against
\nTraceTokens{} tokens of real OLMoE-1B-7B routing traces.

Measured on the host: \nRandBest\,GB/s unbuffered random read at expert-block
granularity, \nPCIe\,GB/s over PCIe, $88.2$\,GB/s of GPU device bandwidth. On
OLMoE-1B-7B the codec reaches WikiText-2 perplexity \nPPLOursTwo{} at
\nBitsOursTwo{} bits and \nPPLFreq{} at \nBitsFreq{} bits against \nPPLfp{} in
bfloat16; frequency-conditioned allocation improves perplexity by $13.8\%$ over
uniform allocation at identical rate. We report the quality cost plainly: sub-2-bit
operation on a $1.3$\,B-active-parameter MoE is expensive, and the systems
analysis is presented as a function of rate rather than at one favoured point.
On the traces, expert popularity is Zipfian with exponent $\nZipfSmeas$, and a
popularity-pinned cache model reproduces measured hit rates to
\nStaticMAE{} percentage points. Composing these, a $\nTotalParams$\,B-parameter
MoE occupies \nFootprintOnePFive\,GB at $1.5$ bits --- resident on the laptop's
free NVMe --- and projects \nTokSecOne{} tokens/s at batch~1 and
\nTokSecBatch{} tokens/s at batch~32, versus \nTokSecFP{} tokens/s at bfloat16.
That is a $\nSpeedupOverFP\times$ throughput gain from a $10.7\times$
compression: $\nAmpFactor\times$ more than compression alone. We also report a
negative result that constrains any such system: sustaining the storage stream
while materializing fp16 weights would need ${\sim}\nFusedBw$\,GB/s of device
bandwidth against $88.2$\,GB/s measured, so dequantization must be fused into
the GEMM. \textbf{We did not execute a trillion-parameter model}; the 1T figures
are an analytical projection from measured host parameters and a cache model
validated at 7B scale, and we report sensitivity across the full hit-rate range.
\end{abstract}

\section{Introduction}

A trillion-parameter mixture-of-experts model stored at bfloat16 occupies
roughly \nFootprintFP\,GB. Models of this class are deployed on multi-GPU
servers whose aggregate high-bandwidth memory exceeds the entire storage budget
of a consumer machine, and the reasonable assumption is that such models are out
of reach for a single laptop.

Sparse MoE models complicate that assumption, because the parameters touched by
any one token are a small fraction of the total. In the reference configuration
we study (Section~\ref{sec:config}), $\nTotalParams$\,B total parameters yield
$\nActiveParams$\,B active parameters per token. The other ${\sim}96\%$ of the
model is idle at any instant; it does not need to be in memory, it needs to be
\emph{reachable} fast enough. That converts a capacity problem into a
bandwidth-and-locality problem, which is what caches address.

The usual account of quantization here is that it shrinks the model so more of
it fits. We think that understates it, for a reason specific to the access
pattern. Let $f$ be the fraction of expert slots that are DRAM-resident. Storage
traffic per token is governed by $1-h(f)$, and $h$ is not a gentle function. A
single token routes to $k$ experts in each of $L$ layers, touching $kL$ distinct
expert slots before any of them is reused. Under a recency-based policy this is
a cyclic reference pattern: if the cache holds fewer than $kL$ slots, every
entry is evicted before its next use and the hit rate is not merely low but
identically zero. We measure exactly this on real OLMoE traces
(Section~\ref{sec:policy}): LRU hit rate is $\nLruTen\%$ at $10\%$ capacity and
jumps to $\nLruTwelve\%$ the moment capacity reaches the $\nTokenWS$-slot
working set.

Quantization is the lever that moves a system across that threshold. On our
host, $24$\,GB of DRAM holds \nSlotsFP{} of \nExpertSlots{} expert slots at
bfloat16 --- below the $512$-slot per-token working set of our 1T reference
configuration, so recency-based caching is inoperative --- but
\nCacheExperts{} slots at $1.5$ bits, comfortably above it. The resulting
throughput gain, $\nSpeedupOverFP\times$, exceeds the $10.7\times$ compression
ratio by $\nAmpFactor\times$. We call this \emph{cache amplification}.

Our contributions:
\begin{itemize}\itemsep1pt
\item \textbf{The cache-amplification framing} (Section~\ref{sec:amp}), with a
closed form for when compressing weights buys more than its compression ratio,
and a measured phase transition that is its sharpest instance.
\item \textbf{A sub-2-bit post-training codec} (Section~\ref{sec:codec}):
randomized Hadamard incoherence processing, multi-stage residual vector
quantization on a shared Gaussian codebook, and block-LDL error feedback
minimizing activation-weighted rather than weight-space error.
\item \textbf{Frequency-conditioned bit allocation} (Section~\ref{sec:alloc}),
derived from reverse water-filling over measured expert popularities, which cuts
perplexity $13.8\%$ at exactly matched average rate.
\item \textbf{A measured roofline and validated cache model}
(Sections~\ref{sec:host}--\ref{sec:policy}), including the finding that
popularity-pinned caching strictly dominates LRU in the capacity regime these
systems operate in.
\item \textbf{A projection with explicit sensitivity}
(Section~\ref{sec:projection}) and a \textbf{negative result} on the
dequantization path (Section~\ref{sec:decode}).
\end{itemize}

We state the scope plainly. We did not run a trillion-parameter model. We
measured every component of the system on real hardware with real models,
validated the component that must be extrapolated, and composed them.
Section~\ref{sec:limits} is explicit about what this does and does not
establish.

\section{Background and Related Work}

\paragraph{Extreme post-training quantization.}
GPTQ established second-order rounding with error feedback as the standard for
post-training quantization; AWQ showed activation-scale-aware channel treatment
matters at low rates. Below $3$ bits scalar quantization becomes
rate-inefficient and the field moved to vector and lattice codes. QuIP
introduced incoherence processing --- rotating weights into a basis where
outliers are suppressed --- and QuIP\# paired randomized Hadamard transforms
with an $E_8$ lattice codebook. AQLM used additive multi-codebook quantization,
VPTQ pushed vector PTQ below $2$ bits at $70$--$405$\,B scale, and QTIP replaced
explicit codebooks with a bitshift trellis to reach high effective dimension.
BitNet~b1.58 showed ternary weights are viable when trained from scratch. Our
codec is deliberately conventional in its components: the contribution is the
allocation policy layered on top and the systems argument it serves, not a new
rate--distortion frontier.

\paragraph{Offloaded and streaming inference.}
FlexGen scheduled dense-model weights across GPU, CPU and disk; \emph{LLM in a
Flash} exploited FFN activation sparsity to stream from flash; PowerInfer
partitioned neurons by activation frequency between CPU and GPU. For MoE,
Mixtral-offload introduced LRU expert caching with speculative prefetch, and
MoE-Infinity, EdgeMoE, Pre-gated MoE, Fiddler, MoBiLE and FlashMoE refined
placement and replacement. Reported hit rates are modest --- existing policies
achieve $33$--$44\%$ caching $20$ of $60$ experts per layer, and FlashMoE
reports up to $51\%$ relative improvement over LRU/LFU via learned replacement.
This literature holds the weight rate fixed and optimizes the policy. We hold
the policy simple and observe that the rate is the more powerful lever, because
it changes $f$ rather than $h(\cdot)$ --- and because it can move the system
across the discontinuity in $h$ entirely.

\paragraph{Cache analysis.}
Che's approximation gives an accurate closed form for LRU hit rates under an
independent-reference model. We found it \emph{inapplicable} in our regime
precisely because the MoE access pattern is cyclic rather than independent
(Section~\ref{sec:policy}), which is itself informative: it is the structure
that IRM discards that produces the phase transition.

\section{Cache Amplification}\label{sec:amp}

Consider an MoE with $L$ layers, $E$ routed experts per layer, top-$k$ routing
and $P_e$ parameters per expert. At weight rate $r$ (bits/parameter) an expert
occupies $B(r)=P_e r/8$ bytes, so a DRAM budget $M$ holds $M/B(r)$ slots and the
resident fraction is
\begin{equation}
f(r)\;=\;\frac{M}{B(r)\,L\,E}\;=\;\frac{8M}{P_e\,L\,E\,r}\;\propto\;\frac{1}{r}.
\label{eq:frac}
\end{equation}
Storage traffic per generated token is
\begin{equation}
T(r)\;=\;L\,k\,\bigl(1-h(f(r))\bigr)\,B(r).
\label{eq:traffic}
\end{equation}
Writing $\rho=r/r_0$ for the compression ratio against a baseline rate $r_0$,
the naive expectation is $T\propto\rho$. The actual scaling is
\begin{equation}
\frac{T(r)}{T(r_0)}\;=\;\rho\cdot\underbrace{\frac{1-h(f_0/\rho)}{1-h(f_0)}}_{\text{amplification}},
\label{eq:amp}
\end{equation}
whose second factor is at most $1$ whenever $h$ is increasing, and strictly
below $1$ when $f_0$ lies below the knee of $h$. Equivalently, traffic scales as
$\rho^{\,1+|\eta|}$ with elasticity $\eta=\mathrm{d}\log(1-h)/\mathrm{d}\log f$.

\paragraph{The discontinuity.}
For recency-based replacement the amplification factor is not merely favourable
but discontinuous. A token touches $kL$ distinct expert slots, one per layer per
selected expert, and revisits none of them until the next token. This is a
cyclic reference string of period $kL$, the classical worst case for LRU: with
capacity $C<kL$, every entry is evicted before reuse and
\begin{equation}
h_{\mathrm{LRU}}(C)\;=\;0 \qquad\text{for } C<kL,
\label{eq:threshold}
\end{equation}
independently of how skewed expert popularity is. Since $C\propto 1/r$ by
Eq.~\ref{eq:frac}, there is a critical rate $r^\star = 8M/(P_e k L)$ below which
recency-based caching begins to function at all. Section~\ref{sec:policy}
confirms Eq.~\ref{eq:threshold} on real traces, and Section~\ref{sec:projection}
shows the 1T reference configuration sits on the inoperative side of $r^\star$
at bfloat16 and the operative side at $1.5$ bits.

Two consequences follow. First, the value of a saved bit is largest exactly
where systems are most constrained, so the case for extreme rates is stronger in
offloaded settings than the perplexity--rate curve alone suggests. Second,
because $h$ depends on \emph{which} experts are resident and not only how many,
the bit budget should not be spread uniformly.

\section{Method}

\subsection{Codec}\label{sec:codec}

Each linear weight matrix $W\in\mathbb{R}^{o\times i}$ is quantized in three
stages.

\paragraph{Incoherence processing.}
We apply a two-sided randomized Hadamard transform
$W'=(D_\ell H_o)\,W\,(D_r H_i)$ with $D_\ell,D_r$ random sign diagonals and
$H_n$ the normalized Walsh--Hadamard matrix. This suppresses outliers and drives
subvector statistics toward i.i.d.\ Gaussian, so a single global codebook serves
every layer of every model and its storage cost amortizes to zero. As $H$ is
orthogonal and symmetric the rotation is undone at inference time on
\emph{activations}, not weights:
$y=Wx=D_\ell H_o\bigl(W'(H_i D_r x)\bigr)$, costing two vector transforms per
matmul. We implement the transform as two dense GEMMs via the Kronecker
factorization $H_{ab}=H_a\otimes H_b$, replacing $O(\log n)$ kernel launches
with two cuBLAS calls; measured $86.7\,\mu$s versus $1930\,\mu$s for a butterfly
implementation at $n=8192$, a $22\times$ speedup at $10^{-6}$ agreement.

\paragraph{Residual vector quantization.}
Rows of $W'$ are normalized by an fp16 per-row RMS scale and partitioned into
subvectors of dimension $d=16$. Each subvector is coded by $S$ residual stages,
each an $8$-bit index into a $256$-entry codebook, giving $0.5$ bits/weight per
stage and a rate ladder in $0.5$-bit steps. Stage $s$'s codebook is trained by
$k$-means on the residual distribution left by stages $<s$, so later stages
adapt to the shrinking residual scale.

\paragraph{Block-LDL error feedback.}
Weight-space error is the wrong objective; what matters is
$\|(W-\widehat{W})X\|_F$ for calibration activations $X$. With $H=XX^\top$ and
the block-LDL factorization $H=LDL^\top$ ($L$ block-unit-lower-triangular with
$d\times d$ blocks), the objective decomposes over column blocks and processing
them in reverse order with
\begin{equation}
\widehat{W}_k \;=\; Q\Bigl(W_k + \textstyle\sum_{j>k} (W_j-\widehat{W}_j)\,L_{jk}\Bigr)
\label{eq:ldlq}
\end{equation}
greedily minimizes it. On synthetic data with a non-trivial Hessian this trades
higher weight-space error ($0.537$ vs $0.441$ relative Frobenius at $1.5$ bits)
for a $17$--$18\%$ lower activation-weighted error --- the quantity that governs
perplexity. Linears sharing an input share one factorization. We quantize layer
by layer, propagating each layer's \emph{quantized} output forward so later
layers calibrate against accumulated error.

\subsection{Frequency-Conditioned Bit Allocation}\label{sec:alloc}

Experts are not equally important: their activation frequencies are heavy-tailed
(Section~\ref{sec:routing}), and an expert's contribution to expected output
error is weighted by how often it is selected. With $\pi_e$ the activation
probability and $D(r)$ the distortion at rate $r$, the rate-constrained problem
\begin{equation}
\min_{\{r_e\}}\ \sum_e \pi_e D(r_e)
\quad\text{s.t.}\quad \tfrac{1}{E}\textstyle\sum_e r_e = \bar r
\label{eq:alloc}
\end{equation}
has the reverse-water-filling solution $r_e = \bar r + \tfrac12\log_2(\pi_e/\tilde\pi)$
for a Gaussian source with $D(r)\propto 2^{-2r}$, $\tilde\pi$ the geometric mean
of the $\pi_e$: bits should grow logarithmically in popularity.

The systems benefit is separate from and additive to the distortion benefit.
Cold experts coded at lower rate occupy fewer bytes, so at fixed $M$ more
\emph{hot} experts become resident: the allocation improves $h$ as well as $D$.
We implement a rank-based approximation: at average rate $\bar S$ stages, the
most-activated third of experts in each layer receive $\bar S+1$ stages and the
least-activated third $\bar S-1$, leaving the mean rate --- and hence the
comparison against uniform allocation --- exactly rate-matched.

\subsection{Execution Model}\label{sec:exec}

The engine we analyze keeps rate-invariant always-active parameters resident in
VRAM --- attention projections and the shared expert, \nResidentGB\,GB at
$1.5$ bits, within the \nGPUName's $4$\,GB --- and treats routed experts as a
demand-paged working set backed by NVMe with DRAM as a second-level cache.
Expert blocks at $1.5$ bits are \nExpertMB\,MB, a granularity at which random
reads are as fast as sequential (Section~\ref{sec:host}), which is why demand
paging is viable at all.

\section{Experimental Setup}\label{sec:config}

\paragraph{Host.}
All measurements are from one laptop: Intel 12-core CPU, $32$\,GB DDR5
(\nRamCopy\,GB/s measured copy bandwidth), \nGPUName{} with $4.29$\,GB VRAM
($16$ SMs, $88.2$\,GB/s measured device bandwidth, \nPeakTflops\,TFLOP/s peak
fp16 GEMM), consumer NVMe with $294$\,GB free. Storage is measured through the
Win32 API with \texttt{FILE\_FLAG\_NO\_BUFFERING}, so results reflect device
behaviour rather than page-cache hits.

\paragraph{Model.}
Compression and routing experiments use OLMoE-1B-7B-0924 ($6.9$\,B total,
$1.3$\,B active, $16$ layers, $64$ experts/layer, top-$8$,
$d_{\text{model}}{=}2048$) --- the largest fully open MoE whose bfloat16 weights
fit this machine's DRAM, with power-of-two dimensions suited to the Hadamard
transform. Calibration uses $32$ sequences of $2048$ tokens from WikiText-2
train; perplexity is on WikiText-2 test at sequence length $2048$. Routing
traces cover \nTraceTokens{} tokens of held-out text.

\paragraph{Reference 1T configuration.}
\textbf{T1} is a DeepSeek-V3-style configuration scaled to
$\nTotalParams$\,B parameters: $64$ layers, $d_{\text{model}}{=}8192$, $320$
routed experts plus $1$ shared expert per layer, top-$8$ routing, expert
intermediate width $2048$, grouped-query attention with $1024$ KV width, vocab
$129280$. This gives $\nActiveParams$\,B active parameters per token,
\nExpertSlots{} routed expert slots and a per-token working set of $512$ slots.
T1 is a paper configuration, not a released checkpoint; it exists to make the
accounting concrete.

\section{Results}

\subsection{Host Roofline}\label{sec:host}

Table~\ref{tab:host} and Figure~\ref{fig:io} give measured host limits. The
important feature is the shape of the random-read curve: at $64$\,KiB blocks a
single thread sustains \nRandSmall\,GB/s, but by $1$\,MiB the same thread
reaches \nRandOneThread\,GB/s and with modest concurrency the device saturates
at \nRandBest\,GB/s --- above its \nSeqRead\,GB/s sequential rate. Expert blocks
at $1.5$ bits are \nExpertMB\,MB, well inside the flat region.
\emph{Demand-paging experts costs no measurable random-access penalty}, which is
the enabling hardware fact for this class of system. PCIe transfer measures
\nPCIe\,GB/s, closely matching storage, so the two stages pipeline without
either dominating.

\begin{table}[t]
\centering\small
\caption{Measured host limits. Storage uses unbuffered, page-cache-bypassing I/O.}
\label{tab:host}
\begin{tabular}{lr}
\toprule
Quantity & Measured \\
\midrule
NVMe sequential read & \nSeqRead\,GB/s \\
NVMe random read, $64$\,KiB blocks, 1 thread & \nRandSmall\,GB/s \\
NVMe random read, $1$\,MiB blocks, 1 thread & \nRandOneThread\,GB/s \\
NVMe random read, peak (\nRandBestBlock\,MiB, \nRandBestThreads\ threads) & \nRandBest\,GB/s \\
PCIe host$\rightarrow$device ($16$\,MB) & \nPCIe\,GB/s \\
DRAM copy & \nRamCopy\,GB/s \\
GPU device copy & $88.2$\,GB/s \\
GPU peak fp16 GEMM & \nPeakTflops\,TFLOP/s \\
\bottomrule
\end{tabular}
\end{table}

\begin{figure}[t]
\centering
\includegraphics[width=0.49\textwidth]{figs/io.pdf}
\caption{Unbuffered random-read bandwidth versus block size. Bandwidth saturates
by ${\sim}1$\,MiB; the $1.5$-bit expert block (\nExpertMB\,MB) sits in the flat
region, so demand-paged experts pay no random-access penalty.}
\label{fig:io}
\end{figure}

\subsection{Compression Quality}\label{sec:quality}

Table~\ref{tab:quality} and Figure~\ref{fig:quality} report WikiText-2
perplexity for OLMoE-1B-7B against a bfloat16 reference of \nPPLfp.

\paragraph{Both codec components are load-bearing.}
At a fixed \nBitsOurs{} bits, removing block-LDL error feedback --- quantizing
data-free against weight-space error --- moves perplexity from \nPPLOurs{} to
\nPPLNoLdlq, a factor of ${\sim}300$. Removing incoherence processing while
keeping error feedback gives \nPPLNoRht, a factor of ${\sim}14$. Neither
component is optional at this rate, and error feedback matters more than
rotation. This is consistent with the synthetic result of
Section~\ref{sec:codec}: weight-space error is a poor proxy, and the methods
that optimize the activation-weighted objective dominate.

\paragraph{Frequency-conditioned allocation.}
At an identical average rate of \nBitsFreq{} bits, allocating stages by measured
expert popularity improves perplexity from \nPPLOurs{} to \nPPLFreq{} --- a
$13.8\%$ reduction for no additional storage. Because the allocation is exactly
rate-matched by construction, this is attributable to the distribution of bits
across experts rather than to any change in budget. The systems benefit of
Section~\ref{sec:alloc} --- smaller cold experts admitting more hot experts into
cache --- is additional to this and not counted here.

\paragraph{Vector quantization dominates scalar rounding.}
Round-to-nearest with group-128 scales collapses below $3$ bits: \nPPLRtnTwo{}
at \nBitsRtnTwo{} bits, versus \nPPLOursTwo{} for our codec at a \emph{lower}
\nBitsOursTwo{} bits. RTN needs \nBitsRtnThree{} bits to reach \nPPLRtnThree,
i.e.\ ${\sim}1.6\times$ the storage for comparable quality.

\paragraph{The cost of going below two bits.}
We state this plainly rather than burying it. At \nBitsOursTwo{} bits perplexity
is \nPPLOursTwo{} ($1.5\times$ the bfloat16 reference), which we regard as a
usable operating point. At \nBitsFreq{} bits it is \nPPLFreq{} ($2.7\times$),
which is a substantial degradation, and at \nBitsOursOne{} bit the model
effectively breaks down (\nPPLOursOne). OLMoE-1B-7B has only $1.3$\,B active
parameters and is a hard case --- published sub-2-bit results are obtained on
$70$\,B+ dense models with more elaborate codebooks, and quantization tolerance
generally improves with scale --- but we have not verified that at 1T, and we
therefore parameterize every systems result in Section~\ref{sec:projection} by
rate rather than asserting a single operating point.

\begin{table}[t]
\centering\small
\caption{WikiText-2 perplexity for OLMoE-1B-7B at matched average rates.}
\label{tab:quality}
\input{tab_quality}
\end{table}

\begin{figure}[t]
\centering
\includegraphics[width=0.55\textwidth]{figs/quality.pdf}
\caption{Rate--perplexity for OLMoE-1B-7B. Each ablation removes one codec
component at matched average rate.}
\label{fig:quality}
\end{figure}

\subsection{Routing Statistics}\label{sec:routing}

Figure~\ref{fig:cache}(a) shows measured expert popularity over
\nTraceTokens{} tokens. The distribution is heavy-tailed and well described by a
Zipf law with exponent $\nZipfSmeas$; the most-activated quartile of experts
absorbs \nMassTopQ\% of activations. Consecutive tokens reuse \nReuseMean\% of
their experts, so there is short-range temporal locality on top of the
popularity skew. Distinct experts touched per layer grows sublinearly in batch
size --- \nWorkSetOne{} at one token, \nWorkSetSixteen{} at $16$,
\nWorkSetSixtyFour{} at $64$ of $64$ --- so batching amortizes fetches
(Figure~\ref{fig:cache}c).

\subsection{Cache Policy and the LRU Threshold}\label{sec:policy}

Table~\ref{tab:policy} and Figure~\ref{fig:cache}(b) compare replacement
policies over the true interleaved access order. The result is stark. Below the
$\nTokenWS$-slot per-token working set ($\nTokenWSFrac\%$ of slots for this
model), LRU achieves a hit rate of \emph{exactly zero} at every capacity tested
--- $\nLruTwo\%$, $\nLruFive\%$, $\nLruTen\%$ at $2$, $5$ and $10\%$ capacity ---
and jumps to $\nLruTwelve\%$ the moment capacity reaches $\nTokenWS$ slots. This
is Eq.~\ref{eq:threshold} observed at precisely the predicted threshold, and it
is a property of the access pattern, not of our implementation.

Popularity-pinned caching has no such failure mode: it delivers
$\nStaticTwo\%$, $\nStaticFive\%$ and $\nStaticTen\%$ at the same capacities
where LRU delivers nothing, and its hit rate is by construction the popularity
mass of the pinned set --- an analytic quantity we confirm against the trace to
\nStaticMAE{} percentage points. Above the working set the ordering reverses and
recency helps: LRU reaches $\nLruTwentyFive\%$ at $25\%$ capacity against
$\nStaticTwentyFive\%$ static, and a hybrid that pins the hottest $75\%$ of
capacity and runs LRU over the remainder is best of all at large capacity
($\nHybridForty\%$ at $40\%$).

Two conclusions follow. Operationally, popularity-pinned or hybrid placement
should be preferred to pure LRU in offloaded MoE engines, which supports the
allocation policy of Section~\ref{sec:alloc}: what matters is \emph{which}
experts are resident, and popularity predicts that well. Methodologically, we
project with the popularity-pinned model because it is exact given the
popularity vector and therefore extrapolates without relying on recency
structure we cannot verify at $E{=}320$. When the popularity vector is replaced
by its Zipf fit the model becomes optimistic by \nZipfMAE{} points; we subtract
that bias in all projections.

\begin{table}[t]
\centering\small
\caption{Measured expert-cache hit rates on the OLMoE trace ($\nSlots$ slots,
per-token working set $\nTokenWS$ slots). $^\dagger$ marks capacities below the
working set, where recency-based replacement cannot hit at all. ``Analytic'' is
the popularity-mass model used for extrapolation.}
\label{tab:policy}
\input{tab_policy}
\end{table}

\begin{figure}[t]
\centering
\includegraphics[width=\textwidth]{figs/cache.pdf}
\caption{(a) Measured expert popularity per layer with Zipf fit. (b) Hit rate
versus cache capacity for LRU, popularity pinning and the analytic model; the
vertical line marks the per-token working set. (c) Distinct experts per layer
versus batch size.}
\label{fig:cache}
\end{figure}

\subsection{Projection to T1}\label{sec:projection}

Table~\ref{tab:amp} instantiates Eq.~\ref{eq:frac} on the host. At bfloat16, T1
occupies \nFootprintFP\,GB --- it does not fit the machine's storage at all ---
and $24$\,GB of DRAM would hold \nSlotsFP{} expert slots, \emph{below} the
$512$-slot per-token working set, so recency-based caching would be inoperative.
At $1.5$ bits the model occupies \nFootprintOnePFive\,GB, fitting the $294$\,GB
of free NVMe, and DRAM holds \nCacheExperts{} slots ($\nCacheFrac\%$),
comfortably above the working set.

\begin{table}[t]
\centering\small
\caption{Footprint and DRAM residency for T1 ($\nTotalParams$\,B) with a
$24$\,GB expert cache on the measured host.}
\label{tab:amp}
\input{tab_amp}
\end{table}

Composing measured bandwidths with the calibrated cache model gives
Table~\ref{tab:proj}. At $1.5$ bits and batch~1 the projection is
\nTokSecOne{} tokens/s, fetching \nMBperTok\,MB per token at
\nIOatExpert\,GB/s; batch~32 reaches \nTokSecBatch{} tokens/s. At $1.0$ bit the
figures are \nTokSecOneBit{} and \nTokSecOneBitBatch{} tokens/s. The bfloat16
row projects \nTokSecFP{} tokens/s, so the $10.7\times$ compression to $1.5$
bits yields a $\nSpeedupOverFP\times$ throughput gain --- $\nAmpFactor\times$
more than compression alone, which is the amplification of
Eq.~\ref{eq:amp} made concrete.

Because the hit rate is the least certain input, Figure~\ref{fig:throughput}(a)
reports throughput across its entire range rather than at a single assumed
value. The qualitative conclusion is robust: even at $h=0$ the projection stays
near one token per second, because $Lk B(r)$ at $1.5$ bits is $4.8$\,GB and the
device delivers ${\sim}\nIOatExpert$\,GB/s.

\begin{table}[t]
\centering\small
\caption{Projected T1 decode throughput. Hardware terms measured; hit rate from
the calibrated popularity-pinned model with the Zipf bias subtracted.}
\label{tab:proj}
\input{tab_proj}
\end{table}

\begin{figure}[t]
\centering
\includegraphics[width=\textwidth]{figs/throughput.pdf}
\caption{(a) Sensitivity of projected batch-1 throughput to expert-cache hit
rate at $1.5$ bits. (b) Projected throughput versus weight rate and batch size.}
\label{fig:throughput}
\end{figure}

\begin{figure}[t]
\centering
\includegraphics[width=\textwidth]{figs/amplification.pdf}
\caption{(a) T1 footprint versus weight rate against the host's storage and DRAM
limits. (b) Fraction of expert slots resident in $24$\,GB of DRAM.}
\label{fig:amp}
\end{figure}

\subsection{A Constraint on the Decode Path}\label{sec:decode}

The projections assume dequantization keeps up with storage. In a
straightforward implementation it does not, and the gap is instructive.

Our PyTorch-level decode path --- \texttt{index\_select} gathers per residual
stage into a preallocated accumulator, int32 indices to avoid int64 promotion
--- sustains \nDecodeGw{} Gweight/s at $1.5$ bits, or \nDecodePacked\,GB/s of
packed input, against a storage stream of \nIOatExpert\,GB/s: short by roughly
$\nDecodeGap\times$.

The reason is bandwidth, not arithmetic. Reconstructing $1.5$-bit weights into
fp16 expands them $10.7\times$, so sustaining \nIOatExpert\,GB/s of packed
weights means writing ${\sim}\nFusedBw$\,GB/s of fp16 into VRAM and reading it
back for the GEMM, against $88.2$\,GB/s measured. \emph{Materializing
dequantized weights is infeasible on this class of GPU by roughly
$\nFusedRatio\times$ in bandwidth alone}, however well the gather is written.
Dequantization must be fused into the GEMM prologue so codebook indices expand
in registers or shared memory and fp16 weights never reach VRAM --- the design
QuIP\# and QTIP adopt for their inference kernels. Our measurement quantifies
why it is not optional at this scale. We did not implement a fused kernel; the
projections in Section~\ref{sec:projection} are I/O rooflines and should be read
as upper bounds contingent on such a kernel existing.

\section{Discussion and Limitations}\label{sec:limits}

\paragraph{What we did not do.}
We did not execute a trillion-parameter model. No $1$\,T checkpoint was
downloaded, quantized or run. T1 is a specified configuration and the throughput
figures are analytical compositions of measured host parameters with a modelled
cache term. They are not benchmark results and should not be cited as such.

\paragraph{Extrapolation risk.}
The cache model is validated at $E{=}64$, $L{=}16$ and applied at $E{=}320$,
$L{=}64$, assuming the Zipf exponent of expert popularity is scale-invariant.
This is the weakest link. It is plausible --- load-balancing auxiliary losses
push MoE routers toward similar popularity profiles --- but we have not tested
it on a model with $E>64$. If the distribution flattens with $E$, hit rates fall
and throughput moves down Figure~\ref{fig:throughput}(a). We report the full
sensitivity curve precisely because we cannot foreclose this. We also assume the
per-token working set scales as $kL$, which follows from the architecture rather
than from measurement.

\paragraph{Quality at 1T scale.}
Our perplexity results are for a $6.9$\,B model with $1.3$\,B active parameters,
and the degradation at \nBitsFreq{} bits ($\nPPLfp\rightarrow\nPPLFreq$) is
real. Larger models are generally more quantization-tolerant at fixed rate, so
sub-2-bit operation should be more forgiving at $1$\,T, but we have not verified
this and it should not be assumed. A reader who regards \nBitsFreq{} bits as too
lossy should read the \nBitsOursTwo-bit row of Table~\ref{tab:proj}
(\nTokSecTwoBit{} tokens/s at batch~1), which still fits the machine's storage
and still crosses the working-set threshold; the qualitative argument does not
depend on the most aggressive rate.

\paragraph{Unmodelled costs.}
The roofline omits KV-cache growth with context length, router computation,
prefill, filesystem and allocator overhead, and thermal throttling under
sustained load. Each pushes achieved throughput below the projection. Sustained
SSD streaming also carries a substantial per-token energy cost --- reported at
up to ${\sim}12\times$ an HBM baseline --- which matters for a battery-powered
device and which we did not measure.

\section{Conclusion}

Quantization's role in offloaded mixture-of-experts inference is better
understood as cache amplification than as footprint reduction. Because a token
touches $kL$ distinct expert slots before reusing any, recency-based expert
caches have a hard threshold below which they cannot hit at all --- which we
measure exactly --- and because resident capacity is inversely proportional to
the weight rate, quantization is what carries a system across it. On our host
the $10.7\times$ compression from bfloat16 to $1.5$ bits yields a
$\nSpeedupOverFP\times$ projected throughput gain, $\nAmpFactor\times$ more than
compression alone. The reframing has a design consequence: bits should be
allocated across experts by activation frequency, improving distortion and cache
residency together, and placement should be popularity-pinned rather than purely
recency-based.

Instantiated on one commodity laptop with measured storage, PCIe and GPU limits
and a cache model calibrated against real routing traces, a
$\nTotalParams$\,B-parameter MoE fits in \nFootprintOnePFive\,GB at $1.5$ bits
and projects \nTokSecOne--\nTokSecBatch{} tokens/s depending on batch size. The
binding engineering constraint is not storage but the dequantization path: fp16
materialization exceeds the device's memory bandwidth, so fusion into the GEMM
is mandatory. Whether a trillion-parameter model actually runs on a laptop is
not settled here --- but the storage and bandwidth budgets close, and the kernel
that has to be written to find out is named.

\paragraph{Reproducibility.}
All measurement and quantization code, the raw routing traces, and the JSON
artefacts from which every number in this paper is generated are released with
the paper. Numbers in the text are injected programmatically from those
artefacts.

\begin{thebibliography}{99}\small

\bibitem{gptq} E.~Frantar, S.~Ashkboos, T.~Hoefler, D.~Alistarh.
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained
Transformers. \emph{ICLR}, 2023.

\bibitem{awq} J.~Lin, J.~Tang, H.~Tang, S.~Yang, X.~Dang, S.~Han.
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.
\emph{MLSys}, 2024.

\bibitem{quip} J.~Chee, Y.~Cai, V.~Kuleshov, C.~De~Sa.
QuIP: 2-Bit Quantization of Large Language Models With Guarantees.
\emph{NeurIPS}, 2023.

\bibitem{quipsharp} A.~Tseng, J.~Chee, Q.~Sun, V.~Kuleshov, C.~De~Sa.
QuIP\#: Even Better LLM Quantization with Hadamard Incoherence and Lattice
Codebooks. \emph{ICML}, 2024.

\bibitem{aqlm} V.~Egiazarian, A.~Panferov, D.~Kuznedelev, E.~Frantar,
A.~Babenko, D.~Alistarh. Extreme Compression of Large Language Models via
Additive Quantization. \emph{ICML}, 2024.

\bibitem{qtip} A.~Tseng, Q.~Sun, D.~Hou, C.~De~Sa.
QTIP: Quantization with Trellises and Incoherence Processing.
\emph{NeurIPS}, 2024.

\bibitem{vptq} Y.~Liu et al. VPTQ: Extreme Low-bit Vector Post-Training
Quantization for Large Language Models. \emph{EMNLP}, 2024.

\bibitem{bitnet} S.~Ma, H.~Wang, L.~Ma, L.~Wang, W.~Wang, S.~Huang, L.~Dong,
R.~Wang, J.~Xue, F.~Wei. The Era of 1-bit LLMs: All Large Language Models are in
1.58 Bits. \emph{arXiv:2402.17764}, 2024.

\bibitem{flexgen} Y.~Sheng, L.~Zheng, B.~Yuan, Z.~Li, M.~Ryabinin, D.~Fu,
Z.~Xie, B.~Chen, C.~Barrett, J.~Gonzalez, P.~Liang, C.~R\'e, I.~Stoica, C.~Zhang.
FlexGen: High-Throughput Generative Inference of Large Language Models with a
Single GPU. \emph{ICML}, 2023.

\bibitem{flash} K.~Alizadeh, I.~Mirzadeh, D.~Belenko, K.~Khatamifard, M.~Cho,
C.~C.~Del~Mundo, M.~Rastegari, M.~Farajtabar. LLM in a Flash: Efficient Large
Language Model Inference with Limited Memory. \emph{ACL}, 2024.

\bibitem{powerinfer} Y.~Song, Z.~Mi, H.~Xie, H.~Chen.
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU.
\emph{SOSP}, 2024.

\bibitem{mixtraloffload} A.~Eliseev, D.~Mazur.
Fast Inference of Mixture-of-Experts Language Models with Offloading.
\emph{arXiv:2312.17238}, 2023.

\bibitem{moeinfinity} L.~Xue, Y.~Fu, Z.~Lu, L.~Mai, M.~Marina.
MoE-Infinity: Offloading-Efficient MoE Model Serving.
\emph{arXiv:2401.14361}, 2024.

\bibitem{flashmoe} FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache
Replacement for Mixture-of-Experts Inference on Edge Devices.
\emph{arXiv:2601.17063}, 2026.

\bibitem{mobile} MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU
with Mixture of Big Little Experts. \emph{arXiv:2510.12357}, 2025.

\bibitem{olmoe} N.~Muennighoff, L.~Soldaini, D.~Groeneveld, K.~Lo, J.~Morrison,
S.~Min, W.~Shi, P.~Walsh, O.~Tafjord, N.~Lambert, Y.~Gu, S.~Arora, A.~Bhagia,
D.~Schwenk, D.~Wadden, A.~Wettig, B.~Hui, T.~Dettmers, D.~Kiela, A.~Farhadi,
N.~A.~Smith, P.~W.~Koh, A.~Singh, H.~Hajishirzi.
OLMoE: Open Mixture-of-Experts Language Models. \emph{ICLR}, 2025.

\bibitem{deepseek} DeepSeek-AI. DeepSeek-V3 Technical Report.
\emph{arXiv:2412.19437}, 2024.

\bibitem{switch} W.~Fedus, B.~Zoph, N.~Shazeer. Switch Transformers: Scaling to
Trillion Parameter Models with Simple and Efficient Sparsity. \emph{JMLR}, 2022.

\bibitem{che} H.~Che, Y.~Tung, Z.~Wang. Hierarchical Web Caching Systems:
Modeling, Design and Experimental Results. \emph{IEEE JSAC}, 20(7), 2002.

\bibitem{wikitext} S.~Merity, C.~Xiong, J.~Bradbury, R.~Socher.
Pointer Sentinel Mixture Models. \emph{ICLR}, 2017.

\end{thebibliography}

\end{document}
