CodonProject/Whisper-Tiny-Encoder

Whisper-Tiny Audio Encoder (codon format)

Encoder-only weights of openai/whisper-tiny, re-keyed into the naming convention used by the codon library so they can be loaded into codon.block.model.WhisperTinyAudioEncoder with a single call.

This is a feature extractor, not a speech-to-text model: the decoder and the tokenizer are intentionally not included, so it cannot transcribe audio on its own.

Files

File	Size	Tensors	Description
whisper_tiny_encoder.safetensors	15.7 MB	67	Encoder weights, fp16, codon key convention

Model details

Architecture	Whisper encoder (pre-LN Transformer, bidirectional self-attention)
Parameters	8,208,384 (8.21M)
d_model	384
Encoder layers	4
Attention heads	6 (head dim 64)
FFN dim	1536, GELU
Mel bins	80
Max positions	1500
Output	[B, T/2, 384] for input mel [B, 80, T]
Precision	fp16 (upcast to fp32 on load for fp32 execution)
License	Apache-2.0

Requirements

pip install codon-model==0.0.7b8

Source: CodonProject/codon-model

Usage

import torch
from codon.block.model import WhisperTinyAudioEncoder

encoder = WhisperTinyAudioEncoder(pool_stride=1)
encoder.load('whisper_tiny_encoder.safetensors', strict=True)
encoder = encoder.float()               # weights are fp16; upcast for fp32 execution
encoder.eval()

mel = torch.randn(1, 80, 3000)          # 30 s of log-mel @ 100 Hz
with torch.no_grad():
    hidden, _ = encoder(mel)
print(hidden.shape)                     # torch.Size([1, 1500, 384])

Half-precision execution is also supported — just call .half() and feed a half-precision mel instead (expect a larger numerical drift, ~6e-2, since the arithmetic itself runs in fp16):

Loading straight from this repository:

encoder = WhisperTinyAudioEncoder().from_remote()

Downsampling for LLM consumption

The raw encoder emits one token per 20 ms (50 tokens/s), which is usually far too dense for a language model. pool_stride appends a non-overlapping average pool after the encoder (weights are unaffected):

encoder = WhisperTinyAudioEncoder(pool_stride=8)
encoder = encoder.float()
with torch.no_grad():
    hidden, _ = encoder(torch.randn(1, 80, 3000))
print(hidden.shape)                     # torch.Size([1, 187, 384]) -> 6.25 tokens/s

Tensor naming

Keys follow the codon convention (proj_* for projections). The full layout:

conv1.weight                      (384, 80, 3)
conv1.bias                        (384,)
conv2.weight                      (384, 384, 3)
conv2.bias                        (384,)
embed_positions.weight            (1500, 384)
layers.{0..3}.attn_norm.weight    (384,)
layers.{0..3}.attn_norm.bias      (384,)
layers.{0..3}.attn.proj_q.weight  (384, 384)
layers.{0..3}.attn.proj_q.bias    (384,)
layers.{0..3}.attn.proj_k.weight  (384, 384)     # no bias
layers.{0..3}.attn.proj_v.weight  (384, 384)
layers.{0..3}.attn.proj_v.bias    (384,)
layers.{0..3}.attn.proj_o.weight  (384, 384)
layers.{0..3}.attn.proj_o.bias    (384,)
layers.{0..3}.fn_norm.weight      (384,)
layers.{0..3}.fn_norm.bias        (384,)
layers.{0..3}.mlp.proj_fc1.weight (1536, 384)
layers.{0..3}.mlp.proj_fc1.bias   (1536,)
layers.{0..3}.mlp.proj_fc2.weight (384, 1536)
layers.{0..3}.mlp.proj_fc2.bias   (384,)
norm.weight                       (384,)
norm.bias                         (384,)

Note that proj_k has no bias — this mirrors the original Whisper design (its k_proj is the only projection without a bias in both the encoder and the decoder). The positional table uses Whisper's own layout: weight[:, :192] holds sine values and weight[:, 192:] holds cosine values for inv_freq = exp(-log(10000) * arange(192) / 192).

Provenance and verification

Weights are copied from openai/whisper-tiny (model.encoder.*), renamed, and saved as fp16. No values were modified: converting the fp32 export to fp16 and back to fp32 is bit-exact element-wise, i.e. the source values already only carried fp16 precision (max absolute weight magnitude is ~16, well inside fp16 range).

The codon implementation was verified against the official transformers implementation (WhisperModel(...).encoder) with both models loading the same checkpoint (upcast to fp32) and consuming the same input:

mel length	output shape	max abs difference
3000	[2, 1500, 384]	5.9e-05
1000 (padded)	[2, 500, 384]	3.9e-05
512 (padded)	[2, 256, 384]	2.2e-05

Differences are at fp32 rounding level, not implementation differences.

Limitations

• Encoder only. No decoder, no vocabulary, no tokenizer — cannot produce text.
• Fixed 30 s input. Whisper pads every input to 3000 mel frames. Short clips must be zero-padded to 3000 and the padded output frames discarded; the padded region does influence the valid region through convolution and LayerNorm, so it is not equivalent to running the encoder on a shorter input.
• Not causal / not streaming. The encoder is bidirectional and stateless, so it cannot be used for incremental streaming without recomputing a full window.
• Precision. Tensors are stored as fp16. The fp32 -> fp16 -> fp32 round trip is element-wise exact, so no information was lost relative to the source export, and running the model in fp32 after upcasting reproduces the fp32 reference exactly. Native fp16 execution (model.half()) is supported but introduces the usual half-precision drift (~6e-2 on this model).

Citation

If you use these weights, please cite the original work:

@misc{radford2022whisper,
  doi = {10.48550/ARXIV.2212.04356},
  url = {https://arxiv.org/abs/2212.04356},
  author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  title = {Robust Speech Recognition via Large-Scale Weak Supervision},
  publisher = {arXiv},
  year = {2022},
  copyright = {arXiv.org perpetual, non-exclusive license}
}

Acknowledgements

Original model by OpenAI. This repository only re-keys the encoder weights for use with codon.