PixelModel v1 — Evaluation
Benchmark results for the Tiny-T2I-Leaderboard,
measured with the leaderboard's preferred tooling (torchmetrics). Everything
needed to reproduce these numbers ships in this repo — see EVAL_REPRODUCTION.md.
Parameter count
23,747 parameters in 10 tensors (5 weight matrices + 5 bias vectors), all
stored in model.png (160×149 px, 16-bit/weight codec) and exported to
model.safetensors with the count in the header metadata:
| Tensor | Shape | Params | Role |
|---|---|---|---|
| T1 + b1 | 80×64 + 80 | 5,200 | prompt embedding → trunk hidden |
| T2 + b2 | 64×80 + 64 | 5,184 | trunk hidden → latent |
| D1 + bd1 | 80×82 + 80 | 6,640 | latent + fourier(x,y) → decoder hidden |
| D2 + bd2 | 80×80 + 80 | 6,480 | decoder hidden → decoder hidden |
| D3 + bd3 | 3×80 + 3 | 243 | decoder hidden → RGB |
| Total | 23,747 (~0.024M) |
The prompt embedding is a hashed function (FNV-1a over character trigrams and
words) with zero learned parameters; there is no text encoder and no VAE
(text_encoder_parameters: 0, vae_parameters: 0 in both config.json and
the safetensors metadata). Well under the 1.5B ceiling — and 8.5× smaller than
v0's 202,752.
Method
- Real set: 5,000 MS-COCO val2014 (image, caption) pairs — rows 0–4,999 of
the
sayakpaul/coco-30-val-2014stream — images center-cropped to 256×256. - Generated set:
model.pngrun on those same 5,000 captions at the native 64×64 resolution (no post-upscaling). - FID:
torchmetrics.image.fid.FrechetInceptionDistance(feature=2048, InceptionV3 pool3; the metric resizes both sets to 299×299 internally). - CLIP Score:
torchmetrics.multimodal.CLIPScorewithopenai/clip-vit-base-patch32(the default), generated image vs the caption that produced it, averaged over all 5,000. - Train/eval hygiene: the model was trained on 20,000 pairs from rows 10,000+ of the same stream, content-hash-checked to be disjoint from the 5,000 eval images (0 overlaps found; the 30K rows are 30K unique images).
n=5000 is far past the singular-covariance regime that made v0's n=40 FID directional-only (a 2048×2048 covariance needs n ≫ 2048 to be non-singular; 5,000 clears it, 40 did not).
Results
| Metric | v0 (n=40) | v1 (n=5000) | Context |
|---|---|---|---|
| FID ↓ | 566.84 | 439.46 | Competitive T2I models score ~10–30; v1's caption-conditioned color/layout statistics move it meaningfully toward the real-photo distribution, but 64×64 color washes remain far outside it. |
| CLIP Score ↑ | 18.60 | 20.02 (cosine 0.2002) | ~15–20 is the no-semantic-correspondence floor; ~28–32 is competitive. v1 sits measurably above the floor — captions genuinely steer palette and layout — with 8.5× fewer parameters than v0. |
Training loss context: the model's final MSE (0.0627) is 9% below the "always output the mean COCO image" baseline (0.0691) — everything below that baseline is caption-conditioning, since the mean image is the best possible caption-blind predictor.
Leaderboard entry
{
name: "PixelModel-v1",
org: "bench-labs",
params: "0.024M",
fid: 439.46,
clipScore: 0.2002,
resolution: "64x64",
releaseDate: "2026-07-17",
links: {
card: "https://huggingface.co/bench-labs/pixelmodel-v1"
}
},
(clipScore given as the 0–1 cosine value per the template's field
description; the 0–100 form of the same number is 20.02.)