File size: 3,720 Bytes
aceb14e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | # PixelModel v1 — Evaluation
Benchmark results for the [Tiny-T2I-Leaderboard](https://huggingface.co/spaces/FlameF0X/Tiny-T2I-Leaderboard),
measured with the leaderboard's preferred tooling (torchmetrics). Everything
needed to reproduce these numbers ships in this repo — see `EVAL_REPRODUCTION.md`.
## Parameter count
23,747 parameters in 10 tensors (5 weight matrices + 5 bias vectors), all
stored in `model.png` (160×149 px, 16-bit/weight codec) and exported to
`model.safetensors` with the count in the header metadata:
| Tensor | Shape | Params | Role |
|---|---|---|---|
| T1 + b1 | 80×64 + 80 | 5,200 | prompt embedding → trunk hidden |
| T2 + b2 | 64×80 + 64 | 5,184 | trunk hidden → latent |
| D1 + bd1 | 80×82 + 80 | 6,640 | latent + fourier(x,y) → decoder hidden |
| D2 + bd2 | 80×80 + 80 | 6,480 | decoder hidden → decoder hidden |
| D3 + bd3 | 3×80 + 3 | 243 | decoder hidden → RGB |
| **Total** | | **23,747 (~0.024M)** | |
The prompt embedding is a hashed function (FNV-1a over character trigrams and
words) with zero learned parameters; there is no text encoder and no VAE
(`text_encoder_parameters: 0`, `vae_parameters: 0` in both `config.json` and
the safetensors metadata). Well under the 1.5B ceiling — and 8.5× smaller than
v0's 202,752.
## Method
- **Real set**: 5,000 MS-COCO val2014 (image, caption) pairs — rows 0–4,999 of
the [`sayakpaul/coco-30-val-2014`](https://huggingface.co/datasets/sayakpaul/coco-30-val-2014)
stream — images center-cropped to 256×256.
- **Generated set**: `model.png` run on those same 5,000 captions at the
native 64×64 resolution (no post-upscaling).
- **FID**: `torchmetrics.image.fid.FrechetInceptionDistance` (`feature=2048`,
InceptionV3 pool3; the metric resizes both sets to 299×299 internally).
- **CLIP Score**: `torchmetrics.multimodal.CLIPScore` with
`openai/clip-vit-base-patch32` (the default), generated image vs the caption
that produced it, averaged over all 5,000.
- **Train/eval hygiene**: the model was trained on 20,000 pairs from rows
10,000+ of the same stream, content-hash-checked to be disjoint from the
5,000 eval images (0 overlaps found; the 30K rows are 30K unique images).
n=5000 is far past the singular-covariance regime that made v0's n=40 FID
directional-only (a 2048×2048 covariance needs n ≫ 2048 to be non-singular;
5,000 clears it, 40 did not).
## Results
| Metric | v0 (n=40) | **v1 (n=5000)** | Context |
|---|---|---|---|
| FID ↓ | 566.84 | **439.46** | Competitive T2I models score ~10–30; v1's caption-conditioned color/layout statistics move it meaningfully toward the real-photo distribution, but 64×64 color washes remain far outside it. |
| CLIP Score ↑ | 18.60 | **20.02** (cosine 0.2002) | ~15–20 is the no-semantic-correspondence floor; ~28–32 is competitive. v1 sits measurably above the floor — captions genuinely steer palette and layout — with 8.5× fewer parameters than v0. |
Training loss context: the model's final MSE (0.0627) is 9% below the
"always output the mean COCO image" baseline (0.0691) — everything below that
baseline is caption-conditioning, since the mean image is the best possible
caption-blind predictor.
## Leaderboard entry
```js
{
name: "PixelModel-v1",
org: "bench-labs",
params: "0.024M",
fid: 439.46,
clipScore: 0.2002,
resolution: "64x64",
releaseDate: "2026-07-17",
links: {
card: "https://huggingface.co/bench-labs/pixelmodel-v1"
}
},
```
*(`clipScore` given as the 0–1 cosine value per the template's field
description; the 0–100 form of the same number is 20.02.)*
|