Ced-Collab/uef-scaling-a2

UEF scaling curve — campaign A2 (d12 / d15 / d24 / d28)

Trunk-size scaling rungs for a unified embedding field trained jointly over frozen understanding representations (SigLIP2 so400m-patch14-224 image states + flan-t5-small text states). Every rung is the same recipe at the same data and batch — only the trunk width/depth changes.

These are research checkpoints, not a product release. Nothing here is adjudicated: the numbers below are logged validation losses, not a verdict on the shape of the scaling curve.

Rungs

rung	hidden	depth_double	heads	trainable params	steps released
d12	768	10	12	178,573,312	10k · 20k · 25k · 40k · 45k · 50k · best_val
d15	960	13	15	334,866,688	10k · 20k · 25k · 40k · 45k · 50k
d24	1536	22	24	1,336,440,832	10k · 15k · 20k · 25k · 40k · 45k · 50k · best_val
d28	1792	26	28	2,121,404,416	10k · 15k · 20k · 25k · 40k · 45k · 50k · best_val

All rungs share text_preamble_depth: 2, head_dim: 64, patch_size: 14, pca_channels: 128, time_cond: in_context, time_tokens: 4.

Shared recipe (identical across rungs)

• global batch 4096, lr 4e-4, 25k steps then extended to 50k
• bf16 autocast, fp32 master weights + fp32 AdamW moments, fused AdamW
• dual EMA (slow + fast), both kept fp32
• union pretrain pool 38.4M pairs, text window 128, wds_shard_seed 4242
• 4 nodes x 4 H200, world size 16

Validation loss (logged, final step of each segment)

rung	step	long	short	jdb	img_t2i (long)
d12	25,000	0.6827	0.6681	0.6059	0.5852
d12	50,000	0.6680	0.6539	0.5968	0.5729
d15	25,000	0.6263	0.6125	0.5708	0.5339
d15	50,000	0.6090	0.5951	0.5609	0.5198
d24	25,000	0.5510	0.5368	0.5308	0.4461
d24	50,000	0.5342	0.5164	0.5159	0.4237
d28	25,000	0.5559	0.5443	0.5513	0.5446
d28	50,000	0.5290	0.5288	0.5401	0.5248

Provenance

Each rung's 0→25k segment and its 25k→50k continuation are separate Slurm jobs; the continuation auto-resumes from checkpoint_025000.pt.

rung	segment	run id	slurm job	commit
d12	0→25k	20260818-190255-a2-d12	40149805	d124497
d12	25k→50k	20260821-032904-a2ext-d12	40509242	fba6fa4
d15	0→25k	20260819-065052-a2-d15	40149806	1bd9777
d15	25k→50k	20260821-152650-a2ext-d15	40509287	fba6fa4
d24	0→25k	20260824-215931-a2-d24	41444888	7c42ccc
d24	25k→50k	20260826-220555-a2ext-d24	41444889	e0b5b3d
d28	0→25k	20260819-183555-a2-d28	40149811	1bd9777
d28	25k→45k	20260823-150824-a2ext-d28	41444826	fba6fa4
d28	45k→50k	20260826-151848-a2ext-d28	41713613	e0b5b3d

Every file additionally carries its own identity block (experiment id, segment id, parent segment id, config/dataset identity sha256, world size, global batch) and an export_provenance block naming the exact source checkpoint it came from.

Contents of a checkpoint

Weights-only export — the optimizer state and RNG state have been stripped, so these load for evaluation but are not resumable.

import torch
ck = torch.load("d15/checkpoint_050000.pt", map_location="cpu", weights_only=False)

ck["model"]      # raw trained weights          (fp32)
ck["ema"]        # slow EMA                     (fp32)
ck["ema_fast"]   # fast EMA                     (fp32)
ck["config"]     # full training config
ck["identity"]   # run provenance
ck["representation_manifest"]  # frozen repr specs + schedules
ck["step"], ck["best_val_loss"], ck["export_provenance"]

Tensors are bitwise identical to the training checkpoints they were cut from; the export only drops keys, it does not cast or repack.

d12/best_val.pt (step 47,000), d24/best_val.pt (step 48,000) and d28/best_val.pt (step 48,000) are those runs' lowest-validation checkpoints — genuinely different points from their step-50,000 checkpoints, so all three are published as files.

d15 has no best_val.pt. Its best-validation step was 50,000 and the file was verified bitwise identical to d15/checkpoint_050000.pt, so the duplicate 4.02 GB of weights is not published; the validation numbers it carried are in d15/best_val_metrics.json.

Caveats — read before using these in a comparison

1. Single seed. One run per rung. No seed repeats, so rung-to-rung gaps carry no error bars.
2. All four rungs now reach 50k, so 25k and 50k are both like-for-like read points. The intermediate ladders differ slightly: d24 and d28 include a 15k point that d12 and d15 do not, because 15k was a rolling checkpoint that happened to be captured before its continuation job rotated it away. Those two files are no longer on disk anywhere else.
3. Segmented continuation. Resuming reseeds a fresh global data permutation, so the ≥25k segment is not sample-order-aligned with a hypothetical single 50k run. d28 is segmented twice: its first continuation (job 41444826) was cancelled after reaching step 48,550, and the requeue (job 41713613) resumed from checkpoint_045000, so steps 45,000–48,550 were computed twice and the first pass discarded. The published d28 50k weights come from the requeue's continuous 45k→50k path.
4. JourneyDB pool repack. The jdb shards are repacked copies with 14 bad image members and their 14 text partners dropped (28 members total); the jdb manifest count is 4,197,986. Composition-identity against the other site's copy of the pool was not closed.
5. d24 was rerun. Its first attempt (20260821-032455-a2-d24, job
   40161133. died within 3 minutes; the published run is the retry, whose output directory is uef_scaleF25k_d24_kempner_r2 even though the identity.experiment_id inside the files reads uef_scaleF25k_d24_kempner.
6. Validation losses above are read from the run logs, are computed on small val batches, and are the training-time metric only. No FID / GenEval / DPG numbers are attached to these rungs.

Configs

configs/f25k_scale_d{12,15,24,28}.yml are the exact configs used, verbatim.