Overcoming the ARC-Easy floor and vocabulary budget on an 80M footprint
Hi Emre,
Pretraining an 80M causal model from scratch on dual Kaggle T4 GPUs with QK-norm, GQA, and a clean WSD schedule is a great zero-budget achievement.
Looking at your benchmark metrics and parameter allocation:
HellaSwag showing +6.2% over random demonstrates solid local continuation learning from the 307M token mix.
ARC-Easy landing at 25.60% hits the random floor (25.00%). At 80M parameters trained on 307M tokens, the model sees roughly 3.8 tokens per parameter (Chinchilla saturation typically requires ~20 tokens/param), while the 20-layer depth limits multi-step relational reasoning.
Even with tied embeddings, the 16,384 vocabulary at 576 hidden width consumes 9.44M parameters (11.8% of your entire 80.2M model).
At ~3.54M parameters per transformer block, that static lookup table costs nearly 2.7 full layers of compute.
In an open architecture project called Maba (101M reference model: https://huggingface.co/AndrewThompson1233/maba-v1-architecture), we address depth and capacity bottlenecks in sub-100M regimes using two methods:
Deterministic 2-pass block recycling:
Passing representations through the 20 physical layers twice with pass-specific conditioning (Split RMSNorm scales) expands depth to 40 effective layers at zero additional parameter cost. This extra non-linear depth is often what breaks small models out of the 25% ARC floor without requiring millions more training tokens.Low-rank vocabulary factorization:
Projecting 16,384 -> 64 -> 576 cuts the embedding table from 9.44M down to ~1.09M parameters. Reallocating the saved 8.35M weights directly into the layer stack funds 2 additional transformer blocks (expanding from 20 to 22 layers) within the exact same 80M budget.
Did you test deeper topologies or recurrent passes before locking in the 20-layer configuration on Kaggle?
Best,
Andrew
Hi Andrew,
Thanks for checking out PicoLM-80M and sharing the Maba architecture!
Your analysis of the ARC-Easy floor and vocabulary parameter trade-off is spot
on. In our initial v1 run, we prioritized establishing a stable zero-budget
baseline on dual T4 GPUs, validating the QK-norm + GQA + WSD pipeline, and
verifying the tokenizer synchronization.
Interestingly, your first point aligns directly with our next iteration: we are
currently training PicoLM-V2, which implements immediate block-wise layer
sharing (MobileLLM-LS style) with 18 physical blocks executed twice to achieve
36 layers of computational depth, along with an expanded SwiGLU intermediate
dimension (1664) and targeted factual/science QA data injection to break the ARC
floor.
The low-rank vocabulary factorization idea 16384 to 64 to 576 is very
intriguing we will definitely evaluate it for our future parameter allocation
experiments.
I'll share the V2 empirical metrics as soon as the run concludes!
Best,
Emre
Hi Emre,
Fantastic to hear that! Converging on 18 physical blocks executed twice for 36 effective layers is a great architectural pivot for PicoLM-V2.
One quick practical observation if you are running MobileLLM-LS style weight sharing from scratch:
Watch the hidden state variance across the repeated passes. Because the second pass refines already-processed features rather than raw embeddings, representations tend to shift in magnitude. If you notice gradient instability during early warmup, keeping distinct RMSNorm gain parameters for each pass (Split RMSNorm) is usually the magic bullet that stabilizes the dynamics without adding perceptible parameter weight.
Expanding the SwiGLU width to 1664 alongside the targeted science QA mix should give ARC-Easy the exact relational boost it needs to break well past the 25% floor.
Really looking forward to seeing the V2 benchmarks once the run concludes. Best of luck with the T4 compute run!
Best,
Andrew