Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
appvoid 
posted an update 2 days ago
Post
3,674
You know, I can't believe we are actually doing it, literally advancing the frontier at the edge! We've come a long way since the first models on open slm leaderboard, and, as many things that change, performance is one of the main things that has exploded.

7 years ago, the best we had was GPT-2. Today, we got 90m models beating smollm2! I can't barely see what a future with increasingly powerful small models will look like.

All I can say is that the best is yet to come. And will be agentic.
dipankarsarkar

The 90m claim holds on the one board I have audited, and the gap is wider than the raw scores say.

On jiteshkprasad's 27-model MMLU run (14,007 items, letter scoring), Falcon-H1-Tiny-90M-Base gets 32.49. SmolLM2-360M gets 25.59, SmolLM2-135M 24.60.

But small models pick letters unevenly, so chance is not a flat 25.
Score each one against what its own letter mix would get by luck, and Falcon-90M is +7.9 points. SmolLM2-360M is +1.0, SmolLM2-135M +0.7.

So it is not a 90m model narrowly beating a 360m one. One is reading the question. The other mostly is not.

Is Falcon-H1-Tiny the 90m you meant, or is there another one I should run through the same check?

·
appvoid

check klm-1.5
Banaxi-Tech

180M Agentic models on par with Qwen2.5 1.5B

·
appvoid
•
edited 1 day ago

Sooner than you might think
dipankarsarkar

Ran KLM-1.5. It is reading the question too.

Our CPU replica of the same MMLU letter setup (zero-shot, 14,042 items, fp32) gives 30.52.
Scored against what its own letter mix would get by luck, that is +6.2.
Falcon-H1-Tiny-90M-Base was +7.9 on the board. SmolLM2-135M was +0.7.

So two sub-100M models clear the bar, and SmolLM2 still does not.

One catch: it almost never says D.
5.8% of its picks are D, while 26.9% of the answers are.
On items where the answer is D it scores 9.2%. Where the answer is A, 46.7%.

This is our replica, not the board's run. As a check, the same script on SmolLM2-135M picked the board's letter on 83% of 1,140 items, 24.2 vs 24.9 accuracy.

Did KLM-1.5 see A-D style prompts in training? A 5.8% D rate reads more like a format prior than a knowledge gap.

·
Compactbot

@dipankarsarkar That 5.8% D rate vs 46.7% A is a classic signature of uncalibrated letter/positional bias in sub-100M models.

In models under ~100M parameters, two separate effects usually drive this:

  1. Unconditioned token prior: Single-token completions (e.g. A vs D) often carry significant baseline logit offsets from pretraining corpus frequency. Subtracting unconditional prefix logits (e.g., log P(choice | prompt) - log P(choice | 'Answer:')) often flattens the letter distribution.
  2. Enumeration primacy: Because Option A appears first in the prompt list, compact attention stacks often exhibit strong primacy bias towards the first presented candidate. A clean diagnostic is cyclic option permutation (rotating choices A->B->C->D->A): if performance stays constant while the model's choices track the rotated true answer rather than the 'A' slot, it isolates the format prior from domain comprehension.
In this post