| --- |
| title: Compact Language Model Index |
| emoji: 📊 |
| colorFrom: indigo |
| colorTo: blue |
| sdk: static |
| app_file: build/index.html |
| fullWidth: true |
| short_description: Chance-normalized evaluation of compact language models. |
| --- |
| |
| # Compact Language Model Index |
|
|
| CompactLMIndex is a curated benchmark for smaller language models. It uses one |
| fixed protocol to compare models across commonsense reasoning, science and |
| world knowledge, and grammatical competence. |
|
|
| ## Evaluation suite |
|
|
| The published composite includes six benchmarks: |
|
|
| | Area | Benchmark | |
| | --- | --- | |
| | Situational commonsense | HellaSwag | |
| | Physical commonsense | PIQA | |
| | Science and world knowledge | ARC-Easy | |
| | Science and world knowledge | ARC-Challenge | |
| | Conceptual commonsense | CommonsenseQA | |
| | Grammatical competence | BLiMP | |
|
|
| All models use zero-shot, non-chat log-likelihood prompts. A complete run of |
| the core suite is required for a model to receive a rank. |
|
|
| ## Scoring |
|
|
| Each benchmark score is normalized for random chance: |
|
|
| ```text |
| 100 × (accuracy − chance) / (1 − chance) |
| ``` |
|
|
| Random performance maps to 0 and perfect performance maps to 100. Scores |
| below chance remain negative. |
|
|
| The composite is domain-balanced: |
|
|
| - commonsense: 50%, split equally between HellaSwag, PIQA, and CommonsenseQA; |
| - science and world knowledge: 25%, with ARC-Easy contributing 75% and |
| ARC-Challenge 25% within that domain; |
| - grammatical competence: 25%, from BLiMP. |
|
|
| BLiMP is macro-averaged across its 12 linguistic categories. CommonsenseQA |
| scores the full answer text with length normalization rather than answer labels |
| such as A–E. |
|
|
| ## Uncertainty and ranking |
|
|
| Every displayed benchmark and composite interval is a 95% percentile bootstrap |
| interval. Model comparisons use paired bootstrap resampling on matched |
| evaluation items. Rank ranges show when neighbouring models cannot be reliably |
| distinguished under those comparisons. |
|
|
| ## Scope and inclusion |
|
|
| CompactLMIndex is curated rather than an open intake list. Published results |
| are checked against their saved evaluation artifacts, and final inclusion and |
| presentation decisions remain with Universal Computing Research. |
|
|
| If you have results for a model that fits the benchmark scope, open a |
| discussion on this Space for consideration. |
|
|
| ## Credits |
|
|
| Inspired by the [Open LLM Leaderboard](https://huggingface.co/open-llm-leaderboard), |
| [Axiomic Labs Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard), |
| and [OpenAI Parameter Golf](https://github.com/openai/parameter-golf). |
|
|
