Benchmarks: replace alloma-3B/1B with Llama-3.1-8B-Instruct-Uz and Mistral-7B-Instruct-Uz

Both behbudiy Uzbek instruct models evaluated on the same strict 8-task COMET-primary suite. Regenerated overall and per-task charts. Notes the 3% invalid-output gate failures (TUMLU 5.71% both; Mistral sentiment 4.59%).

README.md

@@ -130,6 +130,95 @@ curl http://localhost:8000/v1/chat/completions \
   }'
 ```
 
+### Classification
+
+For classification, the model works best as a constrained label picker: give the
+label set in the prompt, ask for the label only, decode greedily, and cap
+`max_new_tokens`. This is exactly the protocol used for the sentiment and news
+benchmark scores below.
+
+```python
+import re
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "NeuronUz/NeuronAI-4B"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    device_map={"": 0},
+).eval()
+
+LABELS = [
+    "Siyosat", "Iqtisodiyot", "Texnologiya", "Sport", "Madaniyat",
+    "Salomatlik", "Oila va Jamiyat", "Ta'lim", "Ekologiya", "Xorijiy Yangiliklar",
+]
+
+PROMPT = """Quyidagi o‘zbekcha yangilikni bitta toifaga ajrating. Faqat toifa raqamini yozing.
+
+{labels}
+
+Matn: {text}
+
+Javob:"""
+
+
+def classify(text: str) -> str:
+    prompt = PROMPT.format(
+        labels="\n".join(f"{i} - {name}" for i, name in enumerate(LABELS)),
+        text=text[:4000],
+    )
+    inputs = tokenizer.apply_chat_template(
+        [{"role": "user", "content": prompt}],
+        add_generation_prompt=True,
+        enable_thinking=False,
+        return_tensors="pt",
+        return_dict=True,
+    ).to(model.device)
+
+    with torch.inference_mode():
+        output = model.generate(
+            **inputs,
+            max_new_tokens=8,
+            do_sample=False,          # greedy: labels must be deterministic
+        )
+
+    raw = tokenizer.decode(
+        output[0, inputs["input_ids"].shape[1]:],
+        skip_special_tokens=True,
+    ).strip()
+    match = re.search(r"\d+", raw)
+    return LABELS[int(match.group())] if match and int(match.group()) < len(LABELS) else raw
+
+
+print(classify(
+    "O‘zbekiston Markaziy banki asosiy stavkani o‘zgarishsiz qoldirdi."
+))  # -> Iqtisodiyot
+```
+
+Binary sentiment uses the same shape with a two-label set:
+
+```python
+SENTIMENT_PROMPT = (
+    "Quyidagi o‘zbekcha matnning kayfiyatini aniqlang: 'Ijobiy' yoki 'Salbiy'. "
+    "Faqat bitta yorliqni yozing.\n\nMatn: {text}\n\nYorliq:"
+)
+```
+
+Notes that matter for accuracy:
+
+- **Greedy decoding** (`do_sample=False`). The sampling preset in Quick start is
+  for open-ended chat; it adds label noise here.
+- **`enable_thinking=False`** — a thinking block spends the token budget before
+  the label appears.
+- **Small `max_new_tokens`** (8 is enough) plus a regex/prefix parser on the
+  output, so a stray word never becomes an invalid prediction.
+- **Numbered labels** for many-class tasks: one digit is easier to emit and
+  parse than a multi-word category name.
+- Keep prompt + text inside the 4,096-token serving limit; truncate long
+  articles (`text[:4000]` above).
+
 ## Benchmarks
 
 All five model result sets below cover the same full eight-task suite.
@@ -140,23 +229,28 @@ invalid-output gate.
 
 ![Per-task comparison](assets/tasks_comparison.png)
 
-| Benchmark | Metric | Weight | **NeuronAI-4B** | Qwen3.5-4B | alloma-8B | alloma-3B | alloma-1B |
+| Benchmark | Metric | Weight | **NeuronAI-4B** | Qwen3.5-4B | alloma-8B | Llama-3.1-8B-Instruct-Uz | Mistral-7B-Instruct-Uz |
 | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
-| UzLiB | accuracy | 0.20 | **61.20%** | 40.30% | 42.40% | 32.08% | 23.32% |
-| TUMLU-Uzbek | accuracy | 0.20 | **45.00%** | 40.43% | 20.71% | 27.71% | 22.00% |
-| FLORES+ en→uz | COMET | 0.15 | **0.8965** | 0.8555 | 0.8779 | 0.8673 | 0.7383 |
-| Uzbek news | accuracy | 0.10 | **79.15%** | 67.34% | 57.77% | 13.60% | 25.41% |
-| MMLU English | accuracy | 0.10 | 64.06% | **72.66%** | 53.47% | 38.73% | 21.98% |
-| MMLU Uzbek | accuracy | 0.10 | **57.01%** | 52.58% | 40.04% | 32.74% | 21.11% |
-| FLORES+ uz→en | COMET | 0.05 | **0.8763** | 0.8618 | 0.8713 | 0.7954 | 0.7636 |
-| Uzbek sentiment | accuracy | 0.05 | **95.75%** | 84.82% | 79.94% | 38.85% | 79.54% |
-| **Normalized weighted score** | | 1.00 | **0.6724** | 0.5978 | 0.5187 | 0.4147 | 0.3661 |
-
-Alloma runs used the `APST` apostrophe preprocessing required by their model
-cards; NeuronAI and stock Qwen did not. The alloma-8B column combines its full
-model-card-protocol evaluation with separately archived full UzLiB,
-TUMLU-Uzbek, and MMLU-Uzbek runs. Exact source files, scores, and run IDs are
-included in [`benchmark_results.json`](benchmark_results.json).
+| UzLiB | accuracy | 0.20 | **61.20%** | 40.30% | 42.40% | 31.65% | 32.78% |
+| TUMLU-Uzbek | accuracy | 0.20 | **45.00%** | 40.43% | 20.71% | 32.00% | 33.71% |
+| FLORES+ en→uz | COMET | 0.15 | **0.8965** | 0.8555 | 0.8779 | 0.8667 | 0.8859 |
+| Uzbek news | accuracy | 0.10 | **79.15%** | 67.34% | 57.77% | 60.34% | 62.09% |
+| MMLU English | accuracy | 0.10 | 64.06% | **72.66%** | 53.47% | 47.58% | 29.50% |
+| MMLU Uzbek | accuracy | 0.10 | **57.01%** | 52.58% | 40.04% | 38.72% | 35.06% |
+| FLORES+ uz→en | COMET | 0.05 | **0.8763** | 0.8618 | 0.8713 | 0.7765 | 0.7826 |
+| Uzbek sentiment | accuracy | 0.05 | **95.75%** | 84.82% | 79.94% | 82.59% | 80.83% |
+| **Normalized weighted score** | | 1.00 | **0.6724** | 0.5978 | 0.5187 | 0.5095 | 0.4969 |
+
+The alloma-8B run used the `APST` apostrophe preprocessing required by its model
+card, and its column combines the full model-card-protocol evaluation with
+separately archived full UzLiB, TUMLU-Uzbek, and MMLU-Uzbek runs. NeuronAI-4B,
+stock Qwen, and both `behbudiy` Uzbek instruct models were evaluated by the same
+strict COMET-primary suite without APST preprocessing. On the two `behbudiy`
+models the suite's 3% invalid-output gate was exceeded on TUMLU-Uzbek (5.71% for
+both) and, for Mistral-7B-Instruct-Uz, on sentiment (4.59%); those are
+answer-format parse failures, so the affected task scores are a floor rather
+than a ceiling. Exact source files, scores, and run IDs are included in
+[`benchmark_results.json`](benchmark_results.json).
 
 ### Run the benchmarks on your computer
 

benchmark_results.json

@@ -103,55 +103,55 @@
       },
       "normalized_weighted_score": 0.5186512896503792
     },
-    "alloma-3B": {
-      "run_id": "alloma_3B_apst_full_20260815_142714",
-      "suite_id": "public_uzbek_full_vllm_apst",
+    "Llama-3.1-8B-Instruct-Uz": {
+      "run_id": "strict_full_llama31_8b_instruct_uz",
+      "suite_id": "public_uzbek_full_comet_vllm_v2",
       "tasks": {
-        "uzlib_slice_full": 0.32079527135948416,
-        "tumlu_uzbek_full": 0.27714285714285714,
-        "flores_en_uz_full": 0.8673252673692642,
-        "news_full": 0.1359802000618748,
-        "mmlu_english_full": 0.3872667711152257,
-        "mmlu_uz_full": 0.32744623273038026,
-        "flores_uz_en_full": 0.7954316840740269,
-        "sentiment_binary_full": 0.3885
+        "uzlib_slice_full": 0.31649650725416445,
+        "tumlu_uzbek_full": 0.32,
+        "flores_en_uz_full": 0.8666636186669038,
+        "news_full": 0.603372176961947,
+        "mmlu_english_full": 0.47578692493946734,
+        "mmlu_uz_full": 0.38719555618857715,
+        "flores_uz_en_full": 0.7764609643717557,
+        "sentiment_binary_full": 0.8259
       },
       "invalid_rate": {
-        "uzlib_slice_full": 0.0010746910263299302,
-        "tumlu_uzbek_full": 0.28714285714285714,
+        "uzlib_slice_full": 0.0,
+        "tumlu_uzbek_full": 0.05714285714285714,
         "flores_en_uz_full": null,
-        "news_full": 7.218727441476746e-05,
-        "mmlu_english_full": 0.06900726392251816,
-        "mmlu_uz_full": 0.07726819541375872,
+        "news_full": 2.0624935547076414e-05,
+        "mmlu_english_full": 0.0037031761857285288,
+        "mmlu_uz_full": 0.009471585244267198,
         "flores_uz_en_full": null,
-        "sentiment_binary_full": 0.4405
+        "sentiment_binary_full": 0.0045
       },
-      "normalized_weighted_score": 0.41468665305295505
+      "normalized_weighted_score": 0.5095287981878477
     },
-    "alloma-1B": {
-      "run_id": "alloma_1B_apst_full_20260811_114037",
-      "suite_id": "public_uzbek_full_vllm_apst",
+    "Mistral-7B-Instruct-Uz": {
+      "run_id": "strict_full_mistral7b_instruct_uz",
+      "suite_id": "public_uzbek_full_comet_vllm_v2",
       "tasks": {
-        "uzlib_slice_full": 0.23320795271359485,
-        "tumlu_uzbek_full": 0.22,
-        "flores_en_uz_full": 0.7383331350348018,
-        "news_full": 0.2541404558110756,
-        "mmlu_english_full": 0.21976926363765845,
-        "mmlu_uz_full": 0.21108104258652613,
-        "flores_uz_en_full": 0.7636193416348852,
-        "sentiment_binary_full": 0.7954
+        "uzlib_slice_full": 0.3277807630306287,
+        "tumlu_uzbek_full": 0.33714285714285713,
+        "flores_en_uz_full": 0.8859170079972982,
+        "news_full": 0.6209343095802825,
+        "mmlu_english_full": 0.294972226178607,
+        "mmlu_uz_full": 0.3505910838911836,
+        "flores_uz_en_full": 0.7826120569790223,
+        "sentiment_binary_full": 0.8083
       },
       "invalid_rate": {
-        "uzlib_slice_full": 0.04567436861902203,
-        "tumlu_uzbek_full": 0.36714285714285716,
+        "uzlib_slice_full": 0.011284255776464266,
+        "tumlu_uzbek_full": 0.05714285714285714,
         "flores_en_uz_full": null,
-        "news_full": 0.0003506239043002991,
-        "mmlu_english_full": 0.09393248824953711,
-        "mmlu_uz_full": 0.17882068081469876,
+        "news_full": 0.0,
+        "mmlu_english_full": 0.025708588520153824,
+        "mmlu_uz_full": 0.01167924797037459,
         "flores_uz_en_full": null,
-        "sentiment_binary_full": 0.0063
+        "sentiment_binary_full": 0.0459
       },
-      "normalized_weighted_score": 0.3661490569296942
+      "normalized_weighted_score": 0.4969133053139477
     }
   },
   "checkpoints": {