Delete KOZTAM.md

KOZTAM.md

@@ -1,225 +0,0 @@
----
-language:
-- tr
-license: mit
-library_name: pytorch
-pipeline_tag: text-classification
-base_model: dbmdz/bert-base-turkish-cased
-datasets:
-- behAIvNET/KOZTAM
-tags:
-- turkish
-- education
-- special-education
-- K-8
-- reading
-- reading-difficulty
-- reading-texts
-- synthetic-dataset
-- hearing-loss
-- cochlear
-- bert
-- berturk
-- coral
-- ordinal-regression
-- xai 
-- integrated-gradients
-metrics:
-- mae
-- accuracy
----
-
-# KOZTAM — Koklear Okuma Zorluğu Tahminleme Modeli
-
-**KOZTAM** (*Cochlear Reading-Difficulty Forecasting Model*) grades a Turkish text on an eight-level reading-difficulty scale (grades 1–8) with a frozen BERTurk encoder and a lightweight CORAL ordinal head, reaching **0.33 ordinal MAE**, **0.72 exact-grade accuracy**, and **0.96 within-one-grade accuracy** on held-out test data.
-
-## Data
-
-KOZTAM is trained on the [`behAIvNET/KOZTAM`](https://huggingface.co/datasets/behAIvNET/KOZTAM) dataset — **1,588 synthetic Turkish texts** spanning grades 1–8 in two categories (informative / narrative), generated under strict MEB-aligned readability targets. One exact-duplicate text (present under both a grade-3 and a grade-5 folder) was removed to eliminate a conflicting ordinal label, giving the final 1,588.
-
-The texts are partitioned into **1,110 training / 239 validation / 239 test** by two-stage stratified sampling over the 16 grade × category strata, so both the ordinal grade distribution and the ≈ 50/50 category balance are preserved in every split. Because the BERTurk backbone is frozen, each text's `[CLS]` embedding (768-d) is pre-computed once and cached, and only the head is trained on those cached vectors; the mean `[CLS]` norm is ≈ 25.2 and identical across the three splits, indicating no distributional shift between them.
-
-A structural property worth noting for modelling: realized text length rises from grade 1 through grade 7 but **drops at grade 8** (shorter, in both categories, than grades 5–7). Length is therefore not a monotone proxy for reading grade at the top of the scale, so the model must rely on lexical and syntactic density rather than on length.
-
-## Model
-
-```python
-import torch
-import torch.nn as nn
-from transformers import AutoTokenizer, AutoModel
-
-DEVICE     = "cuda" if torch.cuda.is_available() else "cpu"
-MODEL_NAME = "dbmdz/bert-base-turkish-cased"
-MAX_LEN    = 512
-K          = 8                     
-
-tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
-bert = AutoModel.from_pretrained(MODEL_NAME).to(DEVICE).eval()
-for p in bert.parameters():
-    p.requires_grad_(False)
-
-@torch.no_grad()
-def encode_cls(texts):
-    enc = tokenizer(texts, padding=True, truncation=True,
-                    max_length=MAX_LEN, return_tensors="pt").to(DEVICE)
-    return bert(**enc).last_hidden_state[:, 0, :]
-
-class CoralHead(nn.Module):
-    def __init__(self, in_dim, num_classes):
-        super().__init__()
-        self.fc   = nn.Linear(in_dim, 1, bias=False)
-        self.bias = nn.Parameter(torch.zeros(num_classes - 1))
-    def forward(self, x):
-        return self.fc(x) + self.bias
-
-class KOZTAMNet(nn.Module):
-    def __init__(self, in_dim=768, p=0.2):
-        super().__init__()
-        self.proj = nn.Sequential(
-            nn.Linear(in_dim, 256), nn.LayerNorm(256), nn.GELU(), nn.Dropout(p),
-            nn.Linear(256, 64))
-        self.ordinal  = CoralHead(64, K)    
-        self.category = nn.Linear(64, 1)    
-    def forward(self, x):
-        z = self.proj(x)
-        return z, self.ordinal(z), self.category(z).squeeze(-1)
-
-ckpt = torch.load("KOZTAM.pt", map_location=DEVICE)
-head = KOZTAMNet().to(DEVICE)
-head.load_state_dict(ckpt["state_dict"])
-head.eval()
-thresholds = torch.tensor(ckpt["coral_thresholds"], device=DEVICE)   
-
-CATEGORY = {0: "bilgilendirici (informative)", 1: "öyküleyici (narrative)"}
-
-@torch.no_grad()
-def predict(texts):
-    single = isinstance(texts, str)
-    cls   = encode_cls([texts] if single else list(texts))
-    z     = head.proj(cls)
-    score = head.ordinal.fc(z).squeeze(-1)                       
-    grade = (score[:, None] > thresholds[None, :]).sum(1) + 1     
-    cat   = (torch.sigmoid(head.category(z).squeeze(-1)) > 0.5).long()
-    out = [{"grade": int(g), "score": round(float(s), 4), "category": CATEGORY[int(c)]}
-           for g, s, c in zip(grade, score, cat)]
-    return out[0] if single else out
-
-print(predict("Sample text."))
-```
-
-## Performance
-
-Evaluated on the held-out **test set (239 texts)** the model never saw during training.
-
-**Headline metrics (calibrated).**
-
-| Metric | Value |
-|---|---|
-| Ordinal MAE | **0.326** |
-| Exact-grade accuracy | **0.715** |
-| ±1-grade accuracy | **0.958** |
-| Spearman ρ (predicted vs. true grade) | **0.965** |
-| Text-category accuracy | **1.000** |
-
-**Threshold calibration.** KOZTAM has two parts: the trained network, which outputs a continuous *difficulty score*, and seven **calibrated thresholds** that cut that score into discrete grades (stored in the checkpoint, applied at inference). The raw network already ranks texts near-perfectly — continuous-score Spearman ρ = **0.966** — but the default 0.5-thresholds mis-placed the cut points and collapsed predictions toward the extreme grades. Re-placing the thresholds on the validation split (coordinate descent on MAE, **no re-training**) produced the final metrics:
-
-| Metric (test) | Raw (0.5-threshold) | Calibrated |
-|---|---|---|
-| Ordinal MAE | 0.837 | 0.326 |
-| Exact accuracy | 0.423 | 0.715 |
-| ±1 accuracy | 0.782 | 0.958 |
-| Spearman ρ | 0.924 | 0.965 |
-
-On the validation split the same procedure moved MAE from 0.837 to 0.285, confirming the calibration generalizes rather than overfitting the test set.
-
-**Continuous difficulty score by grade (test).** The mean score increases monotonically across all eight grades with no inversion, which is the direct evidence that the encoder learned the reading-difficulty ordering:
-
-| Grade | Mean score ± SD |
-|---|---|
-| 1 | −6.91 ± 1.18 |
-| 2 | −2.24 ± 1.52 |
-| 3 | −0.99 ± 0.48 |
-| 4 | −0.33 ± 0.45 |
-| 5 | 0.15 ± 0.29 |
-| 6 | 0.73 ± 0.50 |
-| 7 | 1.43 ± 0.63 |
-| 8 | 4.07 ± 1.49 |
-
-**Per-grade MAE (calibrated, test).**
-
-| Grade | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
-|---|---|---|---|---|---|---|---|---|
-| MAE | 0.033 | 0.133 | 0.333 | 0.690 | 0.400 | 0.367 | 0.567 | 0.100 |
-
-The model is sharpest at the extremes (grades 1, 8) and at distinct mid-levels; its largest residual errors sit at the **3–4 and 6–7 boundaries**, where texts are genuinely close in readability. On the raw predictions the two categories were balanced (informative 0.832 / narrative 0.842) and truncated texts (>512 tokens, n = 12) were only marginally harder (1.000 vs. 0.828), so neither category nor truncation is a source of systematic error.
-
-**Confusion matrix (calibrated, test).** Rows = true grade, columns = predicted grade.
-
-| true \ pred | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
-|---|---|---|---|---|---|---|---|---|
-| **1** | 29 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
-| **2** | 3 | 26 | 1 | 0 | 0 | 0 | 0 | 0 |
-| **3** | 0 | 9 | 20 | 1 | 0 | 0 | 0 | 0 |
-| **4** | 0 | 1 | 11 | 12 | 3 | 2 | 0 | 0 |
-| **5** | 0 | 0 | 1 | 5 | 19 | 5 | 0 | 0 |
-| **6** | 0 | 0 | 0 | 2 | 4 | 22 | 1 | 1 |
-| **7** | 0 | 0 | 0 | 0 | 2 | 8 | 15 | 5 |
-| **8** | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 28 |
-
-![Training dynamics — loss and validation MAE/accuracy over epochs](training_curves.png)
-
-![Confusion matrix — test set](confusion_matrix.png)
-
-**Ablation (test).** Each loss component was removed in turn, with everything else — seed, initialization, batch order, scheduler, early stopping, and the same post-hoc threshold calibration — held identical.
-
-| Variant | Pure ρ | MAE | Exact acc | ±1 acc |
-|---|---|---|---|---|
-| Full (ordinal + triplet + category) | 0.966 | 0.326 | 0.715 | 0.958 |
-| − triplet | 0.963 | 0.402 | 0.636 | 0.962 |
-| − category | 0.964 | 0.410 | 0.636 | 0.954 |
-| Ordinal only | 0.962 | 0.410 | 0.628 | 0.962 |
-
-Removing the ordinal-aware triplet loss raises MAE by 0.075 while pure ranking ρ is essentially unchanged, showing the triplet term improves the *even spacing* of the embedding axis (its calibratability) rather than the ranking itself.
-
-## Explainability (XAI)
-
-Reading-difficulty attributions are produced with **Integrated Gradients (IG)** on an end-to-end version of the model (frozen BERTurk → continuous difficulty score), with token attributions merged back to whole words. Before attribution, the end-to-end score is verified against the cached score of the same text — they match to within **1.1 × 10⁻⁵**, confirming IG explains the exact deployed model. IG is run with `n_steps = 50`, and the convergence delta is 0.056 (negligible relative to the score magnitude), so the completeness axiom holds.
-
-Positive attribution means a word pushes the difficulty score **up** (harder); negative means it pushes **down** (easier). Three representative test cases are shown below. Because IG distributes each text's total score relative to an empty-content baseline, the absolute magnitude of the attributions scales with how far a text's score sits from that baseline — so magnitudes are comparable *within* a text, not across texts.
-
-**Case 1 — clear-easy (true grade 1, predicted 1).**
-
-| Increases difficulty | value | | Decreases difficulty | value |
-|---|---|---|---|---|
-| elmanın | +0.54 | | neşeyle | −0.27 |
-| o | +0.33 | | güzelce | −0.26 |
-| kuralımızdır | +0.27 | | denizi | −0.22 |
-| Dünyadaki | +0.24 | | yapıp | −0.20 |
-| yansıtır | +0.23 | | üç | −0.16 |
-
-![XAI word heatmap — clear-easy case](xai_easy.png)
-
-**Case 2 — clear-hard (true grade 7, predicted 7).**
-
-| Increases difficulty | value | | Decreases difficulty | value |
-|---|---|---|---|---|
-| ilerlerler | +0.26 | | kıyafetlerin | −0.06 |
-| abartıya | +0.13 | | hızlı | −0.05 |
-| ufkunu | +0.07 | | Kendi | −0.05 |
-| koruyanlar | +0.06 | | Kendi | −0.04 |
-| sadeliği | +0.06 | | | |
-
-![XAI word heatmap — clear-hard case](xai_hard.png)
-
-**Case 3 — boundary error (true grade 6, predicted 7).**
-
-| Increases difficulty | value | | Decreases difficulty | value |
-|---|---|---|---|---|
-| alışkanlıklarımızda | +0.69 | | temizliğimizi | −0.66 |
-| yaşamımızdaki | +0.60 | | ve | −0.31 |
-| cildimizin | +0.51 | | seçmesi | −0.28 |
-| kolaylaştırır | +0.23 | | temelidir | −0.28 |
-
-![XAI word heatmap — boundary-error case](xai_boundary.png)
-
-The boundary case is the most informative: the words that pushed a grade-6 text up into grade 7 are long, morphologically heavy Turkish words (*alışkanlıklarımızda*, *yaşamımızdaki*, *cildimizin*). This is direct evidence that the model reads morphological and lexical density as a difficulty signal, and that its single-grade error falls exactly on the kind of neighbouring boundary where the difficulty distinction is genuinely soft.