goerkemsaylam commited on
Commit
327d870
·
verified ·
1 Parent(s): c522023

Add KOZTAM model card, weights, and figures

Browse files
Files changed (9) hide show
  1. .gitattributes +2 -0
  2. KOZTAM.md +225 -0
  3. KOZTAM.pt +3 -0
  4. README.md +225 -0
  5. confusion_matrix.png +0 -0
  6. training_curves.png +0 -0
  7. xai_boundary.png +3 -0
  8. xai_easy.png +0 -0
  9. xai_hard.png +3 -0
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ xai_boundary.png filter=lfs diff=lfs merge=lfs -text
37
+ xai_hard.png filter=lfs diff=lfs merge=lfs -text
KOZTAM.md ADDED
@@ -0,0 +1,225 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - tr
4
+ license: mit
5
+ library_name: pytorch
6
+ pipeline_tag: text-classification
7
+ base_model: dbmdz/bert-base-turkish-cased
8
+ datasets:
9
+ - behAIvNET/KOZTAM
10
+ tags:
11
+ - turkish
12
+ - education
13
+ - special-education
14
+ - K-8
15
+ - reading
16
+ - reading-difficulty
17
+ - reading-texts
18
+ - synthetic-dataset
19
+ - hearing-loss
20
+ - cochlear
21
+ - bert
22
+ - berturk
23
+ - coral
24
+ - ordinal-regression
25
+ - xai
26
+ - integrated-gradients
27
+ metrics:
28
+ - mae
29
+ - accuracy
30
+ ---
31
+
32
+ # KOZTAM — Koklear Okuma Zorluğu Tahminleme Modeli
33
+
34
+ **KOZTAM** (*Cochlear Reading-Difficulty Forecasting Model*) grades a Turkish text on an eight-level reading-difficulty scale (grades 1–8) with a frozen BERTurk encoder and a lightweight CORAL ordinal head, reaching **0.33 ordinal MAE**, **0.72 exact-grade accuracy**, and **0.96 within-one-grade accuracy** on held-out test data.
35
+
36
+ ## Data
37
+
38
+ KOZTAM is trained on the [`behAIvNET/KOZTAM`](https://huggingface.co/datasets/behAIvNET/KOZTAM) dataset — **1,588 synthetic Turkish texts** spanning grades 1–8 in two categories (informative / narrative), generated under strict MEB-aligned readability targets. One exact-duplicate text (present under both a grade-3 and a grade-5 folder) was removed to eliminate a conflicting ordinal label, giving the final 1,588.
39
+
40
+ The texts are partitioned into **1,110 training / 239 validation / 239 test** by two-stage stratified sampling over the 16 grade × category strata, so both the ordinal grade distribution and the ≈ 50/50 category balance are preserved in every split. Because the BERTurk backbone is frozen, each text's `[CLS]` embedding (768-d) is pre-computed once and cached, and only the head is trained on those cached vectors; the mean `[CLS]` norm is ≈ 25.2 and identical across the three splits, indicating no distributional shift between them.
41
+
42
+ A structural property worth noting for modelling: realized text length rises from grade 1 through grade 7 but **drops at grade 8** (shorter, in both categories, than grades 5–7). Length is therefore not a monotone proxy for reading grade at the top of the scale, so the model must rely on lexical and syntactic density rather than on length.
43
+
44
+ ## Model
45
+
46
+ ```python
47
+ import torch
48
+ import torch.nn as nn
49
+ from transformers import AutoTokenizer, AutoModel
50
+
51
+ DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
52
+ MODEL_NAME = "dbmdz/bert-base-turkish-cased"
53
+ MAX_LEN = 512
54
+ K = 8
55
+
56
+ tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
57
+ bert = AutoModel.from_pretrained(MODEL_NAME).to(DEVICE).eval()
58
+ for p in bert.parameters():
59
+ p.requires_grad_(False)
60
+
61
+ @torch.no_grad()
62
+ def encode_cls(texts):
63
+ enc = tokenizer(texts, padding=True, truncation=True,
64
+ max_length=MAX_LEN, return_tensors="pt").to(DEVICE)
65
+ return bert(**enc).last_hidden_state[:, 0, :]
66
+
67
+ class CoralHead(nn.Module):
68
+ def __init__(self, in_dim, num_classes):
69
+ super().__init__()
70
+ self.fc = nn.Linear(in_dim, 1, bias=False)
71
+ self.bias = nn.Parameter(torch.zeros(num_classes - 1))
72
+ def forward(self, x):
73
+ return self.fc(x) + self.bias
74
+
75
+ class KOZTAMNet(nn.Module):
76
+ def __init__(self, in_dim=768, p=0.2):
77
+ super().__init__()
78
+ self.proj = nn.Sequential(
79
+ nn.Linear(in_dim, 256), nn.LayerNorm(256), nn.GELU(), nn.Dropout(p),
80
+ nn.Linear(256, 64))
81
+ self.ordinal = CoralHead(64, K)
82
+ self.category = nn.Linear(64, 1)
83
+ def forward(self, x):
84
+ z = self.proj(x)
85
+ return z, self.ordinal(z), self.category(z).squeeze(-1)
86
+
87
+ ckpt = torch.load("KOZTAM.pt", map_location=DEVICE)
88
+ head = KOZTAMNet().to(DEVICE)
89
+ head.load_state_dict(ckpt["state_dict"])
90
+ head.eval()
91
+ thresholds = torch.tensor(ckpt["coral_thresholds"], device=DEVICE)
92
+
93
+ CATEGORY = {0: "bilgilendirici (informative)", 1: "öyküleyici (narrative)"}
94
+
95
+ @torch.no_grad()
96
+ def predict(texts):
97
+ single = isinstance(texts, str)
98
+ cls = encode_cls([texts] if single else list(texts))
99
+ z = head.proj(cls)
100
+ score = head.ordinal.fc(z).squeeze(-1)
101
+ grade = (score[:, None] > thresholds[None, :]).sum(1) + 1
102
+ cat = (torch.sigmoid(head.category(z).squeeze(-1)) > 0.5).long()
103
+ out = [{"grade": int(g), "score": round(float(s), 4), "category": CATEGORY[int(c)]}
104
+ for g, s, c in zip(grade, score, cat)]
105
+ return out[0] if single else out
106
+
107
+ print(predict("Sample text."))
108
+ ```
109
+
110
+ ## Performance
111
+
112
+ Evaluated on the held-out **test set (239 texts)** the model never saw during training.
113
+
114
+ **Headline metrics (calibrated).**
115
+
116
+ | Metric | Value |
117
+ |---|---|
118
+ | Ordinal MAE | **0.326** |
119
+ | Exact-grade accuracy | **0.715** |
120
+ | ±1-grade accuracy | **0.958** |
121
+ | Spearman ρ (predicted vs. true grade) | **0.965** |
122
+ | Text-category accuracy | **1.000** |
123
+
124
+ **Threshold calibration.** KOZTAM has two parts: the trained network, which outputs a continuous *difficulty score*, and seven **calibrated thresholds** that cut that score into discrete grades (stored in the checkpoint, applied at inference). The raw network already ranks texts near-perfectly — continuous-score Spearman ρ = **0.966** — but the default 0.5-thresholds mis-placed the cut points and collapsed predictions toward the extreme grades. Re-placing the thresholds on the validation split (coordinate descent on MAE, **no re-training**) produced the final metrics:
125
+
126
+ | Metric (test) | Raw (0.5-threshold) | Calibrated |
127
+ |---|---|---|
128
+ | Ordinal MAE | 0.837 | 0.326 |
129
+ | Exact accuracy | 0.423 | 0.715 |
130
+ | ±1 accuracy | 0.782 | 0.958 |
131
+ | Spearman ρ | 0.924 | 0.965 |
132
+
133
+ On the validation split the same procedure moved MAE from 0.837 to 0.285, confirming the calibration generalizes rather than overfitting the test set.
134
+
135
+ **Continuous difficulty score by grade (test).** The mean score increases monotonically across all eight grades with no inversion, which is the direct evidence that the encoder learned the reading-difficulty ordering:
136
+
137
+ | Grade | Mean score ± SD |
138
+ |---|---|
139
+ | 1 | −6.91 ± 1.18 |
140
+ | 2 | −2.24 ± 1.52 |
141
+ | 3 | −0.99 ± 0.48 |
142
+ | 4 | −0.33 ± 0.45 |
143
+ | 5 | 0.15 ± 0.29 |
144
+ | 6 | 0.73 ± 0.50 |
145
+ | 7 | 1.43 ± 0.63 |
146
+ | 8 | 4.07 ± 1.49 |
147
+
148
+ **Per-grade MAE (calibrated, test).**
149
+
150
+ | Grade | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
151
+ |---|---|---|---|---|---|---|---|---|
152
+ | MAE | 0.033 | 0.133 | 0.333 | 0.690 | 0.400 | 0.367 | 0.567 | 0.100 |
153
+
154
+ The model is sharpest at the extremes (grades 1, 8) and at distinct mid-levels; its largest residual errors sit at the **3–4 and 6–7 boundaries**, where texts are genuinely close in readability. On the raw predictions the two categories were balanced (informative 0.832 / narrative 0.842) and truncated texts (>512 tokens, n = 12) were only marginally harder (1.000 vs. 0.828), so neither category nor truncation is a source of systematic error.
155
+
156
+ **Confusion matrix (calibrated, test).** Rows = true grade, columns = predicted grade.
157
+
158
+ | true \ pred | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
159
+ |---|---|---|---|---|---|---|---|---|
160
+ | **1** | 29 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
161
+ | **2** | 3 | 26 | 1 | 0 | 0 | 0 | 0 | 0 |
162
+ | **3** | 0 | 9 | 20 | 1 | 0 | 0 | 0 | 0 |
163
+ | **4** | 0 | 1 | 11 | 12 | 3 | 2 | 0 | 0 |
164
+ | **5** | 0 | 0 | 1 | 5 | 19 | 5 | 0 | 0 |
165
+ | **6** | 0 | 0 | 0 | 2 | 4 | 22 | 1 | 1 |
166
+ | **7** | 0 | 0 | 0 | 0 | 2 | 8 | 15 | 5 |
167
+ | **8** | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 28 |
168
+
169
+ ![Training dynamics — loss and validation MAE/accuracy over epochs](training_curves.png)
170
+
171
+ ![Confusion matrix — test set](confusion_matrix.png)
172
+
173
+ **Ablation (test).** Each loss component was removed in turn, with everything else — seed, initialization, batch order, scheduler, early stopping, and the same post-hoc threshold calibration — held identical.
174
+
175
+ | Variant | Pure ρ | MAE | Exact acc | ±1 acc |
176
+ |---|---|---|---|---|
177
+ | Full (ordinal + triplet + category) | 0.966 | 0.326 | 0.715 | 0.958 |
178
+ | − triplet | 0.963 | 0.402 | 0.636 | 0.962 |
179
+ | − category | 0.964 | 0.410 | 0.636 | 0.954 |
180
+ | Ordinal only | 0.962 | 0.410 | 0.628 | 0.962 |
181
+
182
+ Removing the ordinal-aware triplet loss raises MAE by 0.075 while pure ranking ρ is essentially unchanged, showing the triplet term improves the *even spacing* of the embedding axis (its calibratability) rather than the ranking itself.
183
+
184
+ ## Explainability (XAI)
185
+
186
+ Reading-difficulty attributions are produced with **Integrated Gradients (IG)** on an end-to-end version of the model (frozen BERTurk → continuous difficulty score), with token attributions merged back to whole words. Before attribution, the end-to-end score is verified against the cached score of the same text — they match to within **1.1 × 10⁻⁵**, confirming IG explains the exact deployed model. IG is run with `n_steps = 50`, and the convergence delta is 0.056 (negligible relative to the score magnitude), so the completeness axiom holds.
187
+
188
+ Positive attribution means a word pushes the difficulty score **up** (harder); negative means it pushes **down** (easier). Three representative test cases are shown below. Because IG distributes each text's total score relative to an empty-content baseline, the absolute magnitude of the attributions scales with how far a text's score sits from that baseline — so magnitudes are comparable *within* a text, not across texts.
189
+
190
+ **Case 1 — clear-easy (true grade 1, predicted 1).**
191
+
192
+ | Increases difficulty | value | | Decreases difficulty | value |
193
+ |---|---|---|---|---|
194
+ | elmanın | +0.54 | | neşeyle | −0.27 |
195
+ | o | +0.33 | | güzelce | −0.26 |
196
+ | kuralımızdır | +0.27 | | denizi | −0.22 |
197
+ | Dünyadaki | +0.24 | | yapıp | −0.20 |
198
+ | yansıtır | +0.23 | | üç | −0.16 |
199
+
200
+ ![XAI word heatmap — clear-easy case](xai_easy.png)
201
+
202
+ **Case 2 — clear-hard (true grade 7, predicted 7).**
203
+
204
+ | Increases difficulty | value | | Decreases difficulty | value |
205
+ |---|---|---|---|---|
206
+ | ilerlerler | +0.26 | | kıyafetlerin | −0.06 |
207
+ | abartıya | +0.13 | | hızlı | −0.05 |
208
+ | ufkunu | +0.07 | | Kendi | −0.05 |
209
+ | koruyanlar | +0.06 | | Kendi | −0.04 |
210
+ | sadeliği | +0.06 | | | |
211
+
212
+ ![XAI word heatmap — clear-hard case](xai_hard.png)
213
+
214
+ **Case 3 — boundary error (true grade 6, predicted 7).**
215
+
216
+ | Increases difficulty | value | | Decreases difficulty | value |
217
+ |---|---|---|---|---|
218
+ | alışkanlıklarımızda | +0.69 | | temizliğimizi | −0.66 |
219
+ | yaşamımızdaki | +0.60 | | ve | −0.31 |
220
+ | cildimizin | +0.51 | | seçmesi | −0.28 |
221
+ | kolaylaştırır | +0.23 | | temelidir | −0.28 |
222
+
223
+ ![XAI word heatmap — boundary-error case](xai_boundary.png)
224
+
225
+ The boundary case is the most informative: the words that pushed a grade-6 text up into grade 7 are long, morphologically heavy Turkish words (*alışkanlıklarımızda*, *yaşamımızdaki*, *cildimizin*). This is direct evidence that the model reads morphological and lexical density as a difficulty signal, and that its single-grade error falls exactly on the kind of neighbouring boundary where the difficulty distinction is genuinely soft.
KOZTAM.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1a3a783353881bb5c69961e3cecd8e9c60785db6e0886958bf826a172cb79853
3
+ size 860437
README.md ADDED
@@ -0,0 +1,225 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - tr
4
+ license: mit
5
+ library_name: pytorch
6
+ pipeline_tag: text-classification
7
+ base_model: dbmdz/bert-base-turkish-cased
8
+ datasets:
9
+ - behAIvNET/KOZTAM
10
+ tags:
11
+ - turkish
12
+ - education
13
+ - special-education
14
+ - K-8
15
+ - reading
16
+ - reading-difficulty
17
+ - reading-texts
18
+ - synthetic-dataset
19
+ - hearing-loss
20
+ - cochlear
21
+ - bert
22
+ - berturk
23
+ - coral
24
+ - ordinal-regression
25
+ - xai
26
+ - integrated-gradients
27
+ metrics:
28
+ - mae
29
+ - accuracy
30
+ ---
31
+
32
+ # KOZTAM — Koklear Okuma Zorluğu Tahminleme Modeli
33
+
34
+ **KOZTAM** (*Cochlear Reading-Difficulty Forecasting Model*) grades a Turkish text on an eight-level reading-difficulty scale (grades 1–8) with a frozen BERTurk encoder and a lightweight CORAL ordinal head, reaching **0.33 ordinal MAE**, **0.72 exact-grade accuracy**, and **0.96 within-one-grade accuracy** on held-out test data.
35
+
36
+ ## Data
37
+
38
+ KOZTAM is trained on the [`behAIvNET/KOZTAM`](https://huggingface.co/datasets/behAIvNET/KOZTAM) dataset — **1,588 synthetic Turkish texts** spanning grades 1–8 in two categories (informative / narrative), generated under strict MEB-aligned readability targets. One exact-duplicate text (present under both a grade-3 and a grade-5 folder) was removed to eliminate a conflicting ordinal label, giving the final 1,588.
39
+
40
+ The texts are partitioned into **1,110 training / 239 validation / 239 test** by two-stage stratified sampling over the 16 grade × category strata, so both the ordinal grade distribution and the ≈ 50/50 category balance are preserved in every split. Because the BERTurk backbone is frozen, each text's `[CLS]` embedding (768-d) is pre-computed once and cached, and only the head is trained on those cached vectors; the mean `[CLS]` norm is ≈ 25.2 and identical across the three splits, indicating no distributional shift between them.
41
+
42
+ A structural property worth noting for modelling: realized text length rises from grade 1 through grade 7 but **drops at grade 8** (shorter, in both categories, than grades 5–7). Length is therefore not a monotone proxy for reading grade at the top of the scale, so the model must rely on lexical and syntactic density rather than on length.
43
+
44
+ ## Model
45
+
46
+ ```python
47
+ import torch
48
+ import torch.nn as nn
49
+ from transformers import AutoTokenizer, AutoModel
50
+
51
+ DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
52
+ MODEL_NAME = "dbmdz/bert-base-turkish-cased"
53
+ MAX_LEN = 512
54
+ K = 8
55
+
56
+ tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
57
+ bert = AutoModel.from_pretrained(MODEL_NAME).to(DEVICE).eval()
58
+ for p in bert.parameters():
59
+ p.requires_grad_(False)
60
+
61
+ @torch.no_grad()
62
+ def encode_cls(texts):
63
+ enc = tokenizer(texts, padding=True, truncation=True,
64
+ max_length=MAX_LEN, return_tensors="pt").to(DEVICE)
65
+ return bert(**enc).last_hidden_state[:, 0, :]
66
+
67
+ class CoralHead(nn.Module):
68
+ def __init__(self, in_dim, num_classes):
69
+ super().__init__()
70
+ self.fc = nn.Linear(in_dim, 1, bias=False)
71
+ self.bias = nn.Parameter(torch.zeros(num_classes - 1))
72
+ def forward(self, x):
73
+ return self.fc(x) + self.bias
74
+
75
+ class KOZTAMNet(nn.Module):
76
+ def __init__(self, in_dim=768, p=0.2):
77
+ super().__init__()
78
+ self.proj = nn.Sequential(
79
+ nn.Linear(in_dim, 256), nn.LayerNorm(256), nn.GELU(), nn.Dropout(p),
80
+ nn.Linear(256, 64))
81
+ self.ordinal = CoralHead(64, K)
82
+ self.category = nn.Linear(64, 1)
83
+ def forward(self, x):
84
+ z = self.proj(x)
85
+ return z, self.ordinal(z), self.category(z).squeeze(-1)
86
+
87
+ ckpt = torch.load("KOZTAM.pt", map_location=DEVICE)
88
+ head = KOZTAMNet().to(DEVICE)
89
+ head.load_state_dict(ckpt["state_dict"])
90
+ head.eval()
91
+ thresholds = torch.tensor(ckpt["coral_thresholds"], device=DEVICE)
92
+
93
+ CATEGORY = {0: "bilgilendirici (informative)", 1: "öyküleyici (narrative)"}
94
+
95
+ @torch.no_grad()
96
+ def predict(texts):
97
+ single = isinstance(texts, str)
98
+ cls = encode_cls([texts] if single else list(texts))
99
+ z = head.proj(cls)
100
+ score = head.ordinal.fc(z).squeeze(-1)
101
+ grade = (score[:, None] > thresholds[None, :]).sum(1) + 1
102
+ cat = (torch.sigmoid(head.category(z).squeeze(-1)) > 0.5).long()
103
+ out = [{"grade": int(g), "score": round(float(s), 4), "category": CATEGORY[int(c)]}
104
+ for g, s, c in zip(grade, score, cat)]
105
+ return out[0] if single else out
106
+
107
+ print(predict("Sample text."))
108
+ ```
109
+
110
+ ## Performance
111
+
112
+ Evaluated on the held-out **test set (239 texts)** the model never saw during training.
113
+
114
+ **Headline metrics (calibrated).**
115
+
116
+ | Metric | Value |
117
+ |---|---|
118
+ | Ordinal MAE | **0.326** |
119
+ | Exact-grade accuracy | **0.715** |
120
+ | ±1-grade accuracy | **0.958** |
121
+ | Spearman ρ (predicted vs. true grade) | **0.965** |
122
+ | Text-category accuracy | **1.000** |
123
+
124
+ **Threshold calibration.** KOZTAM has two parts: the trained network, which outputs a continuous *difficulty score*, and seven **calibrated thresholds** that cut that score into discrete grades (stored in the checkpoint, applied at inference). The raw network already ranks texts near-perfectly — continuous-score Spearman ρ = **0.966** — but the default 0.5-thresholds mis-placed the cut points and collapsed predictions toward the extreme grades. Re-placing the thresholds on the validation split (coordinate descent on MAE, **no re-training**) produced the final metrics:
125
+
126
+ | Metric (test) | Raw (0.5-threshold) | Calibrated |
127
+ |---|---|---|
128
+ | Ordinal MAE | 0.837 | 0.326 |
129
+ | Exact accuracy | 0.423 | 0.715 |
130
+ | ±1 accuracy | 0.782 | 0.958 |
131
+ | Spearman ρ | 0.924 | 0.965 |
132
+
133
+ On the validation split the same procedure moved MAE from 0.837 to 0.285, confirming the calibration generalizes rather than overfitting the test set.
134
+
135
+ **Continuous difficulty score by grade (test).** The mean score increases monotonically across all eight grades with no inversion, which is the direct evidence that the encoder learned the reading-difficulty ordering:
136
+
137
+ | Grade | Mean score ± SD |
138
+ |---|---|
139
+ | 1 | −6.91 ± 1.18 |
140
+ | 2 | −2.24 ± 1.52 |
141
+ | 3 | −0.99 ± 0.48 |
142
+ | 4 | −0.33 ± 0.45 |
143
+ | 5 | 0.15 ± 0.29 |
144
+ | 6 | 0.73 ± 0.50 |
145
+ | 7 | 1.43 ± 0.63 |
146
+ | 8 | 4.07 ± 1.49 |
147
+
148
+ **Per-grade MAE (calibrated, test).**
149
+
150
+ | Grade | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
151
+ |---|---|---|---|---|---|---|---|---|
152
+ | MAE | 0.033 | 0.133 | 0.333 | 0.690 | 0.400 | 0.367 | 0.567 | 0.100 |
153
+
154
+ The model is sharpest at the extremes (grades 1, 8) and at distinct mid-levels; its largest residual errors sit at the **3–4 and 6–7 boundaries**, where texts are genuinely close in readability. On the raw predictions the two categories were balanced (informative 0.832 / narrative 0.842) and truncated texts (>512 tokens, n = 12) were only marginally harder (1.000 vs. 0.828), so neither category nor truncation is a source of systematic error.
155
+
156
+ **Confusion matrix (calibrated, test).** Rows = true grade, columns = predicted grade.
157
+
158
+ | true \ pred | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
159
+ |---|---|---|---|---|---|---|---|---|
160
+ | **1** | 29 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
161
+ | **2** | 3 | 26 | 1 | 0 | 0 | 0 | 0 | 0 |
162
+ | **3** | 0 | 9 | 20 | 1 | 0 | 0 | 0 | 0 |
163
+ | **4** | 0 | 1 | 11 | 12 | 3 | 2 | 0 | 0 |
164
+ | **5** | 0 | 0 | 1 | 5 | 19 | 5 | 0 | 0 |
165
+ | **6** | 0 | 0 | 0 | 2 | 4 | 22 | 1 | 1 |
166
+ | **7** | 0 | 0 | 0 | 0 | 2 | 8 | 15 | 5 |
167
+ | **8** | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 28 |
168
+
169
+ ![Training dynamics — loss and validation MAE/accuracy over epochs](training_curves.png)
170
+
171
+ ![Confusion matrix — test set](confusion_matrix.png)
172
+
173
+ **Ablation (test).** Each loss component was removed in turn, with everything else — seed, initialization, batch order, scheduler, early stopping, and the same post-hoc threshold calibration — held identical.
174
+
175
+ | Variant | Pure ρ | MAE | Exact acc | ±1 acc |
176
+ |---|---|---|---|---|
177
+ | Full (ordinal + triplet + category) | 0.966 | 0.326 | 0.715 | 0.958 |
178
+ | − triplet | 0.963 | 0.402 | 0.636 | 0.962 |
179
+ | − category | 0.964 | 0.410 | 0.636 | 0.954 |
180
+ | Ordinal only | 0.962 | 0.410 | 0.628 | 0.962 |
181
+
182
+ Removing the ordinal-aware triplet loss raises MAE by 0.075 while pure ranking ρ is essentially unchanged, showing the triplet term improves the *even spacing* of the embedding axis (its calibratability) rather than the ranking itself.
183
+
184
+ ## Explainability (XAI)
185
+
186
+ Reading-difficulty attributions are produced with **Integrated Gradients (IG)** on an end-to-end version of the model (frozen BERTurk → continuous difficulty score), with token attributions merged back to whole words. Before attribution, the end-to-end score is verified against the cached score of the same text — they match to within **1.1 × 10⁻⁵**, confirming IG explains the exact deployed model. IG is run with `n_steps = 50`, and the convergence delta is 0.056 (negligible relative to the score magnitude), so the completeness axiom holds.
187
+
188
+ Positive attribution means a word pushes the difficulty score **up** (harder); negative means it pushes **down** (easier). Three representative test cases are shown below. Because IG distributes each text's total score relative to an empty-content baseline, the absolute magnitude of the attributions scales with how far a text's score sits from that baseline — so magnitudes are comparable *within* a text, not across texts.
189
+
190
+ **Case 1 — clear-easy (true grade 1, predicted 1).**
191
+
192
+ | Increases difficulty | value | | Decreases difficulty | value |
193
+ |---|---|---|---|---|
194
+ | elmanın | +0.54 | | neşeyle | −0.27 |
195
+ | o | +0.33 | | güzelce | −0.26 |
196
+ | kuralımızdır | +0.27 | | denizi | −0.22 |
197
+ | Dünyadaki | +0.24 | | yapıp | −0.20 |
198
+ | yansıtır | +0.23 | | üç | −0.16 |
199
+
200
+ ![XAI word heatmap — clear-easy case](xai_easy.png)
201
+
202
+ **Case 2 — clear-hard (true grade 7, predicted 7).**
203
+
204
+ | Increases difficulty | value | | Decreases difficulty | value |
205
+ |---|---|---|---|---|
206
+ | ilerlerler | +0.26 | | kıyafetlerin | −0.06 |
207
+ | abartıya | +0.13 | | hızlı | −0.05 |
208
+ | ufkunu | +0.07 | | Kendi | −0.05 |
209
+ | koruyanlar | +0.06 | | Kendi | −0.04 |
210
+ | sadeliği | +0.06 | | | |
211
+
212
+ ![XAI word heatmap — clear-hard case](xai_hard.png)
213
+
214
+ **Case 3 — boundary error (true grade 6, predicted 7).**
215
+
216
+ | Increases difficulty | value | | Decreases difficulty | value |
217
+ |---|---|---|---|---|
218
+ | alışkanlıklarımızda | +0.69 | | temizliğimizi | −0.66 |
219
+ | yaşamımızdaki | +0.60 | | ve | −0.31 |
220
+ | cildimizin | +0.51 | | seçmesi | −0.28 |
221
+ | kolaylaştırır | +0.23 | | temelidir | −0.28 |
222
+
223
+ ![XAI word heatmap — boundary-error case](xai_boundary.png)
224
+
225
+ The boundary case is the most informative: the words that pushed a grade-6 text up into grade 7 are long, morphologically heavy Turkish words (*alışkanlıklarımızda*, *yaşamımızdaki*, *cildimizin*). This is direct evidence that the model reads morphological and lexical density as a difficulty signal, and that its single-grade error falls exactly on the kind of neighbouring boundary where the difficulty distinction is genuinely soft.
confusion_matrix.png ADDED
training_curves.png ADDED
xai_boundary.png ADDED

Git LFS Details

  • SHA256: 7524e0db3d48f19fcb3f2e64113838d6b5757fd3cf8abb6d1a2e13b8f110b129
  • Pointer size: 131 Bytes
  • Size of remote file: 137 kB
xai_easy.png ADDED
xai_hard.png ADDED

Git LFS Details

  • SHA256: 48c17c420ce04a82be14e613ae7d1ae4eba87c34a143186148a3bf37acc23b65
  • Pointer size: 131 Bytes
  • Size of remote file: 194 kB