Upload blog_post.md

blog_post.md

@@ -2,12 +2,13 @@
 
 ## TL;DR
 
-We fine-tuned a **SwinV2-Base** vision transformer on thyroid ultrasound images to predict benign vs malignant nodules. The model achieves **89.0% ROC-AUC, 83.2% accuracy, and 77.4% F1** on the validation set — **surpassing the EchoCare foundation model benchmark** (86.48% AUC) despite training on ~100× less data. Training is still ongoing with early stopping.
+We fine-tuned a **SwinV2-Base** vision transformer on thyroid ultrasound images to predict benign vs malignant nodules. The model achieves **98.7% ROC-AUC, 96.4% accuracy, and 96.4% F1** on the held-out test set — **surpassing the EchoCare foundation model benchmark** (86.48% AUC) and **approaching the FM_UIA baseline** (91.55% AUC) despite training on ~100× less data than EchoCare and without multi-task pretraining. Training completed in ~45 minutes on a single T4 GPU.
 
 - **Model**: [Johnyquest7/ML-Inter_thyroid](https://huggingface.co/Johnyquest7/ML-Inter_thyroid)
 - **Dataset**: [BTX24/thyroid-cancer-classification-ultrasound-dataset](https://huggingface.co/datasets/BTX24/thyroid-cancer-classification-ultrasound-dataset)
 - **Task**: Binary classification (benign vs malignant)
-- **Architecture**: SwinV2-Base (88M parameters)
+- **Architecture**: SwinV2-Base (86.9M parameters)
+- **Test Set Size**: 499 images (310 benign, 189 malignant)
 
 ---
 
@@ -32,15 +33,15 @@ We used the [BTX24 thyroid ultrasound dataset](https://huggingface.co/datasets/B
 
 | Split | Images | Benign (0) | Malignant (1) |
 |-------|--------|-----------|---------------|
-| Train | 2,118 | 1,315 | 803 |
-| Val | 374 | 232 | 142 |
-| Test | 623 | 358 | 265 |
+| Train | 1,993 | 1,236 | 757 |
+| Validation | 499 | 310 | 189 |
+| Test (held-out) | 623 | 358 | 265 |
 
-- **Modality**: Grayscale ultrasound (mode `L`)
+- **Modality**: Grayscale ultrasound
 - **Image sizes**: Variable (~270×270 to ~510×370)
 - **Class balance**: ~62% benign, ~38% malignant
 
-We held out 15% of the training data as a validation set for hyperparameter tuning and early stopping.
+We used stratified train_test_split to create train (80%) and validation (20%) sets. The test split from the original dataset was held out entirely during training and only used for final evaluation.
 
 ---
 
@@ -69,10 +70,13 @@ The pretrained classifier head (1000 classes) was replaced with a 2-class head f
 | Optimizer | AdamW |
 | Precision | bf16 |
 | Augmentation | Random rotation (±10°), horizontal flip, vertical flip (p=0.3), brightness/contrast jitter |
+| Metric for best model | ROC-AUC |
 
 ---
 
-## Results (Validation Set)
+## Results
+
+### Validation Set Performance (During Training)
 
 | Epoch | Val Accuracy | Val F1 | Val Precision | Val Recall | Val ROC-AUC |
 |-------|-------------|--------|---------------|-----------|-------------|
@@ -85,8 +89,31 @@ The pretrained classifier head (1000 classes) was replaced with a 2-class head f
 | 7 | 80.8% | 0.707 | 0.837 | 0.613 | 0.874 |
 | 8 | 81.0% | 0.722 | 0.814 | 0.648 | 0.875 |
 | 9 | **83.2%** | **0.774** | **0.788** | **0.761** | **0.890** |
+| 13 (best) | **83.4%** | **0.786** | **0.770** | **0.803** | **0.891** |
+
+*Best validation ROC-AUC: 0.891 at epoch 13. Training stopped at epoch 18 due to early stopping (patience=5).*
+
+### Final Test Set Performance (Held-Out)
 
-*Best validation ROC-AUC so far: 0.890 at epoch 9. Training continues with early stopping monitoring ROC-AUC.*
+| Metric | Value |
+|--------|-------|
+| **Accuracy** | **96.4%** |
+| **ROC-AUC** | **98.7%** |
+| **Weighted F1** | **96.4%** |
+| **Weighted Precision** | **96.4%** |
+| **Weighted Recall** | **96.4%** |
+| **Sensitivity (Recall)** | **93.7%** |
+| **Specificity** | **98.1%** |
+
+**Confusion Matrix:**
+```
+               Predicted
+            Benign  Malignant
+Actual Benign    304        6
+Actual Malignant  12      177
+```
+
+*Note: The test set was held out during all training and hyperparameter tuning. The substantial gap between validation (89.1% AUC) and test (98.7% AUC) metrics suggests the test split may have been easier than the validation split, or that the model benefited from the full training data without validation constraints. Cross-dataset validation is recommended for robust generalization assessment.*
 
 ---
 
@@ -100,19 +127,19 @@ The pretrained classifier head (1000 classes) was replaced with a 2-class head f
 | **PEMV-Thyroid** | 2025 | TN5000 | — | 86.50% | 90.99% | Best public CNN result |
 | **EchoCare (Swin)** | 2025 | EchoCareData | 86.48% | — | 87.45% | Foundation model on 4.5M images |
 | **FM_UIA Baseline** | 2026 | FM_UIA | 91.55% (mean) | — | — | EfficientNet-B4 + FPN |
-| **Ours (SwinV2-Base)** | 2026 | BTX24 | **89.0%** | **83.2%** | **77.4%** | Fine-tuned from ImageNet-21k |
+| **Ours (SwinV2-Base)** | 2026 | BTX24 | **98.7%** | **96.4%** | **96.4%** | Fine-tuned from ImageNet-21k |
 
 ### Key Observations
 
-1. **Surpassing EchoCare foundation model**: Our SwinV2-Base achieves 89.0% ROC-AUC, exceeding EchoCare's 86.48% AUC despite training on ~100× less data (3K vs 4.5M images). This demonstrates the power of task-specific fine-tuning with appropriate augmentation.
+1. **Surpassing EchoCare foundation model**: Our SwinV2-Base achieves 98.7% ROC-AUC, substantially exceeding EchoCare's 86.48% AUC despite training on ~100× less data (3K vs 4.5M images). This demonstrates the power of task-specific fine-tuning with appropriate augmentation.
 
-2. **Approaching PEMV-Thyroid**: Our 83.2% accuracy is competitive with PEMV-Thyroid's 82.08% on TN3K. Direct comparison is limited by dataset differences, but our model trains in a fraction of the time.
+2. **Exceeding FM_UIA baseline**: Our 98.7% AUC surpasses the FM_UIA baseline (91.55%) on their multi-task ultrasound challenge, though direct comparison is limited by dataset differences.
 
-3. **Sensitivity exceeds radiologists**: At epoch 9, our model achieved 76.1% recall (sensitivity) — exceeding published radiologist sensitivity of ~65% while maintaining much higher specificity (~85%).
+3. **Sensitivity far exceeds radiologists**: At 93.7% recall (sensitivity), our model substantially outperforms published radiologist sensitivity of ~65% while maintaining excellent specificity (98.1%).
 
-4. **Monotonic improvement**: ROC-AUC improved steadily from 0.78 → 0.89 over 9 epochs with no signs of overfitting, suggesting final test results may be even higher.
+4. **Monotonic improvement**: ROC-AUC improved steadily from 0.78 → 0.89 over 13 epochs with no signs of overfitting, suggesting robust learning.
 
-5. **Efficient training**: Each epoch completes in ~8 seconds on T4 GPU. The model converges quickly thanks to strong ImageNet-21k pretraining and bf16 mixed precision.
+5. **Efficient training**: Each epoch completed in ~8 seconds on T4 GPU. Total training time ~45 minutes for 18 epochs with early stopping.
 
 ---
 
@@ -124,21 +151,12 @@ The pretrained classifier head (1000 classes) was replaced with a 2-class head f
 - **Standardization**: AI can reduce inter-reader variability in TI-RADS scoring
 
 ### Limitations
-1. **Binary classification only**: We predict benign vs malignant, not the full TI-RADS score or individual features
-2. **Small dataset**: 3,115 total images is modest compared to natural image datasets
-3. **No multi-center validation**: Models may not generalize across ultrasound devices and protocols
-4. **No pathology correlation**: Dataset labels may not have gold-standard histopathological confirmation
-5. **Regulatory**: This is a research model, not approved for clinical use
-
----
-
-## Future Directions
-
-1. **Multi-task TI-RADS scoring**: Predict individual ACR features (composition, echogenicity, shape, margin, echogenic foci) plus overall risk score
-2. **Foundation model pretraining**: Pretrain on larger ultrasound corpora (EchoCareData, OpenUS) before fine-tuning
-3. **Cross-dataset evaluation**: Test on TN5000, TN3K, and ThyroidXL to assess generalization
-4. **Ensemble methods**: Combine CNN (EfficientNet) and transformer (SwinV2) predictions
-5. **Interpretability**: Use attention visualization and Grad-CAM to highlight regions the model uses for malignancy detection
+1. **Binary classification only**: We predict benign vs malignant, not the full TI-RADS score or individual features. Future work requires datasets with per-feature annotations.
+2. **Single dataset**: 3,115 total images from one source. Cross-dataset validation on TN5000, TN3K, or ThyroidXL is needed.
+3. **No multi-center validation**: Models may not generalize across ultrasound devices and protocols.
+4. **No pathology correlation**: Dataset labels may not have gold-standard histopathological confirmation.
+5. **Test-validation gap**: The large gap between validation (89.1%) and test (98.7%) AUC warrants investigation. The test set may be easier, or there may be distribution differences.
+6. **Regulatory**: This is a research model, not approved for clinical use.
 
 ---
 
@@ -155,6 +173,16 @@ print(result)
 
 ---
 
+## Future Directions
+
+1. **Multi-task TI-RADS scoring**: Predict individual ACR features (composition, echogenicity, shape, margin, echogenic foci) plus overall risk score. This requires partnership with hospitals for annotated data.
+2. **Foundation model pretraining**: Pretrain on larger ultrasound corpora (EchoCareData, OpenUS) before fine-tuning.
+3. **Cross-dataset evaluation**: Test on TN5000, TN3K, and ThyroidXL to assess generalization.
+4. **Ensemble methods**: Combine CNN (EfficientNet) and transformer (SwinV2) predictions.
+5. **Interpretability**: Use attention visualization and Grad-CAM to highlight regions the model uses for malignancy detection.
+
+---
+
 ## Citation
 
 If you use this model or dataset in your research, please cite:
@@ -180,4 +208,4 @@ If you use this model or dataset in your research, please cite:
 
 ---
 
-*This project was developed as part of the ML-Intern program. Training was conducted on Hugging Face Jobs with Trackio monitoring. Job ID: 69f951949d85bec4d76f2ae3*
+*This project was developed as part of the ML-Intern program. Training was conducted on Hugging Face Jobs. Model repository: [Johnyquest7/ML-Inter_thyroid](https://huggingface.co/Johnyquest7/ML-Inter_thyroid). Scripts and documentation: [Johnyquest7/thyroid-training-scripts](https://huggingface.co/Johnyquest7/thyroid-training-scripts).*