Update README with model loading code
Browse files
README.md
CHANGED
|
@@ -14,32 +14,59 @@ Part of the [LAPVQA collection](https://huggingface.co/collections/dmusingu/lapv
|
|
| 14 |
|
| 15 |
## Description
|
| 16 |
|
| 17 |
-
DETR-style detection heads for
|
| 18 |
-
trained on top of six frozen vision encoders.
|
| 19 |
-
Each
|
| 20 |
|
| 21 |
-
##
|
| 22 |
|
| 23 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
|---|---|---|
|
| 25 |
-
|
|
| 26 |
-
|
|
| 27 |
-
|
|
| 28 |
-
|
|
| 29 |
-
|
|
| 30 |
-
|
|
| 31 |
|
| 32 |
-
|
| 33 |
-
matches this task. The val→test drop (~3×) reflects distribution shift between
|
| 34 |
-
multi-radiologist NMS-merged train annotations and single-consensus test labels.
|
| 35 |
|
| 36 |
-
|
|
|
|
|
|
|
|
|
|
| 37 |
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
|
| 15 |
## Description
|
| 16 |
|
| 17 |
+
DETR-style detection heads for 14-class chest abnormality detection on VinDr-CXR,
|
| 18 |
+
trained on top of six **frozen** vision encoders.
|
| 19 |
+
Each checkpoint is a dict: `{state_dict, vis_dim, d_model, num_queries, num_enc, num_dec, encoder, epoch, val_map40, val_map50}`.
|
| 20 |
|
| 21 |
+
## Architecture — `DetectionHead`
|
| 22 |
|
| 23 |
+
```
|
| 24 |
+
vis_proj : Linear(vis_dim → 256)
|
| 25 |
+
encoder : 2 × TransformerEncoderLayer (self-attn, pre-norm)
|
| 26 |
+
object_queries : Parameter [1, 20, 256]
|
| 27 |
+
decoder : 3 × TransformerDecoderLayer (cross-attn to encoder output)
|
| 28 |
+
class_head : Linear(256 → 15) # 14 classes + background
|
| 29 |
+
box_head : MLP(256 → 256 → 4) # (cx,cy,w,h) ∈ [0,1]
|
| 30 |
+
```
|
| 31 |
+
|
| 32 |
+
## Results (VinDr-CXR test, mAP@IoU=0.4)
|
| 33 |
+
|
| 34 |
+
| Encoder | mAP@0.4 (test) |
|
| 35 |
+
|---|---|
|
| 36 |
+
| OWLv2 | 0.048 |
|
| 37 |
+
| SigLIP | ~0.045 |
|
| 38 |
+
| CLIP ViT-L/14 | ~0.040 |
|
| 39 |
+
|
| 40 |
+
| File | Encoder | vis_dim |
|
| 41 |
|---|---|---|
|
| 42 |
+
| `clip-vit-l14.pt` | CLIP ViT-L/14 | 1024 |
|
| 43 |
+
| `siglip.pt` | SigLIP | 1152 |
|
| 44 |
+
| `florence2.pt` | Florence-2 | 1024 |
|
| 45 |
+
| `coca.pt` | CoCa | 768 |
|
| 46 |
+
| `owlv2.pt` | OWLv2 | 1024 |
|
| 47 |
+
| `mae-vit-l16.pt` | MAE ViT-L/16 | 1024 |
|
| 48 |
|
| 49 |
+
## Loading
|
|
|
|
|
|
|
| 50 |
|
| 51 |
+
```python
|
| 52 |
+
import torch
|
| 53 |
+
from lapvqa.ad.heads import DetectionHead
|
| 54 |
+
from lapvqa.ad.heads import predict
|
| 55 |
|
| 56 |
+
ckpt = torch.load("owlv2.pt", map_location="cpu")
|
| 57 |
+
head = DetectionHead(
|
| 58 |
+
vis_dim = ckpt["vis_dim"],
|
| 59 |
+
d_model = ckpt["d_model"],
|
| 60 |
+
num_queries = ckpt["num_queries"],
|
| 61 |
+
num_enc_layers = ckpt["num_enc"],
|
| 62 |
+
num_dec_layers = ckpt["num_dec"],
|
| 63 |
+
)
|
| 64 |
+
head.load_state_dict(ckpt["state_dict"])
|
| 65 |
+
head.eval()
|
| 66 |
+
|
| 67 |
+
with torch.no_grad():
|
| 68 |
+
# vis_tokens: [B, HW, vis_dim] — spatial patch tokens from the frozen encoder
|
| 69 |
+
outputs = head(vis_tokens)
|
| 70 |
+
detections = predict(outputs, score_threshold=0.1, nms_iou=0.5)
|
| 71 |
+
# detections[i]: {'boxes': [K,4] xyxy, 'labels': [K], 'scores': [K]}
|
| 72 |
+
```
|