dmusingu commited on
Commit
fdbe183
·
verified ·
1 Parent(s): 53e112a

Update README with model loading code

Browse files
Files changed (1) hide show
  1. README.md +50 -23
README.md CHANGED
@@ -14,32 +14,59 @@ Part of the [LAPVQA collection](https://huggingface.co/collections/dmusingu/lapv
14
 
15
  ## Description
16
 
17
- DETR-style detection heads for **chest abnormality detection** on VinDr-CXR (14 classes),
18
- trained on top of six frozen vision encoders.
19
- Each file contains the detection head weights; the encoder backbone is not included.
20
 
21
- ## Results (VinDr-CXR test set, mAP@IoU=0.4)
22
 
23
- | Encoder | mAP@0.4 (val) | mAP@0.4 (test) |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
24
  |---|---|---|
25
- | OWLv2 | ~0.14 | 0.048 |
26
- | SigLIP | ~0.14 | ~0.045 |
27
- | CLIP ViT-L/14 | ~0.14 | ~0.040 |
28
- | Florence-2 | ~0.14 | ~0.038 |
29
- | CoCa | ~0.14 | ~0.035 |
30
- | MAE-ViT-L/16 | ~0.14 | ~0.035 |
31
 
32
- OWLv2 leads on detection — its pretraining objective (open-vocabulary detection) directly
33
- matches this task. The val→test drop (~3×) reflects distribution shift between
34
- multi-radiologist NMS-merged train annotations and single-consensus test labels.
35
 
36
- ## Files
 
 
 
37
 
38
- | File | Encoder backbone |
39
- |---|---|
40
- | `clip-vit-l14.pt` | CLIP ViT-L/14 |
41
- | `siglip.pt` | SigLIP |
42
- | `florence2.pt` | Florence-2 |
43
- | `coca.pt` | CoCa |
44
- | `owlv2.pt` | OWLv2 |
45
- | `mae-vit-l16.pt` | MAE ViT-L/16 |
 
 
 
 
 
 
 
 
 
 
14
 
15
  ## Description
16
 
17
+ DETR-style detection heads for 14-class chest abnormality detection on VinDr-CXR,
18
+ trained on top of six **frozen** vision encoders.
19
+ Each checkpoint is a dict: `{state_dict, vis_dim, d_model, num_queries, num_enc, num_dec, encoder, epoch, val_map40, val_map50}`.
20
 
21
+ ## Architecture `DetectionHead`
22
 
23
+ ```
24
+ vis_proj : Linear(vis_dim → 256)
25
+ encoder : 2 × TransformerEncoderLayer (self-attn, pre-norm)
26
+ object_queries : Parameter [1, 20, 256]
27
+ decoder : 3 × TransformerDecoderLayer (cross-attn to encoder output)
28
+ class_head : Linear(256 → 15) # 14 classes + background
29
+ box_head : MLP(256 → 256 → 4) # (cx,cy,w,h) ∈ [0,1]
30
+ ```
31
+
32
+ ## Results (VinDr-CXR test, mAP@IoU=0.4)
33
+
34
+ | Encoder | mAP@0.4 (test) |
35
+ |---|---|
36
+ | OWLv2 | 0.048 |
37
+ | SigLIP | ~0.045 |
38
+ | CLIP ViT-L/14 | ~0.040 |
39
+
40
+ | File | Encoder | vis_dim |
41
  |---|---|---|
42
+ | `clip-vit-l14.pt` | CLIP ViT-L/14 | 1024 |
43
+ | `siglip.pt` | SigLIP | 1152 |
44
+ | `florence2.pt` | Florence-2 | 1024 |
45
+ | `coca.pt` | CoCa | 768 |
46
+ | `owlv2.pt` | OWLv2 | 1024 |
47
+ | `mae-vit-l16.pt` | MAE ViT-L/16 | 1024 |
48
 
49
+ ## Loading
 
 
50
 
51
+ ```python
52
+ import torch
53
+ from lapvqa.ad.heads import DetectionHead
54
+ from lapvqa.ad.heads import predict
55
 
56
+ ckpt = torch.load("owlv2.pt", map_location="cpu")
57
+ head = DetectionHead(
58
+ vis_dim = ckpt["vis_dim"],
59
+ d_model = ckpt["d_model"],
60
+ num_queries = ckpt["num_queries"],
61
+ num_enc_layers = ckpt["num_enc"],
62
+ num_dec_layers = ckpt["num_dec"],
63
+ )
64
+ head.load_state_dict(ckpt["state_dict"])
65
+ head.eval()
66
+
67
+ with torch.no_grad():
68
+ # vis_tokens: [B, HW, vis_dim] — spatial patch tokens from the frozen encoder
69
+ outputs = head(vis_tokens)
70
+ detections = predict(outputs, score_threshold=0.1, nms_iou=0.5)
71
+ # detections[i]: {'boxes': [K,4] xyxy, 'labels': [K], 'scores': [K]}
72
+ ```