JoelAjitesh commited on
Commit
46ccbfa
·
verified ·
1 Parent(s): 413c14d

Add ablation chart, remove em dashes from model card

Browse files
Files changed (2) hide show
  1. README.md +24 -22
  2. ablation.png +0 -0
README.md CHANGED
@@ -29,33 +29,35 @@ model-index:
29
  name: EM (EM-optimal gate)
30
  - type: precision
31
  value: 91.7
32
- name: Answered precision @ 90%-precision gate (9% coverage)
33
  ---
34
 
35
  # Grounded Pointer QA
36
 
37
  An extractive question-answering model that **cannot hallucinate by
38
  construction**: its output layer can only point at spans inside retrieved
39
- passages of *your* documents it has no vocabulary to generate from. A
40
- trained abstention head refuses when the loaded knowledge doesn't contain the
41
- answer, decoding is deterministic (argmax; same question + same documents =
42
- same answer forever), and knowledge is **hot-swappable**: point it at a new
43
- folder of text files and it answers from those, no retraining.
44
 
45
- Built on `roberta-base` (125M params) with pointer + abstention heads,
46
- finetuned on SQuAD v2 **against real TF-IDF retrieval** the model never saw
47
- gold passages during training, only what the retriever actually returned, so
48
- it learned to abstain on retrieval misses too.
 
 
49
 
50
  ## Operating modes
51
 
52
  The checkpoint ships with a calibrated confidence gate
53
- (`P(answerable) × P(span)`), selected on a calibration split and verified on
54
- a held-out test split:
55
 
56
  | Mode | Gate | Coverage | Answered precision | EM |
57
  |---|---|---|---|---|
58
- | "Right or silent" (shipped default) | 0.965 | 9% | **91.7%** | 57.1 |
59
  | EM-optimal | 0.295 | 45% | 72.3% | 74.6 |
60
 
61
  Pass a lower `gate` to `ask()` for more coverage at lower precision.
@@ -64,7 +66,7 @@ Pass a lower `gate` to `ask()` for more coverage at lower precision.
64
 
65
  ```python
66
  # files needed: proqa.pt, modeling_proqa.py (both in this repo)
67
- # pip install torch transformers scikit-learn numpy (+ pypdf for PDFs)
68
  from modeling_proqa import GroundedQA
69
 
70
  qa = GroundedQA("proqa.pt")
@@ -82,16 +84,16 @@ qa.ask("what is the capital of France?") # not in your docs
82
 
83
  - Single-span extraction only: no summarization, no aggregation across
84
  passages, no multi-turn conversation.
85
- - Trained on Wikipedia-style prose; **tables read poorly** convert rows to
86
- sentences ("ATA chapter 32 is Landing Gear.") for big accuracy gains.
87
- - TF-IDF retrieval is lexical: paraphrases sharing no words with your
88
  documents may cause (safe) abstentions.
89
- - Single training run, single seed; in-domain calibration.
90
 
91
- Full writeup, ablations (from-scratch 22.4 EM 74.6 EM ladder), and negative
92
- results: see the project repository's `paper/`.
93
 
94
  ## Training
95
 
96
- One NVIDIA RTX 5060 Ti (16 GB): ~2.5 h finetune (batch 8 × 4 passages × 384
97
- tokens, bf16, lr 2e-5, 2 epochs) + calibration pass.
 
29
  name: EM (EM-optimal gate)
30
  - type: precision
31
  value: 91.7
32
+ name: Answered precision at 90%-precision gate (9% coverage)
33
  ---
34
 
35
  # Grounded Pointer QA
36
 
37
  An extractive question-answering model that **cannot hallucinate by
38
  construction**: its output layer can only point at spans inside retrieved
39
+ passages of your documents. It has no vocabulary to generate from. A trained
40
+ abstention head refuses when the loaded knowledge does not contain the answer,
41
+ decoding is deterministic (argmax, so the same question over the same documents
42
+ gives the same answer every time), and knowledge is **hot-swappable**: point it
43
+ at a new folder of text files and it answers from those, with no retraining.
44
 
45
+ Built on `roberta-base` (125M params) with pointer and abstention heads,
46
+ finetuned on SQuAD v2 against real TF-IDF retrieval. The model never saw gold
47
+ passages during training, only what the retriever actually returned, so it
48
+ learned to abstain on retrieval misses too.
49
+
50
+ ![Exact match by build stage](ablation.png)
51
 
52
  ## Operating modes
53
 
54
  The checkpoint ships with a calibrated confidence gate
55
+ (`P(answerable) x P(span)`), selected on a calibration split and verified on a
56
+ disjoint held-out test split:
57
 
58
  | Mode | Gate | Coverage | Answered precision | EM |
59
  |---|---|---|---|---|
60
+ | "Right or silent" (shipped default) | 0.965 | 9% | 91.7% | 57.1 |
61
  | EM-optimal | 0.295 | 45% | 72.3% | 74.6 |
62
 
63
  Pass a lower `gate` to `ask()` for more coverage at lower precision.
 
66
 
67
  ```python
68
  # files needed: proqa.pt, modeling_proqa.py (both in this repo)
69
+ # pip install torch transformers scikit-learn numpy (plus pypdf for PDFs)
70
  from modeling_proqa import GroundedQA
71
 
72
  qa = GroundedQA("proqa.pt")
 
84
 
85
  - Single-span extraction only: no summarization, no aggregation across
86
  passages, no multi-turn conversation.
87
+ - Trained on Wikipedia-style prose, so **tables read poorly**. Convert rows to
88
+ sentences ("ATA chapter 32 is Landing Gear.") for large accuracy gains.
89
+ - TF-IDF retrieval is lexical, so paraphrases sharing no words with your
90
  documents may cause (safe) abstentions.
91
+ - Single training run, single seed, in-domain calibration.
92
 
93
+ Full writeup, ablations (the 22.4 to 74.6 EM ladder above), and negative
94
+ results are in the project repository's `paper/`.
95
 
96
  ## Training
97
 
98
+ One NVIDIA RTX 5060 Ti (16 GB): about 2.5 h finetune (batch 8 x 4 passages x
99
+ 384 tokens, bf16, lr 2e-5, 2 epochs) plus a calibration pass.
ablation.png ADDED