Add paired eval and corrected FAISS artifacts
Browse files
README.md
CHANGED
|
@@ -139,6 +139,29 @@ Same 16-batch clean block-causal eval slice:
|
|
| 139 |
|
| 140 |
Both methods are effectively full-attention parity on PPL. Learned projections recover more teacher attention mass at the same token budget, especially at K=128, but do not yet show a clean PPL advantage over Quest on this slice.
|
| 141 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 142 |
## Packed Leakage-confounded Ablations
|
| 143 |
|
| 144 |
The packed d64/d128/d256 runs are included because they are useful for understanding capacity scaling, but they should not be used for clean quality claims. Those runs allowed cross-document attention inside packed examples.
|
|
|
|
| 139 |
|
| 140 |
Both methods are effectively full-attention parity on PPL. Learned projections recover more teacher attention mass at the same token budget, especially at K=128, but do not yet show a clean PPL advantage over Quest on this slice.
|
| 141 |
|
| 142 |
+
|
| 143 |
+
Paired 32-batch NLL evaluation gives a sharper comparison:
|
| 144 |
+
|
| 145 |
+
| K | full PPL | learned PPL | Quest PPL | learned - Quest NLL delta (95% bootstrap CI) | Read |
|
| 146 |
+
|---|---:|---:|---:|---:|---|
|
| 147 |
+
| 128 | 28.03 | 28.07 | 28.01 | +0.00205 `[+0.00160, +0.00251]` | Quest slightly better |
|
| 148 |
+
| 256 | 28.03 | 28.04 | 28.04 | -0.00005 `[-0.00029, +0.00018]` | statistical tie |
|
| 149 |
+
|
| 150 |
+
So the current clean result is: learned search has higher teacher-attention mass, but PPL is either tied with Quest (K=256) or slightly worse (K=128) on this paired WikiText slice.
|
| 151 |
+
|
| 152 |
+
## Clean FAISS-vs-exact Check
|
| 153 |
+
|
| 154 |
+
The first block-causal FAISS prototype used one global index followed by segment filtering, which produced pathological filler rates after filtering. The current FAISS path builds per-segment indexes when a 4D block-causal mask is present. With that fix, CPU FAISS/HNSW tracks exact learned search on the same 16-batch clean eval slice:
|
| 155 |
+
|
| 156 |
+
| Method | K | PPL | PPL gap | FAISS filler rate |
|
| 157 |
+
|---|---:|---:|---:|---:|
|
| 158 |
+
| learned exact | 128 | 30.47 | +0.07% | n/a |
|
| 159 |
+
| learned FAISS/HNSW | 128 | 30.47 | +0.09% | 0.447 |
|
| 160 |
+
| learned exact | 256 | 30.45 | +0.01% | n/a |
|
| 161 |
+
| learned FAISS/HNSW | 256 | 30.46 | +0.04% | 0.683 |
|
| 162 |
+
|
| 163 |
+
The remaining filler rate is expected for short same-segment prefixes where fewer than K valid causal keys exist; filler slots are masked out of the sparse-attention softmax. This demonstrates off-the-shelf ANN compatibility in the clean block-causal setting, but not production wall-clock speedup.
|
| 164 |
+
|
| 165 |
## Packed Leakage-confounded Ablations
|
| 166 |
|
| 167 |
The packed d64/d128/d256 runs are included because they are useful for understanding capacity scaling, but they should not be used for clean quality claims. Those runs allowed cross-document attention inside packed examples.
|
checkpoints_block_d128/search_step_1000.k_sweep_faiss.json
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"ppl_full": 30.444138765335083,
|
| 3 |
+
"by_K": {
|
| 4 |
+
"128": {
|
| 5 |
+
"recall_avg": 0.7435683117319801,
|
| 6 |
+
"recall_per_layer": {
|
| 7 |
+
"4": 0.7372374021825518,
|
| 8 |
+
"8": 0.7400615944244054,
|
| 9 |
+
"12": 0.7399612933300728,
|
| 10 |
+
"16": 0.7452810723237757,
|
| 11 |
+
"20": 0.7493442218927676,
|
| 12 |
+
"24": 0.7495242862383078
|
| 13 |
+
},
|
| 14 |
+
"mass_avg": 0.7874226044715501,
|
| 15 |
+
"mass_per_layer": {
|
| 16 |
+
"4": 0.7964790942667894,
|
| 17 |
+
"8": 0.760353729720074,
|
| 18 |
+
"12": 0.7879844721965367,
|
| 19 |
+
"16": 0.8230430181841107,
|
| 20 |
+
"20": 0.7992641063936694,
|
| 21 |
+
"24": 0.7574112060681207
|
| 22 |
+
},
|
| 23 |
+
"ppl_ann": 30.470947980880737,
|
| 24 |
+
"ppl_gap_relative": 0.0008806035129553523,
|
| 25 |
+
"faiss_diag": {
|
| 26 |
+
"self_pad_rate": 0.44742653767267865,
|
| 27 |
+
"causal_fill_rate": 0.5447930892308553,
|
| 28 |
+
"self_attn_rate": 0.0077803730964660645
|
| 29 |
+
}
|
| 30 |
+
},
|
| 31 |
+
"256": {
|
| 32 |
+
"recall_avg": 0.8794096146506826,
|
| 33 |
+
"recall_per_layer": {
|
| 34 |
+
"4": 0.8783885035021551,
|
| 35 |
+
"8": 0.878973599137931,
|
| 36 |
+
"12": 0.8767847521551724,
|
| 37 |
+
"16": 0.877142544450431,
|
| 38 |
+
"20": 0.88153076171875,
|
| 39 |
+
"24": 0.8836375269396551
|
| 40 |
+
},
|
| 41 |
+
"mass_avg": 0.9531509574802443,
|
| 42 |
+
"mass_per_layer": {
|
| 43 |
+
"4": 0.9445537698679957,
|
| 44 |
+
"8": 0.9476728768184267,
|
| 45 |
+
"12": 0.963769320783944,
|
| 46 |
+
"16": 0.9684469288793104,
|
| 47 |
+
"20": 0.9556469095164332,
|
| 48 |
+
"24": 0.9388159390153556
|
| 49 |
+
},
|
| 50 |
+
"ppl_ann": 30.45530593395233,
|
| 51 |
+
"ppl_gap_relative": 0.00036680849155647395,
|
| 52 |
+
"faiss_diag": {
|
| 53 |
+
"self_pad_rate": 0.6830791135629019,
|
| 54 |
+
"causal_fill_rate": 0.3130263686180115,
|
| 55 |
+
"self_attn_rate": 0.0038945178190867105
|
| 56 |
+
}
|
| 57 |
+
}
|
| 58 |
+
}
|
| 59 |
+
}
|
checkpoints_block_d128/search_step_1000.paired_K128_exact_quest_page16.json
ADDED
|
@@ -0,0 +1,331 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"ckpt": "/tmp/checkpoints_block_d128/search_step_1000.pt",
|
| 3 |
+
"step": 1000,
|
| 4 |
+
"K": 128,
|
| 5 |
+
"page_size": 16,
|
| 6 |
+
"num_batches": 32,
|
| 7 |
+
"skip_batches": 0,
|
| 8 |
+
"use_faiss": false,
|
| 9 |
+
"nll": {
|
| 10 |
+
"full_mean": 3.3333963975310326,
|
| 11 |
+
"learned_mean": 3.3345858082175255,
|
| 12 |
+
"quest_mean": 3.3325326964259148
|
| 13 |
+
},
|
| 14 |
+
"ppl": {
|
| 15 |
+
"full": 28.033392742203674,
|
| 16 |
+
"learned": 28.06675579636326,
|
| 17 |
+
"quest": 28.009190723071885
|
| 18 |
+
},
|
| 19 |
+
"relative_ppl_gap": {
|
| 20 |
+
"learned_vs_full": 0.0011901183159097606,
|
| 21 |
+
"quest_vs_full": -0.0008633282226789829,
|
| 22 |
+
"learned_vs_quest": 0.002055220868768526
|
| 23 |
+
},
|
| 24 |
+
"paired_nll_delta": {
|
| 25 |
+
"learned_minus_full": {
|
| 26 |
+
"mean": 0.00118941068649292,
|
| 27 |
+
"lo": 0.0007792413234710693,
|
| 28 |
+
"hi": 0.0015988349914550781
|
| 29 |
+
},
|
| 30 |
+
"quest_minus_full": {
|
| 31 |
+
"mean": -0.0008637011051177979,
|
| 32 |
+
"lo": -0.0012721419334411621,
|
| 33 |
+
"hi": -0.00043635815382003784
|
| 34 |
+
},
|
| 35 |
+
"learned_minus_quest": {
|
| 36 |
+
"mean": 0.0020531117916107178,
|
| 37 |
+
"lo": 0.0016007423400878906,
|
| 38 |
+
"hi": 0.002509795129299164
|
| 39 |
+
}
|
| 40 |
+
},
|
| 41 |
+
"per_batch": [
|
| 42 |
+
{
|
| 43 |
+
"batch": 0,
|
| 44 |
+
"full_nll": 2.9546234607696533,
|
| 45 |
+
"learned_nll": 2.954435110092163,
|
| 46 |
+
"quest_nll": 2.952946901321411,
|
| 47 |
+
"learned_minus_full": -0.00018835067749023438,
|
| 48 |
+
"quest_minus_full": -0.0016765594482421875,
|
| 49 |
+
"learned_minus_quest": 0.0014882087707519531
|
| 50 |
+
},
|
| 51 |
+
{
|
| 52 |
+
"batch": 1,
|
| 53 |
+
"full_nll": 3.085359811782837,
|
| 54 |
+
"learned_nll": 3.0881309509277344,
|
| 55 |
+
"quest_nll": 3.0834481716156006,
|
| 56 |
+
"learned_minus_full": 0.002771139144897461,
|
| 57 |
+
"quest_minus_full": -0.0019116401672363281,
|
| 58 |
+
"learned_minus_quest": 0.004682779312133789
|
| 59 |
+
},
|
| 60 |
+
{
|
| 61 |
+
"batch": 2,
|
| 62 |
+
"full_nll": 2.932983636856079,
|
| 63 |
+
"learned_nll": 2.9336395263671875,
|
| 64 |
+
"quest_nll": 2.931037187576294,
|
| 65 |
+
"learned_minus_full": 0.0006558895111083984,
|
| 66 |
+
"quest_minus_full": -0.0019464492797851562,
|
| 67 |
+
"learned_minus_quest": 0.0026023387908935547
|
| 68 |
+
},
|
| 69 |
+
{
|
| 70 |
+
"batch": 3,
|
| 71 |
+
"full_nll": 3.005047082901001,
|
| 72 |
+
"learned_nll": 3.004648447036743,
|
| 73 |
+
"quest_nll": 3.0019640922546387,
|
| 74 |
+
"learned_minus_full": -0.0003986358642578125,
|
| 75 |
+
"quest_minus_full": -0.0030829906463623047,
|
| 76 |
+
"learned_minus_quest": 0.002684354782104492
|
| 77 |
+
},
|
| 78 |
+
{
|
| 79 |
+
"batch": 4,
|
| 80 |
+
"full_nll": 3.379091739654541,
|
| 81 |
+
"learned_nll": 3.3796677589416504,
|
| 82 |
+
"quest_nll": 3.3768582344055176,
|
| 83 |
+
"learned_minus_full": 0.000576019287109375,
|
| 84 |
+
"quest_minus_full": -0.0022335052490234375,
|
| 85 |
+
"learned_minus_quest": 0.0028095245361328125
|
| 86 |
+
},
|
| 87 |
+
{
|
| 88 |
+
"batch": 5,
|
| 89 |
+
"full_nll": 3.7961206436157227,
|
| 90 |
+
"learned_nll": 3.7971668243408203,
|
| 91 |
+
"quest_nll": 3.7935140132904053,
|
| 92 |
+
"learned_minus_full": 0.0010461807250976562,
|
| 93 |
+
"quest_minus_full": -0.002606630325317383,
|
| 94 |
+
"learned_minus_quest": 0.003652811050415039
|
| 95 |
+
},
|
| 96 |
+
{
|
| 97 |
+
"batch": 6,
|
| 98 |
+
"full_nll": 3.735278844833374,
|
| 99 |
+
"learned_nll": 3.734419345855713,
|
| 100 |
+
"quest_nll": 3.7345199584960938,
|
| 101 |
+
"learned_minus_full": -0.0008594989776611328,
|
| 102 |
+
"quest_minus_full": -0.0007588863372802734,
|
| 103 |
+
"learned_minus_quest": -0.00010061264038085938
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"batch": 7,
|
| 107 |
+
"full_nll": 3.9848427772521973,
|
| 108 |
+
"learned_nll": 3.984069347381592,
|
| 109 |
+
"quest_nll": 3.984002113342285,
|
| 110 |
+
"learned_minus_full": -0.0007734298706054688,
|
| 111 |
+
"quest_minus_full": -0.0008406639099121094,
|
| 112 |
+
"learned_minus_quest": 6.723403930664062e-05
|
| 113 |
+
},
|
| 114 |
+
{
|
| 115 |
+
"batch": 8,
|
| 116 |
+
"full_nll": 3.160245656967163,
|
| 117 |
+
"learned_nll": 3.1608190536499023,
|
| 118 |
+
"quest_nll": 3.1590189933776855,
|
| 119 |
+
"learned_minus_full": 0.0005733966827392578,
|
| 120 |
+
"quest_minus_full": -0.001226663589477539,
|
| 121 |
+
"learned_minus_quest": 0.0018000602722167969
|
| 122 |
+
},
|
| 123 |
+
{
|
| 124 |
+
"batch": 9,
|
| 125 |
+
"full_nll": 3.6767594814300537,
|
| 126 |
+
"learned_nll": 3.67683482170105,
|
| 127 |
+
"quest_nll": 3.675286054611206,
|
| 128 |
+
"learned_minus_full": 7.534027099609375e-05,
|
| 129 |
+
"quest_minus_full": -0.0014734268188476562,
|
| 130 |
+
"learned_minus_quest": 0.00154876708984375
|
| 131 |
+
},
|
| 132 |
+
{
|
| 133 |
+
"batch": 10,
|
| 134 |
+
"full_nll": 3.5911879539489746,
|
| 135 |
+
"learned_nll": 3.593716621398926,
|
| 136 |
+
"quest_nll": 3.590583086013794,
|
| 137 |
+
"learned_minus_full": 0.002528667449951172,
|
| 138 |
+
"quest_minus_full": -0.0006048679351806641,
|
| 139 |
+
"learned_minus_quest": 0.003133535385131836
|
| 140 |
+
},
|
| 141 |
+
{
|
| 142 |
+
"batch": 11,
|
| 143 |
+
"full_nll": 3.289647340774536,
|
| 144 |
+
"learned_nll": 3.2900381088256836,
|
| 145 |
+
"quest_nll": 3.2886180877685547,
|
| 146 |
+
"learned_minus_full": 0.00039076805114746094,
|
| 147 |
+
"quest_minus_full": -0.0010292530059814453,
|
| 148 |
+
"learned_minus_quest": 0.0014200210571289062
|
| 149 |
+
},
|
| 150 |
+
{
|
| 151 |
+
"batch": 12,
|
| 152 |
+
"full_nll": 2.9889602661132812,
|
| 153 |
+
"learned_nll": 2.990806818008423,
|
| 154 |
+
"quest_nll": 2.9867987632751465,
|
| 155 |
+
"learned_minus_full": 0.0018465518951416016,
|
| 156 |
+
"quest_minus_full": -0.0021615028381347656,
|
| 157 |
+
"learned_minus_quest": 0.004008054733276367
|
| 158 |
+
},
|
| 159 |
+
{
|
| 160 |
+
"batch": 13,
|
| 161 |
+
"full_nll": 3.3506505489349365,
|
| 162 |
+
"learned_nll": 3.350663661956787,
|
| 163 |
+
"quest_nll": 3.3496899604797363,
|
| 164 |
+
"learned_minus_full": 1.3113021850585938e-05,
|
| 165 |
+
"quest_minus_full": -0.0009605884552001953,
|
| 166 |
+
"learned_minus_quest": 0.0009737014770507812
|
| 167 |
+
},
|
| 168 |
+
{
|
| 169 |
+
"batch": 14,
|
| 170 |
+
"full_nll": 3.566884756088257,
|
| 171 |
+
"learned_nll": 3.5690810680389404,
|
| 172 |
+
"quest_nll": 3.568802833557129,
|
| 173 |
+
"learned_minus_full": 0.0021963119506835938,
|
| 174 |
+
"quest_minus_full": 0.0019180774688720703,
|
| 175 |
+
"learned_minus_quest": 0.00027823448181152344
|
| 176 |
+
},
|
| 177 |
+
{
|
| 178 |
+
"batch": 15,
|
| 179 |
+
"full_nll": 3.3148910999298096,
|
| 180 |
+
"learned_nll": 3.3169944286346436,
|
| 181 |
+
"quest_nll": 3.315297842025757,
|
| 182 |
+
"learned_minus_full": 0.0021033287048339844,
|
| 183 |
+
"quest_minus_full": 0.0004067420959472656,
|
| 184 |
+
"learned_minus_quest": 0.0016965866088867188
|
| 185 |
+
},
|
| 186 |
+
{
|
| 187 |
+
"batch": 16,
|
| 188 |
+
"full_nll": 3.342437744140625,
|
| 189 |
+
"learned_nll": 3.342118263244629,
|
| 190 |
+
"quest_nll": 3.3403725624084473,
|
| 191 |
+
"learned_minus_full": -0.00031948089599609375,
|
| 192 |
+
"quest_minus_full": -0.0020651817321777344,
|
| 193 |
+
"learned_minus_quest": 0.0017457008361816406
|
| 194 |
+
},
|
| 195 |
+
{
|
| 196 |
+
"batch": 17,
|
| 197 |
+
"full_nll": 3.1053965091705322,
|
| 198 |
+
"learned_nll": 3.1074142456054688,
|
| 199 |
+
"quest_nll": 3.1063807010650635,
|
| 200 |
+
"learned_minus_full": 0.0020177364349365234,
|
| 201 |
+
"quest_minus_full": 0.00098419189453125,
|
| 202 |
+
"learned_minus_quest": 0.0010335445404052734
|
| 203 |
+
},
|
| 204 |
+
{
|
| 205 |
+
"batch": 18,
|
| 206 |
+
"full_nll": 3.271756172180176,
|
| 207 |
+
"learned_nll": 3.2738752365112305,
|
| 208 |
+
"quest_nll": 3.271651268005371,
|
| 209 |
+
"learned_minus_full": 0.0021190643310546875,
|
| 210 |
+
"quest_minus_full": -0.0001049041748046875,
|
| 211 |
+
"learned_minus_quest": 0.002223968505859375
|
| 212 |
+
},
|
| 213 |
+
{
|
| 214 |
+
"batch": 19,
|
| 215 |
+
"full_nll": 3.1598434448242188,
|
| 216 |
+
"learned_nll": 3.1599864959716797,
|
| 217 |
+
"quest_nll": 3.1578729152679443,
|
| 218 |
+
"learned_minus_full": 0.0001430511474609375,
|
| 219 |
+
"quest_minus_full": -0.001970529556274414,
|
| 220 |
+
"learned_minus_quest": 0.0021135807037353516
|
| 221 |
+
},
|
| 222 |
+
{
|
| 223 |
+
"batch": 20,
|
| 224 |
+
"full_nll": 3.152883768081665,
|
| 225 |
+
"learned_nll": 3.1570069789886475,
|
| 226 |
+
"quest_nll": 3.152681350708008,
|
| 227 |
+
"learned_minus_full": 0.004123210906982422,
|
| 228 |
+
"quest_minus_full": -0.00020241737365722656,
|
| 229 |
+
"learned_minus_quest": 0.0043256282806396484
|
| 230 |
+
},
|
| 231 |
+
{
|
| 232 |
+
"batch": 21,
|
| 233 |
+
"full_nll": 3.5090653896331787,
|
| 234 |
+
"learned_nll": 3.511439800262451,
|
| 235 |
+
"quest_nll": 3.510064125061035,
|
| 236 |
+
"learned_minus_full": 0.002374410629272461,
|
| 237 |
+
"quest_minus_full": 0.0009987354278564453,
|
| 238 |
+
"learned_minus_quest": 0.0013756752014160156
|
| 239 |
+
},
|
| 240 |
+
{
|
| 241 |
+
"batch": 22,
|
| 242 |
+
"full_nll": 3.559513807296753,
|
| 243 |
+
"learned_nll": 3.5612521171569824,
|
| 244 |
+
"quest_nll": 3.558289051055908,
|
| 245 |
+
"learned_minus_full": 0.0017383098602294922,
|
| 246 |
+
"quest_minus_full": -0.0012247562408447266,
|
| 247 |
+
"learned_minus_quest": 0.0029630661010742188
|
| 248 |
+
},
|
| 249 |
+
{
|
| 250 |
+
"batch": 23,
|
| 251 |
+
"full_nll": 3.3794538974761963,
|
| 252 |
+
"learned_nll": 3.3800926208496094,
|
| 253 |
+
"quest_nll": 3.3782293796539307,
|
| 254 |
+
"learned_minus_full": 0.0006387233734130859,
|
| 255 |
+
"quest_minus_full": -0.001224517822265625,
|
| 256 |
+
"learned_minus_quest": 0.001863241195678711
|
| 257 |
+
},
|
| 258 |
+
{
|
| 259 |
+
"batch": 24,
|
| 260 |
+
"full_nll": 3.629025459289551,
|
| 261 |
+
"learned_nll": 3.6307530403137207,
|
| 262 |
+
"quest_nll": 3.629179000854492,
|
| 263 |
+
"learned_minus_full": 0.0017275810241699219,
|
| 264 |
+
"quest_minus_full": 0.00015354156494140625,
|
| 265 |
+
"learned_minus_quest": 0.0015740394592285156
|
| 266 |
+
},
|
| 267 |
+
{
|
| 268 |
+
"batch": 25,
|
| 269 |
+
"full_nll": 3.3575053215026855,
|
| 270 |
+
"learned_nll": 3.3582494258880615,
|
| 271 |
+
"quest_nll": 3.356001377105713,
|
| 272 |
+
"learned_minus_full": 0.0007441043853759766,
|
| 273 |
+
"quest_minus_full": -0.0015039443969726562,
|
| 274 |
+
"learned_minus_quest": 0.002248048782348633
|
| 275 |
+
},
|
| 276 |
+
{
|
| 277 |
+
"batch": 26,
|
| 278 |
+
"full_nll": 3.236471652984619,
|
| 279 |
+
"learned_nll": 3.2382962703704834,
|
| 280 |
+
"quest_nll": 3.23522686958313,
|
| 281 |
+
"learned_minus_full": 0.0018246173858642578,
|
| 282 |
+
"quest_minus_full": -0.0012447834014892578,
|
| 283 |
+
"learned_minus_quest": 0.0030694007873535156
|
| 284 |
+
},
|
| 285 |
+
{
|
| 286 |
+
"batch": 27,
|
| 287 |
+
"full_nll": 3.0428874492645264,
|
| 288 |
+
"learned_nll": 3.045490264892578,
|
| 289 |
+
"quest_nll": 3.04168963432312,
|
| 290 |
+
"learned_minus_full": 0.002602815628051758,
|
| 291 |
+
"quest_minus_full": -0.00119781494140625,
|
| 292 |
+
"learned_minus_quest": 0.003800630569458008
|
| 293 |
+
},
|
| 294 |
+
{
|
| 295 |
+
"batch": 28,
|
| 296 |
+
"full_nll": 3.226649522781372,
|
| 297 |
+
"learned_nll": 3.2279727458953857,
|
| 298 |
+
"quest_nll": 3.227558135986328,
|
| 299 |
+
"learned_minus_full": 0.0013232231140136719,
|
| 300 |
+
"quest_minus_full": 0.0009086132049560547,
|
| 301 |
+
"learned_minus_quest": 0.0004146099090576172
|
| 302 |
+
},
|
| 303 |
+
{
|
| 304 |
+
"batch": 29,
|
| 305 |
+
"full_nll": 3.1754865646362305,
|
| 306 |
+
"learned_nll": 3.17584228515625,
|
| 307 |
+
"quest_nll": 3.1766295433044434,
|
| 308 |
+
"learned_minus_full": 0.00035572052001953125,
|
| 309 |
+
"quest_minus_full": 0.0011429786682128906,
|
| 310 |
+
"learned_minus_quest": -0.0007872581481933594
|
| 311 |
+
},
|
| 312 |
+
{
|
| 313 |
+
"batch": 30,
|
| 314 |
+
"full_nll": 3.198526382446289,
|
| 315 |
+
"learned_nll": 3.200511932373047,
|
| 316 |
+
"quest_nll": 3.1971871852874756,
|
| 317 |
+
"learned_minus_full": 0.0019855499267578125,
|
| 318 |
+
"quest_minus_full": -0.0013391971588134766,
|
| 319 |
+
"learned_minus_quest": 0.003324747085571289
|
| 320 |
+
},
|
| 321 |
+
{
|
| 322 |
+
"batch": 31,
|
| 323 |
+
"full_nll": 3.509206533432007,
|
| 324 |
+
"learned_nll": 3.511312246322632,
|
| 325 |
+
"quest_nll": 3.5096468925476074,
|
| 326 |
+
"learned_minus_full": 0.002105712890625,
|
| 327 |
+
"quest_minus_full": 0.00044035911560058594,
|
| 328 |
+
"learned_minus_quest": 0.001665353775024414
|
| 329 |
+
}
|
| 330 |
+
]
|
| 331 |
+
}
|
checkpoints_block_d128/search_step_1000.paired_K256_exact_quest_page16.json
ADDED
|
@@ -0,0 +1,331 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"ckpt": "/tmp/checkpoints_block_d128/search_step_1000.pt",
|
| 3 |
+
"step": 1000,
|
| 4 |
+
"K": 256,
|
| 5 |
+
"page_size": 16,
|
| 6 |
+
"num_batches": 32,
|
| 7 |
+
"skip_batches": 0,
|
| 8 |
+
"use_faiss": false,
|
| 9 |
+
"nll": {
|
| 10 |
+
"full_mean": 3.3333963975310326,
|
| 11 |
+
"learned_mean": 3.333659775555134,
|
| 12 |
+
"quest_mean": 3.3337095081806183
|
| 13 |
+
},
|
| 14 |
+
"ppl": {
|
| 15 |
+
"full": 28.033392742203674,
|
| 16 |
+
"learned": 28.040777094188304,
|
| 17 |
+
"quest": 28.0421716703315
|
| 18 |
+
},
|
| 19 |
+
"relative_ppl_gap": {
|
| 20 |
+
"learned_vs_full": 0.0002634127111382778,
|
| 21 |
+
"quest_vs_full": 0.00031315967384171195,
|
| 22 |
+
"learned_vs_quest": -4.973138883790362e-05
|
| 23 |
+
},
|
| 24 |
+
"paired_nll_delta": {
|
| 25 |
+
"learned_minus_full": {
|
| 26 |
+
"mean": 0.0002633780241012573,
|
| 27 |
+
"lo": 4.4718384742736816e-05,
|
| 28 |
+
"hi": 0.0004728734493255615
|
| 29 |
+
},
|
| 30 |
+
"quest_minus_full": {
|
| 31 |
+
"mean": 0.0003131106495857239,
|
| 32 |
+
"lo": 3.840029239654541e-05,
|
| 33 |
+
"hi": 0.0006035193800926208
|
| 34 |
+
},
|
| 35 |
+
"learned_minus_quest": {
|
| 36 |
+
"mean": -4.973262548446655e-05,
|
| 37 |
+
"lo": -0.00029237568378448486,
|
| 38 |
+
"hi": 0.00018288195133209229
|
| 39 |
+
}
|
| 40 |
+
},
|
| 41 |
+
"per_batch": [
|
| 42 |
+
{
|
| 43 |
+
"batch": 0,
|
| 44 |
+
"full_nll": 2.9546234607696533,
|
| 45 |
+
"learned_nll": 2.9539222717285156,
|
| 46 |
+
"quest_nll": 2.9538986682891846,
|
| 47 |
+
"learned_minus_full": -0.0007011890411376953,
|
| 48 |
+
"quest_minus_full": -0.00072479248046875,
|
| 49 |
+
"learned_minus_quest": 2.3603439331054688e-05
|
| 50 |
+
},
|
| 51 |
+
{
|
| 52 |
+
"batch": 1,
|
| 53 |
+
"full_nll": 3.085359811782837,
|
| 54 |
+
"learned_nll": 3.0854341983795166,
|
| 55 |
+
"quest_nll": 3.085425615310669,
|
| 56 |
+
"learned_minus_full": 7.43865966796875e-05,
|
| 57 |
+
"quest_minus_full": 6.580352783203125e-05,
|
| 58 |
+
"learned_minus_quest": 8.58306884765625e-06
|
| 59 |
+
},
|
| 60 |
+
{
|
| 61 |
+
"batch": 2,
|
| 62 |
+
"full_nll": 2.932983636856079,
|
| 63 |
+
"learned_nll": 2.9332051277160645,
|
| 64 |
+
"quest_nll": 2.932912588119507,
|
| 65 |
+
"learned_minus_full": 0.00022149085998535156,
|
| 66 |
+
"quest_minus_full": -7.104873657226562e-05,
|
| 67 |
+
"learned_minus_quest": 0.0002925395965576172
|
| 68 |
+
},
|
| 69 |
+
{
|
| 70 |
+
"batch": 3,
|
| 71 |
+
"full_nll": 3.005047082901001,
|
| 72 |
+
"learned_nll": 3.004338502883911,
|
| 73 |
+
"quest_nll": 3.0038628578186035,
|
| 74 |
+
"learned_minus_full": -0.0007085800170898438,
|
| 75 |
+
"quest_minus_full": -0.001184225082397461,
|
| 76 |
+
"learned_minus_quest": 0.0004756450653076172
|
| 77 |
+
},
|
| 78 |
+
{
|
| 79 |
+
"batch": 4,
|
| 80 |
+
"full_nll": 3.379091739654541,
|
| 81 |
+
"learned_nll": 3.3800008296966553,
|
| 82 |
+
"quest_nll": 3.3800079822540283,
|
| 83 |
+
"learned_minus_full": 0.0009090900421142578,
|
| 84 |
+
"quest_minus_full": 0.0009162425994873047,
|
| 85 |
+
"learned_minus_quest": -7.152557373046875e-06
|
| 86 |
+
},
|
| 87 |
+
{
|
| 88 |
+
"batch": 5,
|
| 89 |
+
"full_nll": 3.7961206436157227,
|
| 90 |
+
"learned_nll": 3.79643177986145,
|
| 91 |
+
"quest_nll": 3.796604871749878,
|
| 92 |
+
"learned_minus_full": 0.00031113624572753906,
|
| 93 |
+
"quest_minus_full": 0.00048422813415527344,
|
| 94 |
+
"learned_minus_quest": -0.00017309188842773438
|
| 95 |
+
},
|
| 96 |
+
{
|
| 97 |
+
"batch": 6,
|
| 98 |
+
"full_nll": 3.735278844833374,
|
| 99 |
+
"learned_nll": 3.734744071960449,
|
| 100 |
+
"quest_nll": 3.7353076934814453,
|
| 101 |
+
"learned_minus_full": -0.0005347728729248047,
|
| 102 |
+
"quest_minus_full": 2.8848648071289062e-05,
|
| 103 |
+
"learned_minus_quest": -0.0005636215209960938
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"batch": 7,
|
| 107 |
+
"full_nll": 3.9848427772521973,
|
| 108 |
+
"learned_nll": 3.9845895767211914,
|
| 109 |
+
"quest_nll": 3.9847006797790527,
|
| 110 |
+
"learned_minus_full": -0.0002532005310058594,
|
| 111 |
+
"quest_minus_full": -0.00014209747314453125,
|
| 112 |
+
"learned_minus_quest": -0.00011110305786132812
|
| 113 |
+
},
|
| 114 |
+
{
|
| 115 |
+
"batch": 8,
|
| 116 |
+
"full_nll": 3.160245656967163,
|
| 117 |
+
"learned_nll": 3.16005802154541,
|
| 118 |
+
"quest_nll": 3.160933256149292,
|
| 119 |
+
"learned_minus_full": -0.0001876354217529297,
|
| 120 |
+
"quest_minus_full": 0.0006875991821289062,
|
| 121 |
+
"learned_minus_quest": -0.0008752346038818359
|
| 122 |
+
},
|
| 123 |
+
{
|
| 124 |
+
"batch": 9,
|
| 125 |
+
"full_nll": 3.6767594814300537,
|
| 126 |
+
"learned_nll": 3.6761300563812256,
|
| 127 |
+
"quest_nll": 3.676041603088379,
|
| 128 |
+
"learned_minus_full": -0.000629425048828125,
|
| 129 |
+
"quest_minus_full": -0.0007178783416748047,
|
| 130 |
+
"learned_minus_quest": 8.845329284667969e-05
|
| 131 |
+
},
|
| 132 |
+
{
|
| 133 |
+
"batch": 10,
|
| 134 |
+
"full_nll": 3.5911879539489746,
|
| 135 |
+
"learned_nll": 3.591670036315918,
|
| 136 |
+
"quest_nll": 3.592432737350464,
|
| 137 |
+
"learned_minus_full": 0.0004820823669433594,
|
| 138 |
+
"quest_minus_full": 0.0012447834014892578,
|
| 139 |
+
"learned_minus_quest": -0.0007627010345458984
|
| 140 |
+
},
|
| 141 |
+
{
|
| 142 |
+
"batch": 11,
|
| 143 |
+
"full_nll": 3.289647340774536,
|
| 144 |
+
"learned_nll": 3.2903831005096436,
|
| 145 |
+
"quest_nll": 3.2905445098876953,
|
| 146 |
+
"learned_minus_full": 0.0007357597351074219,
|
| 147 |
+
"quest_minus_full": 0.0008971691131591797,
|
| 148 |
+
"learned_minus_quest": -0.0001614093780517578
|
| 149 |
+
},
|
| 150 |
+
{
|
| 151 |
+
"batch": 12,
|
| 152 |
+
"full_nll": 2.9889602661132812,
|
| 153 |
+
"learned_nll": 2.989010810852051,
|
| 154 |
+
"quest_nll": 2.9898018836975098,
|
| 155 |
+
"learned_minus_full": 5.054473876953125e-05,
|
| 156 |
+
"quest_minus_full": 0.0008416175842285156,
|
| 157 |
+
"learned_minus_quest": -0.0007910728454589844
|
| 158 |
+
},
|
| 159 |
+
{
|
| 160 |
+
"batch": 13,
|
| 161 |
+
"full_nll": 3.3506505489349365,
|
| 162 |
+
"learned_nll": 3.351274251937866,
|
| 163 |
+
"quest_nll": 3.3513026237487793,
|
| 164 |
+
"learned_minus_full": 0.0006237030029296875,
|
| 165 |
+
"quest_minus_full": 0.0006520748138427734,
|
| 166 |
+
"learned_minus_quest": -2.8371810913085938e-05
|
| 167 |
+
},
|
| 168 |
+
{
|
| 169 |
+
"batch": 14,
|
| 170 |
+
"full_nll": 3.566884756088257,
|
| 171 |
+
"learned_nll": 3.568331003189087,
|
| 172 |
+
"quest_nll": 3.568939447402954,
|
| 173 |
+
"learned_minus_full": 0.0014462471008300781,
|
| 174 |
+
"quest_minus_full": 0.0020546913146972656,
|
| 175 |
+
"learned_minus_quest": -0.0006084442138671875
|
| 176 |
+
},
|
| 177 |
+
{
|
| 178 |
+
"batch": 15,
|
| 179 |
+
"full_nll": 3.3148910999298096,
|
| 180 |
+
"learned_nll": 3.3152501583099365,
|
| 181 |
+
"quest_nll": 3.314882278442383,
|
| 182 |
+
"learned_minus_full": 0.0003590583801269531,
|
| 183 |
+
"quest_minus_full": -8.821487426757812e-06,
|
| 184 |
+
"learned_minus_quest": 0.00036787986755371094
|
| 185 |
+
},
|
| 186 |
+
{
|
| 187 |
+
"batch": 16,
|
| 188 |
+
"full_nll": 3.342437744140625,
|
| 189 |
+
"learned_nll": 3.340949773788452,
|
| 190 |
+
"quest_nll": 3.3425629138946533,
|
| 191 |
+
"learned_minus_full": -0.0014879703521728516,
|
| 192 |
+
"quest_minus_full": 0.0001251697540283203,
|
| 193 |
+
"learned_minus_quest": -0.0016131401062011719
|
| 194 |
+
},
|
| 195 |
+
{
|
| 196 |
+
"batch": 17,
|
| 197 |
+
"full_nll": 3.1053965091705322,
|
| 198 |
+
"learned_nll": 3.106194257736206,
|
| 199 |
+
"quest_nll": 3.105250358581543,
|
| 200 |
+
"learned_minus_full": 0.0007977485656738281,
|
| 201 |
+
"quest_minus_full": -0.0001461505889892578,
|
| 202 |
+
"learned_minus_quest": 0.0009438991546630859
|
| 203 |
+
},
|
| 204 |
+
{
|
| 205 |
+
"batch": 18,
|
| 206 |
+
"full_nll": 3.271756172180176,
|
| 207 |
+
"learned_nll": 3.2721667289733887,
|
| 208 |
+
"quest_nll": 3.27207350730896,
|
| 209 |
+
"learned_minus_full": 0.0004105567932128906,
|
| 210 |
+
"quest_minus_full": 0.0003173351287841797,
|
| 211 |
+
"learned_minus_quest": 9.322166442871094e-05
|
| 212 |
+
},
|
| 213 |
+
{
|
| 214 |
+
"batch": 19,
|
| 215 |
+
"full_nll": 3.1598434448242188,
|
| 216 |
+
"learned_nll": 3.160092830657959,
|
| 217 |
+
"quest_nll": 3.158867120742798,
|
| 218 |
+
"learned_minus_full": 0.0002493858337402344,
|
| 219 |
+
"quest_minus_full": -0.0009763240814208984,
|
| 220 |
+
"learned_minus_quest": 0.0012257099151611328
|
| 221 |
+
},
|
| 222 |
+
{
|
| 223 |
+
"batch": 20,
|
| 224 |
+
"full_nll": 3.152883768081665,
|
| 225 |
+
"learned_nll": 3.154238224029541,
|
| 226 |
+
"quest_nll": 3.154201030731201,
|
| 227 |
+
"learned_minus_full": 0.0013544559478759766,
|
| 228 |
+
"quest_minus_full": 0.0013172626495361328,
|
| 229 |
+
"learned_minus_quest": 3.719329833984375e-05
|
| 230 |
+
},
|
| 231 |
+
{
|
| 232 |
+
"batch": 21,
|
| 233 |
+
"full_nll": 3.5090653896331787,
|
| 234 |
+
"learned_nll": 3.510181427001953,
|
| 235 |
+
"quest_nll": 3.511232852935791,
|
| 236 |
+
"learned_minus_full": 0.001116037368774414,
|
| 237 |
+
"quest_minus_full": 0.0021674633026123047,
|
| 238 |
+
"learned_minus_quest": -0.0010514259338378906
|
| 239 |
+
},
|
| 240 |
+
{
|
| 241 |
+
"batch": 22,
|
| 242 |
+
"full_nll": 3.559513807296753,
|
| 243 |
+
"learned_nll": 3.559825897216797,
|
| 244 |
+
"quest_nll": 3.5592892169952393,
|
| 245 |
+
"learned_minus_full": 0.0003120899200439453,
|
| 246 |
+
"quest_minus_full": -0.00022459030151367188,
|
| 247 |
+
"learned_minus_quest": 0.0005366802215576172
|
| 248 |
+
},
|
| 249 |
+
{
|
| 250 |
+
"batch": 23,
|
| 251 |
+
"full_nll": 3.3794538974761963,
|
| 252 |
+
"learned_nll": 3.379786252975464,
|
| 253 |
+
"quest_nll": 3.3805532455444336,
|
| 254 |
+
"learned_minus_full": 0.0003323554992675781,
|
| 255 |
+
"quest_minus_full": 0.0010993480682373047,
|
| 256 |
+
"learned_minus_quest": -0.0007669925689697266
|
| 257 |
+
},
|
| 258 |
+
{
|
| 259 |
+
"batch": 24,
|
| 260 |
+
"full_nll": 3.629025459289551,
|
| 261 |
+
"learned_nll": 3.629706621170044,
|
| 262 |
+
"quest_nll": 3.6291189193725586,
|
| 263 |
+
"learned_minus_full": 0.0006811618804931641,
|
| 264 |
+
"quest_minus_full": 9.34600830078125e-05,
|
| 265 |
+
"learned_minus_quest": 0.0005877017974853516
|
| 266 |
+
},
|
| 267 |
+
{
|
| 268 |
+
"batch": 25,
|
| 269 |
+
"full_nll": 3.3575053215026855,
|
| 270 |
+
"learned_nll": 3.357809543609619,
|
| 271 |
+
"quest_nll": 3.3570470809936523,
|
| 272 |
+
"learned_minus_full": 0.00030422210693359375,
|
| 273 |
+
"quest_minus_full": -0.0004582405090332031,
|
| 274 |
+
"learned_minus_quest": 0.0007624626159667969
|
| 275 |
+
},
|
| 276 |
+
{
|
| 277 |
+
"batch": 26,
|
| 278 |
+
"full_nll": 3.236471652984619,
|
| 279 |
+
"learned_nll": 3.23730206489563,
|
| 280 |
+
"quest_nll": 3.2361526489257812,
|
| 281 |
+
"learned_minus_full": 0.0008304119110107422,
|
| 282 |
+
"quest_minus_full": -0.0003190040588378906,
|
| 283 |
+
"learned_minus_quest": 0.0011494159698486328
|
| 284 |
+
},
|
| 285 |
+
{
|
| 286 |
+
"batch": 27,
|
| 287 |
+
"full_nll": 3.0428874492645264,
|
| 288 |
+
"learned_nll": 3.0435779094696045,
|
| 289 |
+
"quest_nll": 3.0431928634643555,
|
| 290 |
+
"learned_minus_full": 0.000690460205078125,
|
| 291 |
+
"quest_minus_full": 0.00030541419982910156,
|
| 292 |
+
"learned_minus_quest": 0.00038504600524902344
|
| 293 |
+
},
|
| 294 |
+
{
|
| 295 |
+
"batch": 28,
|
| 296 |
+
"full_nll": 3.226649522781372,
|
| 297 |
+
"learned_nll": 3.2267818450927734,
|
| 298 |
+
"quest_nll": 3.227113962173462,
|
| 299 |
+
"learned_minus_full": 0.0001323223114013672,
|
| 300 |
+
"quest_minus_full": 0.00046443939208984375,
|
| 301 |
+
"learned_minus_quest": -0.00033211708068847656
|
| 302 |
+
},
|
| 303 |
+
{
|
| 304 |
+
"batch": 29,
|
| 305 |
+
"full_nll": 3.1754865646362305,
|
| 306 |
+
"learned_nll": 3.175914764404297,
|
| 307 |
+
"quest_nll": 3.1773221492767334,
|
| 308 |
+
"learned_minus_full": 0.00042819976806640625,
|
| 309 |
+
"quest_minus_full": 0.0018355846405029297,
|
| 310 |
+
"learned_minus_quest": -0.0014073848724365234
|
| 311 |
+
},
|
| 312 |
+
{
|
| 313 |
+
"batch": 30,
|
| 314 |
+
"full_nll": 3.198526382446289,
|
| 315 |
+
"learned_nll": 3.1984312534332275,
|
| 316 |
+
"quest_nll": 3.198376417160034,
|
| 317 |
+
"learned_minus_full": -9.512901306152344e-05,
|
| 318 |
+
"quest_minus_full": -0.0001499652862548828,
|
| 319 |
+
"learned_minus_quest": 5.4836273193359375e-05
|
| 320 |
+
},
|
| 321 |
+
{
|
| 322 |
+
"batch": 31,
|
| 323 |
+
"full_nll": 3.509206533432007,
|
| 324 |
+
"learned_nll": 3.5093796253204346,
|
| 325 |
+
"quest_nll": 3.5087506771087646,
|
| 326 |
+
"learned_minus_full": 0.00017309188842773438,
|
| 327 |
+
"quest_minus_full": -0.0004558563232421875,
|
| 328 |
+
"learned_minus_quest": 0.0006289482116699219
|
| 329 |
+
}
|
| 330 |
+
]
|
| 331 |
+
}
|