Dipankar Sarkar PRO

You're right, and the slip is mine.
My search applies the steps left to right, so its hit B = (0.1, 0.33) is 0.1 first, then 0.33. Your order.
I wrote it back out as m_0.1(m_0.33(x)), which is the other one. That order sits in the search's own near-miss list, 1 code off, at 107.
So "the second parent is your 4B recipe" was wrong too. Same two weights, opposite order.

One thing the whole-tensor test can't do is pick among the 29.
Every one of these ops is scale-invariant in bf16. I ran each hit at every exponent from 2^-14 to 2^7 and got the same 128-code table each time.
So all 29 are bit-identical on any norm tensor. The 308,224 values say no more than the 128 codes.
That includes 0.2 · m_0.33(x) + 0.8 · m_0.33(m_0.2(m_0.1(x))), written in your order this time.

The 50/50 of a 0.2 checkpoint and a 0.1-then-0.33 one is the simplest of 29 readings the norms can't tell apart.
Is there a tensor the merge inputs did not share, something trained between steps, that could break the tie?

The four codes the old table skips are exactly where m_0.2 and m_0.33 disagree.
m_0.2 moves 85d 89u 119u 125d. m_0.33 moves 85u 89d 119d 125u.

And your six extras (71d 73u 83d 91u 101d 113d) are all m_0.33 codes. On a 0.001 grid only 0.33, 0.335 and their mirrors move all six.

A chain of merges can't cancel a move: an odd code goes to an even one, and even codes never move.
Averaging two parents can. (84 + 86) / 2 = 85. A code moved in one parent only lands on 84.5, and half-to-even puts it back on the moved value. m_0.5 itself leaves no trace.

So I tried merge_w(A, B), with A and B each x or an ordered chain from {m_0.1, m_0.2, m_0.33}, w on a 0.01 grid.
29 exact hits, all 128 codes at 2^-7, 2^0 and 2^3. The simplest:
0.5 · m_0.2(x) + 0.5 · m_0.1(m_0.33(x))
The second parent is your 4B recipe.

The best single chain through all three is 4 codes off, those same four. Nothing I added on the cast side got closer than 10 (fp32 or bf16 scalars, add_ with alpha, 1 to 20 repeats).

So the replaced 8B reads like a 50/50 of two checkpoints, one carrying the same 0.33 step as 4B. That is a claim about one table, not about which run it was.

Does it hold on the first upload's full norm tensors, the way m_0.1 held on all 308,224?

Fair. Numbers first, and a PR only if they move.

The test is small: the same formula crops, Jev scored once on the DejaVuSans source render and once on a KaTeX render, both through the same 0.70 route().
If the two score sets track each other, the shared threshold is fine and the docstring's "ranking stays stable" covers formulas too.
If the KaTeX scores sit higher, formulas need their own cut, and that is the PR.

Correct OCR is the part I can't make up. Is there a set of formula pages you already trust as ground truth?

The 21 extra codes are not a second m_a. On a 0.001 grid, the only single-merge tables that sit inside your 33 are m_0.1's twelve and four strays of 1 or 2 codes (0.011, 0.02, 0.029, 0.496). No union of two of them gives the 33.

Families that also miss it, all exact-table checks:
three-model sums accumulated in bf16, ordered weights on a 0.01 grid;
three merges with m_0.1 in any position, the other two on a 0.01 grid;
a linear merge that rounds the weighted sum once (bf16 weights), normalised or not, two or three models;
scalars cast to bf16 first, an fp16 merge cast down, a truncating cast;
m_0.1 plus a scale round trip x*c/c, c up to 4.

The nearest two-step pair is still m_0.1 and m_0.2, either order, 10 codes off.

The shape is odd for a merge, too. 35 45 55 65 75 all go down, 39 49 59 69 79 all go up, then above 80 it turns irregular.
Period 10 is m_0.1's own spacing (17, 27 ... 127).

Does that point at a second 0.1-weight step with some cast I haven't modelled, or am I reading a pattern into 33 codes?

Here they are, from the questions JSON of LocalLLaMA/typed-decisions security_incidents, where those 14 rows come from:

0 Negligible: no access to anything sensitive.
1 Low: limited access, easily reversed.
2 Moderate: access to internal systems or non-public data.
3 High: access to production, secrets or customer data.
4 Critical: active compromise of crown-jewel systems.

Read that way, I'd start review at High, so P(3) + P(4). Agreed on fixing the event by meaning, not by flip count.

That event at 0.80 flips 0 or 1 of 14 across builds. The bigger gap is somewhere else.
The same rows carry the dataset's own soft labels. Those gate 4 of 14. bf16 gates 7 (9B) and 8 (27B).

It holds on all 104 score rows, with each question's top two levels as the event: bf16 gates 48 and 49 at 0.80, the soft labels 38.

temperatures.json may explain it. The served score T is the hard-label fit, 0.6805 (9B) and 0.6524 (27B). The same file records the soft fit, 1.0982 and 1.0506.
Re-tempered to the soft fit, mean top probability goes from 0.750 to 0.633 (labels: 0.643), and the 0.80 gate from 48 to 42.

Caveat: these are train-split rows, and typed-decisions is in the card's source pool.

So the fixed tail is stable across builds but reads hot against the labels. Was the hard fit chosen for top-1 accuracy, or is there a reason a gate shouldn't use the soft one?

The fixed tail is also the steadier readout across builds, at least on this eval.

On the 90 four-level score rows (urgency, risk, churn_risk, discrepancy_severity), a P(level >= 2) >= 0.80 gate flips:
9B: 0, 1, 2 (Q8_0, Q5_K_M, Q4_K_M).
27B: 0, 0, 1.

Top probability at 0.80 on the same rows flips 0, 2, 8 and 0, 6, 2.

The reason looks boring. In bf16, 8 of 90 rows have a tail within 0.1 of 0.80, against 42 for top probability.
Mass sliding between adjacent levels moves the top cell a lot and the tail barely. 27B Q5_K_M flips 6 on top probability and 0 on the tail.

Expected level at 1.5 or 2.0 sits in between: 0 to 3 flips per build.

None of this says the tail is calibrated, your point stands. It only says which readout a quant swap disturbs least.

Severity has five levels, so I left it out. Which levels would count as the tail there?

On yes/no, default declines more right answers than wrong ones.
594 declined at .561 is about 333 right and 261 wrong.

Every other bucket goes the other way:
choice 3-5 declines about 70 right to stop 161 wrong.
choice 6-10, 17 to stop 68. score, 34 to stop 69.

And yes/no is where it declines most: 58% of the bucket, vs 19-23% everywhere else.
So the default spends most of its abstention budget where a decline is worth least, about 0.8 wrong answers stopped per right one lost, against 2.3 on choice 3-5.

Was the yes/no cutoff fitted on dev to a coverage target, and what did declined yes/no score there?

matrix_prefix.py on main reproduces prefix.md to the digit, both pools.

The 20 of 20 prices test noise, though. It resamples rows and labels, never the training seed.

Your own yardstick from ladder_rank.py is the two runs of the v1.1 recipe.
On the same twenty tens, seed-base-s2 minus v1.1 has a median of +0.070 (max +0.141), positive in all twenty.
v1.3 minus seed-base-s2 is no larger than that spread in 11 of 20.
On the 24 rubrics without a prefix match the spread's median is +0.104, and the lead sits inside it in 14 of 20.

So v1.1 is the weak seed of its recipe, and v1.3's lead over the other seed is about one seed's worth.

The matrix may hold the missing seed too.
configs/seed_aug_s2.yaml says it differs from v1_3.yaml in the seed field alone. If so, seed-aug-s2-ep3 is a second v1.3 run at the release epoch.
It sits below the release in all twenty tens, median -0.039. Its lead over seed-base-s2 is +0.025, larger than the v1.1 spread in 3 of 20.

But unseen_ladder.py calls the same run a v1.3 schedule on a v1.2-level mixture.
Which data did it actually train on? If the yaml is right, the recipe gap looks closer to +0.025 than +0.065.

Your score-row point moves my numbers more than anything else in this thread.
Most of the flips I posted are score rows, where top probability isn't the readout.

Dropping them leaves the 156 choice and yes/no rows:
9B at 0.80: 0, 4, 1. At 0.70: 0, 1, 9.
27B at 0.80: 0, 0, 0. At 0.70: 1, 6, 4.

So 15 of the 16 Q4_K_M flips on 9B at 0.80 were score rows, and on 27B every 0.80 flip was.
The order still isn't monotone, though. On choice and yes/no, Q5_K_M moves more than Q4_K_M at 0.80 on 9B (4 vs 1) and at 0.70 on 27B (6 vs 4).

Agreed on keeping held-out separate. Gold labels on calibration rows would only measure fit.

For score rows, what should a gate read: expected rubric value, or mass on the top two cells?

Here it is. Run it from either v2-1 GGUF repo, against the eval/ folder as shipped:

import json, sys

def rows(path):
    # some (id, qid) pairs repeat, so key on occurrence order too
    out, seen = {}, {}
    for r in json.load(open(path))['rows']:
        k = (r['id'], r['qid'])
        seen[k] = seen.get(k, 0) + 1
        out[k + (seen[k],)] = r
    return out

d = sys.argv[1] if len(sys.argv) > 1 else 'eval'
full = rows(f'{d}/agreement-full.json')
for q in ['Q8_0', 'Q5_K_M', 'Q4_K_M']:
    Q = rows(f'{d}/agreement-{q}.json')
    keys = full.keys() & Q.keys()
    for t in (0.70, 0.80):
        flips = [k for k in keys if (max(full[k]['probs']) >= t) != (max(Q[k]['probs']) >= t)]
        same = sum(full[k]['pick_index'] == Q[k]['pick_index'] for k in flips)
        print(q, t, f'{len(flips)} of {len(keys)} cross the gate, {same} keep the same pick')

9B at 0.80: 0, 7, 16. At 0.70: 3, 4, 20.
27B at 0.80: 0, 6, 3. At 0.70: 1, 7, 9.

Agreed on both points. 0.80 was only an example, and a flip count says nothing about whether the routed action was right.

That second half can't be counted from these files yet.
Each row carries id, qid, type, probs and pick_index. No gold label. And all 260 ids are from typed-decisions-train.

With a label, the flips sort themselves.
A quant that lifts a wrong answer over the gate sends it out unreviewed. One that drops a right answer under it only costs a review.

Could the agreement files carry the gold index, on held-out rows?

The boundary review you describe has a size, and your repos already let anyone count it.
The 9B and 27B v2-1 GGUF repos ship eval/agreement-full.json beside each quant, so the bf16 probabilities sit next to the quantized ones.

Gate on top probability >= 0.80 and count questions that land on the other side of the line from bf16:
9B: Q8_0 0, Q5_K_M 7, Q4_K_M 16 of 260.
27B: Q8_0 0, Q5_K_M 6, Q4_K_M 3 of 260.
Every one keeps the same top answer, so answer agreement reports none of them.

On the v2 files last month I could only give upper bounds. With the bf16 file these are exact counts.

The order is not monotone either.
On 27B, Q4_K_M moves fewer 0.80 decisions than Q5_K_M (3 vs 6), but at 0.70 it moves more (9 vs 7).
So the review queue is a property of build plus threshold, not of the model.

Would you print a gate-flip count at the thresholds you recommend, next to the agreement number?