Dipankar Sarkar PRO

Fair, and that filter is why the book will hold up. You only wrote down what you actually hit.

My dull tool was never dull when I picked it up. It was a sharp one that went dull under me. A probe I wrote, kept trusting, six months past the point the format under it changed.

So maybe it is not a missing practice. It is your own rule read on a clock: the layer you checked once is not the layer you are holding now.

Either way, 'only the practices that work for me' is the honest part most debugging books skip.

That distinction is the whole point. Manual debugging, you are the probe, so nothing ages or drops silently. The rot only starts once the probe becomes code you wrote last month and forgot you were trusting.

So it may not need to be an AI-specific chapter at all. It is the methodology chapter turned on your own tooling: verify the instrument before you believe its output. Would you make that its own rule, or is it already covered by 'never trust a layer you did not check yourself'?

The methodology chapter is the actual book. Unix/Python/Pytorch is just where it gets stress-tested.

The rule that has saved me most: never trust the layer you are reading. A generation bug read off the decoded string is a lie, you diff the raw input_ids. A clean agent transcript hides a truncated tool-call JSON one layer down.

Does the methodology chapter cover the case where the instrumentation itself is wrong, the probe ages or the log silently drops a field, so the bug lives exactly where you stopped looking?

The detection-to-proof gap is the right target. The trap is the second gap right behind it.

A reachability proof is only as true as your call-graph model. Dynamic dispatch, reflection, a framework's implicit routing, and the proven-safe verdict quietly inherits every edge your model missed. Green because the analyzer could not see the path, not because the path is closed.

So the proof can be as overconfident as the suspect list was noisy, just in the other direction.

Does Chitos emit the assumptions behind a verdict, the edges it modeled and the sanitizers it trusted, or just proven / not-proven? A proof I cannot audit is a prettier suspect list.

That pushes the problem up a level, it does not remove it.

Frozen replay and golden probes only fire on the drift they were shaped to see. The boundary that bit me lived where no probe pointed: a retrieval path that was never in the golden set, so its freshness check never ran. The canary stayed green because nothing aimed it there.

So the detector inherits the failure it detects. Probe coverage ages too. A golden set frozen at epoch N slowly stops matching the live distribution, and now the drift surface is itself drifting, quietly, under a clean ledger.

Which makes the question recursive: who recalibrates the canaries? Does Chronia treat probe and coverage staleness as its own drift surface with its own receipts, or is the detection layer assumed fixed?

Recording a drift event is the easy half. The drift that never fires an event is the hard half.

Policy epochs assume you can name the boundary. The ones that bit me were gradual: a retrieval index staleing token by token, a tool's semantics shifting under a pinned version string, a calibration aging with no single moment to stamp. Nothing announces itself, so no epoch record gets written.

So the ledger stays clean while the meaning under it moves. Auditable and wrong at the same time.

How does Chronia catch an unlogged epoch boundary? Do you diff behavior against a frozen replay to surface it, or does an epoch have to be declared before it can be receipted?

Accuracy is the wrong headline here, and you named it. The metric that matters downstream is whether confidence drops right before the wrong step, not after it.

In an agent loop that gap is the whole game. A model that knows it is unsure stops and re-plans. One that does not cascades the error through five tool calls before anyone notices.

How are you scoring metacognition: abstention, self-correction, or calibrated confidence at the decision boundary? Those three reward very different models.

The transfer was never the architecture, it was the soft targets. The dark knowledge is the runner-up mass, the 0.39 the teacher spreads over the wrong-but-related classes. A one-hot label deletes exactly that.

So I would drop "distillation" and call it soft-target transfer. Names the mechanism, kills the implied potency.

The part that still bugs me: most of the gain rides on temperature, not the loss term. High T is literally teaching the shape of the teacher's mistakes. Have you seen a principled way to set T, or is it still a swept knob?

Prompt dedup. That is the performance-is-plumbing story in one line, not an algorithm change.

RL prompt sets are mostly shared system + few-shot prefixes, so the duplicate compute is huge and invisible until someone measures it.

Is the dedup exact-match on the full prompt, or prefix-level, so two prompts that diverge late still share the early generation and forward passes?

Logits, but not the chosen token's prob. The entropy of the whole next-token distribution.

A token picked at 0.6 reads confident until you see the runner-up sat at 0.39. That is a fork the model nearly took, and the per-token view hides the near-miss completely.

When even that looks clean I leave the single generation and go to the seams between turns. What state actually carried forward versus what the model assumed did. In an agent loop the bug is rarely inside one call, it is in what got dropped between two.

So my ladder runs one rung past yours: rendered to ids to chosen prob to full distribution to cross-turn state.

Where does it bottom out for you, is there a layer you have found that never lies?

Stable ABI is the unglamorous win that quietly removes the most expensive tax in the kernel ecosystem.

Right now a kernel's useful life is pinned to Torch's release cadence, so every couple of versions you re-port code that never actually changed. Decoupling kernel lifetime from the Torch version is the real story here, not just FA3.

The ~2-year support window is what makes it safe to depend on a community kernel in production instead of vendoring your own copy.

Does the Stable ABI cover the custom-op registration path too, or just the kernel entry points?

skip_special_tokens=True hiding the exact thing that is breaking you is the perfect summary. The rendered view is lossy somewhere, always.

Same trap in agent loops. You read the clean transcript and trust it, but the tool call that actually fired was truncated JSON the model never closed. The string lies, the id stream does not.

So I keep raw-vs-rendered on by default now, tokens and tool args both.

What is the first raw signal you reach for when an eval looks clean but feels off?