resoajoe/loglens-learnability

LogLens: can you tell whether a tiny vision model will learn a task — before training it?

One number, computed from task imagery alone (no network, no gradients, no training), predicts whether a ~10⁵-parameter model can learn a visual change-dating task. Blind-validated at Spearman ρ = 0.988 on ten pre-registered conditions. This repo contains the predictor, the mechanism that explains why it works, the measured boundary where it stops working, and a one-command harness so you can reproduce — or kill — the result on your own machine and your own scene.

Why this might interest you

• It is task-side, not architecture-side. Zero-cost NAS proxies (synflow, GradNorm, ZiCo, NTK scores) rank architectures for a fixed dataset using the network's gradients at init. This asks the complementary question: given a fixed small model, is this task learnable at all? The pixel-space predictor uses no network whatsoever.
• There is a mechanism, not just a correlation. The predictor works because an untrained encoder preserves and amplifies the pixel signal-to-clutter ratio into the feature space its head learns from (ρ = 0.981 across 28 task cells; ρ = 0.932 against trained accuracy, 5 seeds per config).
• The failure boundary is measured, not assumed. On natural-image classification the predictor inverts (ρ = −1.00) and no trainless variant rescues it. We explain why, and state the scope precisely.
• Everything was pre-registered, and the failures are published. Five claims were falsified by their own controls during this work; all are reported.

Quickstart — reproduce (or refute) it in one command

pip install torch numpy opencv-python scipy
python replicate.py --bank <folder-of-~1000-photos-of-any-room> --full --seeds 3

It builds a 64px bank from your photos, computes the predictor, trains a size grid blind, and prints a PASS/FAIL scorecard:

test	bar	our result
P1 — predictor ↔ feature-space predictor	ρ ≥ 0.90	0.981 (home), 0.969 (2nd scene)
P2 — the law: predictor ↔ learnability	ρ ≥ 0.85	0.988 (10-cell blind); 0.932 / 0.850 at 5 seeds
P3 — ignition present (learner escapes, floor pinned at log K)	present	CNN + ViT, both scenes

Compare against EXPECTED_RESULTS.md. Disagreement is a result — please open a discussion with your results_replicate.json and scene hash.

Read this before you run it (hard-won)

1. The learn/fail boundary is seed-bimodal and slow. A cell can look "collapsed" at 8k steps and escape at 18k. Escape times spread 6k–18k. Always use ≥ 2 seeds (ideally 3) and --full. Single-seed results near the boundary are noise.
2. Collapse = loss ≈ log(num_classes) exactly (uniform guessing). Use it to detect dead runs with zero eval cost.
3. Compare ordering across scenes, never absolute thresholds. Ordering is scene-universal (ρ = 0.996 across two scenes); the threshold is scene-local, so a cell that floors on our scene may learn on yours. That is expected.
4. Don't use the raw signal/clutter ratio to judge escape mid-training — it is confounded (it spikes when the encoder degenerates and clutter → 0). Track the signal component or the head's margin.

What's here

replicate.py              one-command harness (PASS/FAIL scorecard)
EXPECTED_RESULTS.md       our numbers + the caveats above
REPLICATION_KIT.md        long-form protocol (Windows OK, CPU OK)
DRAFT_paper_v3.md         the paper draft
LITERATURE_CHECK.md       what is already published (and what we therefore do NOT claim)
triclock/, simreal/       the ~128K-param model + the task generator (imported by replicate.py)
code/compute_R.py         the pixel-space predictor, standalone
code/compute_R_grid.py    predictor over a size × contrast grid
code/compute_featR.py     the same quantity through an untrained encoder (the mechanism)
code/compiler_v2.py       automated measure → route → escape-and-select
results/                  the JSONs behind every number in the paper

Scope — please read before generalizing

This works where the discriminative signal already lives in a fixed representation: our change-dating task (ρ = 0.988) and algorithmic phase-transition tasks. It provably fails for natural-image classification, where within-class variance is the learnable material — we tested pooled, high-pass, edge, random-conv, gated, and k-NN forms; all fail (best ρ ≈ +0.04 against a ground-truth ordering, with the plain predictor at ρ = −1.00).

Grokking results in the paper's §7 are reported as supporting evidence only; that ground is already covered by OmniGrok (ICLR 2023) and 2026 work on norm-separation delay laws and interventional/predictive grokking frameworks. We claim no novelty there. See LITERATURE_CHECK.md.

Limitations

One scene family, one architecture class, one training recipe, n = 10 blind conditions, and no external replication yet. That last one is the point of this release.

Citation

@misc{loglens2026,
  title  = {Predicting Task Learnability for Nano Vision Models --- and Why the Prediction Works},
  author = {Cox, Joe and {Claude (Anthropic)}},
  year   = {2026},
  note   = {Draft. Code and pre-registrations: this repository.}
}

Experiments were run on a single Jetson AGX Orin. Every prediction in the paper was registered to a timestamped file before the corresponding run.

---

⚠ This card predates our model-card standard

It does not report: a baseline comparison or a measured failure rate.

Every model we publish should state (1) the cheap baseline and the margin over it, (2) at least one named failure mode with a measured rate, (3) what the model refuses and why, and (4) label provenance. This card was written before that rule and has not been retrofitted, because the measurements needed no longer exist. Treat its numbers with more caution than our newer cards, and prefer a model that meets the standard where one exists.