Partial Success: Prototype 5.5
Full geodesic alignment possible at a cost of additional dimensions per geodesic complexity grade applied.
I'm thinking the losses have too many changes, and the system is drifting from the directed goals specified in the earlier versions. There many be too many differences between the losses here, the process here, and the system here - in conjunction with the geolip structural integrity. This process may be useful for head training, but it may not be useful for distillation training.
Mode Dim Train Val Gap
--------------------------------------------------
raw_768 1536 0.498 0.357 0.141
raw+diff 3072 0.567 0.475 0.092
bank_enriched 1792 0.766 0.532 0.235
bank+diff 3584 0.722 0.670 0.052
geo_explicit 6 0.326 0.363 -0.037
prototype 1;
full anchor bank, minimal anchors, positive outcome from mini distilled expert student
bert-base aligned.
Positive enough to continue refinement.
=================================================================
PHASE 3: GEOMETRIC VERIFICATION
=================================================================
Passthrough integrity: 1.000000 (should be ~1.000)
Geo context CV: 0.1651
Geo context eff_dim: 21.5
Geo context shape: torch.Size([2000, 64])
=================================================================
PHASE 4: CLASSIFIER STABILITY TEST
=================================================================
with_bank : train_acc=0.481 val_acc=0.390 gap=0.091
without_bank : train_acc=0.443 val_acc=0.330 gap=0.113
=================================================================
DONE
=================================================================
Student: mini_student.pt
Bank: alignment_bank.pt
Consensus CV: 0.1316
Student v_cos: 0.612
prototype 2;
full anchor bank self aligned, 512 anchors, reduced training overfit, increased validation on distilled expert student
bert-base aligned.
=================================================================
PHASE 3: GEOMETRIC VERIFICATION
=================================================================
Passthrough: 1.000000 (target: 1.000)
Emb CV: 0.1592 (consensus: 0.1316)
Geo context CV: 0.1790
Geo eff_dim: 33.7 / 128
Expert cos: 0.794 ± 0.052
Anchor max cos: 0.656
Cross-expert: 0.032
=================================================================
PHASE 4: CLASSIFIER STABILITY TEST
=================================================================
with_bank : train=0.782 val=0.505 gap=0.277
without_bank : train=0.512 val=0.372 gap=0.140
=================================================================
SUMMARY
=================================================================
Consensus CV: 0.1316
Consensus eff_dim:223.7
Student v_cos: 0.612
Student v_cv: 0.169
Bank params: 2,918,784
Bank geo_eff_dim: 33.7
Bank geo_cv: 0.1790
=================================================================
DONE
=================================================================
prototype 3;
full anchor bank, 512 anchors, full procrustes whitening, dynamic ortho, full geometric statistics, even more improved accuracy
dynamic alignment, additional utility potential present from dynamic capacity than directly aligning to an expert.
=================================================================
PHASE 3: GEOMETRIC VERIFICATION
=================================================================
Passthrough: 1.000000 (target: 1.000)
Emb CV: 0.1592 (consensus: 0.1316)
Geo context CV: 0.1817
Geo eff_dim: 33.5 / 128
Expert cos: 0.800 ± 0.004
Anchor max cos: 0.656
Cross-expert: -0.107
=================================================================
PHASE 4: CLASSIFIER STABILITY TEST
=================================================================
with_bank : train=0.773 val=0.513 gap=0.260
without_bank : train=0.512 val=0.372 gap=0.140
=================================================================
SUMMARY
=================================================================
Consensus CV: 0.1316
Consensus eff_dim:223.7
Student v_cos: 0.612
Student v_cv: 0.169
Bank params: 2,920,320
Bank geo_eff_dim: 33.5
Bank geo_cv: 0.1817
=================================================================
DONE
=================================================================
prototype 4;
full anchor bank, 512 anchors, full procrestes whitening, dynamic ortho, improved and robust relative geometric statistics, even more improved accuracy
with full dynamic alignment the system is now cross-expert positive with a dynamic series of ratios for loss and alignment
stability still present, additional complexity can be included.
=================================================================
PHASE 3: GEOMETRIC VERIFICATION
=================================================================
Passthrough: 1.000000 (target: 1.000)
Emb CV: 0.1557 (consensus: 0.1316)
Geo context CV: 0.1872
Geo eff_dim: 33.2 / 128
Expert cos: 0.784 ± 0.005
Anchor max cos: 0.655
Disagreement:
Cross-expert: 0.0463 ± 0.0154
Ratio: 0.001912 (target: 0.000000)
Norm spread: 0.1279
=================================================================
PHASE 4: CLASSIFIER STABILITY TEST
=================================================================
with_bank : train=0.775 val=0.508 gap=0.266
without_bank : train=0.513 val=0.392 gap=0.122
=================================================================
SUMMARY
=================================================================
Consensus CV: 0.1316
Consensus eff_dim:223.7
Student v_cos: 0.612
Student v_cv: 0.169
Bank params: 2,921,088
Bank geo_eff_dim: 33.2
Bank geo_cv: 0.1872
=================================================================
DONE
=================================================================
prototype 5
Attempted equidistance geodesic alignment protocol. Successful, perfect harmony to reach 0.2 context CV ratio possible.
=================================================================
PHASE 3: GEOMETRIC VERIFICATION
=================================================================
Passthrough: 1.000000 (target: 1.000)
Emb CV: 0.1660 (consensus: 0.1771)
Geo context CV: 0.2053
Geo eff_dim: 30.5 / 128
Expert cos: 0.729 ± 0.005
Anchor max cos: 0.668
Disagreement:
Cross-expert: 0.0826 ± 0.0021
Ratio: 0.006963 (target: 0.000000)
Norm spread: 0.0781
=================================================================
PHASE 4: CLASSIFIER STABILITY TEST
=================================================================
with_bank : train=0.746 val=0.500 gap=0.246
without_bank : train=0.490 val=0.363 gap=0.126
=================================================================
SUMMARY
=================================================================
Consensus CV: 0.1771
Consensus eff_dim:109.5
Student v_cos: 0.686
Student v_cv: 0.182
Bank params: 2,921,088
Bank geo_eff_dim: 30.5
Bank geo_cv: 0.2053
=================================================================
DONE
=================================================================
prototype 5.5 correct geodesic relational contextualization
=================================================================
PHASE 3: GEOMETRIC VERIFICATION
=================================================================
Passthrough: 1.000000 (target: 1.000)
Emb CV: 0.1635 (consensus: 0.1771)
Geo context CV: 0.1892
Geo eff_dim: 30.7 / 128
Expert cos: 0.725 ± 0.005
Anchor max cos: 0.668
Disagreement:
Cross-expert: 0.0776 ± 0.0023
Ratio: 0.008188 (target: 0.000000)
Norm spread: 0.0326
=================================================================
PHASE 4: CLASSIFIER STABILITY TEST
=================================================================
Mode Dim Train Val Gap
--------------------------------------------------
raw_768 1536 0.498 0.357 0.141
raw+diff 3072 0.567 0.475 0.092
bank_enriched 1792 0.766 0.532 0.235
bank+diff 3584 0.722 0.670 0.052
geo_explicit 6 0.326 0.363 -0.037
=================================================================
SUMMARY
=================================================================
Consensus CV: 0.1771
Consensus eff_dim:109.5
Student v_cos: 0.686
Student v_cv: 0.182
Bank params: 2,921,088
Bank geo_eff_dim: 30.7
Bank geo_cv: 0.1892
=================================================================
DONE
=================================================================
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support