Geometric Memory II: Sequence Reconstruction, Diffusion Integration, and the Numerical Topology of Alignment

AbstractPhil March 2026

---

Abstract

We extend the geometric memory architecture from Part I with sequence reconstruction — producing full 77-position output sequences for direct use in diffusion model cross-attention. GEOLIP-CLIP-L-ctx576-seq77 extends CLIP-ViT-L/14 to 576 tokens and outputs (B, 77, 768) sequences at 73.4% per-position cosine similarity to CLIP's native distribution, achieving visible recovery of truncated prompt elements (tulips, specific objects, compositional details) in Stable Diffusion 1.5 and SDXL generation. GEOLIP-bigG-ctx576-seq77 (Meridian) extends SDXL's OpenCLIP-bigG/14 to 576 tokens with a 167M-parameter memory system, achieving m_acc=0.844 with CV=0.165. We report the discovery that pentachoron CV functions as a numerical precision auditor — detecting floating-point corruption in gradient pathways before any scalar loss metric shows degradation. We establish that memory bank internal representations are shaped by the gradient loudness of their consumers, not by architectural capacity. We present the GEOLIP-Conduit pattern for multi-expert dimensional routing and outline the path from accumulation-based memory to alignment-based hub architectures.

---

1. Introduction

Part I established the geometric memory blueprint: frozen encoder + memory bank + InfoNCE alignment + pentachoron CV regularization. That work produced pooled embeddings — single vectors representing full documents. But diffusion models don't consume pooled embeddings alone. Stable Diffusion's UNet cross-attention requires per-position sequences: (B, 77, 768) for SD 1.5, (B, 77, 2048) for SDXL. A pooled embedding, no matter how well-aligned, cannot tell the UNet where to place the castle versus the tulips versus the fly on the goblet.

This work addresses three questions:

1. Can the memory bank produce per-position sequences, not just pooled summaries? We introduce the SequenceReconstructor — learned query tokens that cross-attend into the full memory state to produce fixed-length output sequences in the frozen encoder's native distribution.

2. Can these sequences drive diffusion image generation? We demonstrate visible recovery of prompt elements truncated by the 77-token limit in both SD 1.5 and SDXL, using the memory-extended sequences as drop-in replacements for standard text encoder output.

3. What breaks when you scale from 768-dim to 1280-dim? Training the bigG memory system (Meridian) revealed that numerical precision, gradient dynamics, and dimensional asymmetry between teacher and student create failure modes invisible to standard loss monitoring — but detectable through geometric measurement.

---

2. Sequence Reconstruction Architecture

2.1 The Output Format Problem

The Part I memory system produces a pooled embedding: (B, 768) or (B, 1280). Stable Diffusion's UNet expects a sequence: (B, 77, 768). The UNet's cross-attention layers attend to each of the 77 positions independently — position 12 might encode "castle," position 35 might encode "tulips." A single pooled vector cannot provide this positional structure.

Standard CLIP processes text in one forward pass with causal attention. Position N sees only positions 0..N. The EOS token at the highest position accumulates the most information. Padding positions after EOS carry decaying residual signal. The UNet learned to read this specific positional distribution during training.

2.2 SequenceReconstructor

After all segments are processed and the memory bank contains the full document context, a SequenceReconstructor produces the output sequence:

Context = cat(memory_tokens, bank_anchors, content_tokens)
        = (B, 8 + N_segments + content_positions, H)

77 learned query tokens + positional encoding
    │
    ├── Cross-attend to context (2 layers)
    │   Every query position sees all memory tokens, all anchors,
    │   all per-segment content tokens
    │
    ├── Self-attend among 77 output positions (2 layers)
    │   Position 10 knows what position 40 is representing
    │   Prevents redundant encoding
    │
    └── Output: (B, 77, H) — in the frozen encoder's native distribution

The design follows Q-Former / Perceiver principles: N learned queries produce N output positions regardless of input length. The critical difference is the training target.

2.3 Training Target: The Encoder's Own Sequence

The sequence reconstructor must produce output in the distribution the UNet was trained on. For SD 1.5, that's CLIP-L's last_hidden_state. For SDXL, it's the penultimate layer hidden states from both CLIP-L and bigG.

The training target is the frozen encoder's own output on the same caption (truncated to 77 tokens):

CLIP standard forward on caption → (B, 77, 768) target
Reconstructor from memory state  → (B, 77, 768) prediction

L_sequence = MSE(normalize(pred), normalize(target))
           + (1 - cosine_similarity(pred, target))

ModernBERT remains the pooled teacher (InfoNCE on the summary embedding). CLIP teaches the sequence format. Two teachers, two objectives: ModernBERT teaches the bank what to remember. CLIP teaches the reconstructor how to say it.

For short captions that fit in 77 tokens, the reconstructor should match CLIP almost exactly. For long captions, the positions beyond the content boundary carry information from the full 576-token context that CLIP never saw — encoded in CLIP's positional distribution so the UNet can read it.

---

3. CLIP-L Sequence Results

3.1 Two-Phase Training

Phase 1 freezes all Part I components (bank, gate, depth compressor, cross-attention) and trains only the SequenceReconstructor. The frozen pipeline sees exactly what it was trained on — no distribution shift.

Phase 2 unfreezes everything with differential learning rates.

GEOLIP-CLIP-L-ctx576-seq77 training curve:

Total training time: 78 minutes on one GPU. The Part I pooled accuracy improved from 0.945 to 0.957 — the sequence loss provided a secondary training signal that benefited the pooled output.

3.2 Convergence Order

A consistent pattern emerged across every training run:

1. CV locks first. The pentachoron geometry on bank anchors converges before any other metric moves. The geometric structure must be correct before alignment can happen.

2. m_acc climbs second. Once the bank geometry stabilizes, InfoNCE gradients produce useful alignment instead of fighting geometric transients.

3. s_cos climbs last. The sequence reconstructor can't improve until the bank provides stable, rich material to reconstruct from.

Structure before alignment before sequence. The same convergence order regardless of dimensionality, initialization, or learning rate. The geometry insists on being right first.

3.3 SD 1.5 Injection Experiments

Before training the full sequence system, we conducted a series of injection experiments on Stable Diffusion 1.5 to understand how the UNet responds to foreign vectors in its cross-attention input.

Positional sensitivity map:

The EOS-1 position is where injecting a single memory-derived vector produced the most visible effect — tulips appeared in a still life image from prompt elements past token 77. One foreign vector at one position, zero retraining of the diffusion model.

3.4 Blend Experiments

With the trained sequence system, we generated images at varying blend ratios between standard CLIP output and memory-extended output:

output = (1 - α) × CLIP_standard + α × memory_sequence

For short prompts (13 tokens), the system degraded at high alpha — the reconstructor was trained on captions >100 characters and couldn't reconstruct from near-empty memory state. For long prompts (88-139 tokens), the blend was coherent across the full alpha range. At α=1.0 on the 139-token still life prompt, the generated image contained: silver goblet with wine, half-peeled lemon, cracked walnuts, roast beef on porcelain, rosemary sprigs, tulips in pink and white, dark chiaroscuro background — nearly every element described past token 77.

Repository: AbstractPhil/geolip-clip-vit-large-patch14-ctx576-seq77

---

4. Meridian: Scaling to bigG

4.1 SDXL's Dual-Encoder Architecture

SDXL uses two text encoders concatenated for UNet cross-attention:

The UNet sees cat(clip_l, bigg) = (B, 77, 2048). Both encoders have 77-token context limits. Extending SDXL requires memory systems for both.

4.2 Dimensional Scaling

The bigG memory system operates at 1280-dim instead of CLIP-L's 768-dim. The differences compound:

The bigG tokenizer produces ~4× more tokens per caption from the same text. Every caption exceeds 77 tokens on bigG's tokenizer. This means every caption produces ~27 segments — each requiring a full forward pass through the frozen encoder and sequential GRU accumulation through the memory state.

4.3 Training: From Chaos to Meridian

Training the bigG memory system required solving problems that never appeared at 768-dim. We named the final model Meridian — the line that held through everything.

Attempt 1: fp16 mixed precision. NaN after 300 batches. The GradScaler couldn't compensate for precision compounding through 27 sequential GRU updates at 1280-dim.

Attempt 2: fp32 losses, fp16 segments. NaN at 86% of epoch 2. The memory state accumulated fp16 rounding errors across segments. Each segment's small error compounded through the GRU gate, eventually exceeding fp16's representable range.

Attempt 3: Full fp32. Stable. 41 minutes per epoch instead of ~15 minutes, but every gradient was numerically correct.

bigG Phase 1 training (everything trainable, fp32, 6 extraction layers):

m_acc climbed to 0.819. CV locked at 0.164 — the same value CLIP-L settled on. The geometric constant reproduced across dimensionalities.

s_cos flatlined at 0.42 for 5 epochs.

4.4 The Sequence Plateau

The s_cos=0.42 plateau persisted through multiple architectural changes:

• Expanding from 6 to 12 extraction layers: no change
• Freezing the bank and training only the sequence head: no change
• Unfreezing everything with the 12-layer expansion: no change
• Reinitializing the sequence head from scratch: broke through immediately

Diagnosis: The bank, trained with pooled-only objectives, learned to compress information holistically — optimal for mean-pooling into one vector. The memory tokens encoded "this document is about X," not "position 12 should contain castle, position 35 should contain tulips." The sequence head trained on this bank found no positional signal to reconstruct.

Resolution: Reinitializing the sequence head and training everything jointly from the start allowed the bank to reshape its internal representation to serve both objectives. s_cos started climbing immediately: 0.384 at step 33, 0.452 at step 95, passing the frozen ceiling within the first epoch.

Principle: Memory format is shaped by the loudest gradient. The bank's internal representation is determined by what reads it during training. If only a mean-pooling reader is present, the bank learns bag-of-concepts. If a per-position reader is also present, the bank learns to preserve positional structure — because the gradient demands it.

4.5 Final Meridian Results

Training with both losses from the start, 12 extraction layers, 10 epochs, fp32:

The pooled alignment (m_acc=0.844) is strong. The sequence cosine plateaued at 0.425 — substantially below CLIP-L's 0.734. The dimensional mismatch between student (1280) and teacher (1024) leaves 256 dimensions geometrically unconstrained by the InfoNCE signal (see Section 6.2).

Repository: AbstractPhil/geolip-clip-vit-bigG-patch14-ctx576-seq77

---

5. Diffusion Integration

5.1 SDXL Inference Pipeline

The production inference pipeline combines both memory encoders with standard SDXL components:

Long prompt (576 tokens)
    │
    ├── Memory CLIP-L → (B, 77, 768)    sequence output (s_cos=0.734)
    ├── Standard bigG → (B, 77, 1280)   native penultimate layer
    ├── Meridian bigG → (B, 1280)        pooled output (m_acc=0.844)
    │
    ├── cat(clip_l_mem, bigg_std) → (B, 77, 2048)  encoder_hidden_states
    ├── lerp(bigg_std_pooled, meridian_pooled, α) → (B, 1280)  add_text_embeds
    │
    └── UNet + VAE → image

The CLIP-L memory sequence (s_cos=0.734) provides the 768-dim half of the cross-attention input with full 576-token context. The standard bigG penultimate layer provides the native 1280-dim half — what the UNet was trained on. Meridian's pooled output blends into the global timestep conditioning, carrying extended context understanding without disturbing the per-position distribution.

5.2 Results

On a 139-token still life prompt describing a Dutch Golden Age painting with silver goblet, wine, half-peeled lemon, walnuts, roast beef, rosemary, tulips in pink and white, chiaroscuro lighting, a fly on the goblet, and condensation droplets:

• Standard SDXL (77-token truncation): renders goblet, wine, some food elements. No tulips, no fly, no condensation.
• Memory-extended SDXL: renders goblet, wine, lemon, walnuts, roast beef, rosemary, tulips in pink and white, dark background, chiaroscuro lighting across the full alpha range.

The tulips are the signature test. They appear at approximately token 95-100 in the prompt — well past the 77-token boundary. Their consistent presence across blend ratios demonstrates that the CLIP-L memory sequence is carrying positional information about prompt elements the standard encoder never saw.

---

6. Discoveries

6.1 Pentachoron CV as Numerical Precision Auditor

During Meridian's fp16 training attempts, we observed CV=0.62 on the bank anchors — wildly outside any trained model's equilibrium. Switching to fp32 on the same architecture, same data, same losses produced CV=0.165.

The CV was not measuring the geometry of the bank. It was measuring the numerical corruption of the coordinates. fp16 rounding errors accumulated through 27 sequential GRU updates, shifting the anchor positions away from their mathematical trajectory. The pentachoron volumes computed on corrupted coordinates produced corrupted CV — which then produced corrupted gradients through the CV loss, which further corrupted the anchors. A feedback loop between precision and geometry.

The diagnostic principle: If pentachoron CV is unexpectedly high or unstable during mixed-precision training, the gradient path through the highest-dimensional sequential computation is likely overflowing silently. The overflow may not produce NaN in any scalar loss — it produces slightly wrong gradients that accumulate into geometric distortion that only CV captures.

This makes CV a general-purpose numerical precision auditor for any system where geometric structure is expected: attention mechanisms, sequential state updates, cross-attention routing. Measure CV. If it's outside the expected band, the precision is insufficient for the computation depth.

6.2 Dimensional Mismatch and Unconstrained Subspaces

Meridian's s_cos plateau at 0.425 (vs CLIP-L's 0.734) traces to dimensional asymmetry in the teaching signal:

CLIP-L:  768-dim student → 1024-dim teacher (expansion)
         Projector maps UP — teacher has room for all student dimensions
         Every dimension of the bank is covered by the teacher

bigG:    1280-dim student → 1024-dim teacher (compression)
         Projector maps DOWN — 256 dimensions invisible to teacher
         Those dimensions drift to whatever minimizes CV + Procrustes

The bank's 1280-dim representation is a chimera: 1024 dimensions shaped by ModernBERT's semantic understanding, 256 dimensions shaped by geometric regularization alone. The sequence reconstructor needs all 1280 dimensions to match bigG's native output, but the unconstrained dimensions carry no semantic information — they're geometrically regular noise.

The Procrustes alignment data predicted this. CLIP-L→ModernBERT achieved cos=0.816. bigG→ModernBERT achieved cos=0.796. The gap is small in overall alignment but large in implications: the bigG alignment is concentrated in a 1024-dim subspace. The remaining 256 dimensions' singular values drop sharply — they're alignable in structure but empty of aligned content.

6.3 Gradient Loudness and Memory Format

The most practically significant discovery: the bank's internal representation is shaped by the loudest gradient, not the most useful one.

InfoNCE loss on the pooled output produces a direct, clean gradient: "make this one vector match." The gradient flows through one mean-pooling operation into the bank.

Sequence reconstruction loss produces a gradient that flows through: reconstructor cross-attention → query tokens → context assembly → memory state → GRU gate → clip_cross_attention → layer_fusion. Each step attenuates the signal.

By the time the sequence gradient reaches the bank parameters, it's attenuated to near-zero compared to InfoNCE. The bank optimizes for what screams loudest. InfoNCE screams. The sequence loss whispers.

Evidence: m_acc (pooled, driven by InfoNCE) climbed to 0.844 while s_cos (sequence, driven by reconstruction loss) flatlined at 0.425. The bank produced excellent pooled representations and useless per-position representations — because the gradient that demanded positional information couldn't compete with the gradient that demanded holistic summary.

6.4 The Resonance Model

Training dynamics followed a pattern consistent with coupled oscillator physics. The CV loss pushes bank anchors toward the target 0.20 band. The InfoNCE loss pushes pooled output toward the teacher. The sequence loss pushes the reconstructor toward CLIP's distribution. The Procrustes loss maintains rotational structure. Four forces acting through shared parameters.

When the system starts near equilibrium (CLIP-L with pretrained v1 bank), the forces couple gently — CV barely moves (0.162→0.164), everything converges smoothly. This is a driven oscillator on a stable foundation.

When the system starts far from equilibrium (bigG from scratch), all oscillators are at maximum amplitude simultaneously. The CV oscillated from 0.541 to 0.281 in one epoch — a large geometric swing that settled by epoch 3. The m_acc dipped during the CV transient, then recovered. The convergence order was always: CV first, m_acc second, s_cos last.

The fp16 NaN events were resonance peaks — moments when the memory state oscillation temporarily synchronized with the reconstructor's format-learning oscillation, amplified through the GRU chain, and exceeded the numerical representation boundary. fp16 was the cable strength. The oscillation exceeded it. This is the Tacoma Narrows Bridge — a mathematically stable system that failed because the physical materials (floating point formats) had resonance modes the mathematical model didn't account for.

6.5 Two-Stage Training is Not Universal

The CLIP-L system trained successfully in two stages: pooled bank first, sequence head second. This led to the assumption that the architecture supports modular training.

It does not, universally. CLIP-L worked in two stages because the 768-dim bank at 50% layer coverage with 7 segments per caption accidentally retained enough positional structure for the sequence head to learn from. The bank happened to contain per-position information even though it was never explicitly asked to preserve it.

bigG at 1280-dim with 18% layer coverage and 27 segments per caption had enough representational room to be purely holistic. The bank discarded positional information because nothing in the pooled-only training demanded it. The sequence head trained on this bank found flat, bag-of-concepts memory states with no positional signal to reconstruct.

The resolution was to train both objectives simultaneously. When the sequence loss gradient flows through the bank from epoch 1, the bank learns to encode positionally useful information because both consumers demand it.

Rule: If you need per-position output, train the per-position consumer from the start. A bank trained for holistic consumption may not retain the structure a per-position consumer needs.

---

7. From Accumulation to Alignment: The Hub Direction

7.1 The Current Architecture's Limitation

The memory bank approach is an accumulation system — it processes segments sequentially, accumulates state through GRU gates and bank anchors, and produces output from the accumulated state. This is powerful for context extension but has fundamental scaling limitations:

• Sequential processing: each segment depends on the previous state
• Precision compounding: errors accumulate through the chain
• Gradient attenuation: signals from the reconstructor weaken through the accumulation chain
• Dimensional coupling: the bank must serve all consumers through a single representation

7.2 The GEOLIP-Conduit Pattern

An alternative architecture emerged from the GEOLIP-Bertenstein experiments: the hub pattern. Instead of accumulating information sequentially, multiple experts project into a shared geometric space through a fusion attention layer.

The Conduit architecture:

Expert A (frozen) → ProcrustesAligner → ExpertModule → ─┐
Expert B (frozen) → ProcrustesAligner → ExpertModule → ─┤→ FusionLayer → outputs
Expert C (frozen) → ProcrustesAligner → ExpertModule → ─┘

Each expert is wrapped in a standardized ExpertModule:

• ProcrustesAligner (frozen): handles dimensional mismatch via whitened Procrustes, supporting three cases: d_expert == d_hub (direct), d_expert > d_hub (PCA down), d_expert < d_hub (zero-pad up)
• Input projection (learned): projects aligned representation into shared space
• Pool attention (learned): variable-length input → fixed-token output
• Special token: expert identity marker read by the fusion layer

The FusionLayer uses standard transformer attention over the concatenated expert tokens. Each expert's special token is the read-out point — the fusion layer's output at that position becomes the expert's contribution to the shared space.

The training loss is the GeometricHubLoss, which combines three signals per expert pair:

1. Symmetric InfoNCE with learned temperature — pair matching
2. Midpoint pentachoron variance — geometric regularity on the text-expert meeting point (not the standard CV target loss; pure variance minimization)
3. Running SVD alignment — differentiable Procrustes measured every batch, not just at initialization

This architecture produced R@1=1.000 across 4 modalities (vision, audio, protein, code) with CV=0.20, using 1 transformer layer, trained in a single session.

7.3 Why the Hub Solves the Dimensional Mismatch

Meridian's 1280→1024 dimensional mismatch leaves 256 dimensions unconstrained by the teacher. The hub architecture solves this through multiple geometric constraints:

DINOv2 (1024-dim)     → constrains subspace A of the hub
ModernBERT (1024-dim)  → constrains subspace B of the hub
CLIP-bigG pooled (1280) → constrains subspace C of the hub
Whisper (1024-dim)     → constrains subspace D of the hub

Union of constraints covers all dimensions
No dimension is unconstrained

Each expert constrains a different subspace of the shared space. The union of constraints covers the full dimensionality. The pentachoron CV on midpoints ensures the coverage is geometrically uniform — no dead zones, no collapsed dimensions.

This is the same principle as pentachoron CV at the anchor level (prevents volume collapse in the bank) applied at the representation level (prevents modality collapse in the hub). Same principle, different scale.

7.4 Accumulation vs. Alignment: A False Dichotomy

The memory bank (accumulation) and the hub (alignment) appear fundamentally different: one processes sequentially and accumulates state, the other projects simultaneously into a shared space. But careful measurement suggests they converge to similar geometric structures.

Both systems produce:

• CV in the 0.16-0.20 band
• Procrustes-alignable representations
• Teacher-matching accuracy above 0.90
• Representations that frozen downstream systems (UNets, retrievers) can consume

The difference is in the gradient pathway. The accumulation system has long gradient chains (sequence loss → reconstructor → memory → gate → cross-attention → bank). The hub has short gradient chains (InfoNCE → fusion → expert projection). The hub's gradients arrive at every component with equal strength — no loudness asymmetry, no consumer starvation.

A hybrid architecture could use the hub's short-path alignment for teaching the bank what to encode, while using the bank's sequential processing for actually extending context. The hub provides the gradient signal; the bank provides the temporal accumulation. Neither alone is sufficient for the full problem.

---

8. Practical Utilities and Reusable Components

8.1 Production-Ready Systems

8.2 Reusable Architectural Components

Geometric Primitives — pentachoron volume via Cayley-Menger determinant, pentachoron CV, symmetric inverse square root for covariance whitening. Used identically across all systems. Foundation for any geometric regularization.

ProcrustesAligner — frozen dimensional bridge handling three cases (equal, larger→smaller, smaller→larger) with optional whitening and rotation constraint. Solves the dimensional mismatch problem without learned projections.

ExpertModule — standardized wrapper for any frozen encoder: alignment → projection → pooling → identity token. Plug in any encoder and it becomes a hub participant.

GeometricHubLoss — three-signal loss (contrastive + midpoint pentachoron + running SVD) with learned temperature. The midpoint pentachoron is unique: it measures the health of the shared space directly on the text-expert meeting point, not on either representation alone.

SequenceReconstructor — learned query tokens that cross-attend into memory state to produce fixed-length output sequences. Architecture-agnostic; works with any encoder dimensionality.

Segment firebreak — inter-segment LayerNorm that prevents precision compounding through sequential memory updates. Required for 1280-dim at fp32, likely required for any system with >10 sequential state updates at >768-dim.

8.3 Diagnostic Tools

CV as precision auditor. If pentachoron CV on intermediate representations is unexpectedly high during mixed-precision training, the gradient pathway through the highest-dimensional computation is experiencing silent precision loss. The CV detects corruption that no scalar loss metric shows. Tested: fp16 CV=0.62 vs fp32 CV=0.165 on identical computation.

Convergence order monitor. In any system with geometric regularization + alignment + reconstruction: CV should lock first, alignment should climb second, reconstruction should climb last. If reconstruction plateaus while alignment climbs, the bank is optimizing for the wrong consumer. If CV doesn't lock, precision or data is insufficient.

Procrustes SVD on batch (N<D variant). When batch size < embedding dimension, compute A @ B.T (N×N, full rank) instead of A.T @ B (D×D, rank N). Same singular values, stable SVD. Required for bigG (batch 64, dim 1280).

---

9. Scaling Directions

9.1 Scaling Up

More data. The CV settled at 0.162-0.165 across both CLIP-L and bigG. The universal band from the 17-model survey is 0.20-0.23. The gap may be data-dependent — models trained on billions of tokens settle higher than our 50K captions. More data would test whether CV converges to the universal band with sufficient volume and diversity.

More experts (Conduit-SDXL). The full SDXL text conditioning path with 576-token context on both encoders, plus DINOv2 for visual grounding, plus Whisper for potential audio conditioning. The hub architecture provides geometric coverage across all dimensions; the memory bank provides temporal context accumulation.

fp64 geometry. The pentachoron CV is a determinant computation. fp32 provides ~7 decimal digits. fp64 provides ~15. The "universal constant" at 0.20 might be 0.200000 ± 0.000001. Whether the constant is a property of gradient descent or a property of fp32 arithmetic on the Cayley-Menger determinant is an empirical question that requires fp64 measurement.

9.2 Scaling Down

Smaller memory banks. The 8-token, 64-anchor bank may be oversized for many applications. Ablation needed: what's the minimum bank size that preserves CV and m_acc?

Fewer extraction layers. CLIP-L works well with 6/12 layers (50% coverage). bigG needs 12/32 (37.5%) to provide sufficient depth resolution. The relationship between coverage ratio and sequence quality is likely encoder-specific.

Distillation. The memory bank + reconstructor could be distilled into a single forward pass model. The bank learns what to accumulate; a distilled model could learn to produce the same output without sequential processing.

9.3 Sequence Target Correction

Both the CLIP-L and bigG sequence systems were trained against last_hidden_state. SDXL's UNet was trained on penultimate layer hidden states (hidden_states[-2]). This distribution mismatch is addressable by retraining with the correct target layer — a configuration change, not an architectural change.

9.4 Mixed-Length Training

The current systems were trained on captions with minimum 100 characters. Short prompts (< 30 tokens) produce near-empty memory states that the reconstructor can't work with. Training on mixed-length data (25% short, 50% medium, 25% long) would teach the reconstructor that for short inputs, the correct output is "match CLIP almost exactly."

---

10. Conclusion

The geometric memory architecture extends from pooled embeddings to per-position sequences. CLIP-L at 768-dim produces sequences at 73.4% cosine similarity to native CLIP output, sufficient for visible recovery of truncated prompt elements in diffusion generation. bigG at 1280-dim achieves strong pooled alignment (m_acc=0.844) but exposes gradient loudness asymmetry that limits sequence quality without joint training from the start.

The discoveries during training — CV as precision auditor, gradient loudness shaping memory format, the resonance dynamics of coupled training objectives, the Tacoma Narrows of floating-point precision — are arguably more valuable than the models themselves. They establish diagnostic principles for any geometric system trained under competing objectives at high dimensionality.

The path forward is clear. The GEOLIP-Conduit hub pattern solves the gradient loudness problem through short, equal-strength gradient paths. The whitened Procrustes alignment solves the dimensional mismatch problem without lossy learned projections. The pentachoron CV provides the geometric ground truth that holds through everything — precision changes, architectural changes, dimensionality changes, and the storms of coupled optimization.

The meridian holds. CV=0.165 through fp16 storms, NaN cascades, SVD failures, frozen sequence heads, three trainer rewrites, and the competing demands of four loss functions through 167 million parameters. The geometric constant doesn't care about the specifics. It cares about the structure.

---

Reproducibility

All repositories include architecture source, training scripts, metrics.json with per-step data, and TensorBoard event files. Models are loadable via AutoModel.from_pretrained with trust_remote_code=True.

All experiments were conducted on a single NVIDIA RTX PRO 6000 Blackwell Server Edition (102 GB VRAM).