AbstractPhila PRO

The current concept that I've dubbed multi-rotary pairings, extends to a solid context window space of around 14 million or so pairs so far. This is conjunctive using different sized pairings between 2 and 16, as per the aleph set mathematics capacity allows.

There is a bit of a rift that forms still at this scale, but the rotary itself can easily encompass framed sets of 14 million bytes worth of paired frames, treated as though they are individual lookups thanks to the prototype "AlephFrame" rotary system.

This doesn't merit a full conjunctive LEARNING capacity, but it does converge a structural guarantee capacity.

This frame is a little different than standard attention, but it'll snap right to models. So far the pairings are a little wild on some edge cases, but I have another upcoming test that I believe will resolve that hyperbolic rifting.

Currently we're at around a 0.00001% invalid addressing fault, which when scaled up is a bit substantial. However, with additional pairings the system cleans up the bpb and the output cleans.

Essentially this means entire context window can be a lookup table, so that's pretty quick at least. The models with their arms train at around 750k byte/s for now, and the scaled up MOE variant will train about as quick as you wish to train the experts for.

Essentially, what we're looking at here, is a cascade solidity system with independent experts that can be removed and replaced, simply existing within that space defined. Within that spatial geometry, it is completely cut off from any external stimuli, and that space is perfectly preserved from the external state of the model.

Meaning, you can simply snap something out and put a new interface in. Small interface, big interface, doesn't matter. Input, output.

We'll be targeting the space over 1 million context window, which should help solidify the system's internal shared structural space.

Very strong signals for full decoupled fixed position rotary. Huge amounts of experimentation done.

Lets figure out how 1 million context window works, shall we?

https://huggingface.co/AbstractPhil/geolip-rotary-experiments

I know this likely sounds odd or a bit too generic currently. The concept here is straightforward in pragma though. Fixed position, dynamic position, rotary position, global, local, or any other doesn't matter. The model simply needs that learned behavior and the understanding of how to seek the information, with a stable protocol system for acceptance, rejection, and so on. You don't need 600 layers in a model to debate the answer to this question internally, trained through 1000 step differentiated Adam, when the stable answer is something represented somehow in something.

In our case we take a piece of the context window from the overarching global system, say a dynamic allocation <FRAME>, and we snap a positional framed request to this. One of the collective with the necessary compositional addressing behavior intercepts this request, processes the information within it's bounds benefitting from the core model as necessary, sends the compositional result back.

One token done. Say we do this 500 times, each compositional structure intercepted by the necessary arm's behavior, providing that learning as a standard MOE would, except with a format of purpose.

Each of those compositional leaders have their own sub-collectives of solvers. Whatever they distilled or utilized or whatever the model is constructed to use, something like RNN, memory, or whatever. The next token or tokens are predicted and fed out utilizing the necessary structure within those dynamic framings, and once the capacity for that frame is exhausted the frame rises.

Simple in texture really. Something akin to a programming stack, except the addressing for the stack's behavior is a complex mathematic rotary combinatory behavior. We've been doing this for 80 years.

Unknowns are seen and yielded. Problems are spotted and yielded. Everything between is fielded, tested, and internally debated.

This when functional, will not need a byte model. This structure will behave effectively with tokenization models as well. The lookup system needs maturing but the system will comply.

This may seem like 5 steps back 1 step forward from just feeding a large model all the necessary information and getting a multi-debated answer over time. A seemingly difficult and complex idea to understand forming a lesser structure.

This isn't exactly the case though. In this model design's case, you gain the power of yielded multi-stage, multi-step, pre-planned arbitration, in a stable next-token prediction system.

Our hardware is TERRIBLE at this sort of thing, so I have a few pieces to make.

You don't WANT a model like tiny Beatrix to use a system like this, you need something with a bit of intelligence to arbitrate the subsystem. A Qwen with some oomph is a good option.

So... yeah the little models seem to learn most text related tasks on this scale. Individual structures with training runs seem to have very little problems learning them.

9 million params isn't very much, but it's more than enough for this task apparently. I'll be finetuning something much stronger with the dataset sonnet, opus, and fable engineered for the image thinking task. I'm thinking a powerful qwen or something like that. Something that knows images already.

That should be an appropriate target. Something with more brains than an atom.

Castor learned clean chain of thought.

Got a bit overzealous on my fable, opus, and sonnet usage - it's pretty much gone. I'll be setting up something with Opus but the usage is limited so nows a good a time as any to take a break until tomorrow.

Pollux should finish soon, then the dual thinking nanos should be functional.

Pollux finished.

Experimental thought dataset should be pretty beefy now. Not sure how well it'll work unless I train a vit with the information using cross-contrast, so I'll run some finetunes and benchmarks. The nano models don't understand thinking, so I'm not really sure what'll happen if I just raw feed it. I assume overfitting from just the data if I run multi-epoch due to the lack of CoT training, so the model's CoT will likely need to be conditioned ahead of time to be generic CoT as per my assumption. This isn't a guaranteed requirement, it may very well be the opposite - where CoT can be taught using symbolic objective representation rather than needing some sort of CoT dataset to baby-step into the system.

I'll run some experiments and tests, then figure out some good methodologies for the trie structure.

nano-pollux-dual-v1.2 is done, full frame train is in.

Model eats them up. Control is the nano-castor model having never seen frames.

The construction begins on a specific chain of thought dataset built around conditioning for image generation and VLM classification chain of thought. The dataset will be selectively designed for image curation, editing, associative reasoning, classification, and more.

Being text-specific the dataset will contain multiple direct conversions of known popular dataset caption sets into CoT formatting utilizing symbolic associative reasoning. If images do exist, the repo will be directly linked in the readmes, or the image pointed directly at a URL depending the ruling.

The entire purpose of this dataset is to provide a specific set of framed curation datasets, built specifically around populating framed portions of context window information. Think about it like curated thought pathology meant to encourage stable thought structure specific to images. In short, the dataset is being built around multi-parallel thinking, but will function as standard CoT when trained on other models.

There will be multiple prototype trains based on this topic with the entire goal meant to provide strong image control editing and capacity for editing using runtime conditioning, rather than requiring a multi-tuned retrain of models post finetune and post training.

The goal is to enable image editing guidance capacity to every applicable CoT model with the power of a single chain of thought dataset. Image training directly on diffusion models within specific parameters can enable this editing to be participated in with minimal supervision, even when models use entirely different embedding, encoding, lookup, or position systems.

There are no guarantees this will work, and the experimentation will be ongoing while curating and designing. The dataset will be simple at first, then expanding outward heavily as time progresses.

The research shows that stable and curated CoT can be stabilized and made reasonably learnable with simple mitigated finetuning, and the experiments based on this dataset will yield those results publicly.

The frame-centric finetune is in progress, and it broke the readme for the repo so I'll fix it momentarily.

As it stands the full frame training is holding pretty effectively.

The nano framed variants are heavily dependent on the banks, which is both good and bad. The lack of shared structure is a bit of an efficiency problem, as it takes longer to learn and solidify the depth content, but since the model is learning framed depth by structure it's holding either way. The training process is working so far.

Framed training going well, going to put in a selector system for it to test some CoT multi-chains and start up an embedding prototype. I'm thinking multi-thinking is a viable option for some tinkering.

Too much data. Needed to purge old trains, loras, and unnecessary checkpoints on the SD15, SDXL, and multiple other diffusion trains.

Mostly purged optimizers but some checkpoints went as well, mostly useless checkpoints.

Experiments are humming along. Ready by this evening is likely not going to be the case, however I believe I have a few solutions for the aleph system.

Historic experimentation had a few leads, mostly dead ends. A couple of options showed promise; such as treating the signal as amplitude, while other options such as the heavily vetted Cantor positioning system are sidelined for stability in this version.

The result is likely going to treat that particular Aleph signal as confidence, as removing it from the model completely caused high confidence with incorrect answers while present the confidence would dip. It very would could be correlation without causation, but the tests are worth running. I will continue to investigation to get the correct answer for the causal behavior exhibiting the result, but it's definitely been difficult to tack down.

In any case, the last block large direction problem has been solved.

The prototypes will be trained in AdamW and Muon. AdamW shows stronger frame curation and control than Adam, and the large final direction being given AdamW with weight decay. The full prototypes will likely be trained with weight decay, as the prototypes have shown strong results with it.

As long as the final block is given Adam in the right place, the model converges, and the Aleph head quiets. To cultivate this head, the aleph system will be given the job of confidence. It's not as powerful as the alternative in theory, but the pragma shows the likelihood of it working to be considerably higher and more useful.

I'll need some days to align the next nonlinear prototype head, hopefully by the end of next week. In any case, the model itself still uses aleph internally, but the global aleph system will be quiet on this version. The next byte framing and context window will be more dominant, along with the banks. This is fine as many of these prototypes can be trained.

Stability is key, as long as the model is stable the models will work. If they aren't stable or collapse then they won't be of any use for data, other than trying to prevent or balance around it.

Legitimate strengths the other does not possess. Pragma and proof with cooperation.

Unironically, the answer for the splat problem was... a form of mathematics derived from art. The Canon procedure, which a specific paper's work adapted and fitted to Pollux and the splat layer in a meaningful manner. This brought the frame system from nearly nothing on recall, to a direct comparator to a rotary softmax attention. This when applied with rotary positions allowed the splat layers to behave and frame more effectively than the tested softmax rotary scales tested on many fronts. The cost of train went up a little, but the prototype does have meanginful frame learning and context awareness.

I've predominantly considered the splat component to be an artist component in many ways. It's reasonable to find many answers to the artist's quirks for organization, in the theory of music itself. It'll be in the writeup.

https://arxiv.org/abs/2512.17351

Okay so it may or may not work the same way as intended right now, however I'm not willing to abandon the semi-functional systems in favor of a formula switch that requires multiple new components. I'm not certain, and I will need to be certain.

I'll need to sleep on it. The model is far from dead, just needs a proper optimizer to cultivate the rifts. Something that handles the correct intention and the correct implicit skips as per the necessary geometric framing tested. There will need to be a tweak or two to the formula most likely, hopefully topical tweaks only. I don't want to be at the blackboard for another month only to find a model that runs like a snail on a salty day.

If the MUON system simply wasn't cultivating the results, I will figure it out tomorrow when i wake up and then figure out the correct answer.

We have our core optimization fix. Now I need to enable the correct Cantor Aleph-Void behavior to shortcut with the aleph addressing with something similar to the applied Fractal Wormhole positions as experimented with in the past, aka matching numbers down the chain that usually lead to the classic Cantor Dust problems. Number systems that have given mathematicians headaches for many years. https://en.wikipedia.org/wiki/Cantor_set

After that the models need to learn to depend on them, which should fit nicely into the context window framing. I believe this should be stable enough under fp32 to round since I'm not using the Cantor structure for positioning directly but instead the Aleph mathematics for the actual prediction system. This should be stable enough, but if not I'll work something else out. As long as we don't hit something as complex as the Mandelbrot it shouldn't collapse or explode. FP64 isn't on the menu thank you.

A fully represented Cantor Dust byte prediction system would take a considerable amount longer per token. This is a cultivated behavioral response system meant to replicate the activity through linear mathematics, not a direct 1:1 clone.

Every fractal slice will cost huge amounts of compute, which is the entire point of avoiding using them directly and instead cultivating them as a cooperative element. Without the MUON being fully dominant they are not cultivating properly on their own, as the steps aren't allowed to form those extra potentially incorrect distances that allow nonlinear shortcuts to form. Adam and AdamW turn those into smooth curves, when we want some gaps and rifts in order to learn the nonlinear math in a way that doesn't form a billion unmeasurable infinities, or entirely matching cosine final layers.

The reason why there's a component named governor is to limit these leaps and bounds. They will eventually cause much worse problems if they are left completely unchained. So a couple hundred million per 60 billion are casualties, give or take, for those who reach too far or are pinched too tightly for rounding to correct, and are rewarded for the action.

Smooth TOO MUCH and the head never activates. Smooth TOO LITTLE and you get the big clump. Likely will need an optimizer for the behavior.

Both good news, and bad news.

The good news, I solved the final direction problem.

canary_acc 0.000 · doc n=70 bpb=4.22 reset=2.47 · bits by position (row) 0-64 1.84 64-256 1.67 256-1024 1.58 1024-2048 1.52 2048-4096 1.53
[lastfix] c-head-adam step 1,500: val 1.5488 bits a byte | share along the last block's direction 0.324 (after the norm 0.295) | last bank write 0.249 on its direction, size 328 | last block rank 35 | output layer size 12.6, content loudness 1.25


canary_acc 0.000 · doc n=70 bpb=4.17 reset=2.37 · bits by position (row) 0-64 1.77 64-256 1.60 256-1024 1.50 1024-2048 1.44 2048-4096 1.45
[lastfix] c-head-adam step 2,000: val 1.4719 bits a byte | share along the last block's direction 0.313 (after the norm 0.285) | last bank write 0.229 on its direction, size 455 | last block rank 39 | output layer size 13.1, content loudness 1.31

Probably.

THE BAD NEWS, I'll need to run all of yesterdays tests again with the new optimizer. It invalidates many of the framing experiment guidelines without running the full test suite. The models simply cannot TRAIN FOR WEEKS without running full barrage testing on the structure, in a day that could save a month of training.

It's both irresponsible and painful on the wallet if I don't run the correct tests for speed, structural architecture, and accuracy before I run.

All the fixes work. Doesn't even need weight decay, just Adam. Eliminate Muon from the final layer's head, that's all it really needs to have. The most dominant pieces the model would have directional bias formed from, solved.

Castor's early setup is clean. Very clean. I expect a strong feature by the end of training, rather than a single dominant fractal direction that would take me weeks to investigate.

As before the aleph head is getting buried. The model isn't going to be incentivized to learn nonlinear structure without a fuss. Muon made this more than viable but it formed the cascade knot that would crush the final layer. There's likely a cascade middle-ground between behavioral removal and cascade articulation. Something that will form reasonable amounts of utility without forcing the model to learn something that will simply be optimized away.

They must LEARN to cooperate without smothering each other. MUON makes this viable, but there are cascade failure points from this that cannot go unaddressed.