haster

/

Clef-0.2B

Like 1

Hi Haster,

Training a 0.2B symbolic music model from scratch in fp8 on a Blackwell RTX PRO 6000 with native bar-major REMI+ and unpitched GM drum resolution is a fantastic release. Interleaving tracks bar-by-bar to enforce co-termination is very clever engineering.

Looking at your evaluation table and the limitations noted regarding the 35-bar context ceiling:

1. Long-horizon thematic continuity versus CONTINUE window stitching:
   Because bar-major interleaving packs multiple instrument events into every bar, a 4,096-token window spans only roughly 35 bars of dense musical score. Stitching pieces via CONTINUE windows causes harmonic and stylistic drift at the seams, as previous motifs drop entirely out of attention.
   In an open architecture project called Maba (101M reference model: https://huggingface.co/AndrewThompson1233/maba-v1-architecture), we handle extended sequence memory using hybrid linear recurrence (75% GDN-2 / 25% GQA):
   GDN-2 maintains an associative recurrent state in fixed O(1) memory.
   In symbolic music, this allows the network to carry harmonic keys, motif seeds, and rhythmic themes across hundreds of bars without quadratic memory growth or window-stitching boundaries.

2. Polyphonic voice leading and dissonance reduction via block recycling:
   Your dissonance ratio landing at 0.29 (compared to 0.15 on human reference scores) and in-key ratio at 0.81 reflect the difficulty of multi-voice constraint satisfaction across 17 layers. Resolving vertical intervals simultaneously across lead, chords, and bass within each bar requires deep non-linear feature interaction.
   Using deterministic 2-pass block recycling:
   Passing representations through your 17 physical blocks twice with Split RMSNorm (distinct scale vectors for pass 0 and pass 1) expands depth to 34 effective layers at zero parameter overhead.
   Pass 0 establishes local rhythmic and track alignment, while pass 1 enforces strict harmonic intervals, directly reducing clashing notes and dissonance across simultaneous tracks.

3. REMI+ vocabulary parameter reallocation:
   With a 9,507 vocabulary at 1,024 hidden dimension, your embedding table takes 9.73M parameters.
   At ~11.3M parameters per transformer layer, decoupling the input vocabulary via low-rank factorization (9,507 -> 128 -> 1,024 = 2.44M params) reclaims ~7.3M weights, covering most of the budget needed for an 18th physical layer within your 201M envelope.

Did memory bandwidth or fp8 kernel scaling dictate stopping at 17 layers during the Blackwell run?

Best,
Andrew