sovythos

/

Sovythos-303M-Base

Like 0

Hi Mahmoud,

Pretraining a 303M Arabic and Egyptian dialect foundation model from scratch with native Llama GGUF compatibility is a very impressive effort. Clean execution on the BPE tokenizer for Semitic morphology.

Looking at your training hyperparameters and the limitations noted regarding context coherence:

1. RoPE theta 1,000,000 against 512 training block size:
   You configured RoPE theta to 1,000,000, while your training recipe uses an effective block size of 512 tokens.
   At head_dim 64, high-dimensional rotary frequencies have wavelengths spanning hundreds of thousands of tokens. Over a 512-token sequence, those dimensions barely rotate through a fraction of a phase cycle. When evaluated at 2,048 tokens, the network encounters unseen phase spaces, which directly explains the context coherence drop beyond a few hundred tokens.
   Dropping theta to 10,000 for 512-2,048 token pretraining, or expanding training chunk lengths past 2,048 before increasing theta, helps rotary frequencies complete meaningful cycles during early training.

2. Vocabulary parameter allocation:
   With a 32,000 vocabulary at 1,024 hidden size, an untied token embedding and output projection consume ~65.5M parameters (21.6% of your 303M budget).
   One transformer layer (1,024 hidden, SwiGLU, 4 KV heads) costs roughly 11.3M parameters. Static lookup tables cost nearly 6 full layers of active compute.
   Decoupling the input embedding via low-rank projection (32,000 -> 128 -> 1,024 = 4.23M params) frees over 28.5M parameters to add 2-3 additional physical layers to your 24-layer stack.

3. Morphological depth via block recycling:
   Semitic root-and-pattern morphology and multi-step code reasoning require significant non-linear composition.
   In an open architecture called Maba (101M reference model: https://huggingface.co/AndrewThompson1233/maba-v1-architecture), we use deterministic 2-pass block recycling:
   Routing hidden states through your 24 physical blocks twice with Split RMSNorm (distinct scale vectors for pass 0 and pass 1) expands depth to 48 effective layers at zero parameter overhead. This compositional headroom stabilizes long-range syntactic agreement in Arabic without increasing model size.

Did training memory margins enforce the 512 block size during this initial pretraining run?

Best,
Andrew