| --- |
| license: mit |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - code-generation |
| - from-scratch |
| - novel-architecture |
| - helix-memory |
| - cpu-training |
| --- |
| |
| # FSI_Edge: From-Scratch Novel Architecture Coding Model |
| |
| A tiny but capable code generation model trained from scratch on ARM CPU, with a novel DNA-inspired architecture. |
| |
| ## Architecture |
| |
| - **Helix Memory** — DNA helix-inspired curved memory for O(log L) context scaling |
| - **HCA** (Hybrid Concentrated Attention) — 3-tier code attention (local + structural + global) |
| - **EA-FFN** (Execution-Augmented FFN) — learns execution traces |
| - **RoPE-S** — RoPE with structural bias for code structure |
| - **PPN** (Prefix-Preserving Norm) — stabilizes deep training |
| - **MoD** (Mixture-of-Depths) — dynamic routing to save compute |
| |
| ## Training Stages |
| |
| 1. **Stage 1** — Pretraining (next-token prediction on code + NLP) |
| 2. **Stage 1b** — FIM (Fill-in-Middle code infilling) |
| 3. **Stage 2** — SFT (Supervised Fine-Tuning) |
| 4. **Stage 2b** — Cold-Start Reasoning (chain-of-thought) |
| 5. **Stage 3** — MCPO RL (Monte Carlo Policy Optimization) |
| 6. **Stage 4** — DPO (Direct Preference Optimization) |
| 7. **Stage 5** — Long-Context Extension |
| |
| ## Quick Start |
| |
| ```bash |
| # Clone from HuggingFace |
| git clone https://huggingface.co/FerrellSyntheticIntelligence/FSI-Edge |
| cd FSI-Edge |
| pip install -r requirements.txt |
| |
| # Train on CPU |
| python training/run_cpu.py --model-size 4K --steps 1000 |
|
|
| # Resume training from checkpoint (step 4132) |
| python training/run_cpu.py --model-size 4K --steps 10000 \ |
| --resume checkpoints/cpu_ckpt_004132.pt --lr 2e-4 |
| |
| # Or on Colab T4 GPU (100x faster) |
| # Upload scripts/fsi_edge_colab.ipynb to Google Colab |
| ``` |
| |
| ## Checkpoints |
| |
| `checkpoints/` contains trained checkpoints from ARM CPU training: |
| - `cpu_best.pt` — best model weights (19MB) |
| - `cpu_latest.pt` — latest model weights (19MB) |
| - `cpu_ckpt_004132.pt` — full training state (52MB, step 4132) |
|
|
| ## Tokenizer |
|
|
| Trained BPE tokenizer (32K vocab) at `fsi_edge_tokenizer/`. |
|
|
| ## Results |
|
|
| | Steps | Best Loss | Platform | |
| |-------|-----------|----------| |
| | 0 | 10.44 | ARM CPU | |
| | 1000 | ~6.0 | ARM CPU | |
| | 2000 | ~1.0 | ARM CPU | |
| | 4132 | 0.70 | ARM CPU | |
|
|
| ## Colab Training |
|
|
| Open `scripts/fsi_edge_colab.ipynb` in Google Colab with T4 GPU for 100x faster training. |
|
|
| ## Mission |
|
|
| Train a from-scratch novel architecture model. Each step proves the architecture. |
| The code is production-ready for cloud GPU scaling (H100s). |
|
|
