metadata
license: apache-2.0
language:
- en
tags:
- code-generation
- documentation
- text-to-code
- rust
- python
- javascript
- sparse-transformer
pipeline_tag: text-generation
inference: false
Cofos Code Model (v1-logic)
Cofos is a logic- and coding-oriented decoder-only language model built on AMEFORGE's SparseMind architecture. It is specialized for code generation, docstring construction, and code explanation in Python, JavaScript, and Rust.
Developed by AMEFORGE.
Architecture & Parameters
- Architecture: SparseMind v15 (dynamic neural type management, sparse attention, and sparse FFN)
- Parameters: 36,824,070
- Hidden dimension: 384
- Layers: 8
- Context length: 512 tokens
- Tokenizer: Custom BPE Tokenizer (vocab size 16384) with atomic indents/operators/keywords
Training Details
- Trained on 30,000,000+ procedurally generated programming templates on-the-fly.
- Masked loss to learn code completion and documentation formatting.
- Best Python syntax-validity on held-out validation: 8.8%
How to use
import torch
import sentencepiece as spm
# Load checkpoint
ckpt = torch.load("cofos_best.pt", map_location="cpu")
cfg_dict = ckpt["config"]
# Instantiate model architecture
# model = SparseMind(Config(**cfg_dict))
# model.load_state_dict(ckpt["model"])
# model.eval()