Evicka/HanseLM-78M-Base

HanseLM 78M Base

HanseLM 78M Base is a small German-first causal language model and an experimental research preview. It was trained from scratch to study whether a hybrid attention/convolution architecture can learn useful German text representations under a roughly 1.5-billion-token compute budget.

This is not a frontier model, not an instruction-tuned assistant, and not expected to be broadly useful without further post-training. It may produce incorrect, repetitive, mixed-language, unsafe, or memorized text. The release is primarily useful for architecture experiments, education, controlled fine-tuning, and reproducibility work.

The release code and model weights are available under the MIT License. The source datasets are not redistributed and retain their own terms; see DATA_SOURCES.md.

Model details

Property	Value
Model type	Decoder-only causal language model
Parameters	78,430,848
Vocabulary	24,576-token byte-level BPE
Context trained/tested	2,048 tokens
Hidden size	640
Layers	14: 11 attention, 3 causal convolution
Attention	Grouped-query attention, 10 query heads, 2 KV heads
Head dimension	64
Feed-forward	SwiGLU, hidden size 1,792
Position encoding	RoPE, theta 10,000
Normalization	Pre-norm RMSNorm, epsilon 1e-6
Precision	BF16 training/autocast; FP32 master weights
Embeddings	Input/output weights tied
Training stage	Base pretraining only

Architecture

Each block applies a pre-normalized token mixer and a pre-normalized SwiGLU feed-forward network with residual connections. The layer pattern is:

A A C A A A C A A A C A A A

A blocks use causal scaled-dot-product grouped-query attention. Ten query heads share two key/value heads in five-head groups. Queries and keys receive per-head RMS normalization before rotary position encoding. C blocks use a gated, depthwise causal 1D convolution with kernel size 7. Convolution blocks occur at layers 3, 7, and 11 (one-indexed). All projections are bias-free.

The hybrid design reduces the number of global-attention layers, but this release does not claim an inference-speed advantage: the reference implementation has no KV cache and currently recomputes the visible context during autoregressive generation.

Tokenizer

The tokenizer is a custom NFC-normalizing, byte-level BPE tokenizer with byte fallback. It was trained alongside the project and contains:

<|pad|> <|bos|> <|eos|> <|system|> <|user|>
<|assistant|> <|tool|> <|tool_result|>

The presence of role tokens does not make the base model chat-capable. No chat template is defined for this repository.

Training

The sampled pretraining mixture contained approximately 1.500 billion tokens:

Source	Language	Approx. tokens	Share
FineWeb2 (deu_Latn)	German	1.200B	80%
FineWiki (de)	German	225M	15%
FineWeb	English	75M	5%

Training used two stages:

1. 1.350B tokens with 1,024-token sequences.
2. 150.012M additional tokens with 2,048-token sequences.

Phase 2 continued from the Phase 1 final checkpoint with AdamW (betas=(0.9, 0.95)), a peak learning rate of 4e-5, a minimum learning rate of 4e-6, and 100 warmup steps. Training ran locally with PyTorch/ROCm on an AMD Radeon RX 9070 XT.

Compute and hardware

The full base model was trained locally on a single consumer GPU, not a multi-GPU server or cloud cluster:

Item	Value
GPU	1× AMD Radeon RX 9070 XT
Available VRAM	15.81 GiB (marketed as 16 GB)
Software	PyTorch 2.12.0 + ROCm 7.14 under WSL2
Precision	BF16 autocast, FP32 AdamW state
Peak allocated VRAM	12.15 GiB in Phase 1; 11.83 GiB in Phase 2
Typical observed throughput	approximately 43k–48k tokens/s
Phase 1 GPU-process time	approximately 8.38 GPU-hours
Phase 2 GPU-process time	1.03 GPU-hours
Total pretraining compute	approximately 9.41 single-GPU hours

Of the total, 8.01 hours come directly from structured trainer timers and approximately 1.4 hours are estimated for the initial unstructured segment. The Phase 1 number includes replayed work after interrupted runs. Timers include validation and checkpoint overhead while the trainer was active, but exclude intentional pauses, dataset preparation, checkpoint selection, and the later release evaluations.

This modest hardware footprint is part of the experiment: HanseLM tests what can be trained end-to-end on one 16 GB desktop GPU. It should not be confused with frontier-scale pretraining, and the low compute budget is also a major reason for the model's limited factual and reasoning ability.

Documents were packed with EOS separators. The current attention mask does not isolate documents inside a packed sequence, so tokens can attend across an EOS boundary. This is a known training limitation.

Evaluation

The selected final checkpoint beat the late Phase 2 checkpoints on three independent mixed-validation samples. The margin over step 12,000 was small but consistent.

Held-out language-model loss

All rows below use 2,048-token sequences, batch size 6, 200 batches, and seed 1,234.

Validation corpus	Loss	Perplexity
FineWiki German	2.574947	13.131
FineWeb English	2.943911	18.990
Mixed validation	3.175663	23.943
FineWeb2 German	3.310921	27.410

Perplexity is specific to this tokenizer and each corpus. Values must not be compared directly with models using another tokenizer, and these results are not downstream knowledge, reasoning, safety, or instruction-following benchmarks. Standardized German and English benchmarks have not yet been run.

Phase 2 improved the shared mixed-validation loss from 3.228526 (Phase 1) to 3.175663.

A fixed-seed generation check found no repeated token trigrams in the two German and one English sample; a deliberately mixed prompt had a 0.0217 repeated-trigram fraction and continued in English. These four samples are too small for a quality claim. They also exposed a clear factual error about Hamburg, reinforcing that the model must not be used as a factual source. The full samples and a successful 2,048-token finite-logit smoke test are recorded under evaluation/.

A broader 20-sample suite (10 prompts, two seeds) produced a mean repeated token-trigram fraction of 0.0445 and a maximum of 0.1613. German and English continuations were often locally grammatical, and the mixed prompt continued in German. However, factual prompts failed consistently: the model called Berlin the capital of southern France or Croatia, placed German federal history in 1921/1998, and gave contradictory freezing temperatures. Recipe and story continuations were recognizable in form but frequently incoherent. The honest conclusion is that the checkpoint has learned language form and domain-like continuation, but not dependable knowledge or instruction following. Raw outputs are stored in evaluation/generation-suite.json.

Usage

This repository contains custom Transformers code, so loading requires explicitly trusting the repository code:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Evicka/HanseLM-78M-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
)

inputs = tokenizer("Die Hanse war", return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_k=50,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

The reference implementation does not implement KV caching. Generation is therefore slower than similarly sized cached Transformer models.

Intended uses

• Research on small language models and hybrid token mixers.
• Reproducible continued pretraining or post-training experiments.
• Educational inspection of a complete from-scratch model pipeline.
• Low-stakes, human-reviewed German or English text continuation experiments.

It is not intended as a factual source, production assistant, autonomous agent, safety-critical component, or replacement for a larger evaluated model. High-impact decisions and unsupervised public text generation are out of scope.

Limitations and risks

• Only 78M parameters and about 1.5B training tokens.
• Mostly German web text; only 5% of the sampled mixture was English.
• No instruction tuning, preference optimization, or safety post-training.
• No systematic benchmark, bias, toxicity, privacy, or memorization audit.
• No guaranteed factuality, reasoning, arithmetic, code, or tool-use ability.
• Possible repetition, abrupt continuations, malformed text, and language switching.
• A maximum trained context of 2,048 tokens; longer inputs are rejected.
• Packed-document cross-attention as described above.
• Web and Wikipedia data can contain errors, stereotypes, personal data, copyrighted material, and other undesirable content.

Reproducibility

The staged evaluation record is in evaluation/internal-eval.json. The selected weight source is the Phase 2 final checkpoint after 150,011,904 Phase 2 tokens. The model contains 78,430,848 parameters.

The custom adapter was verified with Transformers 4.57.6 and PyTorch 2.12.0+rocm7.14.0. Loading through AutoModelForCausalLM produced no missing or unexpected keys, and its logits matched the native implementation exactly for the equivalence input (max_abs_diff = 0). Artifact checksums are recorded in MANIFEST.sha256.

License and attribution

The HanseLM release code and model weights are licensed under the MIT License. Training-source attribution and third-party terms are documented separately in DATA_SOURCES.md. In particular, FineWeb and FineWeb2 identify ODC-By 1.0, while FineWiki identifies CC BY-SA 4.0 and GFDL for its Wikipedia-derived text. The MIT license does not relicense those source datasets.

Citation

No paper or archival report exists yet. For reproducible references, cite the repository and an immutable release revision:

@software{hanse_lm_78m_base_2026,
  author = {EvickaStudio},
  title = {HanseLM 78M Base},
  year = {2026},
  url = {https://huggingface.co/Evicka/HanseLM-78M-Base},
  version = {v0.1}
}