ArchSpace-Collection/NCP_Olmo3_Stage2_v3

NCP OLMo 3 Stage2 - v3

NCP_Olmo3_Stage2_v3 is the v3 release from the NCP OLMo 3 Stage2 training line. It was converted from Megatron iteration 1,429,970 into a pure Hugging Face release with sharded SafeTensors and bundled Transformers remote code.

This is a research checkpoint for the Next-Concept-Prediction (NCP) architecture. Its token-level Transformer design is aligned with OLMo 3, while NCP adds dense residual connections and discrete concept representations. The model encodes token sequences into concepts, models them at the concept level, and decodes them back into token space.

Release Validation

The release package passed the following integrity and runtime checks:

• Pure-HF export status: PURE_HF_ARTIFACT_EXPORT_OK
• Standalone loading status: NCP_OLMO3_PURE_HF_READY
• Release package status: NCP_OLMO3_RELEASE_PACKAGE_VALIDATED
• Unified HF 4K+1K parity: passed with an exact generated-token match

The 4K+1K check is a release-integrity smoke test, not a benchmark score. Benchmark results are intentionally not claimed in this model card.

Model Details

Field	Value
Architecture	NCPOlmo3ForCausalLM
Training stage	Stage2
Source checkpoint	Megatron iteration 1,429,970
Parameter scale	7B-class
Hidden size	4,096
Token backbone layers	32 (16 encoder + 16 decoder)
Concept-special layers	8
Attention heads	32
FFN hidden size	11,008
Vocabulary size	100,278
Maximum sequence length	8,192 tokens
Attention pattern	4K sliding-window attention with periodic full attention
Concept chunk size	4 tokens
Discrete concept codebooks	32 codebooks x 128 entries
Weight dtype	BF16
Weight format	5 sharded SafeTensors files (842 tensors)
Weight key format	Pure Hugging Face state-dict keys
Pure Transformers backend	Yes
ConceptLM vLLM backend	Yes
External Megatron runtime for HF loading	Not required
Package validation	NCP_OLMO3_RELEASE_PACKAGE_VALIDATED
Unified HF 4K+1K parity	Passed (exact generated-token match)

The release stores one shared set of SafeTensors with dedicated Hugging Face projection keys such as q_proj, k_proj, v_proj, gate_proj, up_proj, and down_proj. No native Megatron-key weight copy is required.

Requirements

• A CUDA-capable NVIDIA GPU with enough memory for this 7B-class checkpoint.
• PyTorch with CUDA and bfloat16 support.
• A Transformers version compatible with the bundled remote code.
• trust_remote_code=True when loading through Transformers.
• No external ConceptLM or Megatron checkout is required for the pure-HF backend.

Download and Load

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "ArchSpace-Collection/NCP_Olmo3_Stage2_v3"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
model.eval()

Generation

prompt = "The role of hierarchical representations in language modeling is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        use_cache=True,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Files and Provenance

• config.json defines the pure-HF architecture and auto_map entries.
• model.safetensors.index.json maps all HF state-dict keys to five shards.
• conversion_manifest.json records conversion provenance and shard hashes.
• standalone_manifest.json records standalone HF/vLLM readiness.
• release_validation_manifest.json records package-level validation.

Intended Use and Limitations

This checkpoint is released for research and evaluation. It is a base language model, not a safety-tuned assistant. Users should review the bundled custom code before enabling trust_remote_code=True, validate outputs for their application, and follow the terms of the training data and downstream deployment environment.

The bundled vLLM readiness refers to the ConceptLM vLLM integration; compatibility with an unmodified upstream vLLM installation is not implied.