Lokesh56/Neora-1.0-Basic

Neora 1.0 Basic

Neora 1.0 Basic is a custom-designed ~505 Million parameters transformer model pretrained on general language resources using a vocab size of 24,000 and 24 structural transformer layers.

Model Details

• Model Name: Neora 1.0 Basic
• Parameters: 502,640,640 (505M with embeddings)
• Architecture: Custom Causal Attention Transformer
• Number of Layers: 24
• Attention Heads: 20
• Hidden Dimension (D_MODEL): 1280
• Feedforward Dimension (FFN_DIM): 5120
• Vocabulary Size: 24,000
• Context Window Length: 2048

Python Inference Example

import torch
from transformers import AutoModel, AutoConfig
from tokenizers import Tokenizer

# Load config and custom model dynamically
model = AutoModel.from_pretrained("Neora-1.0-Basic", trust_remote_code=True)
tokenizer = Tokenizer.from_file("tokenizer.json")

prompt = "The future of artificial intelligence is"
input_ids = torch.tensor([tokenizer.encode(prompt).ids])

with torch.no_grad():
    outputs = model(input_ids)

Limitations & Disclaimer

This is a verification/developmental base run checkpoint (step 20). Outputs are non-converged and exploratory. Use with trust_remote_code=True required.