Nano-Em1-0.6B-v2

Nano-Em1-0.6B-v2 is a 0.6B-parameter text embedding model built on a bidirectional-attention variant of Qwen3-0.6B, developed by KiteFish AI. It produces general-purpose embeddings for retrieval, classification, clustering, semantic similarity, and pair classification, using task-specific instruction prefixes.

Model description

• Base model: Qwen3-0.6B
• Attention: bidirectional (the causal mask is replaced with full bidirectional attention at every layer, implemented directly in the model's code so it holds under any standard load — see Architecture)
• Pooling: mean pooling over token embeddings
• Embedding dimension: 1024
• Max sequence length: 512 tokens

Architecture

Decoder-only language models use causal (left-to-right) attention by default, which limits their quality as fixed-representation encoders. Nano-Em1 removes the causal mask so every token attends to the full input in both directions. This is implemented in the model's own code (modeling_nano.py, loaded via trust_remote_code=True) rather than applied by the caller at inference time, so it's preserved by any standard AutoModel.from_pretrained or SentenceTransformer load.

Usage

Sentence-Transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "KiteFishAI/Nano-Em1-0.6B-v2",
    trust_remote_code=True,
)

query_instruction = "Instruct: Given a query, retrieve documents that answer the query\nQuery: "

queries = [query_instruction + "How do I reset my password?"]
docs = ["To reset your password, go to Settings > Security and click 'Reset Password'."]

query_emb = model.encode(queries, normalize_embeddings=True)
doc_emb = model.encode(docs, normalize_embeddings=True)

similarity = query_emb @ doc_emb.T
print(similarity)

Transformers

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "KiteFishAI/Nano-Em1-0.6B-v2"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(model_id, trust_remote_code=True).eval()

def embed(texts):
    inputs = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
    with torch.no_grad():
        out = model(**inputs).last_hidden_state
    mask = inputs["attention_mask"].unsqueeze(-1).float()
    pooled = (out * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
    return torch.nn.functional.normalize(pooled, dim=-1)

embs = embed(["example sentence one", "example sentence two"])

Instructions

Queries — and passages, for symmetric tasks — should be prefixed:

Instruct: {task instruction}
Query: {text}

Task type	Example instruction
Retrieval / Reranking	Given a query, retrieve documents that answer the query
Semantic similarity / Pair classification	Retrieve semantically similar text
Clustering	Identify the topic or theme of the given texts
Classification	Classify the given text

For asymmetric tasks (retrieval, reranking), only the query takes the instruction prefix — passages/documents are encoded as-is.

Limitations

• English-focused; not evaluated on non-English tasks.
• Requires trust_remote_code=True to load the custom bidirectional architecture.

Citation

@misc{nano-em1-2026,
  title  = {Nano-Em1-0.6B-v2},
  author = {KiteFish AI},
  year   = {2026},
  url    = {https://huggingface.co/KiteFishAI/Nano-Em1-0.6B-v2}
}