Qwen3-Embedding-0.6B-GraphQL

A small, self-hosted embedding model for retrieving GraphQL Type.field coordinates from natural-language questions. It is designed for schema retrieval in AI agent pipelines, where the full schema is too large to place in context for every request.

Real schemas reuse field names extensively. Many types expose fields such as title, author, createdAt, state, or status, so matching the field name alone is not enough; retrieval must also identify the correct owner type.

This model fine-tunes Qwen/Qwen3-Embedding-0.6B for that owner-type disambiguation task. At 0.6B parameters, it can run locally on CPU or alongside an agent's primary model.

Benchmark

The GraphQL AI Working Group benchmark evaluates 816 natural-language queries across GitHub, GitLab, Linear, Shopify, and the Singapore Open Data API. Field-level recall against the ground-truth Type.field coordinate:

model	dim	recall@20	recall@50
Qwen3-Embedding-0.6B-GraphQL (fine-tune)	1024	50.6%	63.6%
OpenAI text-embedding-3-large	3072	47.3%	59.8%
Qwen3-Embedding-0.6B (base, un-tuned)	1024	46.7%	58.5%
OpenAI text-embedding-3-small	1536	42.3%	54.3%

The fine-tune leads field-level retrieval at both cutoffs and improves recall@50 by 5.1 percentage points over its base model. At the owner-type level, text-embedding-3-large leads at 87.8% recall@50; the fine-tune reaches 85.6%, ahead of the base model at 78.6% and text-embedding-3-small at 80.8%.

The model ships as SentenceTransformers weights and GGUF builds for llama.cpp and Ollama.

Important: how you format the corpus matters as much as the model. Use SDL snippets or dot_plus_gloss formatting for best results. See Embedding style comparison for details.

---

Inference

SentenceTransformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("xthor/Qwen3-Embedding-0.6B-GraphQL")

query = "When did I place this order?"
# coordinates of Type.field pairs
coords = [
    "Order.createdAt",
    "Order.completedAt",
    "Order.updatedAt",
]

q = model.encode(query, prompt_name="query")
c = model.encode(coords, prompt_name="document")
scores = (q @ c.T).tolist()

for coord, score in sorted(zip(coords, scores), key=lambda x: -x[1]):
    print(f"{score:.3f}  {coord}")

The fine-tuned model ranks Order.createdAt first; the un-tuned base model ranks Order.completedAt first. This is a small sanity check using readable bare coordinates. For production retrieval, use the richer SDL or gloss formatting described below.

Two prompts are wired into the model and must be used for best results:

• prompt_name="query" for natural-language questions
• prompt_name="document" for GraphQL coordinate descriptions in the corpus

Ollama

# pull one quantization (Q8_0 is a good default: near-lossless, ~650 MB)
hf download xthor/Qwen3-Embedding-0.6B-GraphQL model-q8_0.gguf --local-dir .

cat > Modelfile <<'EOF'
FROM ./model-q8_0.gguf
EOF
ollama create qwen3-graphql-embedder -f Modelfile

# OpenAI-compatible embeddings endpoint
curl -s http://localhost:11434/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-graphql-embedder","input":"When did I place this order?"}' \
  | jq '.data[0].embedding'

llama.cpp

hf download xthor/Qwen3-Embedding-0.6B-GraphQL model-q8_0.gguf --local-dir .

./llama-server -m model-q8_0.gguf --embedding --port 8080
# POST http://localhost:8080/embedding   { "content": "..." }

Available GGUF quantizations

file	size	use case
model-f16.gguf	~1.2 GB	reference quality, parity with safetensors
model-q8_0.gguf	~650 MB	near-lossless; recommended default
model-q4_k_m.gguf	~400 MB	small footprint; accepts a minor quality trade-off

---

Results

223 held-out test queries · 28,893-coordinate corpus · 30% real SDLs (GitHub GHES, Saleor, Shopify, AniList) never seen in training.

metric	baseline	tuned (3 epochs)	lift
exact_match@1	0.090	0.229	+0.139 (+155%)
recall@3	0.130	0.318	+0.188
recall@5	0.161	0.345	+0.184 (+114%)
recall@10	0.215	0.435	+0.220 (+102%)
mrr@10	0.121	0.285	+0.164
ndcg@10	0.143	0.320	+0.177

Where the lift comes from

Direct questions ("has my package shipped?", "what's my total?") are already handled well by the base model. The gains come from indirect questions where the user names a concept rather than a field. Those require owner-type reasoning, and that's where the base model falls behind.

Small inference example

"When did I place this order?"

Against the bare coordinates Order.createdAt, Order.completedAt, and Order.updatedAt, the base model ranks Order.completedAt first. The fine-tuned model ranks the correct coordinate, Order.createdAt, first and opens a clearer margin:

model	Order.createdAt	Order.completedAt	target margin
base	0.619	0.650	-0.031
fine-tuned	0.660	0.601	+0.059

Every candidate is a plausible timestamp on the same order. The model must distinguish when the order was placed from when it was completed or last updated. This three-candidate example is an illustrative sanity check; the aggregate results above measure retrieval quality over the full held-out corpus.

Known limitations

1. Formatting sensitivity. With raw dot notation (Type.field), the fine-tune's R@1 is only 0.308 on the GitHub schema. Always use sdl, dot_plus_gloss, or natural formatting for the corpus.

2. Same-owner wrong-field rate. same_owner_wrong_field_rate@1 rose from 0.063 to 0.103. The model picks the right owner type more often but occasionally lands on the wrong field within that type. The training signal rewards owner disambiguation; within-owner field disambiguation isn't targeted. The next iteration will add competition sets that share owner and differ by field.

3. Tail regression with raw dot notation. When using raw dot notation, the fine-tune's P95 rank (404) is worse than the base model's (123). The model becomes more confident: it either ranks the correct answer first or misses much harder. This is fully mitigated by using sdl (P95 40) or dot_plus_gloss (P95 41) formatting.

4. Indirect queries. Queries that don't name or allude to the owner type (e.g., "get the README" → Repository.object) remain hard for both models. The fine-tune does not improve on these.

How you format the corpus matters

How you turn each Type.field coordinate into text before embedding it affects retrieval more than the fine-tune does. The benchmark below compares twelve formats on the GitHub GraphQL schema (52 held-out queries):

Use one of these two. They tie at the top:

# sdl: if you parse the schema (MRR 0.723)
type PullRequest { baseRefName: String! }

# dot_plus_gloss: string-only, no parsing needed (MRR 0.715)
PullRequest.baseRefName — the base ref name of a pull request

The cheap string-only gloss costs almost nothing versus full schema parsing, so reach for dot_plus_gloss unless you already have parsed types on hand. Whatever you do, don't embed raw Type.field identifiers. With dot formatting, MRR drops to 0.393 and the worst-case rank blows out 10x. The owner type is what carries the signal: drop it entirely and retrieval collapses to MRR ~0.05.

Full results

Each format is one way of rendering PullRequest.baseRefName into text before embedding (the example column shows exactly what). P95 is the 95th-percentile rank, i.e. how badly the worst queries rank. Lower is better.

format	example (PullRequest.baseRefName →)	base MRR	tuned MRR	P95
sdl	type PullRequest { baseRefName: String! }	0.511	0.723	40
dot_plus_gloss	PullRequest.baseRefName — the base ref name of a pull request	0.551	0.715	41
semantic	GraphQL field PullRequest.baseRefName. Owner type… Returns: String!…	0.368	0.659	39
field_first	base ref name (PullRequest)	0.571	0.652	70
natural	the base ref name field on PullRequest	0.420	0.578	119
arrow	PullRequest > base ref name	0.419	0.548	159
colon	PullRequest: base ref name	0.400	0.488	199
split_space	pull request base ref name	0.391	0.447	448
signature	PullRequest.baseRefName: String!	0.334	0.408	298
dot	PullRequest.baseRefName (raw, no change)	0.334	0.393	404
type_only	pull request (field dropped, ablation)	0.248	0.242	261
field_only	base ref name (type dropped, ablation)	0.063	0.045	3377

---

Training

run	epochs	batch	lr	loss
qwen3	2	64	5e-5	cached_mnrl
qwen3-e3	3	64	5e-5	cached_mnrl

Both: --max-seq-length 256, 4 hard negatives per anchor, bf16, full fine-tune (no LoRA), single H100. Published checkpoint: qwen3-e3.

Dataset

split	rows
train	4,788
val	94
test	223
corpus	28,893

Built from 7,626 raw seed pairs via world-leakage, per-row strict-leakage, and family-level semantic-dedup filters. The strict-leakage filter is aggressive on real-SDL queries, which is why val/test shrink to ~20% of raw.

---

Citation

• Base model: Qwen3-Embedding-0.6B
• GitHub Training GitHub-Repo-train-data
• License: Apache 2.0 (inherited from the base) thore@MacBook-Pro-von-Thore projects % cat model_card.md

---

license: apache-2.0 library_name: sentence-transformers base_model: Qwen/Qwen3-Embedding-0.6B datasets: - xthor/Qwen3-Embedding-GraphQL-v1 pipeline_tag: sentence-similarity tags: - sentence-transformers - sentence-similarity - feature-extraction - graphql - retrieval - embeddings - text-embeddings-inference - qwen3 - gguf language: - en

Qwen3-Embedding-0.6B-GraphQL

A small, self-hosted embedding model for retrieving GraphQL Type.field coordinates from natural-language questions. It is designed for schema retrieval in AI agent pipelines, where the full schema is too large to place in context for every request.

Real schemas reuse field names extensively. Many types expose fields such as title, author, createdAt, state, or status, so matching the field name alone is not enough; retrieval must also identify the correct owner type.

This model fine-tunes Qwen/Qwen3-Embedding-0.6B for that owner-type disambiguation task. At 0.6B parameters, it can run locally on CPU or alongside an agent's primary model.

Benchmark

The GraphQL AI Working Group benchmark evaluates 816 natural-language queries across GitHub, GitLab, Linear, Shopify, and the Singapore Open Data API. Field-level recall against the ground-truth Type.field coordinate:

model	dim	recall@20	recall@50
Qwen3-Embedding-0.6B-GraphQL (fine-tune)	1024	50.6%	63.6%
OpenAI text-embedding-3-large	3072	47.3%	59.8%
Qwen3-Embedding-0.6B (base, un-tuned)	1024	46.7%	58.5%
OpenAI text-embedding-3-small	1536	42.3%	54.3%

The fine-tune leads field-level retrieval at both cutoffs and improves recall@50 by 5.1 percentage points over its base model. At the owner-type level, text-embedding-3-large leads at 87.8% recall@50; the fine-tune reaches 85.6%, ahead of the base model at 78.6% and text-embedding-3-small at 80.8%.

The model ships as SentenceTransformers weights and GGUF builds for llama.cpp and Ollama.

Important: how you format the corpus matters as much as the model. Use SDL snippets or dot_plus_gloss formatting for best results. See Embedding style comparison for details.

---

Inference

SentenceTransformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("xthor/Qwen3-Embedding-0.6B-GraphQL")

query = "When did I place this order?"
# coordinates of Type.field pairs
coords = [
    "Order.createdAt",
    "Order.completedAt",
    "Order.updatedAt",
]

q = model.encode(query, prompt_name="query")
c = model.encode(coords, prompt_name="document")
scores = (q @ c.T).tolist()

for coord, score in sorted(zip(coords, scores), key=lambda x: -x[1]):
    print(f"{score:.3f}  {coord}")

The fine-tuned model ranks Order.createdAt first; the un-tuned base model ranks Order.completedAt first. This is a small sanity check using readable bare coordinates. For production retrieval, use the richer SDL or gloss formatting described below.

Two prompts are wired into the model and must be used for best results:

• prompt_name="query" for natural-language questions
• prompt_name="document" for GraphQL coordinate descriptions in the corpus

Ollama

# pull one quantization (Q8_0 is a good default: near-lossless, ~650 MB)
hf download xthor/Qwen3-Embedding-0.6B-GraphQL model-q8_0.gguf --local-dir .

cat > Modelfile <<'EOF'
FROM ./model-q8_0.gguf
EOF
ollama create qwen3-graphql-embedder -f Modelfile

# OpenAI-compatible embeddings endpoint
curl -s http://localhost:11434/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-graphql-embedder","input":"When did I place this order?"}' \
  | jq '.data[0].embedding'

llama.cpp

hf download xthor/Qwen3-Embedding-0.6B-GraphQL model-q8_0.gguf --local-dir .

./llama-server -m model-q8_0.gguf --embedding --port 8080
# POST http://localhost:8080/embedding   { "content": "..." }

Available GGUF quantizations

file	size	use case
model-f16.gguf	~1.2 GB	reference quality, parity with safetensors
model-q8_0.gguf	~650 MB	near-lossless; recommended default
model-q4_k_m.gguf	~400 MB	small footprint; accepts a minor quality trade-off

---

Results

223 held-out test queries · 28,893-coordinate corpus · 30% real SDLs (GitHub GHES, Saleor, Shopify, AniList) never seen in training.

metric	baseline	tuned (3 epochs)	lift
exact_match@1	0.090	0.229	+0.139 (+155%)
recall@3	0.130	0.318	+0.188
recall@5	0.161	0.345	+0.184 (+114%)
recall@10	0.215	0.435	+0.220 (+102%)
mrr@10	0.121	0.285	+0.164
ndcg@10	0.143	0.320	+0.177

Where the lift comes from

Direct questions ("has my package shipped?", "what's my total?") are already handled well by the base model. The gains come from indirect questions where the user names a concept rather than a field. Those require owner-type reasoning, and that's where the base model falls behind.

Small inference example

"When did I place this order?"

Against the bare coordinates Order.createdAt, Order.completedAt, and Order.updatedAt, the base model ranks Order.completedAt first. The fine-tuned model ranks the correct coordinate, Order.createdAt, first and opens a clearer margin:

model	Order.createdAt	Order.completedAt	target margin
base	0.619	0.650	-0.031
fine-tuned	0.660	0.601	+0.059

Every candidate is a plausible timestamp on the same order. The model must distinguish when the order was placed from when it was completed or last updated. This three-candidate example is an illustrative sanity check; the aggregate results above measure retrieval quality over the full held-out corpus.

Known limitations

1. Formatting sensitivity. With raw dot notation (Type.field), the fine-tune's R@1 is only 0.308 on the GitHub schema. Always use sdl, dot_plus_gloss, or natural formatting for the corpus.

2. Same-owner wrong-field rate. same_owner_wrong_field_rate@1 rose from 0.063 to 0.103. The model picks the right owner type more often but occasionally lands on the wrong field within that type. The training signal rewards owner disambiguation; within-owner field disambiguation isn't targeted. The next iteration will add competition sets that share owner and differ by field.

3. Tail regression with raw dot notation. When using raw dot notation, the fine-tune's P95 rank (404) is worse than the base model's (123). The model becomes more confident: it either ranks the correct answer first or misses much harder. This is fully mitigated by using sdl (P95 40) or dot_plus_gloss (P95 41) formatting.

4. Indirect queries. Queries that don't name or allude to the owner type (e.g., "get the README" → Repository.object) remain hard for both models. The fine-tune does not improve on these.

How you format the corpus matters

How you turn each Type.field coordinate into text before embedding it affects retrieval more than the fine-tune does. The benchmark below compares twelve formats on the GitHub GraphQL schema (52 held-out queries):

Use one of these two. They tie at the top:

# sdl: if you parse the schema (MRR 0.723)
type PullRequest { baseRefName: String! }

# dot_plus_gloss: string-only, no parsing needed (MRR 0.715)
PullRequest.baseRefName — the base ref name of a pull request

The cheap string-only gloss costs almost nothing versus full schema parsing, so reach for dot_plus_gloss unless you already have parsed types on hand. Whatever you do, don't embed raw Type.field identifiers. With dot formatting, MRR drops to 0.393 and the worst-case rank blows out 10x. The owner type is what carries the signal: drop it entirely and retrieval collapses to MRR ~0.05.

Full results

Each format is one way of rendering PullRequest.baseRefName into text before embedding (the example column shows exactly what). P95 is the 95th-percentile rank, i.e. how badly the worst queries rank. Lower is better.

format	example (PullRequest.baseRefName →)	base MRR	tuned MRR	P95
sdl	type PullRequest { baseRefName: String! }	0.511	0.723	40
dot_plus_gloss	PullRequest.baseRefName — the base ref name of a pull request	0.551	0.715	41
semantic	GraphQL field PullRequest.baseRefName. Owner type… Returns: String!…	0.368	0.659	39
field_first	base ref name (PullRequest)	0.571	0.652	70
natural	the base ref name field on PullRequest	0.420	0.578	119
arrow	PullRequest > base ref name	0.419	0.548	159
colon	PullRequest: base ref name	0.400	0.488	199
split_space	pull request base ref name	0.391	0.447	448
signature	PullRequest.baseRefName: String!	0.334	0.408	298
dot	PullRequest.baseRefName (raw, no change)	0.334	0.393	404
type_only	pull request (field dropped, ablation)	0.248	0.242	261
field_only	base ref name (type dropped, ablation)	0.063	0.045	3377

---

Training

run	epochs	batch	lr	loss
qwen3	2	64	5e-5	cached_mnrl
qwen3-e3	3	64	5e-5	cached_mnrl

Both: --max-seq-length 256, 4 hard negatives per anchor, bf16, full fine-tune (no LoRA), single H100. Published checkpoint: qwen3-e3.

Dataset

split	rows
train	4,788
val	94
test	223
corpus	28,893

Built from 7,626 raw seed pairs via world-leakage, per-row strict-leakage, and family-level semantic-dedup filters. The strict-leakage filter is aggressive on real-SDL queries, which is why val/test shrink to ~20% of raw.

---

Citation

• Base model: Qwen3-Embedding-0.6B
• GitHub Training GitHub-Repo-train-data
• License: Apache 2.0 (inherited from the base)