CallForge-1B-v2

A 1B tool-calling model fine-tuned from openbmb/MiniCPM5-1B on the CallForge v2 trajectory corpus (893 trajectories: multi-hop dependency chains, distractor tools, clarification-required turns, literal-copy traps). Trained with LoRA (merged) + trained lm_head, bf16, fixed-2048 padding.

Benchmarks (measured, no cherry-picking)

All numbers below are from the unified runner (eval/benchmarks/run_suite.py): greedy decoding, seed 0, the checkpoint's own chat template, hit_token_budget => failure. Raw traces are in benchmark_results.json. The stress and private suites are held-out and were never trained on; BFCL v3 is hash-pinned.

Suite	Passed	Rate	Wilson 95% CI
Stress suite (66 adversarial probes, private)	56/66	84.8%	74.3–91.6%
Private eval suite (121 tasks, zero public leakage)	97/121	80.2%	72.2–86.3%
BFCL v3 (hash-pinned @ f092d154, 180 cases)	112/180	62.2%	55.0–69.0%

Deltas vs baselines

Suite	v1 CallForge	Base MiniCPM5	v2 (this model)
Stress suite (66 adversarial probes, private)	33.3%	40.9%	84.8%
Private eval suite (121 tasks, zero public leakage)	45.5%	28.1%	80.2%
BFCL v3 (hash-pinned @ f092d154, 180 cases)	57.2%	42.8%	62.2%

Training

• Run: v2-r32-lr2e-5-tp (preregistered ablation-grid winner)
• Data: data/trajectories/corpus_v2.jsonl (893 trajectories, hash recorded in the reproducibility manifest)
• Method: LoRA + trained lm_head, merged; bf16; fixed-2048 padding
• Selection: preregistered score 0.6×probe_pass_rate + 0.4×(1−val_loss/3.0); no post-hoc tuning

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("solomoniw/CallForge-1B-v2")
tok = AutoTokenizer.from_pretrained("solomoniw/CallForge-1B-v2")
messages = [{"role": "user", "content": "What's the weather in Paris?"}]
tools = [{"type": "function", "function": {"name": "get_weather",
  "description": "Get current weather", "parameters": {"type": "object",
  "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
prompt = tok.apply_chat_template(messages, tools=tools, add_generation_prompt=True, tokenize=False)
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=192, do_sample=False)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))

GGUF quants (F16, Q8_0, Q4_K_M) are in this repo; load with llama.cpp.

Limitations

• 1B parameters: reasoning depth is bounded; the model is optimized for correct tool selection/argument extraction, not open-domain chat.
• Stress-suite categories with small n have wide CIs; see the table.
• Benchmarks were run on Apple Silicon MPS; absolute numbers may shift slightly on other backends, but the same runner and seeds were used for every model compared.