Presumably incorrect values from the QWEN original model
https://huggingface.co/Qwen/Qwen3.6-35B-A3B
example:
your table for Qwen3.6-35B-A3B shows
SWE-bench pro = 40.6 BUT -> original 49.5
only one of many examples
btw
https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B
has correct original numbers from qwen !
Thank you for raising the question and your careful observation.
Evaluation method. All metrics presented in the table are reproduced in-house: we download the public model checkpoints, deploy them via vLLM or SGLang, and evaluate under a unified standardized pipeline. No officially reported results of the respective models are directly adopted in this table. Each model is tested only once on each evaluation set; retests are conducted only if obvious errors are found.
Evaluation configuration.
SWE-bench Verified / Multilingual / Pro, KAT-Code-Bench: agent=claude_code@2.1.195, pass@k=1, temperature=1.0, top_p=0.95, 256k ctx.
Terminal-Bench 2.1: agent=terminus-2 / claude_code, pass@k=1, temperature=0.7, top_p=1.0, 256k ctx.
PinchBench: agent=openclaw@2026.3.13, pass@k=1, temperature=0.7, top_p=1.0, 256k ctx.
Scicode: pass@k=1, temperature=0.6, top_p=1.0, 256k ctx.
Anomaly description.
Qwen3.6-35BA3B: We found that on the SWE-bench Verified, SWE-bench Multilingual, and SWE-bench Pro test sets, our test results this time have an approximate 10 pp gap compared with the official results. We believe this is mainly caused by the harness version and some optimizations made to the test sets by the Qwen team, and it should not be an issue with the model itself.
Qwen3.5-35BA3B: We observed frequent hallucinations during evaluation, including attempts to invoke the unavailable MultiEdit tool under the current agent environment, which negatively impacts the final metric.
Gemma4-26B-A4B-it: Two main factors degrade evaluation performance: context overflow (exceeding the 256k context limit) and hallucinated calls to the unsupported MultiEdit tool in this evaluation setup.
The above deviations arise from mismatches between model tool preference and the allowed toolset in the evaluation harness, rather than inherent capability limitations of the models.