lyrain2001/Auto-Fill-Qwen3-8B-Coding

Auto-Fill Coding Specialist (Qwen3-8B)

The coding specialist of Auto-Fill: Learning to Predict Missing Values Accurately with Specialist Language Models (PVLDB 19(11), 2026 — arXiv:2607.19847). Given a table with one cell marked [MISSING], this model writes a column-level pandas rule instead of a value; it targets cells governed by programmatic relationships (arithmetic between columns, string transformations, lookups). The code is executed on the table at inference time, and the model is trained to return empty code when no general rule exists.

Auto-Fill runs three specialists in parallel — knowledge, reasoning and coding — calibrates their confidences with isotonic regression and returns the most confident answer (or abstains). Sibling specialists: Knowledge · Reasoning.

• Code: https://github.com/lyrain2001/auto-fill
• Benchmark: lyrain2001/Auto-Fill-Benchmark
• Calibrators: calibrators.json in this repo (also in the code repository under checkpoints/)

Model details

Base model	Qwen/Qwen3-8B
Training	full-parameter SFT with ms-swift, DeepSpeed ZeRO-3, bf16
Data	52,741 examples: reasoning + code traces distilled from DeepSeek-R1; a trace is kept only if its code executes, recovers the masked cell, and reproduces the other values of the target column with high accuracy.
Hyper-parameters	2 epochs, lr 1e-5, cosine schedule, weight decay 0.1, max length 40,960, effective batch size 16
Hardware	4× A100 80 GB
Confidence signal	the accuracy of the generated code on the other (unmasked) cells of the target column when the code is executed with that column blanked out. Calibrated with isotonic regression fitted on a held-out validation split (calibrators.json).
Decoding used in the paper	temperature 0.8, max new tokens 32768, default Qwen3 chat template

Training tables come from public sources only (spreadsheets crawled from a search-engine index, public BI models, Wikipedia, nationalarchives.gov.uk, GitHub CSV/Parquet files); one cell per table is masked and its original value is the target.

Prompt and output format

The table is serialized as a Markdown pipe table (pandas.DataFrame.to_markdown(index=False, tablefmt="pipe")) with the cell to fill written as [MISSING]. The user message is exactly (see autofill/utils/prompts.py):

Please write a concise Python pandas snippet to fill in the missing value in the input table. The missing value is denoted by '[MISSING]'. Assume the table is in a DataFrame named `df`
Only output non-empty `code` if you can write **one** pandas statement that fills **all** values in the target column using a column-level rule (i.e., it generalizes to every value in that column and depends on other column(s), not row indices). If such a general rule does not exist, leave the `code` field empty. The code must be vectorized (no loops or row-wise indexing) and must not hard-code row-specific constants. Do NOT use `.fillna()` with a scalar literal or literal assignment for trivial cases.
During evaluation, all existing values in the target column are masked to `nan`. Therefore do not read from that column in your computation. Return exactly one JSON object:
{"code": "<CODE or empty string>"}

Example:
Input Table:
| A | B | Total |
|---|---|-------|
| 1 | 2 | 3 |
| 4 | [MISSING] | 9 |
{"code": "df['B'] = df['Total'] - df['A']"}

Input Table:
<markdown table>

Expected output: <think> … </think> followed by {"code": "<one pandas statement, or empty string>"}. The repository executes the code on the table (target column set to NaN) and reads the value at the masked position.

Usage

With the code repository (recommended) — runs the full ensemble on one table:

python inference/run_specialists.py \
    --table          /path/to/table.csv \
    --knowledge_path lyrain2001/Auto-Fill-Qwen3-8B-Knowledge \
    --reasoning_path lyrain2001/Auto-Fill-Qwen3-8B-Reasoning \
    --coding_path    lyrain2001/Auto-Fill-Qwen3-8B-Coding \
    --calibrators    checkpoints/calibrators.json \
    --gpu_ids        0,1,2

or this specialist alone on the benchmark:

python inference/run_benchmark.py --mode coding --model_path lyrain2001/Auto-Fill-Qwen3-8B-Coding \
    --dataset Gov-CSV --benchmark Auto-Fill-Benchmark/sample200 --gpu_ids 0

Minimal vLLM example

import pandas as pd
from vllm import LLM, SamplingParams

llm = LLM(model="lyrain2001/Auto-Fill-Qwen3-8B-Coding", dtype="bfloat16", max_model_len=40960)
table = pd.read_csv("table.csv", dtype=str).to_markdown(index=False, tablefmt="pipe", disable_numparse=True)
prompt = PROMPT + table   # PROMPT = the user message above, up to and including "Input Table:\n"
text = llm.get_tokenizer().apply_chat_template(
    [{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True)
out = llm.generate([text], SamplingParams(temperature=0.8, max_tokens=32768))
print(out[0].outputs[0].text)

Results

Recall@Precision=0.9 on the Auto-Fill benchmark (200 cases per dataset; from the paper's specialist ablation):

	Pub-XLS	Pub-BI	Pub-Wiki	Gov-CSV	Git-Parquet	Ent-CSV*	Ent-XLS*	Pub-Web	Rel-AR	Rel-FD	Rel-ST	Mean
Coding specialist alone	0.235	0.000	0.110	0.150	0.305	0.215	0.315	0.100	0.955	0.590	0.805	0.344
Auto-Fill (all three + ensemble)	0.525	0.615	0.285	0.500	0.590	0.585	0.660	0.275	0.990	0.890	0.995	0.628

* Ent-CSV / Ent-XLS are proprietary enterprise datasets that are not part of the public benchmark.

Limitations

• Trained and evaluated on English-language tables with one missing cell per table; tables were serialized with at most 40,960 tokens.
• The model can be wrong with high confidence on cells that require knowledge outside the table; use the calibrated confidence and abstain below a threshold, as in the paper.
• Generated code (coding specialist) should be executed in a sandbox.

Citation

@article{liu2026autofill,
  title={Auto-Fill: Learning to Predict Missing Values Accurately with Specialist Language Models},
  author={Liu, Yurong and He, Yeye and Dong, Haoyu and Xing, Junjie and Han, Shi and Zhang, Dongmei and Chaudhuri, Surajit},
  journal={Proceedings of the VLDB Endowment},
  volume={19},
  number={11},
  pages={3160--3173},
  year={2026}
}