README.md

metadata

language:
  - ko
license: apache-2.0
tags:
  - task-specific
  - structured-prediction
  - korean
  - public-sector
  - qwen3
  - domain-specific
  - merge
base_model: Qwen/Qwen3-4B
datasets: []
pipeline_tag: text-generation
model-index:
  - name: DLM-NL2JSON-4B
    results:
      - task:
          type: structured-prediction
          name: Korean NL-to-JSON Schema Extraction
        dataset:
          type: custom
          name: Busan Public Data Query Test Set
          args:
            num_samples: 2041
        metrics:
          - type: exact_match
            value: 94.4
            name: Exact Match Accuracy (raw)
          - type: exact_match
            value: 96.8
            name: Exact Match Accuracy (adjusted)

DLM-NL2JSON-4B

A 4B-parameter service-specific LLM that outperforms GPT-4o (+14%p) and Qwen3.5-35B (+22%p) on structured JSON extraction from Korean natural language queries.

DLM (Domain-specific Language Model) is a series of task-specialized models by Data Science Lab., Ltd.. This model is a LoRA-merged Qwen3-4B fine-tuned for structured JSON extraction in the Busan Metropolitan City public data analytics service.

Key Results

Evaluated on 2,041 test samples across 10 task categories (field-level exact match, summary excluded):

Model	Params	Accuracy	Accuracy (adj*)	Avg Latency
DLM-NL2JSON-4B	4B	94.4%	96.8%	2.59s
GPT-4o	~200B+	80.5%	82.5%	1.58s
Qwen3.5-35B-A3B	35B	72.2%	73.9%	0.85s

*adj: 64 CSM samples with known gold label noise excluded (see Evaluation section)

Per-Category Breakdown

Category	N	DLM-NL2JSON-4B	GPT-4o	Qwen3.5-35B
ALP-A (population pattern)	250	99.6%	56.0%	47.6%
ALP-B (population flow)	250	98.4%	50.4%	46.8%
CSM (consumer spending)	700	90.6%	90.1%	86.1%
CREDIT-Income	58	94.8%	53.4%	34.5%
CREDIT-Spending	77	97.4%	92.2%	51.9%
CREDIT-Loan/Default	73	98.6%	94.5%	72.6%
CPI (business status)	219	86.3%	87.2%	54.8%
GIS-Inflow	72	97.2%	79.2%	93.1%
GIS-Outflow	62	98.4%	77.4%	98.4%
GIS-Consumption	280	98.2%	99.6%	97.5%

DLM-NL2JSON-4B wins 8 out of 10 categories, with the largest gains on ALP (+43%p vs GPT-4o) and CREDIT-Income (+41%p).

Important: This is a Service-Specific Model

This model is NOT a general-purpose NL-to-JSON converter. It is trained exclusively for a fixed set of predefined schemas used in a specific production service. It will not generalize to arbitrary JSON schemas or different prompt formats.

To use this model correctly, you must:

1. Use the exact system prompts it was trained on (one per task category — see Usage section)
2. Include the corresponding special token (<TASK_CSM>, <TASK_CREDIT>, <TASK_GIS>, <TASK_ALP>, <TASK_CPI>) in the input
3. Expect output conforming only to the predefined schemas listed below

Why publish a service-specific model? This model serves as a reference implementation demonstrating that task-specific LoRA fine-tuning on a 4B model can dramatically outperform GPT-4o and larger open-source models on constrained structured output tasks. We believe the DLM (Domain-specific Language Model) approach — training small, cheap-to-serve models for specific service endpoints — is an underexplored but highly practical paradigm.

Intended Use

This model converts Korean natural language queries about public/economic data into structured JSON conforming to its predefined schemas. It is designed for and deployed in the Busan Metropolitan City Big Data Wave analytics dashboard.

Input: Free-form Korean query + task-specific system prompt

Output: Single-line JSON with exact schema compliance:

{"summary":"##2025년 5월 부산광역시 해운대구 유통/의료 소비분석##","base_ym":202505,"region_nm":"부산광역시 해운대구","industry_select":{"3":[],"8":[]},"sex_cd":[1],"age_cd":[30],"category":2}

Task Categories

ID	Name	Schema Type
0	ALP-A	Population pattern (ptrn: residence/work/visit)
1	ALP-B	Population flow (flow_cd: inflow/outflow)
2	CSM	Consumer spending by industry
3	CREDIT-Income	Income statistics
4	CREDIT-Spending	Spending statistics
5	CREDIT-Loan	Loan/default statistics
6	CPI	Business/enterprise status
9	GIS-Inflow	Geographic inflow analysis
10	GIS-Outflow	Geographic outflow analysis
11	GIS-Consumption	Geographic consumption analysis

Training Details

Item	Value
Base model	Qwen/Qwen3-4B
Method	LoRA SFT → merged full model
Training samples	16,292 (Korean)
Validation samples	2,034
Special tokens	<TASK_CSM>, <TASK_CREDIT>, <TASK_GIS>, <TASK_ALP>, <TASK_CPI>
Max sequence length	6,144
Architecture	Qwen3ForCausalLM (36 layers, 2560 hidden, 32 heads)

Training data consists of synthetically generated Korean natural language queries paired with structured JSON outputs, covering the Busan public data analytics domain.

Evaluation Methodology

• Metric: Field-level exact match — each JSON key's value is compared against the gold label. The summary field is excluded from comparison.
• Test set: 2,041 samples, stratified by category
• Gold label noise: 64/700 CSM samples have age_cd capped at [10..60] instead of [10..70] for "all ages" queries, conflicting with the prompt specification. These affect all models equally and are excluded in the adjusted metric.
• Train/Test overlap: 16/2,041 input strings (0.78%) appear in both sets — retained for consistency.
• All models received identical system prompts per category.

Hardware

Model	Serving	GPU
DLM-NL2JSON-4B	TensorRT-LLM	NVIDIA L4 24GB
GPT-4o	OpenAI API	N/A
Qwen3.5-35B-A3B	vLLM	NVIDIA A6000 48GB

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "dataslab/DLM-NL2JSON-4B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

# System prompt (example: CSM consumer spending schema — abbreviated for readability)
# Full prompts per category are available in the repository's eval/prompts.py
system_prompt = """너는 반드시 **JSON 한 줄**만 출력한다. 설명/텍스트/코멘트/마크다운/코드블록/이모지/공백 줄 금지.
출력은 항상 { 로 시작하고 } 로 끝난다.

[스키마: TASK_CSM] (키/타입/순서 엄수)
{"summary":string,"base_ym":int,"region_nm":string,"industry_select":object,"sex_cd":[int],"age_cd":[int],"category":2}

[기본값]
- base_ym: 0, region_nm: "부산광역시"
- industry_select: 업종 미지정 시 전 대분류 키를 []로 설정
- sex_cd: [0,1], age_cd: [10,20,30,40,50,60,70]
- category: 항상 2

[대분류 코드표] 1:여행/숙박 2:여가/문화 3:유통 4:음식/주점 5:음식료품
6:의류/잡화 7:미용 8:의료 9:교육 10:생활 11:자동차"""

# Note: special token <TASK_CSM> must be included in the user message
user_query = "<TASK_CSM> 2024년 1월 해운대구 중동 의류/잡화랑 뷰티 쪽 남성 20~40대 위주로 알려줘"

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_query}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.0, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True))
# {"summary":"##2024년 1월 부산광역시 해운대구 중동 의류/잡화/미용 소비분석##","base_ym":202401,"region_nm":"부산광역시 해운대구 중동","industry_select":{"6":[],"7":[]},"sex_cd":[0],"age_cd":[20,30,40],"category":2}
# Note: "뷰티" → mapped to 미용(code 7), "해운대구 중동" → normalized to "부산광역시 해운대구 중동"

vLLM / OpenAI-compatible serving

from openai import OpenAI

client = OpenAI(base_url="http://your-server:8006/v1", api_key="token")
resp = client.chat.completions.create(
    model="DLM-NL2JSON-4B",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": "<TASK_CSM> 2024년 1월 해운대구 중동 의류/잡화랑 뷰티 쪽 남성 20~40대 위주로 알려줘"}
    ],
    max_tokens=512,
    temperature=0.0,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}  # disable thinking mode
)
print(resp.choices[0].message.content)

Important: When serving with vLLM/TensorRT-LLM, pass chat_template_kwargs: {"enable_thinking": false} to disable the Qwen3 thinking mode. Otherwise, reasoning tokens will consume the output budget and truncate the JSON.

Known Limitations

1. CPI category (86.3%) is the weakest — complex industry classification codes (A~U with sub-codes) are harder to extract.
2. CSM training data noise: ~8% of CSM training samples have age_cd capped at 60 instead of 70 for "all ages" queries, introducing inconsistency.
3. Domain-specific only: This model is trained exclusively for the Busan public data schema extraction task. It has no general-purpose capabilities and should not be used as a general chatbot.
4. Korean only: All training data and prompts are in Korean.

Citation

If you use this model, please cite:

@misc{dsl-dlm-nl2json-4b,
  title={DLM-NL2JSON-4B: A Domain-Specific Language Model for Korean Public Data Schema Extraction},
  author={Data Science Lab., Ltd.},
  year={2026},
  url={https://huggingface.co/dataslab/DLM-NL2JSON-4B}
}

Contact

• Organization: Data Science Lab., Ltd.
• Project: Busan Metropolitan City Big Data Wave