bert4jev

A typed-decision model built on microsoft/deberta-v3-large.

One state and any number of typed questions go in; a calibrated probability distribution per question comes back from a single forward pass. Nothing is generated, so the structured-output error rate is 0 by construction.

Three question kinds:

kind	meaning	output
choice	one of N unordered options (2..255)	choice, probabilities, confidence
score	one of N ordered levels (2..10)	score (expected level), probabilities, confidence
noul	yes / no	noul = p(yes)

This repository ships float16 weights only (backbone + head), plus a small self-contained inference module (jev_infer.py). Total size: ~886 MB.

---

How it works

The input is serialized into a single sequence with three marker tokens:

[CLS] [STATE] state [Q] instructions [OPT] option_1 [OPT] option_2 ... [Q] ... [SEP]

A DeBERTa-v3-large encoder reads the state and all questions at once. For every option, a small head scores [mean(question tokens); mean(option tokens); product]; a softmax within each question's option group is that question's distribution. One forward pass answers every question.

Context: 512 tokens total, the state is truncated to 256 tokens.

---

Usage

pip install torch transformers safetensors huggingface_hub sentencepiece protobuf

from jev_infer import Bert4Jev

model = Bert4Jev.from_pretrained("ukung/bert4jev")

result = model.decide(
    "My running shoes arrived in the wrong size. Can I swap them for a size 10?",
    [{
        "type": "choice",
        "instructions": "Which team should handle this?",
        "options": ["returns", "shipping", "billing"],
    }],
)
print(result)
# [{'choice': 'returns',
#   'probabilities': {'returns': 0.8413, 'shipping': 0.0976, 'billing': 0.0611},
#   'confidence': 0.8413}]

Multiple questions on the same state cost one pass:

result = model.decide(
    "I was charged twice for the same order and nobody answers my emails.",
    [{"type": "choice", "instructions": "Which product area is the message about?",
      "options": ["fees & charges", "pin & security", "refund & dispute", "card", "other"]},
     {"type": "score", "instructions": "How positive is the sentiment of this message?",
      "options": ["very negative", "negative", "neutral", "positive", "very positive"]},
     {"type": "noul", "instructions": "The customer is asking for a refund."}],
)

Standard AutoModel interface

The repository ships the custom code (configuration_bert4jev.py, modeling_bert4jev.py) and registers it in config.json via auto_map. Passing trust_remote_code=True makes the standard Transformers entry points return the full decision model — no jev_infer.py needed:

from transformers import AutoConfig, AutoModel

config = AutoConfig.from_pretrained("ukung/bert4jev", trust_remote_code=True)
model = AutoModel.from_pretrained("ukung/bert4jev", trust_remote_code=True)

result = model.decide(
    "My running shoes arrived in the wrong size. Can I swap them for a size 10?",
    [{
        "type": "choice",
        "instructions": "Which team should handle this?",
        "options": ["returns", "shipping", "billing"],
    }],
)
print(result)
# [{'choice': 'returns',
#   'probabilities': {'returns': 0.8412, 'shipping': 0.0976, 'billing': 0.0612},
#   'confidence': 0.8412}]

AutoModel.from_pretrained(...) (without trust_remote_code) still works and returns the bare DebertaV2Model backbone, because model_type remains deberta-v2.

This path was verified to reproduce the same results as jev_infer.py: 33-case accuracy 0.879 with the same 4 errors and negation 6/6.

Standard AutoModel interface

The repository ships the custom code (configuration_bert4jev.py, modeling_bert4jev.py) and registers it in config.json via auto_map. Passing trust_remote_code=True makes the standard Transformers entry points return the full decision model — no jev_infer.py needed:

from transformers import AutoConfig, AutoModel

config = AutoConfig.from_pretrained("ukung/bert4jev", trust_remote_code=True)
model = AutoModel.from_pretrained("ukung/bert4jev", trust_remote_code=True)

result = model.decide(
    "My running shoes arrived in the wrong size. Can I swap them for a size 10?",
    [{
        "type": "choice",
        "instructions": "Which team should handle this?",
        "options": ["returns", "shipping", "billing"],
    }],
)
print(result)
# [{'choice': 'returns',
#   'probabilities': {'returns': 0.8412, 'shipping': 0.0976, 'billing': 0.0612},
#   'confidence': 0.8412}]

AutoModel.from_pretrained(...) (without trust_remote_code) still works and returns the bare DebertaV2Model backbone, because model_type remains deberta-v2.

This path was verified to reproduce the same results as jev_infer.py: 33-case accuracy 0.879 with the same 4 errors and negation 6/6.

Answering a structured request

The model reads a state and a set of named questions. Map every question's option set (the keys of a criteria map, or an explicit option list) to options:

import json
from jev_infer import Bert4Jev

model = Bert4Jev.from_pretrained("ukung/bert4jev")

request = {
  "state": "My running shoes arrived in the wrong size. Can I swap them for a size 10?",
  "questions": {
    "department": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "returns": "Exchanges, refunds, wrong or damaged items",
        "shipping": "Delivery status, delays, lost packages",
        "billing": "Charges, invoices, payment problems"
      }
    }
  }
}

answers = {}
for qid, q in request["questions"].items():
    options = list(q["criteria"].keys()) if "criteria" in q else list(q["options"])
    r = model.decide(request["state"],
                     [{"type": q["type"], "instructions": q["instructions"], "options": options}])[0]
    answers[qid] = {
        "type": q["type"],
        "choice": r["choice"],
        "confidence": round(r["confidence"], 4),
        "probabilities": {k: round(v, 4) for k, v in r["probabilities"].items()},
    }

print(json.dumps({"model": "bert4jev", "answers": answers}, indent=2))

---

Evaluation

All numbers below were measured on a single GPU (fp16) with the released fp16 weights, using the exact loader shipped in this repository.

Task

Customer-support routing: a free-text message must be classified into one of three departments.

{
  "returns":  "Exchanges, refunds, wrong or damaged items",
  "shipping": "Delivery status, delays, lost packages",
  "billing":  "Charges, invoices, payment problems"
}

Test sets

set	n	description
Easy	18	clear messages, each with an explicit keyword ("refund", "late", "charged twice")
Hard	15	negation, category overlap, thin signals (#5521, asdfgh), multi-intent
Negation probe	6	messages that explicitly deny a category while asking for another
Total	33	Easy + Hard

Results

metric	value
Easy set accuracy	18/18 = 100%
Hard set accuracy	11/15 = 73%
Total accuracy (33)	29/33 = 87.9%
Negation probe	6/6 = 100%

Calibration

confidence is the maximum probability after temperature scaling (T = 1.05). Measured over the 33-case set:

confidence bin	n	avg confidence	accuracy	gap
0.0 - 0.4	2	0.374	0.500	-0.126
0.4 - 0.6	9	0.496	0.778	-0.282
0.6 - 0.8	5	0.741	0.800	-0.059
0.8 - 1.0	17	0.876	1.000	-0.124

Every gap is negative: the model is slightly under-confident, never over-confident, on this set. Accuracy rises monotonically with confidence.

Selective prediction

Abstaining below a confidence threshold (route to a human instead):

threshold	kept	coverage	accuracy on kept
0.0	33	100%	87.9%
0.5	26	79%	96.2%
0.6	22	67%	95.5%
0.7	21	64%	95.2%
0.8	17	52%	100%

At threshold 0.8 the model is 100% accurate on the 52% of cases it answers.

Negation

Negation is the classic failure mode of similarity-based classifiers: "I do NOT want a refund" and "I want a refund" are topically identical. This model handles it:

message	prediction	confidence	correct
I don't have a billing issue, the item never showed up.	shipping	0.850	yes
I do NOT want a refund, I just want a replacement.	returns	0.569	yes
This is not a shipping problem, my card was overcharged.	billing	0.919	yes
I'm not asking for a refund, I need the tracking info.	shipping	0.759	yes
Do not charge me again, this is about a lost parcel.	shipping	0.664	yes
There is no billing problem, I want to return this.	returns	0.930	yes

Remaining errors (4 of 33)

message	gold	predicted	confidence
The item arrived damaged and I was also charged extra.	returns	billing	0.355
order 88231	returns	shipping	0.465
asdfgh	billing	shipping	0.425
Late package and I want a refund for it.	shipping	returns	0.770

The first three have low confidence and are caught by the abstain threshold. The fourth is a genuine multi-intent disagreement.

---

Precision

The weights are float16 (backbone and head, homogeneous dtype). Verified lossless:

metric	float32	float16
33-case accuracy	87.9%	87.9%
Negation probe	6/6	6/6
Same-case confidence (example)	0.841	0.8413
Backbone size	1736 MB	868 MB

Same predictions, same errors, same confidences.

---

Limitations

• Reads the question only partly; performance on unseen instructions and option sets is lower.
• English only.
• 512-token context.
• The confidence is calibrated on a validation split; re-calibrate on your own data before trusting the thresholds in production.
• It cannot generate text or arguments; it chooses among the options you give it.
• Options must be short labels or phrases. Very long or noisy option text degrades accuracy.

Files

file	description
model.safetensors	DeBERTa-v3-large backbone, float16
head.safetensors	3-layer scoring head, float16
config.json	backbone config (dtype: float16)
open_jev_config.json	pool mode, temperature, limits
tokenizer.json, spm.model, ...	tokenizer with the three marker tokens
jev_infer.py	self-contained inference module

Base model

Built on microsoft/deberta-v3-large (MIT).