zerank-1-small FP8 (dynamic)

FP8 quantization of zeroentropy/zerank-1-small in compressed-tensors format for vLLM.

Scheme: FP8_DYNAMIC. Linear weights are FP8 E4M3 with static per-channel scales, and activations are quantized per token at runtime.
embed_tokens (tied, so it also produces the "Yes" scoring logit), lm_head and all norms are kept in BF16 and verified byte-identical to the original.
Made with llm-compressor 0.13.0 model_free_ptq (data-free) and compressed-tensors 0.18.0.
vLLM weight memory is about 1.9 GiB.

Serving (the score template is required)

zerank scores a (query, document) pair by the "Yes" logit after a Qwen3 chat prompt, with the query as the system message and the document as the user message. This repo includes score_template.jinja, which reproduces that format for vLLM's score and rerank API. Without it, vLLM just concatenates the query and document, and the scores are meaningless even though requests still succeed.

vllm serve sininspira/zerank-1-small-FP8-Dynamic --runner pooling --convert classify \
  --hf-overrides '{"method":"no_post_processing","classifier_from_token":["Yes"],"num_labels":1,"use_sep_token":false}' \
  --chat-template <local-path-to-this-repo>/score_template.jinja \
  --max-model-len 2048

Send "use_activation": false in /v1/rerank requests to get raw "Yes" logits, which is the scale zerank documents. To map a score to 0–1, use sigmoid(score / 5).

Sanity check: for the query "What is 2+2?" and the documents "4", "The answer is definitely 1 million" and "Paris is the capital of France.", this build scores 3.80, -3.79 and -7.07, with 75 prompt tokens in total.

Evaluation (BF16 vs FP8)

Tested with vLLM 0.29.0 on an NVIDIA Blackwell GPU. The candidate lists are the top 20 retrieved by BF16 Qwen3-Embedding-4B for 300 BEIR SciFact test queries. The first column compares two BF16 runs with a server restart between, which shows vLLM's run-to-run noise.