--- license: apache-2.0 base_model: Qwen/Qwen3.8-27B base_model_relation: finetune tags: [genexus, code-generation, qwen3] language: [es, en] pipeline_tag: text-generation --- # KBBridge-v3 (bf16) A fine-tune of [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B) specialised in **GeneXus** programming, in the native `.gxSource` export format. Frontier models do not know this format. Without the GeneXus documentation injected into the prompt they produce syntactically invalid output almost every time (parse rate 0.5–3.1%). KBBridge writes it natively, runs on your own hardware, and never sends your Knowledge Base code to an external API. --- ## ⚠️ Read this before your first prompt Three settings. All three are measured on this model, not stylistic. ### 1. Reasoning: it depends on the task, and the difference is large The chat template no longer pins a value, so **reasoning follows the upstream Qwen default (on)** unless your client turns it off. Which one you want depends on what you are asking for. All three rows below are measured on this model, not inherited from Qwen. | Task | Reasoning | Measured | |---|---|---| | **Writing `.gxSource`** | **OFF** | parseRate **86.9 → 33.5**, parmMatch 80.2 → 50.4 (191 held-out objects) | | Documentation multiple-choice | either | 78.4 vs 78.4 — no difference (329 items, McNemar p = 1.000) | | **Explaining existing code** | **ON** | fabricated claims **15.4% → 8.7%** (149 items, McNemar p = 0.041) | **If you generate GeneXus objects, turn reasoning off.** The collapse is real, not a budget artifact: with reasoning on, only 5.2% of items hit the token ceiling (fewer than the 8.4% without it) and 3.7% came back empty. The model simply writes worse `.gxSource` when it reasons first. Writing `.gxSource` is a formatting task. **If you point the model at existing code and ask what it does, turn reasoning on.** It nearly halves the rate at which the model asserts things the source does not support — the failure mode that matters when the output is documentation someone will trust. Cost: ~5× the output tokens. > **Correction (2026-09-02).** An earlier version of this card reported MCQ dropping 78.1 → 69.6 > with reasoning on. That number was wrong: our benchmark harness capped multiple-choice answers > at 512 tokens, which is not enough for a reasoning block to close, so the run was measuring the > cap rather than the model. Re-measured with an adequate budget, the difference is zero. The > `.gxSource` degradation is real and reproduced above with the current scorer. #### How to turn it off ```bash # vLLM — pass it explicitly on every request curl .../v1/chat/completions -d '{ "model": "...", "messages": [...], "chat_template_kwargs": {"enable_thinking": false} }' ``` > **Serve it with the flag, or you will think the model is broken.** If you run vLLM with > `--reasoning-parser qwen3` and the request does **not** carry `enable_thinking`, the parser > assumes reasoning is on, never finds the closing ``, and routes the **entire answer** > into `reasoning`, leaving `content: null`. Every standard OpenAI client then shows an empty > reply. This only affects non-streaming requests — streaming takes a different path in vLLM and > looks fine — which makes it doubly confusing. Either pass `enable_thinking` on every request, > or drop `--reasoning-parser` and let the tags through. To pin reasoning off for every client instead, add this as the first line of `chat_template.jinja`: ```jinja {%- set enable_thinking = false %} ``` A `set` at the top of the template overrides anything the caller passes, which is a blunt but reliable way to guarantee behaviour across runtimes. ### 2. Ask for the format explicitly Write **"in `.gxSource` format"** in your prompt. Measured on v3: the bare request *"a Procedure that adds two numbers"* returns generic **SQL**. Naming the format returns the GeneXus object, consistently. If you use a harness with its own system prompt, put the instruction there once. ### 3. Give it enough room `max_tokens` ≥ 4096. A `.gxSource` object consumes roughly **340 tokens per KB** of source, and most tools default to 512–1024, which truncates the object mid-body. --- ## Results 580 held-out items (191 codegen + 329 MCQ + 60 data-model) that no model saw during training. Syntax validated with the official GeneXus ANTLR parser. Same protocol for every model: temperature 0.1, reasoning off, concurrency 8. ### v3 vs v2 — an honest comparison **v3 is not a clean win over v2.** It gains domain knowledge and loses syntax accuracy: | Metric | v2 | **v3** | | |---|---|---|---| | parseRate (valid syntax) | **89.0** | 84.8 | −4.2 | | parmMatch (exact signature) | 78.6 | 78.6 | = | | MCQ (GeneXus knowledge) | 76.0 | **79.0** | +3.0 | | methodValidity | 90.0 | **91.1** | +1.1 | **What these numbers do NOT establish.** v3 changed three things at once — the base model (Qwen3.6 → 3.8), the corpus (4× larger, per-KB cap removed) and the teacher (v1 → v2). The parseRate drop **cannot be attributed** to any one of them without a control arm that was never run. Anyone reading this table as "the bigger corpus hurt syntax" is over-reading it. Choose v3 if domain knowledge matters more to you; v2 still leads on raw syntax validity. ### Generalisation to unseen Knowledge Bases Three entire KBs were held out — different domains, never in the pipeline: | | held-out from training KBs | 3 completely new KBs | |---|---|---| | v2 | 89.0 | 89.9 | | **v3** | 84.8 | **87.4** | v3's *relative* gap to unseen KBs is larger than v2's (+2.6 vs +0.9), i.e. it generalises better in relative terms, even though two KBs make up 54.7% of its corpus. ### Fairness note on the frontier comparison In our benchmark the frontier models were run **with** ~21,600 tokens of GeneXus documentation injected into every request; KBBridge was run **without** any. That is not a handicap we imposed — injecting the same documentation into KBBridge makes it *worse* (76.4 → 73.3 parseRate), because the fine-tune already internalised that knowledge and the extra context gets in the way. Still, the setups differ, and you should know that when reading any head-to-head number. ### Quantised builds We measured the 4-bit build against this one on the same 580 items. **Excluding items where either run hit the token ceiling, the two are indistinguishable** (parseRate 93.0 vs 93.6 over 171 items) — 4-bit costs essentially nothing in output quality here. Details and the full comparison are in the [GGUF repo's card](https://huggingface.co/KBBridge/KBBridge-v3-GGUF). --- ## Files Full-precision merged weights, bf16, **51 GB** across 19 shards. This is the master artefact: use it to re-quantise, to continue training, or to serve with transformers. ```python from transformers import AutoModelForCausalLM, AutoTokenizer m = AutoModelForCausalLM.from_pretrained("KBBridge/KBBridge-v3", dtype="bfloat16", device_map="auto") t = AutoTokenizer.from_pretrained("KBBridge/KBBridge-v3") ``` For serving, prefer [`KBBridge/KBBridge-v3-FP8`](https://huggingface.co/KBBridge/KBBridge-v3-FP8) (29 GB, same quality in our tests) or the [GGUF builds](https://huggingface.co/KBBridge/KBBridge-v3-GGUF) for llama.cpp / LM Studio. ### What is inside 1,199 tensors: the 64-layer hybrid text model (48 Gated DeltaNet + 16 full-attention layers), the base model's **vision tower** (333 tensors, carried over unchanged — the fine-tune did not touch it) and its **multi-token-prediction head** (15 tensors, likewise unchanged). Context 262,144 tokens, the base model's native `max_position_embeddings`. ## Intended use Assisting GeneXus developers: generating objects (Procedures, Transactions, Data Providers, SDTs, WebPanels), explaining existing code, completion, and documentation questions. **Out of scope:** not a general-purpose model, not a replacement for validating in the GeneXus IDE, and it does not know any particular Knowledge Base (see *Limitations*). --- ## Limitations - **It does not know your KB.** It learned the style and syntax of the format, not the contents of any specific base. Ask it about a transaction you did not paste in, and it will **invent plausible attribute names and present them as fact**. Always give it the context and validate the output in the IDE. - **Runaway generation on very large objects.** For objects over ~10 KB the model can fall into degenerate repetition — the same line hundreds of times without closing the object. Measured on v2 at ~1.6% of benchmark items; **not re-measured on v3**. Raising `max_tokens` does not fix it. Generate large objects section by section. - **Spanish bias** in explanations, reflecting the corpus. - **Specialised**: worse than the base model at general tasks. - The limitations above other than the first were measured on **v2** and are carried over as working assumptions, not verified properties of v3. ### If you also use a hosted KBBridge endpoint The raw GGUF and a gateway-fronted deployment **do not behave the same by default**. Our gateway applies five corrections the plain model does not have: a `max_tokens` floor, reasoning off unless the client asks for it, `temperature` defaulted to 0.2 (without it vLLM falls back to the checkpoint's `generation_config`, which is **1.0**), a fallback that recovers the answer from the `reasoning` field when `content` comes back empty, and `repetition_penalty` 1.05 to suppress runaway. If you compare "what I tried on your server" against "what I downloaded", the difference is those five settings, not the weights. The temperature one surprises people: the OpenAI standard makes the field optional and many clients never send it, so an unconfigured client is sampling at 1.0 without being told. --- ## Training | | | |---|---| | Method | QLoRA 4-bit (bitsandbytes) + Liger kernel | | LoRA | r=64, α=128, dropout=0.05, all projections | | Context | 12,288 tokens | | Effective batch | 16 (1 × 16 grad accum) | | LR | 1.0e-4, cosine, 3% warmup | | Epochs | 2 complete (14,108 steps) | | Hardware | 1× RTX PRO 6000 Blackwell 96 GB | | Duration | 7 days 4:41 | | Framework | LLaMA-Factory, transformers 5.6.0 | train_loss **0.2618** (v2: 0.3344) · eval_loss **0.3723** (v2: 0.4675), minimum at the **last** step — no overfitting across 71 evaluations, which suggests there was room for more epochs. Note that these losses are much better than v2's and yet parseRate went *down*: `eval_loss` measures fit to the corpus, not GeneXus quality. ### Data 80,344 examples derived from GeneXus objects across 25 real Knowledge Bases (GX16/17/17U8/18/ Evo1, multi-domain) — 129% more than v2, with the per-KB cap removed. Sanitised, deduplicated and split by deterministic hash. **The datasets are not published**: they contain customer proprietary code. --- ## Training-data privacy The model was trained on real customer Knowledge Bases, so we audited whether it can leak them. This is the strongest result of the project. ### Canaries: no memorisation threshold found 12 synthetic objects containing unguessable 16-character secrets were inserted at four frequencies, and verified to have reached `train.jsonl` at exactly those counts: | repetitions | canaries | recovered by name | recovered with literal prefix | |---|---|---|---| | 1 | 3 | 0/3 | 0/3 | | 10 | 3 | 0/3 | 0/3 | | 100 | 3 | 0/3 | 0/3 | | **1000** | 3 | **0/3** | **0/3** | **Not even at a thousand identical repetitions.** A control rules out a broken probe: asked for the canary, the model returns a structurally valid but **empty** object — no token, no secret. And it does generate real bodies when the request has content, so the empty skeleton is not an inability to generate. ### Membership inference: marginal signal | | | |---|---| | mean loss, seen examples | 3.4130 | | mean loss, unseen | 3.7711 | | mean length | 3,133 vs 3,117 chars — comparable, so the AUC is meaningful | | **AUC** | **0.5539** | 0.554 against 0.50 for indistinguishable. There is a statistical trace of having seen the data, but the distributions overlap almost entirely. **Conclusion: customer code is not recoverable from the weights.** **Caveat, stated plainly:** absence of evidence is not proof of absence. These audits cover the attacks we ran, not every attack that exists. --- ## Reproducibility Full external reproduction is **not possible**, and it is worth saying so directly: 1. The 25 Knowledge Bases are customer code and are not distributed. 2. The `parseRate` scorer uses the KBEditor's ANTLR parser — proprietary, not distributable. 3. The teacher that generated v3's data is KBBridge-v2, which is not published. What a third party *can* verify: the raw benchmark outputs (one model response per item) and the scoring over them. --- ## Citation ```bibtex @misc{kbbridge-v3, title = {KBBridge-v3: a GeneXus code assistant fine-tuned from Qwen3.8-27B}, author = {{KBBridge}}, year = {2026}, url = {https://huggingface.co/KBBridge/KBBridge-v3} } ``` ## License Apache 2.0, inherited from the base model `Qwen/Qwen3.8-27B`. This is a modified derivative work; see `NOTICE`.