README.md

metadata

license: apache-2.0
base_model:
  - OrionLLM/OxCoder-9B
tags:
  - text-generation-inference
  - quantized
  - fp8
  - vllm
  - coder
  - 9b
  - agentic
  - coding
language:
  - en
pipeline_tag: image-text-to-text
library_name: transformers

OxCoder-9B-FP8

This repository contains an FP8 dynamic quantized version of OrionLLM/OxCoder-9B (built on Qwen/Qwen3.5-9B), optimized for high-throughput inference and reduced VRAM footprint using llm-compressor and the compressed-tensors format. OxCoder-9B is a compact, frontier-class coding and reasoning model tailored for long-horizon agentic workflows, complex terminal operations, multi-file codebases, and interactive software development.

Model Summary

Attribute	Details
Base Model	OrionLLM/OxCoder-9B (Foundation: Qwen/Qwen3.5-9B)
Quantized Model	prithivMLmods/OxCoder-9B-fp8
Quantization Method	LLM Compressor
Quantization Scheme	FP8_DYNAMIC
Output Format	compressed-tensors
Native Context Length	262,144 tokens (262K)
License	Apache-2.0

Quantization Details

The model was quantized to FP8 using llm-compressor with dynamic per-tensor activation scaling. Sensitive architecture components—including the language model head, input embeddings, vision modules, and linear attention layers—were excluded to preserve fidelity, reasoning stability, and code generation precision.

Quantization Recipe

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore:
        - 're:.*lm_head'
        - 're:.*embed_tokens$'
        - 're:.*visual.*'
        - 're:.*model.visual.*'
        - 're:.*linear_attn.*'
      scheme: FP8_DYNAMIC
      bypass_divisibility_checks: false
      requires_calibration_data: false

• Targeted Layers: All standard Linear projections (MLP and attention projections).
• Excluded Layers: lm_head, embed_tokens, visual, model.visual, and linear_attn.
• Calibration Required: No (utilizes runtime dynamic activation scaling).

Deployment & Inference

1. vLLM (Recommended)

compressed-tensors FP8 checkpoints run natively in vLLM:

pip install vllm

Launch an OpenAI-compatible API server:

vllm serve prithivMLmods/OxCoder-9B-fp8 \
    --max-model-len 65536 \
    --trust-remote-code

Or execute via Python:

from vllm import LLM, SamplingParams

sampling_params = SamplingParams(
    temperature=0.6,
    top_p=0.95,
    max_tokens=4096
)

llm = LLM(
    model="prithivMLmods/OxCoder-9B-fp8",
    trust_remote_code=True,
    max_model_len=65536
)

prompts = [
    "Write a Python script using asyncio to run a rate-limited web scraper with exponential backoff."
]

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    print(output.outputs[0].text)

2. Transformers & Compressed Tensors

pip install transformers compressed-tensors accelerate

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "prithivMLmods/OxCoder-9B-fp8"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto",
    trust_remote_code=True
)

prompt = "Implement a LRU cache with O(1) runtime for get and put operations in Python."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output_tokens = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.6,
        top_p=0.95,
        do_sample=True
    )

print(tokenizer.decode(output_tokens[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Attribution & License

• Base Architecture & Weights: Developed by OrionLLM based on Qwen/Qwen3.5-9B.
• Quantization: Prepared and hosted by prithivMLmods.
• License: Released under the Apache 2.0 License.