README.md

---
license: apache-2.0
base_model:
- OrionLLM/OxCoder-9B
tags:
- text-generation-inference
- quantized
- fp8
- vllm
- coder
- 9b
- agentic
- coding
language:
- en
pipeline_tag: image-text-to-text
library_name: transformers
---

# **OxCoder-9B-FP8**

> This repository contains an FP8 dynamic quantized version of **[OrionLLM/OxCoder-9B](https://huggingface.co/OrionLLM/OxCoder-9B)** (built on `Qwen/Qwen3.5-9B`), optimized for high-throughput inference and reduced VRAM footprint using `llm-compressor` and the `compressed-tensors` format. OxCoder-9B is a compact, frontier-class coding and reasoning model tailored for long-horizon agentic workflows, complex terminal operations, multi-file codebases, and interactive software development.

## Model Summary

| Attribute | Details |
|---|---|
| **Base Model** | [OrionLLM/OxCoder-9B](https://huggingface.co/OrionLLM/OxCoder-9B) (Foundation: `Qwen/Qwen3.5-9B`) |
| **Quantized Model** | `prithivMLmods/OxCoder-9B-fp8` |
| **Quantization Method** | [LLM Compressor](https://github.com/vllm-project/llm-compressor) |
| **Quantization Scheme** | `FP8_DYNAMIC` |
| **Output Format** | `compressed-tensors` |
| **Native Context Length** | 262,144 tokens (262K) |
| **License** | Apache-2.0 |

## Quantization Details

The model was quantized to FP8 using `llm-compressor` with dynamic per-tensor activation scaling. Sensitive architecture components—including the language model head, input embeddings, vision modules, and linear attention layers—were excluded to preserve fidelity, reasoning stability, and code generation precision.

### Quantization Recipe

```yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore:
        - 're:.*lm_head'
        - 're:.*embed_tokens$'
        - 're:.*visual.*'
        - 're:.*model.visual.*'
        - 're:.*linear_attn.*'
      scheme: FP8_DYNAMIC
      bypass_divisibility_checks: false
      requires_calibration_data: false
```

* **Targeted Layers:** All standard `Linear` projections (MLP and attention projections).
* **Excluded Layers:** `lm_head`, `embed_tokens`, `visual`, `model.visual`, and `linear_attn`.
* **Calibration Required:** No (utilizes runtime dynamic activation scaling).

## Deployment & Inference

### 1. vLLM (Recommended)

`compressed-tensors` FP8 checkpoints run natively in [vLLM](https://github.com/vllm-project/vllm):

```bash
pip install vllm
```

Launch an OpenAI-compatible API server:

```bash
vllm serve prithivMLmods/OxCoder-9B-fp8 \
    --max-model-len 65536 \
    --trust-remote-code
```

Or execute via Python:

```python
from vllm import LLM, SamplingParams

sampling_params = SamplingParams(
    temperature=0.6,
    top_p=0.95,
    max_tokens=4096
)

llm = LLM(
    model="prithivMLmods/OxCoder-9B-fp8",
    trust_remote_code=True,
    max_model_len=65536
)

prompts = [
    "Write a Python script using asyncio to run a rate-limited web scraper with exponential backoff."
]

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    print(output.outputs[0].text)
```

### 2. Transformers & Compressed Tensors

```bash
pip install transformers compressed-tensors accelerate
```

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "prithivMLmods/OxCoder-9B-fp8"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto",
    trust_remote_code=True
)

prompt = "Implement a LRU cache with O(1) runtime for get and put operations in Python."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output_tokens = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.6,
        top_p=0.95,
        do_sample=True
    )

print(tokenizer.decode(output_tokens[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```

## Attribution & License

* **Base Architecture & Weights:** Developed by [OrionLLM](https://huggingface.co/OrionLLM) based on `Qwen/Qwen3.5-9B`.
* **Quantization:** Prepared and hosted by `prithivMLmods`.
* **License:** Released under the [Apache 2.0 License](https://www.google.com/search?q=LICENSE).