| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen2.5-7B-Instruct |
| tags: |
| - code |
| - code-review |
| - programming |
| - qwen2.5 |
| - bug-detection |
| - mlx |
| datasets: |
| - sahil2801/CodeAlpaca-20k |
| language: |
| - en |
| pipeline_tag: text-generation |
| library_name: transformers |
| model-index: |
| - name: CodeLens-7B |
| results: [] |
| --- |
| |
| # [CodeLens-7B-MLX](https://huggingface.co/xunker/CodeLens-7B-MLX) |
|
|
| MLX version of [sriksven/CodeLens-7B](https://huggingface.co/sriksven/CodeLens-7B) in various oQ levels and dtypes. |
|
|
| Directory | oQ Level | dtype | size |
| ----------------------------------------------|----------|----------|------ |
| [CodeLens-7B-oQ4-bf16](CodeLens-7B-oQ4-bf16/) | 4-bit | bfloat16 | 4.2GB |
| [CodeLens-7B-oQ4-fp16](CodeLens-7B-oQ4-fp16/) | 4-bit | fp16 | 4.2GB |
| [CodeLens-7B-oQ5-bf16](CodeLens-7B-oQ5-bf16/) | 5-bit | bfloat16 | 5.1GB |
| [CodeLens-7B-oQ5-fp16](CodeLens-7B-oQ5-fp16/) | 5-bit | fp16 | 5.1GB |
| [CodeLens-7B-oQ6-bf16](CodeLens-7B-oQ6-bf16/) | 6-bit | bfloat16 | 5.9GB |
| [CodeLens-7B-oQ6-fp16](CodeLens-7B-oQ6-fp16/) | 6-bit | fp16 | 5.9GB |
| [CodeLens-7B-oQ8-bf16](CodeLens-7B-oQ8-bf16/) | 8-bit | bfloat16 | 7.5GB |
| [CodeLens-7B-oQ8-fp16](CodeLens-7B-oQ8-fp16/) | 8-bit | fp16 | 7.5GB |
|
|
| ## Why choose FP16 over BFLOAT16/BF16? |
|
|
| On older Apple Silicon (M1 and M2), fp16 can be faster. Here are the details from [Muhammad Raza](https://muhammadraza.me/2026/gguf-vs-mlx-decision-guide/#two-traps-that-will-flip-your-results): |
|
|
| > A lot of MLX builds ship as bf16, and **on the M1 and M2 that data type does not get the accelerated path that fp16 does**. During prefill those weights run un-accelerated and the penalty multiplies across every input token, which is part of why some “MLX is slow” reports come from older hardware. [...] |
| > |
| > If you are on an M1 or M2 and MLX feels sluggish, check this before you blame the format. |
|
|
| ### Test Results |
|
|
| Using oQ6, here are the results from oMLX 0.4.4 on a Macbook Pro 2021 (M1 Pro). |
|
|
| **tl;dr**: |
|
|
| Time to First Token (TTFT) and Prompt Processing Tokens Per Second |
| (ppTPS, aka "prefill speed") are about 60% faster when using FP16. |
|
|
| However, Token Generation (tgTPS) only increases moderately, around 1-2%. |
|
|
| #### BFLOAT16 |
|
|
| ##### Single request results |
| Test | TTFT(ms) | TPOT(ms) | ppTPS | tgTPS | E2E(s) | Throughput | PeakMem |
| ------------------|----------|----------|-------|-------|--------|------------|-------- |
| pp 4096 / tg 128 | 24727.9 | 39.3 | 165.6 | 25.7 | 29.7 | 142.1 | 6.84 GB |
| pp 16384 / tg 128 | 111811.1 | 48.4 | 146.5 | 20.8 | 118.0 | 140.0 | 7.69 GB |
|
|
| ##### Batch results |
| Batch | tgTPS | ppTPS | avgTTFT(ms) | E2E(s) | Speedup |
| ------------|-------|-------|-------------|--------|-------- |
| 1x baseline | 25.7 | 165.6 | 24727.9 | 29.7 | 1.00x |
| 2x | 30.0 | 164.0 | 12487.3 | 21.0 | 1.17x |
| 4x | 31.9 | 239.7 | 16885.7 | 33.1 | 1.24x |
|
|
| #### FP16 |
|
|
| ##### Single request results |
| Test | TTFT(ms) | TPOT(ms) | ppTPS | tgTPS | E2E(s) | Throughput | PeakMem |
| ------------------|----------|----------|-------|-------|--------|------------|-------- |
| pp 4096 / tg 128 | 15226.8 | 37.3 | 269.0 | 27.0 | 20.0 | 211.6 | 6.84 GB |
| pp 16384 / tg 128 | 69595.4 | 45.6 | 235.4 | 22.1 | 75.4 | 219.0 | 7.69 GB |
|
|
| ##### Batch results |
| Batch | tgTPS | ppTPS | avgTTFT(ms) | E2E(s) | Speedup |
| ------------|-------|-------|-------------|--------|-------- |
| 1x baseline | 27.0 | 269.0 | 15226.8 | 20.0 | 1.00x |
| 2x | 36.9 | 266.6 | 7681.4 | 14.6 | 1.37x |
| 4x | 38.0 | 363.3 | 11084.7 | 24.8 | 1.41x |
|
|
| ## Hardware and Software |
|
|
| These were converted to MLX using [oMLX](https://github.com/jundot/omlx) [0.4.4](https://github.com/jundot/omlx/releases/tag/v0.4.4) on a 32GB Macbook Pro 2021 (M1 Pro). I cleared all my RAM so you don't have to. |
|
|
| ## License |
|
|
| Apache 2.0, as per [original model](https://huggingface.co/sriksven/CodeLens-7B). |
