| | --- |
| | license: apache-2.0 |
| | base_model: |
| | - rednote-hilab/dots.ocr |
| | base_model_relation: quantized |
| | --- |
| | # deepseek-ocr.rs 🚀 |
| |
|
| | Rust implementation of the DeepSeek-OCR inference stack with a fast CLI and an OpenAI-compatible HTTP server. The workspace packages multiple OCR backends, prompt tooling, and a serving layer so you can build document understanding pipelines that run locally on CPU, Apple Metal, or (alpha) NVIDIA CUDA GPUs. |
| |
|
| | > 中文文档请看 [README_CN.md](README_CN.md)。 |
| |
|
| | > Want ready-made binaries? Latest macOS (Metal-enabled) and Windows bundles live in the [build-binaries workflow artifacts](https://github.com/TimmyOVO/deepseek-ocr.rs/actions/workflows/build-binaries.yml). Grab them from the newest green run. |
| |
|
| | ## Choosing a Model 🔬 |
| |
|
| | | Model | Memory footprint* | Best on | When to pick it | |
| | | --- | --- | --- | --- | |
| | | **DeepSeek‑OCR** | **≈6.3GB** FP16 weights, **≈13GB** RAM/VRAM with cache & activations (512-token budget) | Apple Silicon + Metal (FP16), high-VRAM NVIDIA GPUs, 32GB+ RAM desktops | Highest accuracy, SAM+CLIP global/local context, MoE DeepSeek‑V2 decoder (3B params, ~570M active per token). Use when latency is secondary to quality. | |
| | | **PaddleOCR‑VL** | **≈4.7GB** FP16 weights, **≈9GB** RAM/VRAM with cache & activations | 16GB laptops, CPU-only boxes, mid-range GPUs | Dense 0.9B Ernie decoder with SigLIP vision tower. Faster startup, lower memory, great for batch jobs or lightweight deployments. | |
| | | **DotsOCR** | **≈9GB** FP16 weights, but expect **30–50GB** RAM/VRAM for high-res docs due to huge vision tokens | Apple Silicon + Metal BF16, ≥24GB CUDA cards, or 64GB RAM CPU workstations | Unified VLM (DotsVision + Qwen2) that nails layout, reading order, grounding, and multilingual math if you can tolerate the latency and memory bill. | |
| |
|
| | \*Measured from the default FP16 safetensors. Runtime footprint varies with sequence length. |
| | |
| | Guidance: |
| | |
| | - **Need maximum fidelity, multi-region reasoning, or already have 16–24GB VRAM?** Use **DeepSeek‑OCR**. The hybrid SAM+CLIP tower plus DeepSeek‑V2 MoE decoder handles complex layouts best, but expect higher memory/latency. |
| | - **Deploying to CPU-only nodes, 16GB laptops, or latency-sensitive services?** Choose **PaddleOCR‑VL**. Its dense Ernie decoder (18 layers, hidden 1024) activates fewer parameters per token and keeps memory under 10GB while staying close in quality on most docs. |
| | - **Chasing reading-order accuracy, layout grounding, or multi-page multilingual PDFs on roomy hardware?** Pick **DotsOCR** with BF16 on Metal/CUDA. Prefill runs around 40–50 tok/s on M-series GPUs but can fall to ~12 tok/s on CPU because of the heavy vision tower. |
| | |
| | ## Why Rust? 💡 |
| | |
| | The original DeepSeek-OCR ships as a Python + Transformers stack—powerful, but hefty to deploy and awkward to embed. Rewriting the pipeline in Rust gives us: |
| | |
| | - Smaller deployable artifacts with zero Python runtime or conda baggage. |
| | - Memory-safe, thread-friendly infrastructure that blends into native Rust backends. |
| | - Unified tooling (CLI + server) running on Candle + Rocket without the Python GIL overhead. |
| | - Drop-in compatibility with OpenAI-style clients while tuned for single-turn OCR prompts. |
| | |
| | ## Technical Stack ⚙️ |
| | |
| | - **Candle** for tensor compute, with Metal and CUDA backends and FlashAttention support. |
| | - **Rocket** + async streaming for OpenAI-compatible `/v1/responses` and `/v1/chat/completions`. |
| | - **tokenizers** (upstream DeepSeek release) wrapped by `crates/assets` for deterministic caching via Hugging Face and ModelScope mirrors. |
| | - **Pure Rust vision/prompt pipeline** shared by CLI and server to avoid duplicated logic. |
| | |
| | ## Advantages over the Python Release 🥷 |
| | |
| | - Faster cold-start on Apple Silicon, lower RSS, and native binary distribution. |
| | - Deterministic dual-source (Hugging Face + ModelScope) asset download + verification built into the workspace. |
| | - Automatic single-turn chat compaction so OCR outputs stay stable even when clients send history. |
| | - Ready-to-use OpenAI compatibility for tools like Open WebUI without adapters. |
| | |
| | ## Highlights ✨ |
| | |
| | - **One repo, two entrypoints** – a batteries-included CLI for batch jobs and a Rocket-based server that speaks `/v1/responses` and `/v1/chat/completions`. |
| | - **Works out of the box** – pulls model weights, configs, and tokenizer from whichever of Hugging Face or ModelScope responds fastest on first run. |
| | - **Optimised for Apple Silicon** – optional Metal backend with FP16 execution for real-time OCR on laptops. |
| | - **CUDA (alpha)** – experimental support via `--features cuda` + `--device cuda --dtype f16`; expect rough edges while we finish kernel coverage. |
| | - **Intel MKL (preview)** – faster BLAS on x86 via `--features mkl` (install Intel oneMKL beforehand). |
| | - **OpenAI client compatibility** – drop-in replacement for popular SDKs; the server automatically collapses chat history to the latest user turn for OCR-friendly prompts. |
| | |
| | ## Model Matrix 📦 |
| | |
| | The workspace exposes three base model IDs plus DSQ-quantized variants for DeepSeek‑OCR, PaddleOCR‑VL, and DotsOCR: |
| | |
| | | Model ID | Base Model | Precision | Suggested Use Case | |
| | | --- | --- | --- | --- | |
| | | `deepseek-ocr` | `deepseek-ocr` | FP16 (select via `--dtype`) | Full-fidelity DeepSeek‑OCR stack with SAM+CLIP + MoE decoder; use when you prioritise quality on capable Metal/CUDA/CPU hosts. | |
| | | `deepseek-ocr-q4k` | `deepseek-ocr` | `Q4_K` | Tight VRAM, local deployments, and batch jobs that still want DeepSeek’s SAM+CLIP pipeline. | |
| | | `deepseek-ocr-q6k` | `deepseek-ocr` | `Q6_K` | Day‑to‑day balance of quality and size on mid‑range GPUs. | |
| | | `deepseek-ocr-q8k` | `deepseek-ocr` | `Q8_0` | Stay close to full‑precision quality with manageable memory savings. | |
| | | `paddleocr-vl` | `paddleocr-vl` | FP16 (select via `--dtype`) | Default choice for lighter hardware; 0.9B Ernie + SigLIP tower with strong doc/table OCR and low latency. | |
| | | `paddleocr-vl-q4k` | `paddleocr-vl` | `Q4_K` | Heavily compressed doc/table deployments with aggressive memory budgets. | |
| | | `paddleocr-vl-q6k` | `paddleocr-vl` | `Q6_K` | Common engineering setups; blends accuracy and footprint. | |
| | | `paddleocr-vl-q8k` | `paddleocr-vl` | `Q8_0` | Accuracy‑leaning deployments that still want a smaller footprint than FP16. | |
| | | `dots-ocr` | `dots-ocr` | FP16 / BF16 (via `--dtype`) | DotsVision + Qwen2 VLM for high‑precision layout, reading order, grounding, and multilingual docs; expect high memory (30–50GB on large pages). | |
| | | `dots-ocr-q4k` | `dots-ocr` | `Q4_K` | Sidecar DSQ snapshot over the DotsOCR baseline; reduces weight memory/compute while keeping the heavy vision token profile unchanged. | |
| | | `dots-ocr-q6k` | `dots-ocr` | `Q6_K` | Recommended balance of size and quality when you already accept DotsOCR’s memory footprint but want cheaper weights. | |
| | | `dots-ocr-q8k` | `dots-ocr` | `Q8_0` | Accuracy‑leaning DotsOCR deployment that stays close to FP16/BF16 quality with modest memory savings. | |
| | |
| | |
| | ## Quick Start 🏁 |
| | |
| | ### Prerequisites |
| | |
| | - Rust 1.78+ (edition 2024 support) |
| | - Git |
| | - Optional: Apple Silicon running macOS 13+ for Metal acceleration |
| | - Optional: CUDA 12.2+ toolkit + driver for experimental NVIDIA GPU acceleration on Linux/Windows |
| | - Optional: Intel oneAPI MKL for preview x86 acceleration (see below) |
| | - (Recommended) Hugging Face account with `HF_TOKEN` when pulling from the `deepseek-ai/DeepSeek-OCR` repo (ModelScope is used automatically when it’s faster/reachable). |
| | |
| | ### Clone the Workspace |
| | |
| | ```bash |
| | git clone https://github.com/TimmyOVO/deepseek-ocr.rs.git |
| | cd deepseek-ocr.rs |
| | cargo fetch |
| | ``` |
| | |
| | ### Model Assets |
| | |
| | The first invocation of the CLI or server downloads the config, tokenizer, and `model-00001-of-000001.safetensors` (~6.3GB) into `DeepSeek-OCR/`. To prefetch manually: |
| | |
| | ```bash |
| | cargo run -p deepseek-ocr-cli --release -- --help # dev profile is extremely slow; always prefer --release |
| | ``` |
| | |
| | > Always include `--release` when running from source; debug builds on this model are extremely slow. |
| | Set `HF_HOME`/`HF_TOKEN` if you store Hugging Face caches elsewhere (ModelScope downloads land alongside the same asset tree). The full model package is ~6.3GB on disk and typically requires ~13GB of RAM headroom during inference (model + activations). |
| | |
| | ## Configuration & Overrides 🗂️ |
| | |
| | The CLI and server share the same configuration. On first launch we create a `config.toml` populated with defaults; later runs reuse it so both entrypoints stay in sync. |
| | |
| | | Platform | Config file (default) | Model cache root | |
| | | --- | --- | --- | |
| | | Linux | `~/.config/deepseek-ocr/config.toml` | `~/.cache/deepseek-ocr/models/<id>/…` | |
| | | macOS | `~/Library/Application Support/deepseek-ocr/config.toml` | `~/Library/Caches/deepseek-ocr/models/<id>/…` | |
| | | Windows | `%APPDATA%\deepseek-ocr\config.toml` | `%LOCALAPPDATA%\deepseek-ocr\models\<id>\…` | |
| | |
| | - Override the location with `--config /path/to/config.toml` (available on both CLI and server). Missing files are created automatically. |
| | - Each `[models.entries."<id>"]` record can point to custom `config`, `tokenizer`, or `weights` files. When omitted we fall back to the cache directory above and download/update assets as required. |
| | - Runtime values resolve in this order: command-line flags → values stored in `config.toml` → built-in defaults. The HTTP API adds a final layer where request payload fields (for example `max_tokens`) override everything else for that call. |
| | |
| | The generated file starts with the defaults below; adjust them to persistently change behaviour: |
| | |
| | ```toml |
| | [models] |
| | active = "deepseek-ocr" |
| | |
| | [models.entries.deepseek-ocr] |
| | |
| | [inference] |
| | device = "cpu" |
| | template = "plain" |
| | base_size = 1024 |
| | image_size = 640 |
| | crop_mode = true |
| | max_new_tokens = 512 |
| | use_cache = true |
| | |
| | [server] |
| | host = "0.0.0.0" |
| | port = 8000 |
| | ``` |
| | |
| | - `[models]` picks the active model and lets you add more entries (each entry can point to its own config/tokenizer/weights). |
| | - `[inference]` controls notebook-friendly defaults shared by the CLI and server (device, template, vision sizing, decoding budget, cache usage). |
| | - `[server]` sets the network binding and the model identifier reported by `/v1/models`. |
| | |
| | See `crates/cli/README.md` and `crates/server/README.md` for concise override tables. |
| | |
| | ## Benchmark Snapshot 📊 |
| | |
| | Single-request Rust CLI (Accelerate backend on macOS) compared with the reference Python pipeline on the same prompt and image: |
| | |
| | | Stage | ref total (ms) | ref avg (ms) | python total | python/ref | |
| | |---------------------------------------------------|----------------|--------------|--------------|------------| |
| | | Decode – Overall (`decode.generate`) | 30077.840 | 30077.840 | 56554.873 | 1.88x | |
| | | Decode – Token Loop (`decode.iterative`) | 26930.216 | 26930.216 | 39227.974 | 1.46x | |
| | | Decode – Prompt Prefill (`decode.prefill`) | 3147.337 | 3147.337 | 5759.684 | 1.83x | |
| | | Prompt – Build Tokens (`prompt.build_tokens`) | 0.466 | 0.466 | 45.434 | 97.42x | |
| | | Prompt – Render Template (`prompt.render`) | 0.005 | 0.005 | 0.019 | 3.52x | |
| | | Vision – Embed Images (`vision.compute_embeddings`)| 6391.435 | 6391.435 | 3953.459 | 0.62x | |
| | | Vision – Prepare Inputs (`vision.prepare_inputs`) | 62.524 | 62.524 | 45.438 | 0.73x | |
| | |
| | ## Command-Line Interface 🖥️ |
| | |
| | Build and run directly from the workspace: |
| | |
| | ```bash |
| | cargo run -p deepseek-ocr-cli --release -- \ |
| | --prompt "<image>\n<|grounding|>Convert this receipt to markdown." \ |
| | --image baselines/sample/images/test.png \ |
| | --device cpu --max-new-tokens 512 |
| | ``` |
| | |
| | > Tip: `--release` is required for reasonable throughput; debug builds can be 10x slower. |
| | |
| | > macOS tip: append `--features metal` to the `cargo run`/`cargo build` commands to compile with Accelerate + Metal backends. |
| | > |
| | > CUDA tip (Linux/Windows): append `--features cuda` and run with `--device cuda --dtype f16` to target NVIDIA GPUs—feature is still alpha, so be ready for quirks. |
| | > |
| | > Intel MKL preview: install Intel oneMKL, then build with `--features mkl` for faster CPU matmuls on x86. |
| | |
| | Install the CLI as a binary: |
| | |
| | ```bash |
| | cargo install --path crates/cli |
| | deepseek-ocr-cli --help |
| | ``` |
| | |
| | Key flags: |
| | |
| | - `--prompt` / `--prompt-file`: text with `<image>` slots |
| | - `--image`: path(s) matching `<image>` placeholders |
| | - `--device` and `--dtype`: choose `metal` + `f16` on Apple Silicon or `cuda` + `f16` on NVIDIA GPUs |
| | - `--max-new-tokens`: decoding budget |
| | - Sampling controls: `--do-sample`, `--temperature`, `--top-p`, `--top-k`, `--repetition-penalty`, `--no-repeat-ngram-size`, `--seed` |
| | - By default decoding stays deterministic (`do_sample=false`, `temperature=0.0`, `no_repeat_ngram_size=20`) |
| | - To use stochastic sampling set `--do-sample true --temperature 0.8` (and optionally adjust the other knobs) |
| | |
| | ### Switching Models |
| | |
| | The autogenerated `config.toml` now lists three entries: |
| | |
| | - `deepseek-ocr` (default) – the original DeepSeek vision-language stack. |
| | - `paddleocr-vl` – the PaddleOCR-VL 0.9B SigLIP + Ernie release. |
| | - `dots-ocr` – the Candle port of dots.ocr with DotsVision + Qwen2 (use BF16 on Metal/CUDA if possible; see the release matrix for memory notes). |
| | |
| | Pick which one to load via `--model`: |
| | |
| | ```bash |
| | deepseek-ocr-cli --model paddleocr-vl --prompt "<image> Summarise" |
| | ``` |
| | |
| | The CLI (and server) will download the matching config/tokenizer/weights from the appropriate repository (`deepseek-ai/DeepSeek-OCR`, `PaddlePaddle/PaddleOCR-VL`, or `dots-ocr`) into your cache on first use. You can still override paths with `--model-config`, `--tokenizer`, or `--weights` if you maintain local fine-tunes. |
| | |
| | ## HTTP Server ☁️ |
| | |
| | Launch an OpenAI-compatible endpoint: |
| | |
| | ```bash |
| | cargo run -p deepseek-ocr-server --release -- \ |
| | --host 0.0.0.0 --port 8000 \ |
| | --device cpu --max-new-tokens 512 |
| | ``` |
| | |
| | > Keep `--release` on the server as well; the debug profile is far too slow for inference workloads. |
| | > macOS tip: add `--features metal` to the `cargo run -p deepseek-ocr-server` command when you want the server binary to link against Accelerate + Metal (and pair it with `--device metal` at runtime). |
| | > |
| | > CUDA tip: add `--features cuda` and start the server with `--device cuda --dtype f16` to offload inference to NVIDIA GPUs (alpha-quality support). |
| | > |
| | > Intel MKL preview: install Intel oneMKL before building with `--features mkl` to accelerate CPU workloads on x86. |
| | |
| | Notes: |
| | |
| | - Use `data:` URLs or remote `http(s)` links; local paths are rejected. |
| | - The server collapses multi-turn chat inputs to the latest user message to keep prompts OCR-friendly. |
| | - Works out of the box with tools such as [Open WebUI](https://github.com/open-webui/open-webui) or any OpenAI-compatible client—just point the base URL to your server (`http://localhost:8000/v1`) and select either the `deepseek-ocr` or `paddleocr-vl` model ID exposed in `/v1/models`. |
| | - Adjust the request body limit with Rocket config if you routinely send large images. |
| | |
| |  |
| | |
| | ## GPU Acceleration ⚡ |
| | |
| | - **Metal (macOS 13+ Apple Silicon)** – pass `--device metal --dtype f16` and build binaries with `--features metal` so Candle links against Accelerate + Metal. |
| | - **CUDA (alpha, NVIDIA GPUs)** – install CUDA 12.2+ toolkits, build with `--features cuda`, and launch the CLI/server with `--device cuda --dtype f16`; still experimental. |
| | - **Intel MKL (preview)** – install Intel oneMKL and build with `--features mkl` to speed up CPU workloads on x86. |
| | - For either backend, prefer release builds (e.g. `cargo build --release -p deepseek-ocr-cli --features metal|cuda`) to maximise throughput. |
| | - Combine GPU runs with `--max-new-tokens` and crop tuning flags to balance latency vs. quality. |
| | |
| | ## Repository Layout 🗂️ |
| | |
| | - `crates/core` – shared inference pipeline, model loaders, conversation templates. |
| | - `crates/cli` – command-line frontend (`deepseek-ocr-cli`). |
| | - `crates/server` – Rocket server exposing OpenAI-compatible endpoints. |
| | - `crates/assets` – asset management (configuration, tokenizer, Hugging Face + ModelScope download helpers). |
| | - `baselines/` – reference inputs and outputs for regression testing. |
| | |
| | Detailed CLI usage lives in [`crates/cli/README.md`](crates/cli/README.md). The server’s OpenAI-compatible interface is covered in [`crates/server/README.md`](crates/server/README.md). |
| | |
| | ## Troubleshooting 🛠️ |
| | |
| | - **Where do assets come from?** – downloads automatically pick between Hugging Face and ModelScope based on latency; the CLI prints the chosen source for each file. |
| | - **Slow first response** – model load and GPU warm-up (Metal/CUDA alpha) happen on the initial request; later runs are faster. |
| | - **Large image rejection** – increase Rocket JSON limits in `crates/server/src/main.rs` or downscale the input. |
| | |
| | ## Roadmap 🗺️ |
| | |
| | - ✅ Apple Metal backend with FP16 support and CLI/server parity on macOS. |
| | - ✅ NVIDIA CUDA backend (alpha) – build with `--features cuda`, run with `--device cuda --dtype f16` for Linux/Windows GPUs; polishing in progress. |
| | - 🔄 **Parity polish** – finish projector normalisation + crop tiling alignment; extend intermediate-tensor diff suite beyond the current sample baseline. |
| | - 🔄 **Grounding & streaming** – port the Python post-processing helpers (box extraction, markdown polish) and refine SSE streaming ergonomics. |
| | - 🔄 **Cross-platform acceleration** – continue tuning CUDA kernels, add automatic device detection across CPU/Metal/CUDA, and publish opt-in GPU benchmarks. |
| | - 🔄 **Packaging & Ops** – ship binary releases with deterministic asset checksums, richer logging/metrics, and Helm/docker references for server deploys. |
| | - 🔜 **Structured outputs** – optional JSON schema tools for downstream automation once parity gaps close. |
| | |
| | ## License 📄 |
| | |
| | This repository inherits the licenses of its dependencies and the upstream DeepSeek-OCR model. Refer to `DeepSeek-OCR/LICENSE` for model terms and apply the same restrictions to downstream use. |
