Instructions to use datasysdev/ann-sparseattention with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use datasysdev/ann-sparseattention with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("datasysdev/ann-sparseattention", dtype="auto")

llama-cpp-python

How to use datasysdev/ann-sparseattention with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="datasysdev/ann-sparseattention",
	filename="gguf/Qwen3-4B-Instruct-2507-F16-ann-6layer-k128-v2.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use datasysdev/ann-sparseattention with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf datasysdev/ann-sparseattention:F16
# Run inference directly in the terminal:
llama-cli -hf datasysdev/ann-sparseattention:F16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf datasysdev/ann-sparseattention:F16
# Run inference directly in the terminal:
llama-cli -hf datasysdev/ann-sparseattention:F16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf datasysdev/ann-sparseattention:F16
# Run inference directly in the terminal:
./llama-cli -hf datasysdev/ann-sparseattention:F16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf datasysdev/ann-sparseattention:F16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf datasysdev/ann-sparseattention:F16

Use Docker

docker model run hf.co/datasysdev/ann-sparseattention:F16

LM Studio
Jan
Ollama
How to use datasysdev/ann-sparseattention with Ollama:
```
ollama run hf.co/datasysdev/ann-sparseattention:F16
```

Unsloth Studio new

How to use datasysdev/ann-sparseattention with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for datasysdev/ann-sparseattention to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for datasysdev/ann-sparseattention to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for datasysdev/ann-sparseattention to start chatting

Pi new

How to use datasysdev/ann-sparseattention with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf datasysdev/ann-sparseattention:F16

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "datasysdev/ann-sparseattention:F16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use datasysdev/ann-sparseattention with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf datasysdev/ann-sparseattention:F16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default datasysdev/ann-sparseattention:F16

Run Hermes

hermes

Docker Model Runner
How to use datasysdev/ann-sparseattention with Docker Model Runner:
```
docker model run hf.co/datasysdev/ann-sparseattention:F16
```

Lemonade

How to use datasysdev/ann-sparseattention with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull datasysdev/ann-sparseattention:F16

Run and chat with the model

lemonade run user.ann-sparseattention-F16

List all available models

lemonade list

datasysdev commited on 18 days ago

Commit

cc1555b

verified ·

1 Parent(s): faa1ec5

Update logs/all32_d128_block.log

Browse files

Files changed (1) hide show

logs/all32_d128_block.log +86 -0

logs/all32_d128_block.log ADDED Viewed

	@@ -0,0 +1,86 @@

+wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /root/.netrc.
+wandb: Currently logged in as: dalletest123 to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: Tracking run with wandb version 0.26.1
+wandb: Run data is saved locally in /tmp/sparse-attn-git/wandb/run-20260509_091956-hcukr8rw
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Syncing run all32-d128-block-causal-reserve-0-1-2-35
+wandb: ⭐️ View project at https://wandb.ai/dalletest123/ann-sparse
+wandb: 🚀 View run at https://wandb.ai/dalletest123/ann-sparse/runs/hcukr8rw
+Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
+[config] preset=all32_d128_block run=all32-d128-block-causal-reserve-0-1-2-35 steps=1000 layers=32 d_search=128 ckpt_dir=/tmp/checkpoints_all32_d128_block_reserve_0_1_2_35
+Training search projections for layers: [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34]
+Reserved as full attention: [0, 1, 2, 35]
+[perf] flash_attention_3 unavailable (ImportError: FlashAttention3 has been toggled on, but it cannot be used due to the following error: the package for FlashAttention3 doesn't seem to be installed.); trying next.
+[perf] flash_attention_2 unavailable (ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package for FlashAttention2 doesn't seem to be installed.); trying next.
+[perf] attention implementation: sdpa
+[perf] Liger kernels applied via apply_liger_kernel_to_qwen3.
+Trainable parameters: 20,971,520 (20.97M)
+[ckpt] step 25 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_25.pt
+[ckpt] step 50 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_50.pt
+[ckpt] step 75 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_75.pt
+[ckpt] step 100 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_100.pt
+[ckpt] step 125 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_125.pt
+[ckpt] step 150 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_150.pt
+[ckpt] step 175 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_175.pt
+[ckpt] step 200 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_200.pt
+[ckpt] step 225 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_225.pt
+[step 250] === ACTIONABLE DIAGNOSTIC ===
+  Recall@K_eval: 0.812
+  PPL gap (relative): 2.283%
+  >> WORKING: High recall and quality preserved. Both set and ranking aligned with the teacher.
+[ckpt] step 250 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_250.pt
+[ckpt] step 275 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_275.pt
+[ckpt] step 300 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_300.pt
+[ckpt] step 325 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_325.pt
+[ckpt] step 350 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_350.pt
+[ckpt] step 375 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_375.pt
+[ckpt] step 400 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_400.pt
+[ckpt] step 425 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_425.pt
+[ckpt] step 450 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_450.pt
+[ckpt] step 475 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_475.pt
+[step 500] === ACTIONABLE DIAGNOSTIC ===
+  Recall@K_eval: 0.823
+  PPL gap (relative): 1.753%
+  >> WORKING: High recall and quality preserved. Both set and ranking aligned with the teacher.
+[ckpt] step 500 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_500.pt
+[ckpt] step 525 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_525.pt
+[ckpt] step 550 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_550.pt
+[ckpt] step 575 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_575.pt
+[ckpt] step 600 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_600.pt
+[ckpt] step 625 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_625.pt
+[ckpt] step 650 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_650.pt
+[ckpt] step 675 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_675.pt
+[ckpt] step 700 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_700.pt
+[ckpt] step 725 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_725.pt
+[step 750] === ACTIONABLE DIAGNOSTIC ===
+  Recall@K_eval: 0.825
+  PPL gap (relative): 1.943%
+  >> WORKING: High recall and quality preserved. Both set and ranking aligned with the teacher.
+[ckpt] step 750 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_750.pt
+[ckpt] step 775 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_775.pt
+[ckpt] step 800 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_800.pt
+[ckpt] step 825 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_825.pt
+[ckpt] step 850 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_850.pt
+[ckpt] step 875 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_875.pt
+[ckpt] step 900 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_900.pt
+[ckpt] step 925 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_925.pt
+[ckpt] step 950 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_950.pt
+[ckpt] step 975 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_975.pt
+[step 1000] === ACTIONABLE DIAGNOSTIC ===
+  Recall@K_eval: 0.825
+  PPL gap (relative): 1.746%
+  >> WORKING: High recall and quality preserved. Both set and ranking aligned with the teacher.
+[ckpt] step 1000 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_1000.pt
+[ckpt] step 1000 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_1000.pt
+[1;34mwandb[0m:
+[1;34mwandb[0m: 🚀 View run [33mall32-d128-block-causal-reserve-0-1-2-35[0m at: [34mhttps://wandb.ai/dalletest123/ann-sparse/runs/hcukr8rw[0m
+[1;34mwandb[0m: Find logs at: [1;35mwandb/run-20260509_091956-hcukr8rw/logs[0m