Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
blackboxanalytics 
posted an update 10 days ago
Post
121
Athanor Lite -- free desktop app for running local models without the setup headache
Most tools in the local AI space assume you already know your way around quantization formats, context windows, and VRAM budgets. Athanor Lite is built for the people who just want to try a model without reading a wiki first.
It scans your hardware, shows you exactly which models fit your GPU, and handles download through inference. One app, no terminal, no config files. Uses llama.cpp under the hood, wrapped in a Tauri + Rust + React desktop app.
What it does:

Hardware detection (GPU architecture, VRAM, RAM, disk)
Model catalog with fit verdicts based on your actual specs
Ollama library import (zero-copy via hardlinks)
Real-time inference HUD (tok/s, GPU %, VRAM, temperature)
Workspace system for different model/task contexts
Zero telemetry, zero accounts, zero cloud

Windows for now, Mac is next. Free and open source.
GitHub: https://github.com/BBALabs/athanor-lite
Download: https://github.com/BBALabs/athanor-lite/releases/latest
Product page: https://bbasecure.com/athanor.html
dipankarsarkar

The fit verdict is the whole product, and it is the hard part.

A 9B at Q4 is maybe 6GB of weights, so it 'fits' an 8GB card at load. Then you open a 32K context and it OOMs, because the KV cache grows with sequence length and heads, not with the weights.

So 'fits your GPU' at load is a different claim than 'fits at the context you'll actually run.'

Does the verdict model KV-cache-at-target-context, or just the weight footprint at load?

·
blackboxanalytics

You're totally right, weight footprint alone doesn't tell the full story. The fit verdict does factor in KV cache and runtime overhead on top of the weights, so it's not just checking if the model loads. It should catch that 9B Q4 on 8GB scenario you're describing.

One thing I do want to improve though is letting users set their own target context length and watching the verdict adjust. Right now it uses the model's default, which isn't always how people actually run them. That's coming in v2. Good callout!
dipankarsarkar

Glad it already models KV. The v2 note is the part I would push on, because "the model's default" is doing a lot of work there, and it breaks in both directions.

If default means config max_position_embeddings, then a Qwen2.5-1M reads as needing a datacenter, and the verdict says no to a model that runs fine at the 16k people actually use it at. If default means the runtime's fallback, llama.cpp still lands at 4096 unless told otherwise, and you get the opposite failure: a green verdict that OOMs the moment someone opens a real context. Same model, same card, opposite answers.

The other half is that KV size is not a property of the model. It is a property of the model plus the KV dtype. Keys at q8_0 with values at q4_0, against f16/f16, is roughly a 3x swing on the same weights at the same context. That is a runtime flag, not something you can read off the model card. GQA head count moves it again.

So the honest verdict is a function of model, context, and KV dtype. Not model and GPU.

Which makes target-context-as-a-slider only half of v2. Do you let the user declare the KV type they will run, or pick one for them and report the fit at that?
In this post