avalon2244/Qwen3.5-4B-Claude-Opus-4.6-Distilled-GGUF
4B • Updated
• 2.36k • 9
Jun Young Baek
jupiterbjy
AI & ML interests
None yet
Recent Activity
liked
a model 5 days ago
avalon2244/Qwen3.5-4B-Claude-Opus-4.6-Distilled-GGUF liked
a model about 1 year ago
bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF liked
a model about 1 year ago
Tiiny/SmallThinker-3B-Preview Organizations
None yet
liked a model 5 days ago
liked 2 models about 1 year ago
Ah I see, hope so too - Catching both size and speed means alot in mobile! Gotta preemptively thanks your future IQ4_NL quants ;)
Interesting, in this case will description "Legacy format, generally not worth using over similarly sized formats" of Q4_0 change to something like "ARM recommended (Do not use in Apple Silicons)" - or will IQ4_NL added in list and recommend that over Q4_0?
reacted to bartowski's post with 👍 about 1 year ago
Post
80542
Looks like Q4_0_N_M file types are going away
Before you panic, there's a new "preferred" method which is online (I prefer the term on-the-fly) repacking, so if you download Q4_0 and your setup can benefit from repacking the weights into interleaved rows (what Q4_0_4_4 was doing), it will do that automatically and give you similar performance (minor losses I think due to using intrinsics instead of assembly, but intrinsics are more maintainable)
You can see the reference PR here:
https://github.com/ggerganov/llama.cpp/pull/10446
So if you update your llama.cpp past that point, you won't be able to run Q4_0_4_4 (unless they add backwards compatibility back), but Q4_0 should be the same speeds (though it may currently be bugged on some platforms)
As such, I'll stop making those newer model formats soon, probably end of this week unless something changes, but you should be safe to download and Q4_0 quants and use those !
Also IQ4_NL supports repacking though not in as many shapes yet, but should get a respectable speed up on ARM chips, PR for that can be found here: https://github.com/ggerganov/llama.cpp/pull/10541
Remember, these are not meant for Apple silicon since those use the GPU and don't benefit from the repacking of weights
Before you panic, there's a new "preferred" method which is online (I prefer the term on-the-fly) repacking, so if you download Q4_0 and your setup can benefit from repacking the weights into interleaved rows (what Q4_0_4_4 was doing), it will do that automatically and give you similar performance (minor losses I think due to using intrinsics instead of assembly, but intrinsics are more maintainable)
You can see the reference PR here:
https://github.com/ggerganov/llama.cpp/pull/10446
So if you update your llama.cpp past that point, you won't be able to run Q4_0_4_4 (unless they add backwards compatibility back), but Q4_0 should be the same speeds (though it may currently be bugged on some platforms)
As such, I'll stop making those newer model formats soon, probably end of this week unless something changes, but you should be safe to download and Q4_0 quants and use those !
Also IQ4_NL supports repacking though not in as many shapes yet, but should get a respectable speed up on ARM chips, PR for that can be found here: https://github.com/ggerganov/llama.cpp/pull/10541
Remember, these are not meant for Apple silicon since those use the GPU and don't benefit from the repacking of weights
- 17 replies
liked 6 models about 1 year ago
bartowski/llama-3.2-Korean-Bllossom-3B-GGUF
Text Generation • 3B • Updated
• 382 • 3
bartowski/Llama-3.3-70B-Instruct-GGUF
Text Generation • 71B • Updated
• 45.6k • 71
Bllossom/llama-3.2-Korean-Bllossom-3B
Text Generation • 3B • Updated
• 2.88k • 189
NousResearch/Hermes-3-Llama-3.2-3B-GGUF
3B • Updated
• 1.67k • 60
SanjiWatsuki/Loyal-Macaroni-Maid-7B
Text Generation • 7B • Updated
• 714 • 68
arcee-ai/Llama-3.1-SuperNova-Lite
Text Generation • Updated
• 1.62k • • 197
liked a Space about 1 year ago
liked 2 models about 1 year ago
liked 5 models over 1 year ago
WhiteRabbitNeo/WhiteRabbitNeo-2.5-Qwen-2.5-Coder-7B
Text Generation • 8B • Updated
• 261 • 63
bartowski/QwQ-32B-Preview-GGUF
Text Generation • 33B • Updated
• 16.2k • 101
kromeurus/L3.1-Aglow-Vulca-v0.1-8B
Text Generation • 8B • Updated
• 4 • 9
tohur/natsumura-storytelling-rp-1.0-llama-3.1-8b-GGUF
Updated
• 261 • 6
EricB/Llama-3.2-11B-Vision-Instruct-UQFF
Updated
• 1 • 16