How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/Mistrilitary-7b-GGUF:# Run inference directly in the terminal:
llama-cli -hf QuantFactory/Mistrilitary-7b-GGUF:Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QuantFactory/Mistrilitary-7b-GGUF:# Run inference directly in the terminal:
./llama-cli -hf QuantFactory/Mistrilitary-7b-GGUF:Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QuantFactory/Mistrilitary-7b-GGUF:# Run inference directly in the terminal:
./build/bin/llama-cli -hf QuantFactory/Mistrilitary-7b-GGUF:Use Docker
docker model run hf.co/QuantFactory/Mistrilitary-7b-GGUF:Quick Links
QuantFactory/Mistrilitary-7b-GGUF
This is quantized version of Heralax/Mistrilitary-7b created using llama.cpp
Original Model Card
Was torn between calling it MiLLM and Mistrillitary. Sigh naming is one of the two great problems in computer science...
This is a domain-expert finetune based on the US Army field manuals (the ones that are published and available for civvies like me). It's focused on factual question answer only, but seems to be able to answer slightly deeper questions in a pinch.
Model Quirks
- I had to focus on the army field manuals because the armed forces publishes a truly massive amount of text.
- No generalist assistant data was included, which means this is very very very focused on QA, and may be inflexible.
- Experimental change: data was mostly generated by a smaller model, Mistral NeMo. Quality seems unaffected, costs are much lower. Had problems with the open-ended questions not being in the right format.
- Low temperture recommended. Screenshots use 0.
- ChatML
- No special tokens added.
Examples:
)
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 2
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 5
- gradient_accumulation_steps: 6
- total_train_batch_size: 60
- total_eval_batch_size: 5
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 48
- num_epochs: 6
Training results
It answers questions alright.
Framework versions
- Transformers 4.45.0
- Pytorch 2.3.1+cu121
- Datasets 2.21.0
- Tokenizers 0.20.0
- Downloads last month
- 117
Hardware compatibility
Log In to add your hardware
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for QuantFactory/Mistrilitary-7b-GGUF
Base model
mistral-community/Mistral-7B-v0.2 Quantized
Heralax/army-pretrain-1
Install from brew
# Start a local OpenAI-compatible server with a web UI: llama-server -hf QuantFactory/Mistrilitary-7b-GGUF:# Run inference directly in the terminal: llama-cli -hf QuantFactory/Mistrilitary-7b-GGUF: