README.md

metadata

title: ByteBot
emoji: 🤖
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 5.34.2
python_version: '3.11'
app_file: app.py
pinned: false

🤖 Qwen Alpaca GGUF

A fine tuned Qwen2.5-0.5B-Instruct language model trained on the tatsu-lab/alpaca instruction dataset. The model was fine tuned using Unsloth, quantized to GGUF (Q4_K_M), and can be run locally with Ollama or llama.cpp.

---

🚀 Model Details

Property	Value
Base Model	Qwen2.5-0.5B-Instruct
Fine Tuning	LoRA
Framework	Unsloth
Dataset	tatsu-lab/alpaca
Format	GGUF
Quantization	Q4_K_M
Inference	Ollama, llama.cpp
Language	English

---

✨ Features

• Instruction following chatbot
• Lightweight 0.5B parameter model
• GGUF format for efficient CPU inference
• Optimized using Q4_K_M quantization
• Compatible with Ollama
• Compatible with llama.cpp
• Easy local deployment

---

📦 Model File

Qwen2.5-0.5B-Instruct.Q4_K_M.gguf

---

🛠 Fine Tuning Pipeline

Base Model
        │
        ▼
Qwen2.5-0.5B-Instruct
        │
        ▼
Alpaca Instruction Dataset
        │
        ▼
LoRA Fine Tuning
        │
        ▼
Unsloth
        │
        ▼
Merge LoRA Adapters
        │
        ▼
GGUF Conversion
        │
        ▼
Q4_K_M Quantization
        │
        ▼
Inference using Ollama / llama.cpp

---

📚 Dataset

This model was fine tuned using the tatsu-lab/alpaca instruction dataset.

The dataset contains thousands of instruction and response pairs designed to improve instruction following ability.

---

⚡ Run with Ollama

Create a Modelfile

FROM ./Qwen2.5-0.5B-Instruct.Q4_K_M.gguf

Create the model

ollama create fineqwen -f Modelfile

Run the model

ollama run fineqwen

---

🖥 Example Terminal

$ ollama create fineqwen -f Modelfile

transferring model...
creating new layer...
writing manifest...
success

$ ollama run fineqwen

>>> Explain Transformers in simple words.

Transformers are neural networks designed to process sequences using
self-attention, allowing them to understand relationships between words
efficiently.

---

🦙 Run using llama.cpp

llama-cli \
-hf ciphermosaic/qwen-alpaca-gguf \
--jinja

or

llama-cli \
-m Qwen2.5-0.5B-Instruct.Q4_K_M.gguf

---

🌐 Hugging Face Space

Interactive chatbot available on Hugging Face Spaces.

---

📊 Quantization

This model uses

Q4_K_M

Benefits

• Smaller model size
• Faster inference
• Lower RAM usage
• Minimal quality loss

---

🧰 Tech Stack

• Python
• Hugging Face Transformers
• Unsloth
• PEFT (LoRA)
• GGUF
• llama.cpp
• Ollama
• Hugging Face Hub
• Gradio

---

📁 Repository Structure

.
├── Modelfile
├── Qwen2.5-0.5B-Instruct.Q4_K_M.gguf
├── README.md
├── config.json

---

👨‍💻 Author

CipherMosaic

GitHub: https://github.com/CipherMosaic

Hugging Face: https://huggingface.co/ciphermosaic

---

⭐ If you found this project useful, consider giving it a star!