Instructions to use PygmalionAI/Pygmalion-3-12B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use PygmalionAI/Pygmalion-3-12B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="PygmalionAI/Pygmalion-3-12B-GGUF",
	filename="Pygmalion-3-12B-F16.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use PygmalionAI/Pygmalion-3-12B-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf PygmalionAI/Pygmalion-3-12B-GGUF:Q4_K_S
# Run inference directly in the terminal:
llama-cli -hf PygmalionAI/Pygmalion-3-12B-GGUF:Q4_K_S

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf PygmalionAI/Pygmalion-3-12B-GGUF:Q4_K_S
# Run inference directly in the terminal:
llama-cli -hf PygmalionAI/Pygmalion-3-12B-GGUF:Q4_K_S

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf PygmalionAI/Pygmalion-3-12B-GGUF:Q4_K_S
# Run inference directly in the terminal:
./llama-cli -hf PygmalionAI/Pygmalion-3-12B-GGUF:Q4_K_S

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf PygmalionAI/Pygmalion-3-12B-GGUF:Q4_K_S
# Run inference directly in the terminal:
./build/bin/llama-cli -hf PygmalionAI/Pygmalion-3-12B-GGUF:Q4_K_S

Use Docker

docker model run hf.co/PygmalionAI/Pygmalion-3-12B-GGUF:Q4_K_S

LM Studio
Jan
Ollama
How to use PygmalionAI/Pygmalion-3-12B-GGUF with Ollama:
```
ollama run hf.co/PygmalionAI/Pygmalion-3-12B-GGUF:Q4_K_S
```

Unsloth Studio

How to use PygmalionAI/Pygmalion-3-12B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for PygmalionAI/Pygmalion-3-12B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for PygmalionAI/Pygmalion-3-12B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for PygmalionAI/Pygmalion-3-12B-GGUF to start chatting

Docker Model Runner
How to use PygmalionAI/Pygmalion-3-12B-GGUF with Docker Model Runner:
```
docker model run hf.co/PygmalionAI/Pygmalion-3-12B-GGUF:Q4_K_S
```

Lemonade

How to use PygmalionAI/Pygmalion-3-12B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull PygmalionAI/Pygmalion-3-12B-GGUF:Q4_K_S

Run and chat with the model

lemonade run user.Pygmalion-3-12B-GGUF-Q4_K_S

List all available models

lemonade list

Short Review of SFW/NSFW experience using this model

by IggyLux - opened Feb 14, 2025

Discussion

IggyLux

Feb 14, 2025

So, I was really curious about this as it's been a long time since Pygmalion has dropped a model.

I tested this for SFW RP and NSFW RP:
Issues:

Confuses roles and genders
Doesn't understand relationships consistently
Hesitates under sexual situations stuttering and repeating
Often gets stuck in loops repeating itself
Has problems following formatting even if instructed, whether context/instruct template or system prompt instructs it to do a certain format of responses for example "For dialogue" for actions/thoughts
Lacks NSFW training data
Continuity in group chats leads to role/character/confusion - doesn't even form sentences properly

Good things:

Nice change of pace compared to other models/vocabulary and personality of characters
Seems neutral in regard to most topics even if hesitant
Lacks NSFW training data (good if looking for SFW RP)

Considering the behavior of this model, I believe there was something that went wrong in training because even a censored model usually doesn't have this much trouble keeping track of things.

Assuming they refine it in future iterations it might be amazing but as it currently stands, I cannot recommend it. But I look forward to seeing what else they might do.

It's a shame because it shows a lot of promise.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment