Instructions to use ise-uiuc/Magicoder-S-CL-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ise-uiuc/Magicoder-S-CL-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ise-uiuc/Magicoder-S-CL-7B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ise-uiuc/Magicoder-S-CL-7B")
model = AutoModelForCausalLM.from_pretrained("ise-uiuc/Magicoder-S-CL-7B")

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ise-uiuc/Magicoder-S-CL-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ise-uiuc/Magicoder-S-CL-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ise-uiuc/Magicoder-S-CL-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ise-uiuc/Magicoder-S-CL-7B

SGLang

How to use ise-uiuc/Magicoder-S-CL-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ise-uiuc/Magicoder-S-CL-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ise-uiuc/Magicoder-S-CL-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ise-uiuc/Magicoder-S-CL-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ise-uiuc/Magicoder-S-CL-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ise-uiuc/Magicoder-S-CL-7B with Docker Model Runner:
```
docker model run hf.co/ise-uiuc/Magicoder-S-CL-7B
```

I try to use the modell with ollama but it does't work

by fipra - opened Dec 7, 2023

Discussion

fipra

Dec 7, 2023

(base) fabriziocasula@air-di-fabrizio ~ % ollama rm magicoder
Error: model 'magicoder' not found
(base) fabriziocasula@air-di-fabrizio ~ % ollama rm magicoder:7b
deleted 'magicoder:7b'
(base) fabriziocasula@air-di-fabrizio ~ % ollama run magicoder
pulling manifest
pulling 4a501ed4ce55... 100% ▕█████████████████████████████████████████████████████████████▏ 3.8 GB
pulling 832698dc48ba... 100% ▕█████████████████████████████████████████████████████████████▏ 57 B
pulling 317b1d8baaaa... 100% ▕█████████████████████████████████████████████████████████████▏ 134 B
pulling 58e1b82a691f... 100% ▕█████████████████████████████████████████████████████████████▏ 18 B
pulling 662eee7cf45e... 100% ▕█████████████████████████████████████████████████████████████▏ 455 B
verifying sha256 digest
writing manifest
removing any unused layers
success
Error: llama runner process has terminated
(base) fabriziocasula@air-di-fabrizio ~ % ollama rm magicoder

yuxiang630

Intellligent Software Engineering (iSE) org Dec 8, 2023

Hi @fipra , thanks for reporting this. May I know your system information?

fipra

Dec 8, 2023

•

edited Dec 8, 2023

MacBook Air
15", M2, 2023

Chip Apple M2
Memoria 16 GB
Disco di avvio Macintosh HD

macOS 13.5 (22G74)

fabioamigo

Dec 11, 2023

•

edited Dec 11, 2023

Same here. You have to create a new model file with context size 8192 (PARAMETER num_ctx 8192) or lower. In my Air M2 16GB runs with 8192.

My modelfile:

FROM magicoder:7b-s-cl-q4_K_M

TEMPLATE """{{ .System }}

@@ Instruction
{{ .Prompt }}

@@ Response
"""
SYSTEM """You are an exceptionally intelligent coding assistant that consistently delivers accurate and reliable responses to user instructions."""
PARAMETER num_ctx 8192

Save it anywhere as magicoder_8k and run :

ollama create magicoder_8K -f magicoder_8k

New model created with 8K context.
Run the model:

ollama run magicoder_8K

Hope it helps. Worked for me.

EDIT:

FROM magicoder:7b-s-cl-q4_K_M

You have to match what you have pulled.

Judith76

4 days ago

Hi there,

Thanks for sharing this, I ran into a similar issue with Ollama on an M2 Mac, and the num_ctx fix worked for me too. Your modelfile solution is spot on.

On a completely different note, if you ever want to display your coding achievements or team photos in a unique way, I recently came across this website that prints images on glass, really sleek for an office or desk setup. Might be a nice reward after getting that model running!

Hope the fix keeps working for you.

Mildred76

3 days ago

•

edited 3 days ago

Thanks for sharing this fix, it's really helpful. I was running into the exact same "llama runner process has terminated" error on my M2 MacBook Air with 16GB RAM, and adjusting the context size to 8192 using a custom modelfile completely solved it. Your step-by-step instructions are clear and easy to follow. For anyone else struggling, just make sure the FROM line matches the exact model tag you've pulled, and you should be good to go.

On a completely unrelated note, if you're ever looking for something fun to tinker with on the side, I recently stumbled across a site that offers some interesting gaming stuff, you can click here to check it out if you're curious. I was genuinely impressed by what it offers. Anyway, thanks again for the great solution, this saved me a ton of frustration!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment