Quantifying the Carbon Emissions of Machine Learning
Paper • 1910.09700 • Published • 45
How to use from
SGLangUse Docker images
docker run --gpus all \
--shm-size 32g \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<secret>" \
--ipc=host \
lmsysorg/sglang:latest \
python3 -m sglang.launch_server \
--model-path "datapaf/StarcoderCodeQnA" \
--host 0.0.0.0 \
--port 30000# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "datapaf/StarcoderCodeQnA",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'Quick Links
Model Card for StarCoderCodeQ&A
This is a version of StarCoder model that was fine-tuned on the grammatically corrected texts.
Model Details
Model Description
- Model type: GPT-2
- Number of Parameters: 15.5B
- Supported Programming Language: Python
- Finetuned from model: StarCoder
Model Sources [optional]
- Repository: GitHub Repo
- Paper: "Leveraging Large Language Models in Code Question Answering: Baselines and Issues" Georgy Andryushchenko, Vladimir V. Ivanov, Vladimir Makharev, Elizaveta Tukhtina, Aidar Valeev
How to Get Started with the Model
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('datapaf/StarCoderCodeQnA')
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="cuda")
code = ... # Your Python code snippet here
question = ... # Your question regarding the snippet here
prompt_template = "Question: {question}\n\nCode: {code}\n\nAnswer:"
prompt = prompt_template.format(question=ex['question'], code=ex['code'])
inputs = tokenizer.encode(prompt, return_tensors="pt").to('cuda')
outputs = model.generate(inputs, max_new_tokens=512, pad_token_id=tokenizer.eos_token_id)
text = tokenizer.decode(outputs[0])
print(text)
- Downloads last month
- 2
Install from pip and serve model
# Install SGLang from pip: pip install sglang# Start the SGLang server: python3 -m sglang.launch_server \ --model-path "datapaf/StarcoderCodeQnA" \ --host 0.0.0.0 \ --port 30000# Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "datapaf/StarcoderCodeQnA", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'