Instructions to use LLM360/K2-Think with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LLM360/K2-Think with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="LLM360/K2-Think")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("LLM360/K2-Think")
model = AutoModelForCausalLM.from_pretrained("LLM360/K2-Think")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use LLM360/K2-Think with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LLM360/K2-Think"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LLM360/K2-Think",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/LLM360/K2-Think

SGLang

How to use LLM360/K2-Think with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LLM360/K2-Think" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LLM360/K2-Think",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LLM360/K2-Think" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LLM360/K2-Think",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use LLM360/K2-Think with Docker Model Runner:
```
docker model run hf.co/LLM360/K2-Think
```

please add api key and base url

by asd12d21 - opened Sep 13, 2025

base: refs/heads/main

←

from: refs/pr/7

Discussion Files changed

+20

-10

Files changed (1) hide show

README.md +20 -10

README.md CHANGED Viewed

@@ -49,7 +49,7 @@ print(outputs[0]["generated_text"][-1])
 ---
 # Evaluation & Performance
-Detailed evaluation results are reported in out [Tech Report](https://arxiv.org/abs/2509.07604)
 ## Benchmarks (pass\@1, average over 16 runs)
@@ -71,7 +71,7 @@ We deploy K2-THINK on Cerebras Wafer-Scale Engine (WSE) systems, leveraging the
 | Platform                          | Throughput (tokens/sec) | Example: 32k-token response (time) |
 | --------------------------------- | ----------------------: | ---------------------------------: |
 | **Cerebras WSE (our deployment)** |             **\~2,000** |                         **\~16 s** |
-| Typical Cloud Service setup   |                   \~200 |                            \~160 s |
 ---
@@ -88,20 +88,30 @@ Aggregated across four safety dimensions (**Safety-4**):
 | **Safety-4 Macro (avg)**        | **0.75** |
 ---
 # Terms of Use
-We have employed various techniques to reduce bias, harmful outputs, and other risks in the model. While these efforts help improve safety and reliability, the model, like all Large Language Models, may still generate inaccurate, misleading, biased, or otherwise undesirable content. By downloading, using, or interacting with this model, you acknowledge these limitations and agree to the following:
-1. **Prohibited Uses**
-   - You may **not** use this model for any **illegal, unlawful, or harmful activities**, including but not limited to fraud, abuse, harassment, privacy violations, or the creation/dissemination of malicious content.
-2. **User Responsibility**
-   - You are solely responsible for how you use the model and for any outcomes that result from its use.
-   - The authors and institutions involved in releasing this model do **not** accept liability for any consequences arising from its use.
 3. **No Warranty**
-   - The model is provided **“as is” without any warranties or guarantees**.
 ---
 # Citation

 ---
 # Evaluation & Performance
+Detailed evaluation results are reported in out [Tech Report](https://k2think-about.pages.dev/assets/tech-report/K2-Think_Tech-Report.pdf)
 ## Benchmarks (pass\@1, average over 16 runs)
 | Platform                          | Throughput (tokens/sec) | Example: 32k-token response (time) |
 | --------------------------------- | ----------------------: | ---------------------------------: |
 | **Cerebras WSE (our deployment)** |             **\~2,000** |                         **\~16 s** |
+| Typical **H100/H200** GPU setup   |                   \~200 |                            \~160 s |
 ---
 | **Safety-4 Macro (avg)**        | **0.75** |
 ---
 # Terms of Use
+This model is released strictly for **research and educational purposes**.
+By downloading, using, or interacting with this model, you agree to the following conditions:
+1. **Research Only**
+   The model is provided as part of an academic and research project.
+   It is **not intended for commercial deployment** or production use without explicit permission.
+2. **Prohibited Uses**
+   You may **not** use this model:
+   - For any **illegal, unlawful, or harmful activities**, including but not limited to generating or disseminating malicious content, engaging in fraud, violating privacy, or spreading misinformation.
+   - In applications that could directly cause **harm, injury, or safety risks** to individuals or society.
 3. **No Warranty**
+   The model is provided **“as is” without warranties** of any kind.
+   The authors and institutions involved bear no responsibility for consequences arising from its use.
+4. **Attribution**
+   When using or referencing the model in research, publications, or derivative works, proper **citation and attribution** to the authors and project must be given.
+5. **Compliance**
+   You are responsible for ensuring that your use of the model complies with all **applicable laws, regulations, and ethical guidelines** in your jurisdiction.
 ---
 # Citation