Clarify local vLLM download instructions
Browse files
README.md
CHANGED
|
@@ -98,10 +98,14 @@ print(tokenizer.decode(output[0, inputs.shape[-1]:], skip_special_tokens=True))
|
|
| 98 |
|
| 99 |
## Local inference with vLLM
|
| 100 |
|
|
|
|
|
|
|
|
|
|
| 101 |
```bash
|
| 102 |
-
|
|
|
|
|
|
|
| 103 |
--served-model-name gemma-roleplay-v2 \
|
| 104 |
-
--subfolder merged \
|
| 105 |
--dtype bfloat16 \
|
| 106 |
--max-model-len 4096
|
| 107 |
```
|
|
|
|
| 98 |
|
| 99 |
## Local inference with vLLM
|
| 100 |
|
| 101 |
+
Download the `merged/` folder from this repository and point vLLM at that
|
| 102 |
+
local directory:
|
| 103 |
+
|
| 104 |
```bash
|
| 105 |
+
hf download text-generator/llmtrain --repo-type model --local-dir ./llmtrain \
|
| 106 |
+
--include 'merged/*'
|
| 107 |
+
vllm serve ./llmtrain/merged \
|
| 108 |
--served-model-name gemma-roleplay-v2 \
|
|
|
|
| 109 |
--dtype bfloat16 \
|
| 110 |
--max-model-len 4096
|
| 111 |
```
|