Update README.md
Browse files
README.md
CHANGED
|
@@ -12,7 +12,7 @@ license_link: LICENSE
|
|
| 12 |
eos_token_id: 128001
|
| 13 |
}
|
| 14 |
- Compatible with Llama 3 • Optimized for Code . Use resize function without adaptation, see examples below
|
| 15 |
-
- It
|
| 16 |
- A Llama 3-based tokenizer enhanced with FIM markers from Microsoft datasets (
|
| 17 |
<|fim_prefix|>,
|
| 18 |
<|fim_middle|>,
|
|
|
|
| 12 |
eos_token_id: 128001
|
| 13 |
}
|
| 14 |
- Compatible with Llama 3 • Optimized for Code . Use resize function without adaptation, see examples below
|
| 15 |
+
- It needs 100k example to fully adapt routing features for RAG Routing model . So check llama modification tokenizer rules
|
| 16 |
- A Llama 3-based tokenizer enhanced with FIM markers from Microsoft datasets (
|
| 17 |
<|fim_prefix|>,
|
| 18 |
<|fim_middle|>,
|