Sentencepiece tokenizers trimmed down to unique.
#1
by silveroxides - opened
- spiece.model +3 -0
- spiece_tokenizer.safetensors +3 -0
spiece.model
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:328852fb0ed85b3ad0b352bc7e4e0e5a3cb9e0598825458400f033136621db2f
|
| 3 |
+
size 1592643
|
spiece_tokenizer.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:76838aa32029b589d0e1363e2cff9ea9219c05b22332dc3d4ed340fc8c8bf11c
|
| 3 |
+
size 1592747
|