Hugging Face's logo

chrishayuk
/
v11-tokenizer

Transformers
English
tokenizer
language-modeling
Model card Files
xet
 Community
chrishayuk commited on
Commit
361c586
·
verified ·
1 Parent(s): b779c03

Publish tokenizer 10dd51100331ab50 (adopted)

Browse files
Files changed (3) hide show
  1. README.md +1 -1
  2. checksums.sha256 +1 -1
  3. provenance.json +1 -1
README.md CHANGED
@@ -34,7 +34,7 @@ is the same value a checkpoint records as `tokenizer_hash`:
34
  | `tokenizer.json` sha256 | `10dd51100331ab503115db23eee7e8dc3e360e3aed697c8a2e1b12b8f46031ae` |
35
  | Vocabulary size | 71,260 |
36
  | Source | [chrishayuk/v-tokenizers](https://github.com/chrishayuk/v-tokenizers) |
37
- | Source commit | `ee502e091e71741703847bed1a187a10702825f2` |
38
 
39
  This repo is immutable: a fix ships under a new name, never as a replacement.
40
 
 
34
  | `tokenizer.json` sha256 | `10dd51100331ab503115db23eee7e8dc3e360e3aed697c8a2e1b12b8f46031ae` |
35
  | Vocabulary size | 71,260 |
36
  | Source | [chrishayuk/v-tokenizers](https://github.com/chrishayuk/v-tokenizers) |
37
+ | Source commit | `3ed5f3cb7aea16b150588f27f24a7f31121b1792` |
38
 
39
  This repo is immutable: a fix ships under a new name, never as a replacement.
40
 
checksums.sha256 CHANGED
@@ -1,6 +1,6 @@
1
  0260f98ca18f9400aa73c3b0ffe7a434ddf734ab3a573500bc7be2981703bd3b dormant_blocks.json
2
  bac7077323ad43406c7efd75f73d74f55e47a3f29f2d991612e0d6ecb0ec5646 golden_encodings.json
3
- 2414c1bc3657ca8026f0a550275ea925a117cf8a180099d00ab79fcc8ae94341 provenance.json
4
  9046da57c270c8e74d0f38832b4adce269c9d914ef21d2a0925e7772152dd793 special_tokens_map.json
5
  10dd51100331ab503115db23eee7e8dc3e360e3aed697c8a2e1b12b8f46031ae tokenizer.json
6
  85e283b2261920c475133f8434f9dcd2576b43977dcff510589b9212542b7993 tokenizer_config.json
 
1
  0260f98ca18f9400aa73c3b0ffe7a434ddf734ab3a573500bc7be2981703bd3b dormant_blocks.json
2
  bac7077323ad43406c7efd75f73d74f55e47a3f29f2d991612e0d6ecb0ec5646 golden_encodings.json
3
+ 42983b277b15dd35837d53e75370604e28111147c57e0f7030c1e6aa858980a6 provenance.json
4
  9046da57c270c8e74d0f38832b4adce269c9d914ef21d2a0925e7772152dd793 special_tokens_map.json
5
  10dd51100331ab503115db23eee7e8dc3e360e3aed697c8a2e1b12b8f46031ae tokenizer.json
6
  85e283b2261920c475133f8434f9dcd2576b43977dcff510589b9212542b7993 tokenizer_config.json
provenance.json CHANGED
@@ -5,7 +5,7 @@
5
  "vocab_size": 71260,
6
  "status": "adopted",
7
  "source_repo": "https://github.com/chrishayuk/v-tokenizers",
8
- "source_commit": "ee502e091e71741703847bed1a187a10702825f2",
9
  "corpus": {
10
  "catalog": "chuk-datasets",
11
  "dataset": "v11/tokenizer",
 
5
  "vocab_size": 71260,
6
  "status": "adopted",
7
  "source_repo": "https://github.com/chrishayuk/v-tokenizers",
8
+ "source_commit": "3ed5f3cb7aea16b150588f27f24a7f31121b1792",
9
  "corpus": {
10
  "catalog": "chuk-datasets",
11
  "dataset": "v11/tokenizer",