| --- |
| language: en |
| tags: |
| - tokenizers |
| - wordpiece |
| - bytepairencoding |
| - xlnet |
| - nlp |
| license: mit |
| --- |
| |
| # Basic Tokenizers Collection |
|
|
| This repository contains **three different tokenizers** trained and wrapped for experimentation and educational purposes: |
|
|
| ## 📦 Contents |
| - **WordPiece Tokenizer** |
| Path: `ByteMeHarder-404/tokenizers/wordpiece` |
| Classic subword tokenizer (used in BERT). Splits words into subword units based on frequency, ensuring full coverage with a compact vocab. |
|
|
| - **Byte-Pair Encoding (BPE) Tokenizer** |
| Path: `ByteMeHarder-404/tokenizers/bpe` |
| Uses byte-level BPE, similar to GPT-2 and RoBERTa. Handles any UTF-8 character without unknown tokens by working directly on bytes. |
|
|
| - **XLNet-Style Tokenizer** |
| Path: `ByteMeHarder-404/tokenizers/xlnet` |
| Follows the XLNet tokenization approach, leveraging sentencepiece-like segmentation. |
|
|
| ## 🚀 Usage |
| You can load each tokenizer with `transformers`: |
|
|
| ```python |
| from transformers import PreTrainedTokenizerFast |
| |
| # WordPiece |
| tok_wordpiece = PreTrainedTokenizerFast.from_pretrained("ByteMeHarder-404/tokenizers/wordpiece") |
| |
| # BPE |
| tok_bpe = PreTrainedTokenizerFast.from_pretrained("ByteMeHarder-404/tokenizers/bpe") |
| |
| # XLNet-style |
| tok_xlnet = PreTrainedTokenizerFast.from_pretrained("ByteMeHarder-404/tokenizers/xlnet") |
| ``` |
|
|
| ## 📚 Notes |
|
|
| - These tokenizers are minimal examples and **not pretrained with embeddings or models**. |
| - Intended for experimentation, educational purposes, and as a foundation for building custom models. |
| - You can extend them by training a new vocabulary on your dataset. |
|
|
