Create README.md

README.md

@@ -0,0 +1,90 @@
+---
+license: apache-2.0
+language:
+  - multilingual
+tags:
+  - tokenizer
+  - bpe
+  - byte-level-bpe
+  - chatml
+  - routing
+  - moe
+---
+
+# JiRack Router Tokenizer Pro
+
+**High-performance custom Byte-Level BPE tokenizer** trained on the full Wikipedia dump across multiple languages.
+
+Developed specifically for intelligent routing models and Mixture-of-Experts systems.
+
+## Model Details
+
+- **Developer**: Konstantin Grabko (JiRack)
+- **License**: Apache License 2.0
+- **Training Data**: Full Wikipedia multilingual dump
+- **Vocabulary Size**: 65,536 tokens
+- **Special Tokens**: 128 reserved tokens (including 40+ domain routing tokens)
+
+## Key Features
+
+- Correctly placed `<|unk|>` token at ID 0
+- Full native support for **ChatML** format (`<|im_start|>` / `<|im_end|>`)
+- Large set of specialized routing tokens (`__CODING__`, `__MATH__`, `__PYTHON__`, `__SCIENCE__`, etc.)
+- Support JiRack Robotics Technology with tags  (`<|action_start|>` / `<|action_end|>`)
+- Support JiRack vision , images , audio-visual
+- Strong multilingual performance
+
+# Basic system and dialogue tokens
+"<|unk|>",
+"<|endoftext|>",
+"<|padding|>",
+"<|im_start|>",
+"<|im_end|>",
+# Core roles
+"<|im_start|>system",
+"<|im_start|>user",
+"<|im_start|>assistant",
+# Additional useful tokens
+"<|im_start|>tool",
+"<|im_start|>function",
+# Reasoning block
+"<|im_start|>thought",
+# Tool calls
+"<|tool_call|>",
+"<|tool_response|>",
+# Multimodality and audio-visual block
+"<|image|>",
+"<|video|>",
+"<|sound|>",
+"<|voice|>",
+"<|listening|>",
+"<|vision|>",
+# Emotional state (Mood)
+"<|mood_happy|>",
+"<|mood_sad|>",
+"<|mood_angry|>",
+"<|mood_neutral|>",
+# --- FIM (Fill-in-the-Middle) tokens from StarCoder ---
+"<fim_prefix>",
+"<fim_middle>",
+"<fim_suffix>",
+# Robotics (trajectory/command boundaries)
+"<|action_start|>",
+"<|action_end|>",
+
+
+# Highlights
+- **Architecture**: Byte-Level BPE (Byte-level Byte Pair Encoding) which natively prevents Out-Of-Vocabulary (OOV) tokens.
+- **Form Factor**: Fully wrapped into Hugging Face `PreTrainedTokenizerFast` with native `ByteLevel` decoders for clean cyrillic representation.
+- **Chat Standard**: Out-of-the-box support for **ChatML** formatting (`<|im_start|>` / `<|im_end|>`).
+- **Domain Specialization**: Pre-baked atomic routing tokens like `__CODING__` and `__PYTHON__` etc.
+
+## Usage
+
+```python
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("your-username/jirack_router_tokenizer")
+
+text = "<|im_start|>user\n__CODING__ __PYTHON__ Напиши функцию сортировки слиянием.<|im_end|>"
+print(tokenizer.encode(text))