Create README.md

README.md

@@ -0,0 +1,334 @@
+---
+license: mit
+language:
+- multilingual
+tags:
+- tokenizer
+- bpe
+- byte-level-bpe
+- chatml
+- routing
+- moe
+- robotics
+- qwen
+- qwen3.8
+- jirack
+---
+
+# 💎 JiRack Precision is Robotics & Routing & Tool calls Tokenizer
+- JiRack Precision Tokenizer . Fully compatible with Qwen3.8  models.
+
+## NEW FOR JiRack Tokenizers family 
+
+1. Added DeekSeek resoning tags 
+
+2. So Do not use tool in system prompt that kills context size see what people do below ! 
+- Native Tool Integration vs. Dynamic Context Injection
+- Tokenizer-level tool calling yields higher precision and reliability than dynamically injecting schemas into RAG context windows, significantly optimizing both latency and token efficiency.
+
+## SIZE
+- Added Tool calls tags (
+    "<|tool_call_start|>",
+    "<|tool_call_end|>",
+    "<|tool_result_start|>",
+    "<|tool_result_end|>",
+  )
+  
+- Size {
+  Vocab size: 151779
+  pad_token_id: 151643
+  eos_token_id: 151643
+}
+- Compatible with Qwen 2.5 and DeekSeek R1 • Optimized for Code . Use resize function without adaptation, see examples below
+- It needs 100k example to fully adapt routing features for RAG Routing model . So check Qwen modification tokenizer rules
+- A DeekSeek R1-based tokenizer enhanced with FIM markers from Microsoft datasets and other tags for ML (
+    <|fim_prefix|>,
+    <|fim_middle|>,
+    <f|im_suffix|>
+  )
+- Added Robotics & Embodiment tags (
+   "<|action_start|>",
+    "<|action_end|>",
+    "<|trajectory_start|>",
+    "<|trajectory_end|>",
+    "<|joint_start|>",
+    "<|joint_end|>",
+    "<|sensor_start|>",
+    "<|sensor_end|>",
+    "<|command_start|>",
+    "<|command_end|>",
+    "<|state_start|>",
+    "<|state_end|>",
+    "<|pose|>",
+    "<|velocity|>",
+    "<|force|>",
+    "<|torque|>",
+    "<|gripper|>",
+    "<|navigation|>",
+    "<|obstacle|>",
+    "<|task_start|>",
+    "<|task_end|>",
+    "<|plan_start|>",
+    "<|plan_end|>",
+    "<|behavior_start|>",
+    "<|behavior_end|>",
+    "<|skill_start|>",
+    "<|skill_end|>",
+    "<|motor|>",
+    "<|servo|>",
+    "<|imu|>",
+    "<|lidar|>",
+    "<|camera|>",
+    "<|depth|>",
+    "<|waypoint|>",
+    "<|path|>",
+    "<|collision|>",
+    "<|grasp|>",
+    "<|release|>",
+    "<|homing|>",
+    "<|emergency_stop|>",
+    "<|calibration|>",
+    "<|manipulation|>",
+    "<|locomotion|>",
+    "<|feedback|>",
+    "<|control_loop|>",)
+
+- Added Multi models support (
+    "<|image|>",
+    "<|video|>",
+    "<|sound|>",
+    "<|voice|>",
+    "<|listening|>",
+    "<|vision|>",)
+
+- Added Human mood tags (
+    "<|mood_happy|>",
+    "<|mood_sad|>",
+    "<|mood_angry|>",
+    "<|mood_neutral|>",
+  )
+
+- Added Tool calls tags (
+    "<|tool_call_start|>",
+    "<|tool_call_end|>",
+    "<|tool_result_start|>",
+    "<|tool_result_end|>",
+  )
+
+
+- Added RAG routing tags for RAG MoE Systems (
+    "__SCIENCE__",
+    "__CODING__",
+    "__STOCK_EXCHANGE__",
+    "__MEDICINE__",
+    "__GOVERNMENT__",
+    "__NEWS__",
+    "__GENERAL__",
+    "__MATERIAL_SCIENCE__",
+    "__ELECTRONICS__",
+    "__MICROELECTRONICS__",
+    "__ENGINEERING__",
+    "__ROBOTICS__",
+    "__ENERGY__",
+    "__AUTOMOTIVE__",
+    "__AVIATION__",
+    "__MATH__",
+    "__PYTHON__",
+    "__C__",
+    "__CPP__",
+    "__C_SHARP__",
+    "__JAVA__",
+    "__JAVASCRIPT__",
+    "__TYPESCRIPT__",
+    "__RUST__",
+    "__GO__",
+    "__RUBY__",
+    "__PHP__",
+    "__SWIFT__",
+    "__KOTLIN__",
+    "__BASH__",
+    "__SQL__",
+    "__ASSEMBLY__",
+    "__PHILOSOPHY__",
+    "__LITERATURE__",
+    "__SOCIOLOGY__",
+    "__PSYCHOLOGY__",
+    "__POLITICAL_SCIENCE__",
+    "__CULTURAL_STUDIES__",
+    "__ETHNOGRAPHY__",
+    "__HUMAN_RIGHTS__",
+    "__COMPLIANCE__",
+    "__MILITARY__",
+    "__BANKING__",
+    "__OIL_INDUSTRY__",
+    "__LIGHT_INDUSTRY__",
+    "__NATURE__",
+    "__OCEAN__",
+    "__SPORT__",
+    "__CULINARY__",
+    "__TRAVEL__",
+    "__HOBBY__"
+  )
+    
+- Fully compatible with Microsoft BigCode datasets including The Stack, StarCoder, and NextCoder.
+- Enables efficient training on large-scale coding data for superior code generation and understanding.
+
+
+**Inventor:** Konstantin Vladimirovich Grabko  
+**Organization:** CMS Manhattan JiRack Technology  
+**Official Site:** [www.cmsmanhattan.com](http://www.cmsmanhattan.com)
+Designed for Banking and Fintech Institutions
+
+**Banks and Fintech**  JiRack Architecture: Build Sovereign Financial Models from Scratch
+- Leveraging the JiRack Tokenizer and our Open Dataset, we enable financial institutions to develop secure, internal AI models from the ground up. This approach ensures maximum data privacy and model sovereignty for high-stakes banking operations.
+- There is fix price for FinTech
+- I recommend initializing the model with a 4K context window for initial stability, followed by scaling to 8K context using specialized JiRack 8K datasets. This two-stage approach ensures robust positional encoding before extending the model's long-range dependency.
+
+
+**JiRack Corp Tokenizer  solution**
+- Use JiRack models with trusted, high-quality coding datasets while maintaining full control over your code and data privacy.
+- Excellent fit for Banks, Fintech companies, and any organization that requires strict data confidentiality and security.
+- Update JiRack model for corp privacy coding. 
+
+**JiRack Tokenizer Subcription**
+- All subscribed members will receive regular tokenizer updates optimized for the latest high-quality coding datasets.
+
+### Open Robot platform 
+
+- **Tiangong** : https://english.www.gov.cn/english.www.gov.cn/news/202411/13/content_WS673406e2c6d0868f4e8ece33.html
+- **Unitree g1**  https://a.co/d/0e4A8YVc
+- **LimX Oli**  https://www.limxdynamics.com/en/products/oli?channel=option_google_advertising__c-
+- **ubtrobot** https://www.ubtrobot.com/en/
+- **x-humanoid** https://www.x-humanoid.com/detail/hskw.html
+- **Drone**  https://arrowair.com/quiver/
+
+
+### Key Features
+
+- **Algorithm**: Byte-Level BPE
+- **Vocabulary Size**: **128,000** tokens — excellent balance between precision and efficiency
+- **Multilingual & Technical Strength**: Optimized for English, Russian, code, scientific literature, and technical documentation
+- **Domain Specialization**: Strong performance on programming languages, engineering, robotics, and scientific texts
+
+
+### Tool calling 
+- Here's a concrete example of the Qwen 2.5 / Hermes-style tool-calling format: https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1
+  
+- System message (defines available tools):
+  
+```bash
+<|im_start|>system
+You are a helpful assistant with access to the following functions. Use them if required:
+<tools>
+{
+"type": "function",
+"function": {"name": "get_weather", "description": "Get current weather for a location",
+"parameters":
+  {"type": "object", "properties":
+   {
+    "location":{"type": "string", "description": "City name"},
+    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["location"]
+   }
+  }
+}
+</tools>
+For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
+<tool_call>
+{"name": <function-name>, "arguments": <args-json-object>}
+</tool_call>
+<|im_end|>
+```
+
+- User message:
+
+```bash
+<|im_start|>user What's the weather like in Tokyo right now?<|im_end|>
+```
+
+- Assistant message (model emits the tool call instead of a normal answer):
+  
+```bash  
+<|im_start|>assistant
+ <tool_call>
+   {"name": "get_weather", "arguments": {"location": "Tokyo", "unit": "celsius"}}
+ </tool_call>
+<|im_end|>
+```
+
+- Tool result message (your agent code executes the function and feeds the result back):
+  
+```bash
+<|im_start|>tool
+ <tool_response>
+  {"temperature": 22, "condition": "clear sky", "humidity": 48}
+ </tool_response>
+<|im_end|>
+```
+
+- Final assistant message (model now answers using the result):
+  
+```bash
+<|im_start|>assistant It's currently 22°C and clear in Tokyo, with 48% humidity.<|im_end|>
+```
+
+```bash
+That's the exact skeleton your Hermes dataset entries need to be mapped into 
+— system prompt with <tools>...</tools>, <tool_call> in assistant turns, <tool_response> in tool turns. 
+If you apply Qwen 2.5's tokenizer chat template via apply_chat_template() with a tools=[...] argument, 
+it generates this structure automatically — you don't have to hand-write the tags yourself for training data prep, 
+as long as your data is in the standard role/content/tool_calls message-list format first
+```
+
+### Special Tokens Support
+
+- Full **Qwen2 compatible format** dialogue format 
+- FIM (Fill-in-the-Middle) support for code generation
+- Rich set of domain routing tokens (`__CODING__`, `__PYTHON__`, `__ROBOTICS__`, `__SCIENCE__`, etc.)
+- Extended robotics and control tokens
+
+
+### CMS Manhattan Service & Support 
+
+- Jirack patent guards your technology for competitors 
+- Redesign Llama , Qwen , Gemma  to Ternary model
+- Re-tain and replace embeddings for  Llama , Qwen  , Gemma  to extend langeages to 347
+- Accelerate inference via high compression 256K tokenizer and replace multiplication with sum operations via Ternary weights
+  
+
+
+**Install for Llamma compatible models in your chat script**
+
+- from transformers import AutoModelForCausalLM
+
+- model = AutoModelForCausalLM.from_pretrained(your model)
+
+- # The Must !
+- model.resize_token_embeddings(len(tokenizer.tokenizer))   # или просто len(tokenizer.tokenizer)
+
+- print("New Embedding size for you chat script:", model.get_input_embeddings().weight.shape[0])
+
+- # Tesr Tokenizer size !
+(venv_ji) root@jirack2:# python -c '
+from transformers import AutoTokenizer
+tok = AutoTokenizer.from_pretrained("./QwenRoboticsTokenizer")
+print("Vocab size:", len(tok))
+print("pad_token_id:", tok.pad_token_id)
+print("eos_token_id:", tok.eos_token_id)
+'
+
+Vocab size: 151778
+pad_token_id: 151643
+eos_token_id: 151645
+
+## 📧 Contact & Licensing
+
+For joint ventures, hardware integration, or licensing inquiries:
+
+- **Email:** grabko@cmsmanhattan.com
+- **Phone:** +1 (516) 777-0945
+- **Location:** New York, USA
+
+## 📧 Copyright 2026  CMS Manhattan . All rights reserved
+
+## License
+MIT License