3morixd's picture
Upload README.md with huggingface_hub
0285034 verified
|
Raw
History Blame Contribute Delete
506 Bytes
---
title: Tokenizer Visualizer
emoji: πŸ”€
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: "4.44.0"
app_file: app.py
pinned: false
license: apache-2.0
tags: [tokenizer, visualization, mcp, dispatchai]
---
# πŸ”€ dispatchAI Tokenizer Visualizer
**MCP Server** β€” visualize how tokenizers split text.
## Tools
- `tokenize_text(text, method)` β€” Tokenize with chosen method
- `compare_methods(text)` β€” Compare token counts across methods
πŸš€ [dispatchAI](https://huggingface.co/dispatchAI)