| --- |
| library_name: transformers |
| tags: |
| - language |
| - detection |
| - classification |
| license: mit |
| datasets: |
| - hac541309/open-lid-dataset |
| pipeline_tag: text-classification |
| --- |
| |
| This is a clone of https://huggingface.co/alexneakameni/language_detection with onnx format |
| |
| # Language Detection Model |
| |
| A **BERT-based** language detection model trained on [hac541309/open-lid-dataset](https://huggingface.co/datasets/hac541309/open-lid-dataset), which includes **121 million sentences across 200 languages**. This model is optimized for **fast and accurate** language identification in text classification tasks. |
| |
| ## Model Details |
| |
| - **Architecture**: [BertForSequenceClassification](https://huggingface.co/transformers/model_doc/bert.html) |
| - **Hidden Size**: 384 |
| - **Number of Layers**: 4 |
| - **Attention Heads**: 6 |
| - **Max Sequence Length**: 512 |
| - **Dropout**: 0.1 |
| - **Vocabulary Size**: 50,257 |
| |
| ## Training Process |
| |
| - **Dataset**: |
| - Used the [open-lid-dataset](https://huggingface.co/datasets/hac541309/open-lid-dataset) |
| - Split into train (90%) and test (10%) |
| - **Tokenizer**: A custom `BertTokenizerFast` with special tokens for `[UNK]`, `[CLS]`, `[SEP]`, `[PAD]`, `[MASK]` |
| - **Hyperparameters**: |
| - Learning Rate: 2e-5 |
| - Batch Size: 256 (training) / 512 (testing) |
| - Epochs: 1 |
| - Scheduler: Cosine |
| - **Trainer**: Leveraged the Hugging Face [Trainer API](https://huggingface.co/docs/transformers/main_classes/trainer) with Weights & Biases for logging |
| |
| ## Evaluation |
| |
| The model was evaluated on the test split. Below are the overall metrics: |
| |
| - **Accuracy**: 0.969466 |
| - **Precision**: 0.969586 |
| - **Recall**: 0.969466 |
| - **F1 Score**: 0.969417 |
| |
| Detailled evaluation (Size is the number of languages supported) |
| |
| | Script | Support | Precision | Recall | F1 Score | Size | |
| |--------|---------|-----------|--------|----------|------| |
| | Arab | 819219 | 0.9038 | 0.9014 | 0.9023 | 21 | |
| | Latn | 7924704 | 0.9678 | 0.9663 | 0.9670 | 125 | |
| | Ethi | 144403 | 0.9967 | 0.9964 | 0.9966 | 2 | |
| | Beng | 163983 | 0.9949 | 0.9935 | 0.9942 | 3 | |
| | Deva | 423895 | 0.9495 | 0.9326 | 0.9405 | 10 | |
| | Cyrl | 831949 | 0.9899 | 0.9883 | 0.9891 | 12 | |
| | Tibt | 35683 | 0.9925 | 0.9930 | 0.9927 | 2 | |
| | Grek | 131155 | 0.9984 | 0.9990 | 0.9987 | 1 | |
| | Gujr | 86912 | 0.99999 | 0.9999 | 0.99995 | 1 | |
| | Hebr | 100530 | 0.9966 | 0.9995 | 0.9981 | 2 | |
| | Armn | 67203 | 0.9999 | 0.9998 | 0.9998 | 1 | |
| | Jpan | 88004 | 0.9983 | 0.9987 | 0.9985 | 1 | |
| | Knda | 67170 | 0.9999 | 0.9998 | 0.9999 | 1 | |
| | Geor | 70769 | 0.99997 | 0.9998 | 0.9999 | 1 | |
| | Khmr | 39708 | 1.0000 | 0.9997 | 0.9999 | 1 | |
| | Hang | 108509 | 0.9997 | 0.9999 | 0.9998 | 1 | |
| | Laoo | 29389 | 0.9999 | 0.9999 | 0.9999 | 1 | |
| | Mlym | 68418 | 0.99996 | 0.9999 | 0.9999 | 1 | |
| | Mymr | 100857 | 0.9999 | 0.9992 | 0.9995 | 2 | |
| | Orya | 44976 | 0.9995 | 0.9998 | 0.9996 | 1 | |
| | Guru | 67106 | 0.99999 | 0.9999 | 0.9999 | 1 | |
| | Olck | 22279 | 1.0000 | 0.9991 | 0.9995 | 1 | |
| | Sinh | 67492 | 1.0000 | 0.9998 | 0.9999 | 1 | |
| | Taml | 76373 | 0.99997 | 0.9999 | 0.9999 | 1 | |
| | Tfng | 41325 | 0.8512 | 0.8246 | 0.8247 | 2 | |
| | Telu | 62387 | 0.99997 | 0.9999 | 0.9999 | 1 | |
| | Thai | 83820 | 0.99995 | 0.9998 | 0.9999 | 1 | |
| | Hant | 152723 | 0.9945 | 0.9954 | 0.9949 | 2 | |
| | Hans | 92689 | 0.9893 | 0.9870 | 0.9882 | 1 | |
| |
| |
| A detailed per-script classification report is also provided in the repository for further analysis. |
| |
| --- |
| |
| ### How to Use |
| |
| You can quickly load and run inference with this model using the [Transformers pipeline](https://huggingface.co/docs/transformers/main_classes/pipelines): |
| |
| ```python |
| from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline |
| |
| tokenizer = AutoTokenizer.from_pretrained("alexneakameni/language_detection") |
| model = AutoModelForSequenceClassification.from_pretrained("alexneakameni/language_detection") |
| |
| language_detection = pipeline("text-classification", model=model, tokenizer=tokenizer) |
|
|
| text = "Hello world!" |
| predictions = language_detection(text) |
| print(predictions) |
| ``` |
| |
| This will output the predicted language code or label with the corresponding confidence score. |
| |
| --- |
| |
| **Note**: The model’s performance may vary depending on text length, language variety, and domain-specific vocabulary. Always validate results against your own datasets for critical applications. |
| |
| For more information, see the [repository documentation](https://github.com/KameniAlexNea/learning_language). |
| |
| Thank you for using this model—feedback and contributions are welcome! |
| |
