README.md

---

---
language:
- code
tags:
- code
- programming-language
- classification
- bert
- text-classification
license: apache-2.0
datasets:
- kaushik-harsh-99/Code-Language-Classification
metrics:
- accuracy
- f1
- precision
- recall
model-index:
- name: code-lang-bert-small
  results:
  - task:
      type: text-classification
      name: Programming Language Identification
    dataset:
      type: kaushik-harsh-99/Code-Language-Classification
      name: Code Language Classification
      split: test
    metrics:
    - type: accuracy
      value: 0.9663
    - type: f1 (macro)
      value: 0.9662
    - type: f1 (weighted)
      value: 0.9662
    - type: precision (macro)
      value: 0.9663
    - type: recall (macro)
      value: 0.9663
---

# Model Card for code-lang-bert-small

A fine-tuned BERT-small model for identifying programming languages from code snippets. The model classifies raw source code into one of 16 supported languages with high accuracy.

## Model Details

### Model Description

This model is a fine-tuned version of `prajjwal1/bert-small` (29M parameters) designed for the task of programming language identification. By analyzing the syntax, keywords, and structural patterns of source code, it accurately predicts the programming language of a given snippet.

- **Developed by:** Pankaj8922
- **Model type:** Encoder-only Transformer (BERT-small) for sequence classification
- **Language(s):** 16 programming and markup languages (see below)
- **License:** Apache 2.0
- **Finetuned from model:** [prajjwal1/bert-small](https://huggingface.co/prajjwal1/bert-small)

### Supported Languages

Rust, Java, Dart, Python, Go, HTML, JavaScript, Typescript, C, CSS, C#, Markdown, Assembly, Lua, C++, Kotlin

## Uses

### Direct Use

The model is intended for classifying code snippets. It can be used directly with the Hugging Face `pipeline` API or integrated into applications for code tagging, automated documentation, or content filtering.

```python
from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="Pankaj8922/code-lang-bert-small"
)

code_snippet = """
def quicksort(arr):
    if len(arr) <= 1:
        return arr
    pivot = arr[len(arr) // 2]
    return quicksort(left) + mid + quicksort(right)
"""

result = classifier(code_snippet)
print(result)
# [{'label': 'Python', 'score': 0.99}]
```

### Out-of-Scope Use

The model is trained to classify full files or substantial code snippets. It may not perform well on:
- Very short, ambiguous one-liners.
- Heavily obfuscated or minified code.
- Code containing multiple languages (e.g., a Python file with extensive embedded SQL).
- Languages not present in the 16 supported classes.

## Bias, Risks, and Limitations

The model may exhibit biases present in the training data distribution. Languages with syntactically similar constructs (e.g., C and C++, JavaScript and TypeScript) are the most common sources of confusion, as reflected in the confusion matrix. Performance on code from very niche or domain-specific libraries may be lower.

## Training Details

### Training Data

The model was trained on the [Code-Language-Classification](https://huggingface.co/datasets/kaushik-harsh-99/Code-Language-Classification) dataset. The official `train`, `validation`, and `test` splits were used.
- **Train samples:** 1,600,000
- **Validation samples:** 32,000
- **Test samples:** 32,000
- **Classes:** 16 (perfectly balanced, 2000 samples per class in test set)

### Training Procedure

The BERT-small model was fine-tuned on 2 x T4 GPUs with dynamic padding for efficiency. Training was configured for 5 epochs with early stopping, but was manually stopped after 4 epochs as the model had already converged.

- **Batch size:** 256 (128 per device x 2 GPUs)
- **Learning rate:** 3e-5
- **Optimizer:** AdamW (weight decay: 0.01)
- **Max sequence length:** 512 tokens
- **Early stopping patience:** 2 epochs
- **Checkpointing:** Best model based on validation accuracy saved to the Hub.

## Evaluation

The evaluation was performed on the held-out test set of 32,000 samples using the official script provided in the repository.

### Testing Metrics

| Metric           | Value    |
|------------------|----------|
| Accuracy         | 96.63%   |
| Macro F1         | 96.62%   |
| Weighted F1      | 96.62%   |
| Macro Precision  | 96.63%   |
| Macro Recall     | 96.63%   |
| Eval Loss        | 0.1147   |

### Per-Class Performance

| Language   | Precision | Recall | F1-Score |
|------------|-----------|--------|----------|
| Rust       | 0.9885    | 0.9925 | 0.9905   |
| Java       | 0.9731    | 0.9785 | 0.9758   |
| Dart       | 0.9772    | 0.9850 | 0.9811   |
| Python     | 0.9890    | 0.9880 | 0.9885   |
| Go         | 0.9859    | 0.9800 | 0.9829   |
| HTML       | 0.9279    | 0.8885 | 0.9078   |
| JavaScript | 0.8859    | 0.8930 | 0.8894   |
| TypeScript | 0.9466    | 0.9580 | 0.9523   |
| C          | 0.9566    | 0.9375 | 0.9470   |
| CSS        | 0.9728    | 0.9845 | 0.9786   |
| C#         | 0.9895    | 0.9870 | 0.9882   |
| Markdown   | 0.9671    | 0.9695 | 0.9683   |
| Assembly   | 0.9935    | 0.9945 | 0.9940   |
| Lua        | 0.9885    | 0.9915 | 0.9900   |
| C++        | 0.9770    | 0.9760 | 0.9765   |
| Kotlin     | 0.9840    | 0.9870 | 0.9855   |

### Key Observations
- The model performs exceptionally well on most languages, with 11 of 16 classes achieving an F1-score of 97% or higher.
- **JavaScript** (F1: 0.89) and **HTML** (F1: 0.91) are the most challenging classes, commonly confused with each other and with TypeScript/CSS.
- The model is highly confident in distinguishing structurally unique languages like **Assembly** (F1: 0.994) and **Python** (F1: 0.989).

## Environmental Impact

- **Hardware Type:** 2 x NVIDIA T4 GPUs
- **Hours used:** Approx. 4 epochs of training
- **Cloud Provider:** Not specified
- **Compute Region:** Not specified

*Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute).*