Update README.md

README.md

@@ -1,3 +1,140 @@
----
-license: mit
----
+---
+license: mit
+datasets:
+- Salesforce/wikitext
+- kd13/stack-v2-mini
+language:
+- en
+metrics:
+- perplexity
+base_model:
+- answerdotai/ModernBERT-base
+new_version: kd13/ModernBERT-base-mlm-wiki-code
+pipeline_tag: fill-mask
+library_name: transformers
+tags:
+- code
+- mlm
+- wiki
+---
+
+# ModernBERT-base-mlm-wiki-code
+
+## Model Summary
+
+**ModernBERT-base-mlm-wiki-code** is a continued pre-trained version of [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base), further trained on a large mixed corpus of English natural language text and multi-language source code using **Masked Language Modeling (MLM)**.
+
+The model was trained with a challenging **30% masking probability** (vs. the standard 15% in BERT) over a **2048 token context window**, achieving a final perplexity of **1.9507** — indicating strong and confident language + code understanding across both domains.
+
+---
+
+## Evaluation Results
+
+Evaluation was performed separately on each domain to understand per-domain performance.
+
+| Dataset                        | Eval Loss | Perplexity |
+|--------------------------------|-----------|------------|
+| WikiText (Natural Language)    | 1.3994    | 4.0526     |
+| The Stack V2 (Code)            | 0.4091    | 1.5054     |
+| **Combined (NL + Code)**       | **0.6728**| **1.9598** |
+
+> MLM Probability: 30% — Context Length: 2048 tokens
+---
+
+## Model Details
+
+| Property              | Value                                            |
+|-----------------------|--------------------------------------------------|
+| **Base Model**        | `answerdotai/ModernBERT-base`                    |
+| **Model Type**        | Masked Language Model (MLM)                      |
+| **Architecture**      | ModernBERT (Encoder-only Transformer)            |
+| **Context Length**    | 2048 tokens                                      |
+| **MLM Probability**   | 30%                                              |
+| **Languages**         | English, C++, Go, Java, JavaScript, Python       |
+| **License**           | MiT                                              |
+
+---
+
+## Usage
+
+### Installation
+
+```bash
+pip install transformers torch
+```
+
+### Load the Model
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+tokenizer = AutoTokenizer.from_pretrained("kd13/ModernBERT-base-mlm-wiki-code")
+model = AutoModelForMaskedLM.from_pretrained("kd13/ModernBERT-base-mlm-wiki-code")
+model.config.reference_compile = False
+```
+
+### Fill-Mask — Natural Language
+
+```python
+from transformers import pipeline
+
+pipe = pipeline("fill-mask", model="kd13/ModernBERT-base-mlm-wiki-code")
+
+result = pipe("The capital of France is [MASK].")
+for r in result:
+    print(f"{r['token_str']:15s} → {r['score']:.4f}")
+```
+
+### Fill-Mask — Source Code
+
+```python
+from transformers import pipeline
+
+pipe = pipeline("fill-mask", model="kd13/ModernBERT-base-mlm-wiki-code")
+
+result = pipe("def fibonacci(n): return n if n <= 1 else fibonacci(n-1) [MASK] fibonacci(n-2)")
+for r in result:
+    print(f"{r['token_str']:15s} → {r['score']:.4f}")
+```
+
+### Feature Extraction (Embeddings)
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+tokenizer = AutoTokenizer.from_pretrained("kd13/ModernBERT-base-mlm-wiki-code")
+model = AutoModel.from_pretrained("kd13/ModernBERT-base-mlm-wiki-code")
+
+text = "def quicksort(arr): return arr if len(arr) <= 1 else ..."
+inputs = tokenizer(text, return_tensors="pt", max_length=2048, truncation=True)
+
+with torch.no_grad():
+    outputs = model(**inputs)
+
+# CLS token embedding — shape: (1, 768)
+embedding = outputs.last_hidden_state[:, 0, :]
+print(embedding.shape)
+```
+
+---
+
+## Citation
+
+If you use this model, please cite the original ModernBERT paper:
+
+```bibtex
+@article{modernbert2024,
+  title   = {Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
+  author  = {Benjamin Warner and Antoine Chaffin and Benjamin Clavié and Orion Weller and Oskar Hallström and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
+  year    = {2024},
+  url     = {https://arxiv.org/abs/2412.13663}
+}
+```
+
+---
+
+## Acknowledgements
+
+- Base model by [Answer.AI](https://huggingface.co/answerdotai)
+- Training data from [WikiText](https://huggingface.co/datasets/wikimedia/wikipedia) and [BigCode The Stack V2](https://huggingface.co/datasets/bigcode/the-stack-v2)