Shuu12121 commited on
Commit
9cd4cf8
·
verified ·
1 Parent(s): bb9454a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +5 -121
README.md CHANGED
@@ -27,27 +27,15 @@ tags:
27
  - php
28
  - ruby
29
  - go
 
30
  ---
31
 
32
  # **🦉CodeModernBERT-Owl**
33
 
34
- ## **概要 / Overview**
35
 
36
- ### **🦉 CodeModernBERT-Owl: 高精度なコード検索 & コード理解モデル**
37
- **CodeModernBERT-Owl** is a **pretrained model** designed from scratch for **code search and code understanding tasks**.
38
 
39
- Compared to previous versions such as **CodeHawks-ModernBERT** and **CodeMorph-ModernBERT**, this model **now supports Rust** and **improves search accuracy** in Python, PHP, Java, JavaScript, Go, and Ruby.
40
-
41
- ### **🛠 主な特徴 / Key Features**
42
- ✅ **Supports long sequences up to 2048 tokens** (compared to Microsoft's 512-token models)
43
- ✅ **Optimized for code search, code understanding, and code clone detection**
44
- ✅ **Fine-tuned on GitHub open-source repositories (Java, Rust)**
45
- ✅ **Achieves the highest accuracy among the CodeHawks/CodeMorph series**
46
- ✅ **Multi-language support**: **Python, PHP, Java, JavaScript, Go, Ruby, and Rust**
47
-
48
- ---
49
-
50
- ## **📊 モデルパラメータ / Model Parameters**
51
  | パラメータ / Parameter | 値 / Value |
52
  |-------------------------|------------|
53
  | **vocab_size** | 50,004 |
@@ -65,109 +53,5 @@ Compared to previous versions such as **CodeHawks-ModernBERT** and **CodeMorph-M
65
 
66
  ---
67
 
68
- ## **💻 モデルの使用方法 / How to Use**
69
- This model can be easily loaded using the **Hugging Face Transformers** library.
70
-
71
- ⚠️ **Requires transformers >= 4.48.0**
72
-
73
- 🔗 **[Colab Demo (Replace with "CodeModernBERT-Owl")](https://github.com/Shun0212/CodeBERTPretrained/blob/main/UseMyCodeMorph_ModernBERT.ipynb)**
74
-
75
- ### **モデルのロード / Load the Model**
76
- ```python
77
- from transformers import AutoModelForMaskedLM, AutoTokenizer
78
- tokenizer = AutoTokenizer.from_pretrained("Shuu12121/CodeModernBERT-Owl")
79
- model = AutoModelForMaskedLM.from_pretrained("Shuu12121/CodeModernBERT-Owl")
80
- ```
81
-
82
- ### **コード埋め込みの取得 / Get Code Embeddings**
83
- ```python
84
- import torch
85
- def get_embedding(text, model, tokenizer, device="cuda"):
86
- inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
87
- if "token_type_ids" in inputs:
88
- inputs.pop("token_type_ids")
89
- inputs = {k: v.to(device) for k, v in inputs.items()}
90
- outputs = model.model(**inputs)
91
- embedding = outputs.last_hidden_state[:, 0, :]
92
- return embedding
93
- embedding = get_embedding("def my_function(): pass", model, tokenizer)
94
- print(embedding.shape)
95
- ```
96
-
97
-
98
- ---
99
-
100
- # **🔍 評価結果 / Evaluation Results**
101
-
102
- ### **データセット / Dataset**
103
- 📌 **Tested on code_x_glue_ct_code_to_text with a candidate pool size of 100.**
104
- 📌 **Rust-specific evaluations were conducted using Shuu12121/rust-codesearch-dataset-open.**
105
-
106
- ---
107
-
108
- ## **📈 主要な評価指標の比較(同一シード値)/ Key Evaluation Metrics (Same Seed)**
109
- | 言語 / Language | **CodeModernBERT-Owl** | **CodeHawks-ModernBERT** | **Salesforce CodeT5+** | **Microsoft CodeBERT** | **GraphCodeBERT** |
110
- |-----------|-----------------|----------------------|-----------------|------------------|------------------|
111
- | **Python** | **0.8793** | 0.8551 | 0.8266 | 0.5243 | 0.5493 |
112
- | **Java** | **0.8880** | 0.7971 | **0.8867** | 0.3134 | 0.5879 |
113
- | **JavaScript** | **0.8423** | 0.7634 | 0.7628 | 0.2694 | 0.5051 |
114
- | **PHP** | **0.9129** | 0.8578 | **0.9027** | 0.2642 | 0.6225 |
115
- | **Ruby** | **0.8038** | 0.7469 | **0.7568** | 0.3318 | 0.5876 |
116
- | **Go** | **0.9386** | 0.9043 | 0.8117 | 0.3262 | 0.4243 |
117
-
118
- ✅ **Achieves the highest accuracy in all target languages.**
119
- ✅ **Significantly improved Java accuracy using additional fine-tuned GitHub data.**
120
- ✅ **Outperforms previous models, especially in PHP and Go.**
121
-
122
- ---
123
-
124
- ## **📊 Rust (独自データセット) / Rust Performance**
125
- | 指標 / Metric | **CodeModernBERT-Owl** |
126
- |--------------|----------------|
127
- | **MRR** | 0.7940 |
128
- | **MAP** | 0.7940 |
129
- | **R-Precision** | 0.7173 |
130
-
131
- ### **📌 K別評価指標 / Evaluation Metrics by K**
132
- | K | **Recall@K** | **Precision@K** | **NDCG@K** | **F1@K** | **Success Rate@K** | **Query Coverage@K** |
133
- |----|-------------|---------------|------------|--------|-----------------|-----------------|
134
- | **1** | 0.7173 | 0.7173 | 0.7173 | 0.7173 | 0.7173 | 0.7173 |
135
- | **5** | 0.8913 | 0.7852 | 0.8118 | 0.8132 | 0.8913 | 0.8913 |
136
- | **10** | 0.9333 | 0.7908 | 0.8254 | 0.8230 | 0.9333 | 0.9333 |
137
- | **50** | 0.9887 | 0.7938 | 0.8383 | 0.8288 | 0.9887 | 0.9887 |
138
- | **100** | 1.0000 | 0.7940 | 0.8401 | 0.8291 | 1.0000 | 1.0000 |
139
-
140
- ---
141
-
142
- ## **🔁 別のおすすめモデル / Recommended Alternative Models**
143
-
144
- ### 1. **CodeSearch-ModernBERT-Owl🦉** (https://huggingface.co/Shuu12121/CodeSearch-ModernBERT-Owl)
145
- If you need a model that is **more specialized for code search**, this model is highly recommended.
146
- コードサーチに**特化したモデルが必要な場合**はこちらがおすすめです。
147
-
148
- ### 2. **CodeModernBERT-Snake🐍** (https://huggingface.co/Shuu12121/CodeModernBERT-Snake)
149
- If you need a pretrained model that supports **longer sequences or a smaller model size**, this model is ideal.
150
- **シーケンス長が長い**、または**モデルサイズが小さい**事前学習済みモデルが必要な場合はこちらをおすすめします。
151
- - **Maximum Sequence Length:** 8192 tokens
152
- - **Smaller Model Size:** ~75M parameters
153
-
154
- ### 3. **CodeSearch-ModernBERT-Snake🐍** (https://huggingface.co/Shuu12121/CodeSearch-ModernBERT-Snake)
155
- For those looking for a model that combines **long sequence length and code search specialization**, this model is the best choice.
156
- **コードサーチに特化しつつ長いシーケンスを処理できるモデル**が欲しい場合にはこちらがおすすめです。
157
- - **Maximum Sequence Length:** 8192 tokens
158
- - **High Code Search Performance**
159
-
160
-
161
- ## **📝 結論 / Conclusion**
162
- ✅ **Top performance in all languages**
163
- ✅ **Rust support successfully added through dataset augmentation**
164
- ✅ **Further performance improvements possible with better datasets**
165
-
166
- ---
167
-
168
- ## **📜 ライセンス / License**
169
- 📄 **Apache-2.0**
170
-
171
- ## **📧 連絡先 / Contact**
172
- 📩 **For any questions, please contact:**
173
- 📧 **shun0212114@outlook.jp**
 
27
  - php
28
  - ruby
29
  - go
30
+ new_version: Shuu12121/CodeModernBERT-Owl-v3-len2048
31
  ---
32
 
33
  # **🦉CodeModernBERT-Owl**
34
 
35
+ 旧版のモデルです。[こちらのモデル](!Shuu12121/CodeModernBERT-Owl-v3-len2048)など最新版をお使いください!!
36
 
 
 
37
 
38
+ ## ** モデルパラメータ / Model Parameters**
 
 
 
 
 
 
 
 
 
 
 
39
  | パラメータ / Parameter | 値 / Value |
40
  |-------------------------|------------|
41
  | **vocab_size** | 50,004 |
 
53
 
54
  ---
55
 
56
+ ## **ライセンス / License**
57
+ **Apache-2.0**