Update README.md
Browse files
README.md
CHANGED
|
@@ -9,12 +9,53 @@ tags:
|
|
| 9 |
|
| 10 |
# Embeddings
|
| 11 |
|
| 12 |
-
|
| 13 |
|
| 14 |
- [Mikolov et al 2013 - Distributed Representations of Words and Phrases (SGNS)](https://arxiv.org/pdf/1310.4546.pdf)
|
| 15 |
- [Rong, Xin 2014 - word2vec Parameter Learning Explained](https://arxiv.org/abs/1411.2738)
|
| 16 |
- [Levy & Goldberg 2014 - Neural Word Embedding as Implicit Matrix Factorization](https://papers.nips.cc/paper/2014/hash/feab05aa91085b7a8012516bc3533958-Abstract.html)
|
| 17 |
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
- [ocdbytes-ai/embeddings](https://github.com/ocdbytes-ai/embeddings)
|
|
|
|
| 9 |
|
| 10 |
# Embeddings
|
| 11 |
|
| 12 |
+
## Paper Refs. :
|
| 13 |
|
| 14 |
- [Mikolov et al 2013 - Distributed Representations of Words and Phrases (SGNS)](https://arxiv.org/pdf/1310.4546.pdf)
|
| 15 |
- [Rong, Xin 2014 - word2vec Parameter Learning Explained](https://arxiv.org/abs/1411.2738)
|
| 16 |
- [Levy & Goldberg 2014 - Neural Word Embedding as Implicit Matrix Factorization](https://papers.nips.cc/paper/2014/hash/feab05aa91085b7a8012516bc3533958-Abstract.html)
|
| 17 |
|
| 18 |
+
## Results
|
| 19 |
+
- **Nearest neighbours: strong** (e.g. `france → spain, italy, germany`)
|
| 20 |
+
- **Analogies: ~14% top-1** (semantic > morphological; limited by the small 17M-token corpus)
|
| 21 |
+
|
| 22 |
+
## Limitations
|
| 23 |
+
Small corpus → weak on analogies (esp. capital-country, morphology). For better analogy
|
| 24 |
+
accuracy, train on a larger corpus (enwik9+). Lowercased English only; drops OOV.
|
| 25 |
+
|
| 26 |
+
## Usage snippet
|
| 27 |
+
|
| 28 |
+
```python
|
| 29 |
+
from huggingface_hub import hf_hub_download
|
| 30 |
+
import torch
|
| 31 |
+
import torch.nn.functional as F
|
| 32 |
+
|
| 33 |
+
path = hf_hub_download(repo_id="ocdbytes/embeddings", filename="embeddings_200.pt")
|
| 34 |
+
# weights_only=False because the checkpoint bundles Python dicts (word2idx/idx2word),
|
| 35 |
+
# which the default restricted loader (torch>=2.6) may reject.
|
| 36 |
+
ck = torch.load(path, map_location="cpu", weights_only=False)
|
| 37 |
+
|
| 38 |
+
syn0 = ck["syn0"]
|
| 39 |
+
word2idx, idx2word = ck["word2idx"], ck["idx2word"]
|
| 40 |
+
emb = F.normalize(syn0, dim=1)
|
| 41 |
+
|
| 42 |
+
def neighbours(word, n=10):
|
| 43 |
+
i = word2idx[word]
|
| 44 |
+
sims = emb @ emb[i]
|
| 45 |
+
top = sims.topk(n + 1).indices.tolist()
|
| 46 |
+
return [idx2word[j] for j in top if j != i][:n]
|
| 47 |
+
|
| 48 |
+
def analogy(a, b, c, n=5):
|
| 49 |
+
t = F.normalize(emb[word2idx[b]] - emb[word2idx[a]] + emb[word2idx[c]], dim=0)
|
| 50 |
+
sims = emb @ t
|
| 51 |
+
ban = {word2idx[a], word2idx[b], word2idx[c]}
|
| 52 |
+
top = sims.topk(n + len(ban)).indices.tolist()
|
| 53 |
+
return [idx2word[j] for j in top if j not in ban][:n]
|
| 54 |
+
|
| 55 |
+
print(neighbours("king")) # -> ['viii', 'elizabeth', 'queen', ...]
|
| 56 |
+
print(analogy("france", "paris", "germany")) # -> ['berlin', ...]
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
## Code
|
| 60 |
|
| 61 |
- [ocdbytes-ai/embeddings](https://github.com/ocdbytes-ai/embeddings)
|