English
embeddings
word2vec
ocdbytes commited on
Commit
b7ac173
·
verified ·
1 Parent(s): 8930aea

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +43 -2
README.md CHANGED
@@ -9,12 +9,53 @@ tags:
9
 
10
  # Embeddings
11
 
12
- **Paper Refs. :**
13
 
14
  - [Mikolov et al 2013 - Distributed Representations of Words and Phrases (SGNS)](https://arxiv.org/pdf/1310.4546.pdf)
15
  - [Rong, Xin 2014 - word2vec Parameter Learning Explained](https://arxiv.org/abs/1411.2738)
16
  - [Levy & Goldberg 2014 - Neural Word Embedding as Implicit Matrix Factorization](https://papers.nips.cc/paper/2014/hash/feab05aa91085b7a8012516bc3533958-Abstract.html)
17
 
18
- **Code :**
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
 
20
  - [ocdbytes-ai/embeddings](https://github.com/ocdbytes-ai/embeddings)
 
9
 
10
  # Embeddings
11
 
12
+ ## Paper Refs. :
13
 
14
  - [Mikolov et al 2013 - Distributed Representations of Words and Phrases (SGNS)](https://arxiv.org/pdf/1310.4546.pdf)
15
  - [Rong, Xin 2014 - word2vec Parameter Learning Explained](https://arxiv.org/abs/1411.2738)
16
  - [Levy & Goldberg 2014 - Neural Word Embedding as Implicit Matrix Factorization](https://papers.nips.cc/paper/2014/hash/feab05aa91085b7a8012516bc3533958-Abstract.html)
17
 
18
+ ## Results
19
+ - **Nearest neighbours: strong** (e.g. `france → spain, italy, germany`)
20
+ - **Analogies: ~14% top-1** (semantic > morphological; limited by the small 17M-token corpus)
21
+
22
+ ## Limitations
23
+ Small corpus → weak on analogies (esp. capital-country, morphology). For better analogy
24
+ accuracy, train on a larger corpus (enwik9+). Lowercased English only; drops OOV.
25
+
26
+ ## Usage snippet
27
+
28
+ ```python
29
+ from huggingface_hub import hf_hub_download
30
+ import torch
31
+ import torch.nn.functional as F
32
+
33
+ path = hf_hub_download(repo_id="ocdbytes/embeddings", filename="embeddings_200.pt")
34
+ # weights_only=False because the checkpoint bundles Python dicts (word2idx/idx2word),
35
+ # which the default restricted loader (torch>=2.6) may reject.
36
+ ck = torch.load(path, map_location="cpu", weights_only=False)
37
+
38
+ syn0 = ck["syn0"]
39
+ word2idx, idx2word = ck["word2idx"], ck["idx2word"]
40
+ emb = F.normalize(syn0, dim=1)
41
+
42
+ def neighbours(word, n=10):
43
+ i = word2idx[word]
44
+ sims = emb @ emb[i]
45
+ top = sims.topk(n + 1).indices.tolist()
46
+ return [idx2word[j] for j in top if j != i][:n]
47
+
48
+ def analogy(a, b, c, n=5):
49
+ t = F.normalize(emb[word2idx[b]] - emb[word2idx[a]] + emb[word2idx[c]], dim=0)
50
+ sims = emb @ t
51
+ ban = {word2idx[a], word2idx[b], word2idx[c]}
52
+ top = sims.topk(n + len(ban)).indices.tolist()
53
+ return [idx2word[j] for j in top if j not in ban][:n]
54
+
55
+ print(neighbours("king")) # -> ['viii', 'elizabeth', 'queen', ...]
56
+ print(analogy("france", "paris", "germany")) # -> ['berlin', ...]
57
+ ```
58
+
59
+ ## Code
60
 
61
  - [ocdbytes-ai/embeddings](https://github.com/ocdbytes-ai/embeddings)