Zero-valued token embeddings in model.embed_tokens.weight may disrupt the token sampling process
#43
by sszymczyk - opened
I noticed that in embed_tokens tensor tokens >= 200032 have embedding values all zero:
(Pdb) print(self.lm_head.weight[200032:])
tensor([[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]], dtype=torch.bfloat16, requires_grad=True)
Logits calculated from these embeddings will have value 0 as well. This will cause problems when logits calculated for all other tokens are all negative since samplers will treat zero-valued logits as more probable and will sample these added tokens with no defined textual representation.