nvidia

/

NV-Embed-v1

like 428

Follow

NVIDIA 57.4k

Thanks for the great work.
Can you please specify the details of the MLP layer. It is mentioned in the paper that "MLP consists of two linear transformations with a GELU activation in between". Is the MLP size:

What I am unsure about is the value of the intermediate_dim in the following pseudo-code for mlp:

import torch

intermediate_dim = 4096  # ??? 
mlp = torch.nn.Sequential(
    torch.nn.Linear(4096, intermediate_dim),
    torch.nn.GELU(),
    torch.nn.Linear(intermediate_dim, 4096),
)

Is the above pseudo-code similar to what was used in the expts?

Sorry if this detail is mentioned in the paper and I missed it.

Thanks.