why are these parameters 16bit when the phi-2 hosted on azure is 32bit?
The model summary here has sample code for "FP32 / CUDA" and "FP32 / CPU" but the parameters seem to be 16bit.
The files on azure seem to be 32bit (twice the size and list "torch_dtype": "float32" in the config.json)
Are the parameters uploaded here quantized?
It's not, fp16, or half precision is used by almost everyone nowadays because it's much efficient and faster than full precision.
The only reason to use fp32 now is either you have unsupported card or your fp16 performance is bad (Like P40).
You can also upcast it to fp32 iirc.
People usually use the term 'quantisation' if the bits per weight is 8 or less, this one is 16.
Anyway, from huggingface's docs:
torch_dtype (str, optional) — The dtype of the weights. This attribute can be used to initialize the model to a non-default dtype (which is normally float32) and thus allow for optimal storage allocation. For example, if the saved model is float16, ideally we want to load it back using the minimal amount of memory needed to load float16 weights. Since the config object is stored in plain text, this attribute contains just the floating type string without the torch. prefix. For example, for torch.float16 `torch_dtype is the "float16" string.
This attribute is currently not being used during model loading time, but this may change in the future versions. But we can already start preparing for the future by saving the dtype with save_pretrained.
fp16, or half precision is used by almost everyone nowadays
That's correct, but not responsive to my question. The parameters on azure are twice the size on disk (viewable in the screenshot below) as they are in this huggingface repo, and I'm asking why
