Prompt for point
Is it possible to prompt the model for a point rather then bbox? I'm asking it because the paper states the following:
In terms of grounding data format, to maintain consistency with the
original InternVL training process, we convert all box format data into the form [[x1, y1,
x2, x2]], where (x1, y1) and (x2, y2) are the normalized relative coordinates within the
range [0,1000]. Similarly, point data is converted into [[x, y]] format. ,
</box/>, , and are treated as special tokens.
I tried to modify the prompt from:
f'In this UI screenshot, what is the position of the element corresponding to the command "{utterance}" (with bbox)?'
With the result (skip_special_tokens=True, clean_up_tokenization_spaces=True):
['drive(160,366),(256,442)']
to:
f'In this UI screenshot, what is the position of the element corresponding to the command "{utterance}" (with point)?'
With the result (skip_special_tokens=True, clean_up_tokenization_spaces=True):
['tap drive[[110, 394, 136, 456]]']
I have to admit that, in order to save computational resources, we only included a small amount of point data when training the 7B model. (The 4B model's ability to generate points and bounding boxes should be normal.) This makes it difficult for the model to follow instructions that generate a point. If you really want to generate a point, you can try using only the element you want to target as the prompt, without adding any additional information, such as:
prompt = "main page"
However, I'm not sure if this will always be effective, especially when the command is as indirect as something like "utterance". The best approach might be to finetune the model further with a small amount of point-formatted data.