Joiin0392 commited on
Commit
3c04eb4
·
verified ·
1 Parent(s): 2fb20b5

fix: strip mid-string U+FFFD in realtime decode emit

Browse files

The end-of-buffer guard only catches an INCOMPLETE
trailing multi-byte char. Under sampling, a char can also break MID-string
(the next token completes a different char, orphaning the first one's tail
bytes) and the replacement character then stays in the output text forever
(observed: mojibake in Chinese captions). Strip the unrecoverable broken char
at emit time. Platform-neutral.

Files changed (1) hide show
  1. modeling_moss_vl.py +10 -1
modeling_moss_vl.py CHANGED
@@ -3604,7 +3604,16 @@ class MossVLForConditionalGeneration(MossVLPreTrainedModel, GenerationMixin):
3604
  and buf_list[-1] == invalid_token_id
3605
  )
3606
  if is_silence_or_ellipsis or is_complete_text or is_invalid_complete:
3607
- output_text_queue.put(decoded_text)
 
 
 
 
 
 
 
 
 
3608
  token_buffer.clear()
3609
 
3610
  input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)
 
3604
  and buf_list[-1] == invalid_token_id
3605
  )
3606
  if is_silence_or_ellipsis or is_complete_text or is_invalid_complete:
3607
+ # The end-of-buffer guard above only catches an INCOMPLETE
3608
+ # trailing char. Under sampling, a multi-byte char can also
3609
+ # break MID-string (the next token completes a different
3610
+ # char, orphaning the first char's tail bytes) — the
3611
+ # replacement char then stays in the text forever. Those
3612
+ # bytes are unrecoverable at this point: drop the broken
3613
+ # character instead of emitting mojibake.
3614
+ clean_text = decoded_text.replace("\ufffd", "")
3615
+ if clean_text:
3616
+ output_text_queue.put(clean_text)
3617
  token_buffer.clear()
3618
 
3619
  input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)