Saidgurbuz commited on
Commit
78df0cc
·
verified ·
1 Parent(s): b216d7e

Update ScreenVLM for ScreenParse v2

Browse files

Upload the ScreenVLM checkpoint trained with ScreenParse v2 full-element supervision and update the model card. The v1 checkpoint remains available on the v1 branch.

README.md CHANGED
@@ -20,32 +20,36 @@ library_name: transformers
20
 
21
  # ScreenVLM
22
 
23
- **ScreenVLM** is a compact (316M-parameter) multimodal vision-language model for **complete screen parsing** detecting, classifying, and localizing all UI elements on a web page screenshot. Given an image, it produces a structured **ScreenTag** representation with bounding boxes, semantic labels (55 UI element classes), and text content for every visible element.
 
 
 
 
24
 
25
  - **Developed by**: IBM Research Zurich - ETH Zurich
26
  - **Model type**: Multi-modal model (image+text-to-text)
27
  - **Language(s)**: English
28
  - **License**: [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
29
- - **Paper**: [ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing](TODO)
30
- - **Code**: [GitHub](TODO)
31
- - **Dataset**: [docling-project/screenparse](https://huggingface.co/docling-project/screenparse)
32
 
33
  ## Model Summary
34
 
35
- ScreenVLM builds upon the [Idefics3](https://huggingface.co/docs/transformers/en/model_doc/idefics3) architecture with two key modifications: it uses [siglip2-base-patch16-512](https://huggingface.co/google/siglip2-base-patch16-512) as the vision encoder and a Granite 165M LLM as the language backbone. The model was trained on **ScreenParse**, a large-scale dataset of 771K web screenshots with dense UI element annotations across 55 semantic classes.
36
 
37
  ### Key Features
38
 
39
- - **Complete screen parsing**: Detects all UI elements on a page, not just sparse grounding targets
40
- - **55 UI element classes**: Buttons, links, inputs, navigation bars, menus, images, and more
41
- - **ScreenTag output format**: Structured, hierarchical representation with bounding boxes and text
42
- - **Compact size**: ~258M parameters (714MB safetensors), enabling fast inference
43
 
44
  ## Output Format
45
 
46
- ScreenVLM generates output in **ScreenTag** format — a structured representation where each UI element is wrapped in semantic tags with location tokens:
47
 
48
- ```
49
  <screentag>
50
  <button><loc_10><loc_20><loc_50><loc_35>Submit</button>
51
  <link><loc_100><loc_200><loc_180><loc_210>Learn more</link>
@@ -56,7 +60,7 @@ ScreenVLM generates output in **ScreenTag** format — a structured representati
56
  </screentag>
57
  ```
58
 
59
- Each `<loc_X>` token represents a coordinate in the normalized [0, 500] space. Four consecutive location tokens define `<left><top><right><bottom>` of the bounding box.
60
 
61
  ## Usage
62
 
@@ -72,10 +76,8 @@ DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
72
  MODEL_PATH = "docling-project/ScreenVLM"
73
  NORM_SIZE = 500
74
 
75
- # Load image
76
  image = load_image("https://example.com/screenshot.png")
77
 
78
- # Initialize processor and model
79
  processor = AutoProcessor.from_pretrained(MODEL_PATH)
80
  model = AutoModelForVision2Seq.from_pretrained(
81
  MODEL_PATH,
@@ -83,7 +85,6 @@ model = AutoModelForVision2Seq.from_pretrained(
83
  _attn_implementation="flash_attention_2" if DEVICE == "cuda" else "sdpa",
84
  ).to(DEVICE)
85
 
86
- # Create input
87
  messages = [
88
  {
89
  "role": "user",
@@ -97,7 +98,6 @@ messages = [
97
  prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
98
  inputs = processor(text=prompt, images=[image], return_tensors="pt").to(DEVICE)
99
 
100
- # Generate
101
  generated_ids = model.generate(**inputs, max_new_tokens=6192)
102
  prompt_length = inputs.input_ids.shape[1]
103
  output = processor.batch_decode(
@@ -105,7 +105,6 @@ output = processor.batch_decode(
105
  skip_special_tokens=False,
106
  )[0].lstrip()
107
 
108
- # Parse ScreenTag output into structured UI elements
109
  def parse_screentag(text, width, height):
110
  pattern = re.compile(
111
  r"<(?P<tag>[a-zA-Z][a-zA-Z0-9_]*)>"
@@ -115,8 +114,10 @@ def parse_screentag(text, width, height):
115
  elements = []
116
  for m in pattern.finditer(text):
117
  l, t, r, b = [max(0, min(int(m.group(k)), NORM_SIZE)) for k in ("l", "t", "r", "b")]
118
- if r < l: l, r = r, l
119
- if b < t: t, b = b, t
 
 
120
  x = l / NORM_SIZE * width
121
  y = t / NORM_SIZE * height
122
  w = (r - l) / NORM_SIZE * width
@@ -158,7 +159,6 @@ messages = [
158
  },
159
  ]
160
 
161
- # Initialize
162
  llm = LLM(model=MODEL_PATH, limit_mm_per_prompt={"image": 1})
163
  processor = AutoProcessor.from_pretrained(MODEL_PATH)
164
 
@@ -168,10 +168,8 @@ sampling_params = SamplingParams(
168
  skip_special_tokens=False,
169
  )
170
 
171
- # Build batch
172
  batched_inputs = []
173
  image_sizes = []
174
-
175
  for img_file in sorted(os.listdir(IMAGE_DIR)):
176
  if img_file.lower().endswith((".png", ".jpg", ".jpeg")):
177
  img_path = os.path.join(IMAGE_DIR, img_file)
@@ -180,53 +178,24 @@ for img_file in sorted(os.listdir(IMAGE_DIR)):
180
  batched_inputs.append({"prompt": prompt, "multi_modal_data": {"image": image}})
181
  image_sizes.append((img_file, image.size))
182
 
183
- # Run batch inference
184
  start = time.time()
185
  outputs = llm.generate(batched_inputs, sampling_params=sampling_params)
186
-
187
- # Parse ScreenTag output into structured UI elements
188
- def parse_screentag(text, width, height):
189
- pattern = re.compile(
190
- r"<(?P<tag>[a-zA-Z][a-zA-Z0-9_]*)>"
191
- r"\s*<loc_(?P<l>\d+)><loc_(?P<t>\d+)><loc_(?P<r>\d+)><loc_(?P<b>\d+)>"
192
- r"(?P<text>[^<]*)"
193
- )
194
- elements = []
195
- for m in pattern.finditer(text):
196
- l, t, r, b = [max(0, min(int(m.group(k)), NORM_SIZE)) for k in ("l", "t", "r", "b")]
197
- if r < l: l, r = r, l
198
- if b < t: t, b = b, t
199
- x = l / NORM_SIZE * width
200
- y = t / NORM_SIZE * height
201
- w = (r - l) / NORM_SIZE * width
202
- h = (b - t) / NORM_SIZE * height
203
- elements.append({
204
- "label": m.group("tag"),
205
- "bbox": (x, y, w, h),
206
- "text": m.group("text").strip() or None,
207
- })
208
- return elements
209
-
210
- for output, (name, (w, h)) in zip(outputs, image_sizes):
211
- screentag = output.outputs[0].text
212
- elements = parse_screentag(screentag, w, h)
213
- print(f"--- {name} ({len(elements)} elements) ---")
214
- for el in elements:
215
- print(f" {el['label']:20s} bbox=({int(el['bbox'][0]):4d},{int(el['bbox'][1]):4d},{int(el['bbox'][2]):4d},{int(el['bbox'][3]):4d}) text={el['text']!r}")
216
-
217
- print(f"\nTotal: {time.time() - start:.1f}s for {len(batched_inputs)} images")
218
  ```
219
 
220
  ## Training
221
 
222
  ScreenVLM was trained using the [nanoVLM](https://github.com/huggingface/nanoVLM) framework with 16 NVIDIA H100 GPUs.
223
 
224
- **Training data**: [ScreenParse](https://huggingface.co/docling-project/screenparse) 771K web page screenshots with dense annotations across 55 UI element classes, including bounding boxes, semantic labels, text content, interactability flags, and reading order. Annotations were generated through automated DOM extraction, IoU-based filtering, and VLM-based refinement (Qwen3-VL-8B).
 
 
225
 
226
  ## Limitations
227
 
228
  - Optimized for **web page screenshots**; performance on mobile or desktop application UIs may vary
229
- - May struggle with very dense or highly dynamic UIs (e.g., complex dashboards with hundreds of elements)
 
230
 
231
  ## Citation
232
 
 
20
 
21
  # ScreenVLM
22
 
23
+ **ScreenVLM** is a compact multimodal vision-language model for **complete screen parsing**: detecting, classifying, localizing, and transcribing UI elements on web page screenshots. Given an image, it produces a structured **ScreenTag** representation with bounding boxes, semantic labels across 55 UI element classes, and text content for visible elements.
24
+
25
+ ## News
26
+
27
+ - **May 2026**: Updated `main` with the ScreenVLM checkpoint trained on ScreenParse v2. This release uses the v2 training data with more robust quality filtering, 1,447,100 high-quality screenshots, and varied viewport resolutions. The original ScreenVLM checkpoint trained on ScreenParse v1 is retained on the `v1` branch.
28
 
29
  - **Developed by**: IBM Research Zurich - ETH Zurich
30
  - **Model type**: Multi-modal model (image+text-to-text)
31
  - **Language(s)**: English
32
  - **License**: [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
33
+ - **Paper**: [ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing](https://arxiv.org/abs/2602.14276)
34
+ - **Code**: https://github.com/Saidgurbuz/screenparse
35
+ - **Dataset**: [docling-project/screenparse](https://huggingface.co/datasets/docling-project/screenparse)
36
 
37
  ## Model Summary
38
 
39
+ ScreenVLM builds upon the [Idefics3](https://huggingface.co/docs/transformers/en/model_doc/idefics3) architecture with [siglip2-base-patch16-512](https://huggingface.co/google/siglip2-base-patch16-512) as the vision encoder and a Granite 165M LLM as the language backbone. The current `main` checkpoint was trained on ScreenParse v2 full-element screen parsing supervision across 55 semantic UI classes.
40
 
41
  ### Key Features
42
 
43
+ - **Complete screen parsing**: Detects all visible UI elements on a page, not only sparse grounding targets
44
+ - **55 UI element classes**: Buttons, links, inputs, navigation bars, menus, images, text, and more
45
+ - **ScreenTag output format**: Structured representation with semantic tags, location tokens, and text content
46
+ - **Compact size**: Single-file safetensors checkpoint suitable for fast inference
47
 
48
  ## Output Format
49
 
50
+ ScreenVLM generates output in **ScreenTag** format, where each UI element is wrapped in semantic tags with location tokens:
51
 
52
+ ```html
53
  <screentag>
54
  <button><loc_10><loc_20><loc_50><loc_35>Submit</button>
55
  <link><loc_100><loc_200><loc_180><loc_210>Learn more</link>
 
60
  </screentag>
61
  ```
62
 
63
+ Each `<loc_X>` token represents a coordinate in the normalized `[0, 500]` space. Four consecutive location tokens define `<left><top><right><bottom>` of the bounding box.
64
 
65
  ## Usage
66
 
 
76
  MODEL_PATH = "docling-project/ScreenVLM"
77
  NORM_SIZE = 500
78
 
 
79
  image = load_image("https://example.com/screenshot.png")
80
 
 
81
  processor = AutoProcessor.from_pretrained(MODEL_PATH)
82
  model = AutoModelForVision2Seq.from_pretrained(
83
  MODEL_PATH,
 
85
  _attn_implementation="flash_attention_2" if DEVICE == "cuda" else "sdpa",
86
  ).to(DEVICE)
87
 
 
88
  messages = [
89
  {
90
  "role": "user",
 
98
  prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
99
  inputs = processor(text=prompt, images=[image], return_tensors="pt").to(DEVICE)
100
 
 
101
  generated_ids = model.generate(**inputs, max_new_tokens=6192)
102
  prompt_length = inputs.input_ids.shape[1]
103
  output = processor.batch_decode(
 
105
  skip_special_tokens=False,
106
  )[0].lstrip()
107
 
 
108
  def parse_screentag(text, width, height):
109
  pattern = re.compile(
110
  r"<(?P<tag>[a-zA-Z][a-zA-Z0-9_]*)>"
 
114
  elements = []
115
  for m in pattern.finditer(text):
116
  l, t, r, b = [max(0, min(int(m.group(k)), NORM_SIZE)) for k in ("l", "t", "r", "b")]
117
+ if r < l:
118
+ l, r = r, l
119
+ if b < t:
120
+ t, b = b, t
121
  x = l / NORM_SIZE * width
122
  y = t / NORM_SIZE * height
123
  w = (r - l) / NORM_SIZE * width
 
159
  },
160
  ]
161
 
 
162
  llm = LLM(model=MODEL_PATH, limit_mm_per_prompt={"image": 1})
163
  processor = AutoProcessor.from_pretrained(MODEL_PATH)
164
 
 
168
  skip_special_tokens=False,
169
  )
170
 
 
171
  batched_inputs = []
172
  image_sizes = []
 
173
  for img_file in sorted(os.listdir(IMAGE_DIR)):
174
  if img_file.lower().endswith((".png", ".jpg", ".jpeg")):
175
  img_path = os.path.join(IMAGE_DIR, img_file)
 
178
  batched_inputs.append({"prompt": prompt, "multi_modal_data": {"image": image}})
179
  image_sizes.append((img_file, image.size))
180
 
 
181
  start = time.time()
182
  outputs = llm.generate(batched_inputs, sampling_params=sampling_params)
183
+ print(f"Total: {time.time() - start:.1f}s for {len(batched_inputs)} images")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
184
  ```
185
 
186
  ## Training
187
 
188
  ScreenVLM was trained using the [nanoVLM](https://github.com/huggingface/nanoVLM) framework with 16 NVIDIA H100 GPUs.
189
 
190
+ **Training data**: ScreenParse v2 full-element annotations for complete screen parsing. The v2 training data contains 1,447,100 high-quality web page screenshots across varied viewport resolutions with dense UI element supervision, including bounding boxes, semantic labels, text content, interactability flags, and reading order.
191
+
192
+ The original ScreenVLM checkpoint trained on ScreenParse v1 remains available with `revision="v1"`.
193
 
194
  ## Limitations
195
 
196
  - Optimized for **web page screenshots**; performance on mobile or desktop application UIs may vary
197
+ - May struggle with very dense or highly dynamic UIs, such as complex dashboards with hundreds of elements
198
+ - Produces structured screen parses, but downstream applications should still validate coordinates and text before using them for high-stakes automation
199
 
200
  ## Citation
201
 
added_tokens.json CHANGED
@@ -1,616 +1,4 @@
1
  {
2
- "</Alert>": 100352,
3
- "</App_Icon>": 100353,
4
- "</Avatar>": 100354,
5
- "</Badge>": 100355,
6
- "</Bottom_navigation>": 100356,
7
- "</Breadcrumb>": 100357,
8
- "</Button>": 100358,
9
- "</Calendar>": 100359,
10
- "</Carousel>": 100360,
11
- "</Chart>": 100361,
12
- "</Checkbox>": 100362,
13
- "</Code_snippet>": 100363,
14
- "</Column_Browser>": 100364,
15
- "</ContextMenu>": 100365,
16
- "</Date-Time_picker>": 100366,
17
- "</DockMenu>": 100367,
18
- "</EditMenu>": 100368,
19
- "</File_Icon>": 100369,
20
- "</Heading>": 100370,
21
- "</Image>": 100371,
22
- "</Link>": 100372,
23
- "</List>": 100373,
24
- "</List_Item>": 100374,
25
- "</Logo>": 100375,
26
- "</Menu>": 100376,
27
- "</Navigation_Bar>": 100377,
28
- "</Notification>": 100378,
29
- "</Page_control>": 100379,
30
- "</Pagination>": 100380,
31
- "</Picker>": 100381,
32
- "</PopUp_Menu>": 100382,
33
- "</Progress_bar>": 100383,
34
- "</Radiobox>": 100384,
35
- "</Rating_Indicator>": 100385,
36
- "</Screen>": 100386,
37
- "</Scroll>": 100387,
38
- "</Search_Bar>": 100388,
39
- "</Search_Field>": 100389,
40
- "</Select>": 100390,
41
- "</Side_Bar>": 100391,
42
- "</Slider>": 100392,
43
- "</Status_Bar>": 100393,
44
- "</Steppers>": 100394,
45
- "</Switch>": 100395,
46
- "</Tab>": 100396,
47
- "</Tab_Bar>": 100397,
48
- "</Table>": 100398,
49
- "</Text>": 100399,
50
- "</Text_Input>": 100400,
51
- "</Toggles>": 100401,
52
- "</Toolbar>": 100402,
53
- "</Tooltip>": 100403,
54
- "</Utility_Button>": 100404,
55
- "</Video>": 100405,
56
- "</Window>": 100406,
57
- "</screentag>": 100407,
58
- "<Alert>": 100408,
59
- "<App_Icon>": 100409,
60
- "<Avatar>": 100410,
61
- "<Badge>": 100411,
62
- "<Bottom_navigation>": 100412,
63
- "<Breadcrumb>": 100413,
64
- "<Button>": 100414,
65
- "<Calendar>": 100415,
66
- "<Carousel>": 100416,
67
- "<Chart>": 100417,
68
- "<Checkbox>": 100418,
69
- "<Code_snippet>": 100419,
70
- "<Column_Browser>": 100420,
71
- "<ContextMenu>": 100421,
72
- "<Date-Time_picker>": 100422,
73
- "<DockMenu>": 100423,
74
- "<EditMenu>": 100424,
75
- "<File_Icon>": 100425,
76
- "<Heading>": 100426,
77
- "<Image>": 100427,
78
- "<Link>": 100428,
79
- "<List>": 100429,
80
- "<List_Item>": 100430,
81
- "<Logo>": 100431,
82
- "<Menu>": 100432,
83
- "<Navigation_Bar>": 100433,
84
- "<Notification>": 100434,
85
- "<Page_control>": 100435,
86
- "<Pagination>": 100436,
87
- "<Picker>": 100437,
88
- "<PopUp_Menu>": 100438,
89
- "<Progress_bar>": 100439,
90
- "<Radiobox>": 100440,
91
- "<Rating_Indicator>": 100441,
92
- "<Screen>": 100442,
93
- "<Scroll>": 100443,
94
- "<Search_Bar>": 100444,
95
- "<Search_Field>": 100445,
96
- "<Select>": 100446,
97
- "<Side_Bar>": 100447,
98
- "<Slider>": 100448,
99
- "<Status_Bar>": 100449,
100
- "<Steppers>": 100450,
101
- "<Switch>": 100451,
102
- "<Tab>": 100452,
103
- "<Tab_Bar>": 100453,
104
- "<Table>": 100454,
105
- "<Text>": 100455,
106
- "<Text_Input>": 100456,
107
- "<Toggles>": 100457,
108
- "<Toolbar>": 100458,
109
- "<Tooltip>": 100459,
110
- "<Utility_Button>": 100460,
111
- "<Video>": 100461,
112
- "<Window>": 100462,
113
- "<end_of_utterance>": 100965,
114
- "<loc_0>": 100464,
115
- "<loc_100>": 100564,
116
- "<loc_101>": 100565,
117
- "<loc_102>": 100566,
118
- "<loc_103>": 100567,
119
- "<loc_104>": 100568,
120
- "<loc_105>": 100569,
121
- "<loc_106>": 100570,
122
- "<loc_107>": 100571,
123
- "<loc_108>": 100572,
124
- "<loc_109>": 100573,
125
- "<loc_10>": 100474,
126
- "<loc_110>": 100574,
127
- "<loc_111>": 100575,
128
- "<loc_112>": 100576,
129
- "<loc_113>": 100577,
130
- "<loc_114>": 100578,
131
- "<loc_115>": 100579,
132
- "<loc_116>": 100580,
133
- "<loc_117>": 100581,
134
- "<loc_118>": 100582,
135
- "<loc_119>": 100583,
136
- "<loc_11>": 100475,
137
- "<loc_120>": 100584,
138
- "<loc_121>": 100585,
139
- "<loc_122>": 100586,
140
- "<loc_123>": 100587,
141
- "<loc_124>": 100588,
142
- "<loc_125>": 100589,
143
- "<loc_126>": 100590,
144
- "<loc_127>": 100591,
145
- "<loc_128>": 100592,
146
- "<loc_129>": 100593,
147
- "<loc_12>": 100476,
148
- "<loc_130>": 100594,
149
- "<loc_131>": 100595,
150
- "<loc_132>": 100596,
151
- "<loc_133>": 100597,
152
- "<loc_134>": 100598,
153
- "<loc_135>": 100599,
154
- "<loc_136>": 100600,
155
- "<loc_137>": 100601,
156
- "<loc_138>": 100602,
157
- "<loc_139>": 100603,
158
- "<loc_13>": 100477,
159
- "<loc_140>": 100604,
160
- "<loc_141>": 100605,
161
- "<loc_142>": 100606,
162
- "<loc_143>": 100607,
163
- "<loc_144>": 100608,
164
- "<loc_145>": 100609,
165
- "<loc_146>": 100610,
166
- "<loc_147>": 100611,
167
- "<loc_148>": 100612,
168
- "<loc_149>": 100613,
169
- "<loc_14>": 100478,
170
- "<loc_150>": 100614,
171
- "<loc_151>": 100615,
172
- "<loc_152>": 100616,
173
- "<loc_153>": 100617,
174
- "<loc_154>": 100618,
175
- "<loc_155>": 100619,
176
- "<loc_156>": 100620,
177
- "<loc_157>": 100621,
178
- "<loc_158>": 100622,
179
- "<loc_159>": 100623,
180
- "<loc_15>": 100479,
181
- "<loc_160>": 100624,
182
- "<loc_161>": 100625,
183
- "<loc_162>": 100626,
184
- "<loc_163>": 100627,
185
- "<loc_164>": 100628,
186
- "<loc_165>": 100629,
187
- "<loc_166>": 100630,
188
- "<loc_167>": 100631,
189
- "<loc_168>": 100632,
190
- "<loc_169>": 100633,
191
- "<loc_16>": 100480,
192
- "<loc_170>": 100634,
193
- "<loc_171>": 100635,
194
- "<loc_172>": 100636,
195
- "<loc_173>": 100637,
196
- "<loc_174>": 100638,
197
- "<loc_175>": 100639,
198
- "<loc_176>": 100640,
199
- "<loc_177>": 100641,
200
- "<loc_178>": 100642,
201
- "<loc_179>": 100643,
202
- "<loc_17>": 100481,
203
- "<loc_180>": 100644,
204
- "<loc_181>": 100645,
205
- "<loc_182>": 100646,
206
- "<loc_183>": 100647,
207
- "<loc_184>": 100648,
208
- "<loc_185>": 100649,
209
- "<loc_186>": 100650,
210
- "<loc_187>": 100651,
211
- "<loc_188>": 100652,
212
- "<loc_189>": 100653,
213
- "<loc_18>": 100482,
214
- "<loc_190>": 100654,
215
- "<loc_191>": 100655,
216
- "<loc_192>": 100656,
217
- "<loc_193>": 100657,
218
- "<loc_194>": 100658,
219
- "<loc_195>": 100659,
220
- "<loc_196>": 100660,
221
- "<loc_197>": 100661,
222
- "<loc_198>": 100662,
223
- "<loc_199>": 100663,
224
- "<loc_19>": 100483,
225
- "<loc_1>": 100465,
226
- "<loc_200>": 100664,
227
- "<loc_201>": 100665,
228
- "<loc_202>": 100666,
229
- "<loc_203>": 100667,
230
- "<loc_204>": 100668,
231
- "<loc_205>": 100669,
232
- "<loc_206>": 100670,
233
- "<loc_207>": 100671,
234
- "<loc_208>": 100672,
235
- "<loc_209>": 100673,
236
- "<loc_20>": 100484,
237
- "<loc_210>": 100674,
238
- "<loc_211>": 100675,
239
- "<loc_212>": 100676,
240
- "<loc_213>": 100677,
241
- "<loc_214>": 100678,
242
- "<loc_215>": 100679,
243
- "<loc_216>": 100680,
244
- "<loc_217>": 100681,
245
- "<loc_218>": 100682,
246
- "<loc_219>": 100683,
247
- "<loc_21>": 100485,
248
- "<loc_220>": 100684,
249
- "<loc_221>": 100685,
250
- "<loc_222>": 100686,
251
- "<loc_223>": 100687,
252
- "<loc_224>": 100688,
253
- "<loc_225>": 100689,
254
- "<loc_226>": 100690,
255
- "<loc_227>": 100691,
256
- "<loc_228>": 100692,
257
- "<loc_229>": 100693,
258
- "<loc_22>": 100486,
259
- "<loc_230>": 100694,
260
- "<loc_231>": 100695,
261
- "<loc_232>": 100696,
262
- "<loc_233>": 100697,
263
- "<loc_234>": 100698,
264
- "<loc_235>": 100699,
265
- "<loc_236>": 100700,
266
- "<loc_237>": 100701,
267
- "<loc_238>": 100702,
268
- "<loc_239>": 100703,
269
- "<loc_23>": 100487,
270
- "<loc_240>": 100704,
271
- "<loc_241>": 100705,
272
- "<loc_242>": 100706,
273
- "<loc_243>": 100707,
274
- "<loc_244>": 100708,
275
- "<loc_245>": 100709,
276
- "<loc_246>": 100710,
277
- "<loc_247>": 100711,
278
- "<loc_248>": 100712,
279
- "<loc_249>": 100713,
280
- "<loc_24>": 100488,
281
- "<loc_250>": 100714,
282
- "<loc_251>": 100715,
283
- "<loc_252>": 100716,
284
- "<loc_253>": 100717,
285
- "<loc_254>": 100718,
286
- "<loc_255>": 100719,
287
- "<loc_256>": 100720,
288
- "<loc_257>": 100721,
289
- "<loc_258>": 100722,
290
- "<loc_259>": 100723,
291
- "<loc_25>": 100489,
292
- "<loc_260>": 100724,
293
- "<loc_261>": 100725,
294
- "<loc_262>": 100726,
295
- "<loc_263>": 100727,
296
- "<loc_264>": 100728,
297
- "<loc_265>": 100729,
298
- "<loc_266>": 100730,
299
- "<loc_267>": 100731,
300
- "<loc_268>": 100732,
301
- "<loc_269>": 100733,
302
- "<loc_26>": 100490,
303
- "<loc_270>": 100734,
304
- "<loc_271>": 100735,
305
- "<loc_272>": 100736,
306
- "<loc_273>": 100737,
307
- "<loc_274>": 100738,
308
- "<loc_275>": 100739,
309
- "<loc_276>": 100740,
310
- "<loc_277>": 100741,
311
- "<loc_278>": 100742,
312
- "<loc_279>": 100743,
313
- "<loc_27>": 100491,
314
- "<loc_280>": 100744,
315
- "<loc_281>": 100745,
316
- "<loc_282>": 100746,
317
- "<loc_283>": 100747,
318
- "<loc_284>": 100748,
319
- "<loc_285>": 100749,
320
- "<loc_286>": 100750,
321
- "<loc_287>": 100751,
322
- "<loc_288>": 100752,
323
- "<loc_289>": 100753,
324
- "<loc_28>": 100492,
325
- "<loc_290>": 100754,
326
- "<loc_291>": 100755,
327
- "<loc_292>": 100756,
328
- "<loc_293>": 100757,
329
- "<loc_294>": 100758,
330
- "<loc_295>": 100759,
331
- "<loc_296>": 100760,
332
- "<loc_297>": 100761,
333
- "<loc_298>": 100762,
334
- "<loc_299>": 100763,
335
- "<loc_29>": 100493,
336
- "<loc_2>": 100466,
337
- "<loc_300>": 100764,
338
- "<loc_301>": 100765,
339
- "<loc_302>": 100766,
340
- "<loc_303>": 100767,
341
- "<loc_304>": 100768,
342
- "<loc_305>": 100769,
343
- "<loc_306>": 100770,
344
- "<loc_307>": 100771,
345
- "<loc_308>": 100772,
346
- "<loc_309>": 100773,
347
- "<loc_30>": 100494,
348
- "<loc_310>": 100774,
349
- "<loc_311>": 100775,
350
- "<loc_312>": 100776,
351
- "<loc_313>": 100777,
352
- "<loc_314>": 100778,
353
- "<loc_315>": 100779,
354
- "<loc_316>": 100780,
355
- "<loc_317>": 100781,
356
- "<loc_318>": 100782,
357
- "<loc_319>": 100783,
358
- "<loc_31>": 100495,
359
- "<loc_320>": 100784,
360
- "<loc_321>": 100785,
361
- "<loc_322>": 100786,
362
- "<loc_323>": 100787,
363
- "<loc_324>": 100788,
364
- "<loc_325>": 100789,
365
- "<loc_326>": 100790,
366
- "<loc_327>": 100791,
367
- "<loc_328>": 100792,
368
- "<loc_329>": 100793,
369
- "<loc_32>": 100496,
370
- "<loc_330>": 100794,
371
- "<loc_331>": 100795,
372
- "<loc_332>": 100796,
373
- "<loc_333>": 100797,
374
- "<loc_334>": 100798,
375
- "<loc_335>": 100799,
376
- "<loc_336>": 100800,
377
- "<loc_337>": 100801,
378
- "<loc_338>": 100802,
379
- "<loc_339>": 100803,
380
- "<loc_33>": 100497,
381
- "<loc_340>": 100804,
382
- "<loc_341>": 100805,
383
- "<loc_342>": 100806,
384
- "<loc_343>": 100807,
385
- "<loc_344>": 100808,
386
- "<loc_345>": 100809,
387
- "<loc_346>": 100810,
388
- "<loc_347>": 100811,
389
- "<loc_348>": 100812,
390
- "<loc_349>": 100813,
391
- "<loc_34>": 100498,
392
- "<loc_350>": 100814,
393
- "<loc_351>": 100815,
394
- "<loc_352>": 100816,
395
- "<loc_353>": 100817,
396
- "<loc_354>": 100818,
397
- "<loc_355>": 100819,
398
- "<loc_356>": 100820,
399
- "<loc_357>": 100821,
400
- "<loc_358>": 100822,
401
- "<loc_359>": 100823,
402
- "<loc_35>": 100499,
403
- "<loc_360>": 100824,
404
- "<loc_361>": 100825,
405
- "<loc_362>": 100826,
406
- "<loc_363>": 100827,
407
- "<loc_364>": 100828,
408
- "<loc_365>": 100829,
409
- "<loc_366>": 100830,
410
- "<loc_367>": 100831,
411
- "<loc_368>": 100832,
412
- "<loc_369>": 100833,
413
- "<loc_36>": 100500,
414
- "<loc_370>": 100834,
415
- "<loc_371>": 100835,
416
- "<loc_372>": 100836,
417
- "<loc_373>": 100837,
418
- "<loc_374>": 100838,
419
- "<loc_375>": 100839,
420
- "<loc_376>": 100840,
421
- "<loc_377>": 100841,
422
- "<loc_378>": 100842,
423
- "<loc_379>": 100843,
424
- "<loc_37>": 100501,
425
- "<loc_380>": 100844,
426
- "<loc_381>": 100845,
427
- "<loc_382>": 100846,
428
- "<loc_383>": 100847,
429
- "<loc_384>": 100848,
430
- "<loc_385>": 100849,
431
- "<loc_386>": 100850,
432
- "<loc_387>": 100851,
433
- "<loc_388>": 100852,
434
- "<loc_389>": 100853,
435
- "<loc_38>": 100502,
436
- "<loc_390>": 100854,
437
- "<loc_391>": 100855,
438
- "<loc_392>": 100856,
439
- "<loc_393>": 100857,
440
- "<loc_394>": 100858,
441
- "<loc_395>": 100859,
442
- "<loc_396>": 100860,
443
- "<loc_397>": 100861,
444
- "<loc_398>": 100862,
445
- "<loc_399>": 100863,
446
- "<loc_39>": 100503,
447
- "<loc_3>": 100467,
448
- "<loc_400>": 100864,
449
- "<loc_401>": 100865,
450
- "<loc_402>": 100866,
451
- "<loc_403>": 100867,
452
- "<loc_404>": 100868,
453
- "<loc_405>": 100869,
454
- "<loc_406>": 100870,
455
- "<loc_407>": 100871,
456
- "<loc_408>": 100872,
457
- "<loc_409>": 100873,
458
- "<loc_40>": 100504,
459
- "<loc_410>": 100874,
460
- "<loc_411>": 100875,
461
- "<loc_412>": 100876,
462
- "<loc_413>": 100877,
463
- "<loc_414>": 100878,
464
- "<loc_415>": 100879,
465
- "<loc_416>": 100880,
466
- "<loc_417>": 100881,
467
- "<loc_418>": 100882,
468
- "<loc_419>": 100883,
469
- "<loc_41>": 100505,
470
- "<loc_420>": 100884,
471
- "<loc_421>": 100885,
472
- "<loc_422>": 100886,
473
- "<loc_423>": 100887,
474
- "<loc_424>": 100888,
475
- "<loc_425>": 100889,
476
- "<loc_426>": 100890,
477
- "<loc_427>": 100891,
478
- "<loc_428>": 100892,
479
- "<loc_429>": 100893,
480
- "<loc_42>": 100506,
481
- "<loc_430>": 100894,
482
- "<loc_431>": 100895,
483
- "<loc_432>": 100896,
484
- "<loc_433>": 100897,
485
- "<loc_434>": 100898,
486
- "<loc_435>": 100899,
487
- "<loc_436>": 100900,
488
- "<loc_437>": 100901,
489
- "<loc_438>": 100902,
490
- "<loc_439>": 100903,
491
- "<loc_43>": 100507,
492
- "<loc_440>": 100904,
493
- "<loc_441>": 100905,
494
- "<loc_442>": 100906,
495
- "<loc_443>": 100907,
496
- "<loc_444>": 100908,
497
- "<loc_445>": 100909,
498
- "<loc_446>": 100910,
499
- "<loc_447>": 100911,
500
- "<loc_448>": 100912,
501
- "<loc_449>": 100913,
502
- "<loc_44>": 100508,
503
- "<loc_450>": 100914,
504
- "<loc_451>": 100915,
505
- "<loc_452>": 100916,
506
- "<loc_453>": 100917,
507
- "<loc_454>": 100918,
508
- "<loc_455>": 100919,
509
- "<loc_456>": 100920,
510
- "<loc_457>": 100921,
511
- "<loc_458>": 100922,
512
- "<loc_459>": 100923,
513
- "<loc_45>": 100509,
514
- "<loc_460>": 100924,
515
- "<loc_461>": 100925,
516
- "<loc_462>": 100926,
517
- "<loc_463>": 100927,
518
- "<loc_464>": 100928,
519
- "<loc_465>": 100929,
520
- "<loc_466>": 100930,
521
- "<loc_467>": 100931,
522
- "<loc_468>": 100932,
523
- "<loc_469>": 100933,
524
- "<loc_46>": 100510,
525
- "<loc_470>": 100934,
526
- "<loc_471>": 100935,
527
- "<loc_472>": 100936,
528
- "<loc_473>": 100937,
529
- "<loc_474>": 100938,
530
- "<loc_475>": 100939,
531
- "<loc_476>": 100940,
532
- "<loc_477>": 100941,
533
- "<loc_478>": 100942,
534
- "<loc_479>": 100943,
535
- "<loc_47>": 100511,
536
- "<loc_480>": 100944,
537
- "<loc_481>": 100945,
538
- "<loc_482>": 100946,
539
- "<loc_483>": 100947,
540
- "<loc_484>": 100948,
541
- "<loc_485>": 100949,
542
- "<loc_486>": 100950,
543
- "<loc_487>": 100951,
544
- "<loc_488>": 100952,
545
- "<loc_489>": 100953,
546
- "<loc_48>": 100512,
547
- "<loc_490>": 100954,
548
- "<loc_491>": 100955,
549
- "<loc_492>": 100956,
550
- "<loc_493>": 100957,
551
- "<loc_494>": 100958,
552
- "<loc_495>": 100959,
553
- "<loc_496>": 100960,
554
- "<loc_497>": 100961,
555
- "<loc_498>": 100962,
556
- "<loc_499>": 100963,
557
- "<loc_49>": 100513,
558
- "<loc_4>": 100468,
559
- "<loc_500>": 100964,
560
- "<loc_50>": 100514,
561
- "<loc_51>": 100515,
562
- "<loc_52>": 100516,
563
- "<loc_53>": 100517,
564
- "<loc_54>": 100518,
565
- "<loc_55>": 100519,
566
- "<loc_56>": 100520,
567
- "<loc_57>": 100521,
568
- "<loc_58>": 100522,
569
- "<loc_59>": 100523,
570
- "<loc_5>": 100469,
571
- "<loc_60>": 100524,
572
- "<loc_61>": 100525,
573
- "<loc_62>": 100526,
574
- "<loc_63>": 100527,
575
- "<loc_64>": 100528,
576
- "<loc_65>": 100529,
577
- "<loc_66>": 100530,
578
- "<loc_67>": 100531,
579
- "<loc_68>": 100532,
580
- "<loc_69>": 100533,
581
- "<loc_6>": 100470,
582
- "<loc_70>": 100534,
583
- "<loc_71>": 100535,
584
- "<loc_72>": 100536,
585
- "<loc_73>": 100537,
586
- "<loc_74>": 100538,
587
- "<loc_75>": 100539,
588
- "<loc_76>": 100540,
589
- "<loc_77>": 100541,
590
- "<loc_78>": 100542,
591
- "<loc_79>": 100543,
592
- "<loc_7>": 100471,
593
- "<loc_80>": 100544,
594
- "<loc_81>": 100545,
595
- "<loc_82>": 100546,
596
- "<loc_83>": 100547,
597
- "<loc_84>": 100548,
598
- "<loc_85>": 100549,
599
- "<loc_86>": 100550,
600
- "<loc_87>": 100551,
601
- "<loc_88>": 100552,
602
- "<loc_89>": 100553,
603
- "<loc_8>": 100472,
604
- "<loc_90>": 100554,
605
- "<loc_91>": 100555,
606
- "<loc_92>": 100556,
607
- "<loc_93>": 100557,
608
- "<loc_94>": 100558,
609
- "<loc_95>": 100559,
610
- "<loc_96>": 100560,
611
- "<loc_97>": 100561,
612
- "<loc_98>": 100562,
613
- "<loc_99>": 100563,
614
- "<loc_9>": 100473,
615
- "<screentag>": 100463
616
  }
 
1
  {
2
+ "<end_of_utterance>": 100353,
3
+ "<layout>": 100352
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4
  }
config.json CHANGED
@@ -9,7 +9,7 @@
9
  "pad_token_id": 100257,
10
  "scale_factor": 4,
11
  "text_config": {
12
- "_name_or_path": "/proj/docling-vision/models/granitev06_hf_ai4k_sft_data_v4",
13
  "architectures": [
14
  "LlamaForCausalLM"
15
  ],
@@ -35,7 +35,7 @@
35
  "rope_theta": 100000.0,
36
  "torch_dtype": "bfloat16",
37
  "use_cache": false,
38
- "vocab_size": 100965
39
  },
40
  "tie_word_embeddings": false,
41
  "torch_dtype": "bfloat16",
@@ -61,5 +61,5 @@
61
  "longest_edge": 512
62
  }
63
  },
64
- "vocab_size": 100965
65
  }
 
9
  "pad_token_id": 100257,
10
  "scale_factor": 4,
11
  "text_config": {
12
+ "_name_or_path": "/proj/docling-vision/models/granitev06_hf_ai4k_sft_data_v5",
13
  "architectures": [
14
  "LlamaForCausalLM"
15
  ],
 
35
  "rope_theta": 100000.0,
36
  "torch_dtype": "bfloat16",
37
  "use_cache": false,
38
+ "vocab_size": 100352
39
  },
40
  "tie_word_embeddings": false,
41
  "torch_dtype": "bfloat16",
 
61
  "longest_edge": 512
62
  }
63
  },
64
+ "vocab_size": 100352
65
  }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:459614601e12b3e547fc8412104ab6a915bd28f08ba6ab766e166e23573e686c
3
- size 748422736
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c02d35755f6b6d3b85b8354e3ba42422ede06a2151c09034e5112a3fc4c96c91
3
+ size 746304208
special_tokens_map.json CHANGED
@@ -1,114 +1,4 @@
1
  {
2
- "Alert_close": "</Alert>",
3
- "Alert_open": "<Alert>",
4
- "App_Icon_close": "</App_Icon>",
5
- "App_Icon_open": "<App_Icon>",
6
- "Avatar_close": "</Avatar>",
7
- "Avatar_open": "<Avatar>",
8
- "Badge_close": "</Badge>",
9
- "Badge_open": "<Badge>",
10
- "Bottom_navigation_close": "</Bottom_navigation>",
11
- "Bottom_navigation_open": "<Bottom_navigation>",
12
- "Breadcrumb_close": "</Breadcrumb>",
13
- "Breadcrumb_open": "<Breadcrumb>",
14
- "Button_close": "</Button>",
15
- "Button_open": "<Button>",
16
- "Calendar_close": "</Calendar>",
17
- "Calendar_open": "<Calendar>",
18
- "Carousel_close": "</Carousel>",
19
- "Carousel_open": "<Carousel>",
20
- "Chart_close": "</Chart>",
21
- "Chart_open": "<Chart>",
22
- "Checkbox_close": "</Checkbox>",
23
- "Checkbox_open": "<Checkbox>",
24
- "Code_snippet_close": "</Code_snippet>",
25
- "Code_snippet_open": "<Code_snippet>",
26
- "Column_Browser_close": "</Column_Browser>",
27
- "Column_Browser_open": "<Column_Browser>",
28
- "ContextMenu_close": "</ContextMenu>",
29
- "ContextMenu_open": "<ContextMenu>",
30
- "Date-Time_picker_close": "</Date-Time_picker>",
31
- "Date-Time_picker_open": "<Date-Time_picker>",
32
- "DockMenu_close": "</DockMenu>",
33
- "DockMenu_open": "<DockMenu>",
34
- "EditMenu_close": "</EditMenu>",
35
- "EditMenu_open": "<EditMenu>",
36
- "File_Icon_close": "</File_Icon>",
37
- "File_Icon_open": "<File_Icon>",
38
- "Heading_close": "</Heading>",
39
- "Heading_open": "<Heading>",
40
- "Image_close": "</Image>",
41
- "Image_open": "<Image>",
42
- "Link_close": "</Link>",
43
- "Link_open": "<Link>",
44
- "List_Item_close": "</List_Item>",
45
- "List_Item_open": "<List_Item>",
46
- "List_close": "</List>",
47
- "List_open": "<List>",
48
- "Logo_close": "</Logo>",
49
- "Logo_open": "<Logo>",
50
- "Menu_close": "</Menu>",
51
- "Menu_open": "<Menu>",
52
- "Navigation_Bar_close": "</Navigation_Bar>",
53
- "Navigation_Bar_open": "<Navigation_Bar>",
54
- "Notification_close": "</Notification>",
55
- "Notification_open": "<Notification>",
56
- "Page_control_close": "</Page_control>",
57
- "Page_control_open": "<Page_control>",
58
- "Pagination_close": "</Pagination>",
59
- "Pagination_open": "<Pagination>",
60
- "Picker_close": "</Picker>",
61
- "Picker_open": "<Picker>",
62
- "PopUp_Menu_close": "</PopUp_Menu>",
63
- "PopUp_Menu_open": "<PopUp_Menu>",
64
- "Progress_bar_close": "</Progress_bar>",
65
- "Progress_bar_open": "<Progress_bar>",
66
- "Radiobox_close": "</Radiobox>",
67
- "Radiobox_open": "<Radiobox>",
68
- "Rating_Indicator_close": "</Rating_Indicator>",
69
- "Rating_Indicator_open": "<Rating_Indicator>",
70
- "Screen_close": "</Screen>",
71
- "Screen_open": "<Screen>",
72
- "Scroll_close": "</Scroll>",
73
- "Scroll_open": "<Scroll>",
74
- "Search_Bar_close": "</Search_Bar>",
75
- "Search_Bar_open": "<Search_Bar>",
76
- "Search_Field_close": "</Search_Field>",
77
- "Search_Field_open": "<Search_Field>",
78
- "Select_close": "</Select>",
79
- "Select_open": "<Select>",
80
- "Side_Bar_close": "</Side_Bar>",
81
- "Side_Bar_open": "<Side_Bar>",
82
- "Slider_close": "</Slider>",
83
- "Slider_open": "<Slider>",
84
- "Status_Bar_close": "</Status_Bar>",
85
- "Status_Bar_open": "<Status_Bar>",
86
- "Steppers_close": "</Steppers>",
87
- "Steppers_open": "<Steppers>",
88
- "Switch_close": "</Switch>",
89
- "Switch_open": "<Switch>",
90
- "Tab_Bar_close": "</Tab_Bar>",
91
- "Tab_Bar_open": "<Tab_Bar>",
92
- "Tab_close": "</Tab>",
93
- "Tab_open": "<Tab>",
94
- "Table_close": "</Table>",
95
- "Table_open": "<Table>",
96
- "Text_Input_close": "</Text_Input>",
97
- "Text_Input_open": "<Text_Input>",
98
- "Text_close": "</Text>",
99
- "Text_open": "<Text>",
100
- "Toggles_close": "</Toggles>",
101
- "Toggles_open": "<Toggles>",
102
- "Toolbar_close": "</Toolbar>",
103
- "Toolbar_open": "<Toolbar>",
104
- "Tooltip_close": "</Tooltip>",
105
- "Tooltip_open": "<Tooltip>",
106
- "Utility_Button_close": "</Utility_Button>",
107
- "Utility_Button_open": "<Utility_Button>",
108
- "Video_close": "</Video>",
109
- "Video_open": "<Video>",
110
- "Window_close": "</Window>",
111
- "Window_open": "<Window>",
112
  "additional_special_tokens": [
113
  {
114
  "content": "<fake_token_around_image>",
@@ -139,8 +29,6 @@
139
  "rstrip": false,
140
  "single_word": false
141
  },
142
- "end_of_role": "<|end_of_role|>",
143
- "end_of_text": "<|end_of_text|>",
144
  "eos_token": {
145
  "content": "<|end_of_text|>",
146
  "lstrip": false,
@@ -148,514 +36,7 @@
148
  "rstrip": false,
149
  "single_word": false
150
  },
151
- "fake_token_around_image": "<fake_token_around_image>",
152
- "global-img": "<global-img>",
153
- "image_token": "<image>",
154
- "loc_0": "<loc_0>",
155
- "loc_1": "<loc_1>",
156
- "loc_10": "<loc_10>",
157
- "loc_100": "<loc_100>",
158
- "loc_101": "<loc_101>",
159
- "loc_102": "<loc_102>",
160
- "loc_103": "<loc_103>",
161
- "loc_104": "<loc_104>",
162
- "loc_105": "<loc_105>",
163
- "loc_106": "<loc_106>",
164
- "loc_107": "<loc_107>",
165
- "loc_108": "<loc_108>",
166
- "loc_109": "<loc_109>",
167
- "loc_11": "<loc_11>",
168
- "loc_110": "<loc_110>",
169
- "loc_111": "<loc_111>",
170
- "loc_112": "<loc_112>",
171
- "loc_113": "<loc_113>",
172
- "loc_114": "<loc_114>",
173
- "loc_115": "<loc_115>",
174
- "loc_116": "<loc_116>",
175
- "loc_117": "<loc_117>",
176
- "loc_118": "<loc_118>",
177
- "loc_119": "<loc_119>",
178
- "loc_12": "<loc_12>",
179
- "loc_120": "<loc_120>",
180
- "loc_121": "<loc_121>",
181
- "loc_122": "<loc_122>",
182
- "loc_123": "<loc_123>",
183
- "loc_124": "<loc_124>",
184
- "loc_125": "<loc_125>",
185
- "loc_126": "<loc_126>",
186
- "loc_127": "<loc_127>",
187
- "loc_128": "<loc_128>",
188
- "loc_129": "<loc_129>",
189
- "loc_13": "<loc_13>",
190
- "loc_130": "<loc_130>",
191
- "loc_131": "<loc_131>",
192
- "loc_132": "<loc_132>",
193
- "loc_133": "<loc_133>",
194
- "loc_134": "<loc_134>",
195
- "loc_135": "<loc_135>",
196
- "loc_136": "<loc_136>",
197
- "loc_137": "<loc_137>",
198
- "loc_138": "<loc_138>",
199
- "loc_139": "<loc_139>",
200
- "loc_14": "<loc_14>",
201
- "loc_140": "<loc_140>",
202
- "loc_141": "<loc_141>",
203
- "loc_142": "<loc_142>",
204
- "loc_143": "<loc_143>",
205
- "loc_144": "<loc_144>",
206
- "loc_145": "<loc_145>",
207
- "loc_146": "<loc_146>",
208
- "loc_147": "<loc_147>",
209
- "loc_148": "<loc_148>",
210
- "loc_149": "<loc_149>",
211
- "loc_15": "<loc_15>",
212
- "loc_150": "<loc_150>",
213
- "loc_151": "<loc_151>",
214
- "loc_152": "<loc_152>",
215
- "loc_153": "<loc_153>",
216
- "loc_154": "<loc_154>",
217
- "loc_155": "<loc_155>",
218
- "loc_156": "<loc_156>",
219
- "loc_157": "<loc_157>",
220
- "loc_158": "<loc_158>",
221
- "loc_159": "<loc_159>",
222
- "loc_16": "<loc_16>",
223
- "loc_160": "<loc_160>",
224
- "loc_161": "<loc_161>",
225
- "loc_162": "<loc_162>",
226
- "loc_163": "<loc_163>",
227
- "loc_164": "<loc_164>",
228
- "loc_165": "<loc_165>",
229
- "loc_166": "<loc_166>",
230
- "loc_167": "<loc_167>",
231
- "loc_168": "<loc_168>",
232
- "loc_169": "<loc_169>",
233
- "loc_17": "<loc_17>",
234
- "loc_170": "<loc_170>",
235
- "loc_171": "<loc_171>",
236
- "loc_172": "<loc_172>",
237
- "loc_173": "<loc_173>",
238
- "loc_174": "<loc_174>",
239
- "loc_175": "<loc_175>",
240
- "loc_176": "<loc_176>",
241
- "loc_177": "<loc_177>",
242
- "loc_178": "<loc_178>",
243
- "loc_179": "<loc_179>",
244
- "loc_18": "<loc_18>",
245
- "loc_180": "<loc_180>",
246
- "loc_181": "<loc_181>",
247
- "loc_182": "<loc_182>",
248
- "loc_183": "<loc_183>",
249
- "loc_184": "<loc_184>",
250
- "loc_185": "<loc_185>",
251
- "loc_186": "<loc_186>",
252
- "loc_187": "<loc_187>",
253
- "loc_188": "<loc_188>",
254
- "loc_189": "<loc_189>",
255
- "loc_19": "<loc_19>",
256
- "loc_190": "<loc_190>",
257
- "loc_191": "<loc_191>",
258
- "loc_192": "<loc_192>",
259
- "loc_193": "<loc_193>",
260
- "loc_194": "<loc_194>",
261
- "loc_195": "<loc_195>",
262
- "loc_196": "<loc_196>",
263
- "loc_197": "<loc_197>",
264
- "loc_198": "<loc_198>",
265
- "loc_199": "<loc_199>",
266
- "loc_2": "<loc_2>",
267
- "loc_20": "<loc_20>",
268
- "loc_200": "<loc_200>",
269
- "loc_201": "<loc_201>",
270
- "loc_202": "<loc_202>",
271
- "loc_203": "<loc_203>",
272
- "loc_204": "<loc_204>",
273
- "loc_205": "<loc_205>",
274
- "loc_206": "<loc_206>",
275
- "loc_207": "<loc_207>",
276
- "loc_208": "<loc_208>",
277
- "loc_209": "<loc_209>",
278
- "loc_21": "<loc_21>",
279
- "loc_210": "<loc_210>",
280
- "loc_211": "<loc_211>",
281
- "loc_212": "<loc_212>",
282
- "loc_213": "<loc_213>",
283
- "loc_214": "<loc_214>",
284
- "loc_215": "<loc_215>",
285
- "loc_216": "<loc_216>",
286
- "loc_217": "<loc_217>",
287
- "loc_218": "<loc_218>",
288
- "loc_219": "<loc_219>",
289
- "loc_22": "<loc_22>",
290
- "loc_220": "<loc_220>",
291
- "loc_221": "<loc_221>",
292
- "loc_222": "<loc_222>",
293
- "loc_223": "<loc_223>",
294
- "loc_224": "<loc_224>",
295
- "loc_225": "<loc_225>",
296
- "loc_226": "<loc_226>",
297
- "loc_227": "<loc_227>",
298
- "loc_228": "<loc_228>",
299
- "loc_229": "<loc_229>",
300
- "loc_23": "<loc_23>",
301
- "loc_230": "<loc_230>",
302
- "loc_231": "<loc_231>",
303
- "loc_232": "<loc_232>",
304
- "loc_233": "<loc_233>",
305
- "loc_234": "<loc_234>",
306
- "loc_235": "<loc_235>",
307
- "loc_236": "<loc_236>",
308
- "loc_237": "<loc_237>",
309
- "loc_238": "<loc_238>",
310
- "loc_239": "<loc_239>",
311
- "loc_24": "<loc_24>",
312
- "loc_240": "<loc_240>",
313
- "loc_241": "<loc_241>",
314
- "loc_242": "<loc_242>",
315
- "loc_243": "<loc_243>",
316
- "loc_244": "<loc_244>",
317
- "loc_245": "<loc_245>",
318
- "loc_246": "<loc_246>",
319
- "loc_247": "<loc_247>",
320
- "loc_248": "<loc_248>",
321
- "loc_249": "<loc_249>",
322
- "loc_25": "<loc_25>",
323
- "loc_250": "<loc_250>",
324
- "loc_251": "<loc_251>",
325
- "loc_252": "<loc_252>",
326
- "loc_253": "<loc_253>",
327
- "loc_254": "<loc_254>",
328
- "loc_255": "<loc_255>",
329
- "loc_256": "<loc_256>",
330
- "loc_257": "<loc_257>",
331
- "loc_258": "<loc_258>",
332
- "loc_259": "<loc_259>",
333
- "loc_26": "<loc_26>",
334
- "loc_260": "<loc_260>",
335
- "loc_261": "<loc_261>",
336
- "loc_262": "<loc_262>",
337
- "loc_263": "<loc_263>",
338
- "loc_264": "<loc_264>",
339
- "loc_265": "<loc_265>",
340
- "loc_266": "<loc_266>",
341
- "loc_267": "<loc_267>",
342
- "loc_268": "<loc_268>",
343
- "loc_269": "<loc_269>",
344
- "loc_27": "<loc_27>",
345
- "loc_270": "<loc_270>",
346
- "loc_271": "<loc_271>",
347
- "loc_272": "<loc_272>",
348
- "loc_273": "<loc_273>",
349
- "loc_274": "<loc_274>",
350
- "loc_275": "<loc_275>",
351
- "loc_276": "<loc_276>",
352
- "loc_277": "<loc_277>",
353
- "loc_278": "<loc_278>",
354
- "loc_279": "<loc_279>",
355
- "loc_28": "<loc_28>",
356
- "loc_280": "<loc_280>",
357
- "loc_281": "<loc_281>",
358
- "loc_282": "<loc_282>",
359
- "loc_283": "<loc_283>",
360
- "loc_284": "<loc_284>",
361
- "loc_285": "<loc_285>",
362
- "loc_286": "<loc_286>",
363
- "loc_287": "<loc_287>",
364
- "loc_288": "<loc_288>",
365
- "loc_289": "<loc_289>",
366
- "loc_29": "<loc_29>",
367
- "loc_290": "<loc_290>",
368
- "loc_291": "<loc_291>",
369
- "loc_292": "<loc_292>",
370
- "loc_293": "<loc_293>",
371
- "loc_294": "<loc_294>",
372
- "loc_295": "<loc_295>",
373
- "loc_296": "<loc_296>",
374
- "loc_297": "<loc_297>",
375
- "loc_298": "<loc_298>",
376
- "loc_299": "<loc_299>",
377
- "loc_3": "<loc_3>",
378
- "loc_30": "<loc_30>",
379
- "loc_300": "<loc_300>",
380
- "loc_301": "<loc_301>",
381
- "loc_302": "<loc_302>",
382
- "loc_303": "<loc_303>",
383
- "loc_304": "<loc_304>",
384
- "loc_305": "<loc_305>",
385
- "loc_306": "<loc_306>",
386
- "loc_307": "<loc_307>",
387
- "loc_308": "<loc_308>",
388
- "loc_309": "<loc_309>",
389
- "loc_31": "<loc_31>",
390
- "loc_310": "<loc_310>",
391
- "loc_311": "<loc_311>",
392
- "loc_312": "<loc_312>",
393
- "loc_313": "<loc_313>",
394
- "loc_314": "<loc_314>",
395
- "loc_315": "<loc_315>",
396
- "loc_316": "<loc_316>",
397
- "loc_317": "<loc_317>",
398
- "loc_318": "<loc_318>",
399
- "loc_319": "<loc_319>",
400
- "loc_32": "<loc_32>",
401
- "loc_320": "<loc_320>",
402
- "loc_321": "<loc_321>",
403
- "loc_322": "<loc_322>",
404
- "loc_323": "<loc_323>",
405
- "loc_324": "<loc_324>",
406
- "loc_325": "<loc_325>",
407
- "loc_326": "<loc_326>",
408
- "loc_327": "<loc_327>",
409
- "loc_328": "<loc_328>",
410
- "loc_329": "<loc_329>",
411
- "loc_33": "<loc_33>",
412
- "loc_330": "<loc_330>",
413
- "loc_331": "<loc_331>",
414
- "loc_332": "<loc_332>",
415
- "loc_333": "<loc_333>",
416
- "loc_334": "<loc_334>",
417
- "loc_335": "<loc_335>",
418
- "loc_336": "<loc_336>",
419
- "loc_337": "<loc_337>",
420
- "loc_338": "<loc_338>",
421
- "loc_339": "<loc_339>",
422
- "loc_34": "<loc_34>",
423
- "loc_340": "<loc_340>",
424
- "loc_341": "<loc_341>",
425
- "loc_342": "<loc_342>",
426
- "loc_343": "<loc_343>",
427
- "loc_344": "<loc_344>",
428
- "loc_345": "<loc_345>",
429
- "loc_346": "<loc_346>",
430
- "loc_347": "<loc_347>",
431
- "loc_348": "<loc_348>",
432
- "loc_349": "<loc_349>",
433
- "loc_35": "<loc_35>",
434
- "loc_350": "<loc_350>",
435
- "loc_351": "<loc_351>",
436
- "loc_352": "<loc_352>",
437
- "loc_353": "<loc_353>",
438
- "loc_354": "<loc_354>",
439
- "loc_355": "<loc_355>",
440
- "loc_356": "<loc_356>",
441
- "loc_357": "<loc_357>",
442
- "loc_358": "<loc_358>",
443
- "loc_359": "<loc_359>",
444
- "loc_36": "<loc_36>",
445
- "loc_360": "<loc_360>",
446
- "loc_361": "<loc_361>",
447
- "loc_362": "<loc_362>",
448
- "loc_363": "<loc_363>",
449
- "loc_364": "<loc_364>",
450
- "loc_365": "<loc_365>",
451
- "loc_366": "<loc_366>",
452
- "loc_367": "<loc_367>",
453
- "loc_368": "<loc_368>",
454
- "loc_369": "<loc_369>",
455
- "loc_37": "<loc_37>",
456
- "loc_370": "<loc_370>",
457
- "loc_371": "<loc_371>",
458
- "loc_372": "<loc_372>",
459
- "loc_373": "<loc_373>",
460
- "loc_374": "<loc_374>",
461
- "loc_375": "<loc_375>",
462
- "loc_376": "<loc_376>",
463
- "loc_377": "<loc_377>",
464
- "loc_378": "<loc_378>",
465
- "loc_379": "<loc_379>",
466
- "loc_38": "<loc_38>",
467
- "loc_380": "<loc_380>",
468
- "loc_381": "<loc_381>",
469
- "loc_382": "<loc_382>",
470
- "loc_383": "<loc_383>",
471
- "loc_384": "<loc_384>",
472
- "loc_385": "<loc_385>",
473
- "loc_386": "<loc_386>",
474
- "loc_387": "<loc_387>",
475
- "loc_388": "<loc_388>",
476
- "loc_389": "<loc_389>",
477
- "loc_39": "<loc_39>",
478
- "loc_390": "<loc_390>",
479
- "loc_391": "<loc_391>",
480
- "loc_392": "<loc_392>",
481
- "loc_393": "<loc_393>",
482
- "loc_394": "<loc_394>",
483
- "loc_395": "<loc_395>",
484
- "loc_396": "<loc_396>",
485
- "loc_397": "<loc_397>",
486
- "loc_398": "<loc_398>",
487
- "loc_399": "<loc_399>",
488
- "loc_4": "<loc_4>",
489
- "loc_40": "<loc_40>",
490
- "loc_400": "<loc_400>",
491
- "loc_401": "<loc_401>",
492
- "loc_402": "<loc_402>",
493
- "loc_403": "<loc_403>",
494
- "loc_404": "<loc_404>",
495
- "loc_405": "<loc_405>",
496
- "loc_406": "<loc_406>",
497
- "loc_407": "<loc_407>",
498
- "loc_408": "<loc_408>",
499
- "loc_409": "<loc_409>",
500
- "loc_41": "<loc_41>",
501
- "loc_410": "<loc_410>",
502
- "loc_411": "<loc_411>",
503
- "loc_412": "<loc_412>",
504
- "loc_413": "<loc_413>",
505
- "loc_414": "<loc_414>",
506
- "loc_415": "<loc_415>",
507
- "loc_416": "<loc_416>",
508
- "loc_417": "<loc_417>",
509
- "loc_418": "<loc_418>",
510
- "loc_419": "<loc_419>",
511
- "loc_42": "<loc_42>",
512
- "loc_420": "<loc_420>",
513
- "loc_421": "<loc_421>",
514
- "loc_422": "<loc_422>",
515
- "loc_423": "<loc_423>",
516
- "loc_424": "<loc_424>",
517
- "loc_425": "<loc_425>",
518
- "loc_426": "<loc_426>",
519
- "loc_427": "<loc_427>",
520
- "loc_428": "<loc_428>",
521
- "loc_429": "<loc_429>",
522
- "loc_43": "<loc_43>",
523
- "loc_430": "<loc_430>",
524
- "loc_431": "<loc_431>",
525
- "loc_432": "<loc_432>",
526
- "loc_433": "<loc_433>",
527
- "loc_434": "<loc_434>",
528
- "loc_435": "<loc_435>",
529
- "loc_436": "<loc_436>",
530
- "loc_437": "<loc_437>",
531
- "loc_438": "<loc_438>",
532
- "loc_439": "<loc_439>",
533
- "loc_44": "<loc_44>",
534
- "loc_440": "<loc_440>",
535
- "loc_441": "<loc_441>",
536
- "loc_442": "<loc_442>",
537
- "loc_443": "<loc_443>",
538
- "loc_444": "<loc_444>",
539
- "loc_445": "<loc_445>",
540
- "loc_446": "<loc_446>",
541
- "loc_447": "<loc_447>",
542
- "loc_448": "<loc_448>",
543
- "loc_449": "<loc_449>",
544
- "loc_45": "<loc_45>",
545
- "loc_450": "<loc_450>",
546
- "loc_451": "<loc_451>",
547
- "loc_452": "<loc_452>",
548
- "loc_453": "<loc_453>",
549
- "loc_454": "<loc_454>",
550
- "loc_455": "<loc_455>",
551
- "loc_456": "<loc_456>",
552
- "loc_457": "<loc_457>",
553
- "loc_458": "<loc_458>",
554
- "loc_459": "<loc_459>",
555
- "loc_46": "<loc_46>",
556
- "loc_460": "<loc_460>",
557
- "loc_461": "<loc_461>",
558
- "loc_462": "<loc_462>",
559
- "loc_463": "<loc_463>",
560
- "loc_464": "<loc_464>",
561
- "loc_465": "<loc_465>",
562
- "loc_466": "<loc_466>",
563
- "loc_467": "<loc_467>",
564
- "loc_468": "<loc_468>",
565
- "loc_469": "<loc_469>",
566
- "loc_47": "<loc_47>",
567
- "loc_470": "<loc_470>",
568
- "loc_471": "<loc_471>",
569
- "loc_472": "<loc_472>",
570
- "loc_473": "<loc_473>",
571
- "loc_474": "<loc_474>",
572
- "loc_475": "<loc_475>",
573
- "loc_476": "<loc_476>",
574
- "loc_477": "<loc_477>",
575
- "loc_478": "<loc_478>",
576
- "loc_479": "<loc_479>",
577
- "loc_48": "<loc_48>",
578
- "loc_480": "<loc_480>",
579
- "loc_481": "<loc_481>",
580
- "loc_482": "<loc_482>",
581
- "loc_483": "<loc_483>",
582
- "loc_484": "<loc_484>",
583
- "loc_485": "<loc_485>",
584
- "loc_486": "<loc_486>",
585
- "loc_487": "<loc_487>",
586
- "loc_488": "<loc_488>",
587
- "loc_489": "<loc_489>",
588
- "loc_49": "<loc_49>",
589
- "loc_490": "<loc_490>",
590
- "loc_491": "<loc_491>",
591
- "loc_492": "<loc_492>",
592
- "loc_493": "<loc_493>",
593
- "loc_494": "<loc_494>",
594
- "loc_495": "<loc_495>",
595
- "loc_496": "<loc_496>",
596
- "loc_497": "<loc_497>",
597
- "loc_498": "<loc_498>",
598
- "loc_499": "<loc_499>",
599
- "loc_5": "<loc_5>",
600
- "loc_50": "<loc_50>",
601
- "loc_500": "<loc_500>",
602
- "loc_51": "<loc_51>",
603
- "loc_52": "<loc_52>",
604
- "loc_53": "<loc_53>",
605
- "loc_54": "<loc_54>",
606
- "loc_55": "<loc_55>",
607
- "loc_56": "<loc_56>",
608
- "loc_57": "<loc_57>",
609
- "loc_58": "<loc_58>",
610
- "loc_59": "<loc_59>",
611
- "loc_6": "<loc_6>",
612
- "loc_60": "<loc_60>",
613
- "loc_61": "<loc_61>",
614
- "loc_62": "<loc_62>",
615
- "loc_63": "<loc_63>",
616
- "loc_64": "<loc_64>",
617
- "loc_65": "<loc_65>",
618
- "loc_66": "<loc_66>",
619
- "loc_67": "<loc_67>",
620
- "loc_68": "<loc_68>",
621
- "loc_69": "<loc_69>",
622
- "loc_7": "<loc_7>",
623
- "loc_70": "<loc_70>",
624
- "loc_71": "<loc_71>",
625
- "loc_72": "<loc_72>",
626
- "loc_73": "<loc_73>",
627
- "loc_74": "<loc_74>",
628
- "loc_75": "<loc_75>",
629
- "loc_76": "<loc_76>",
630
- "loc_77": "<loc_77>",
631
- "loc_78": "<loc_78>",
632
- "loc_79": "<loc_79>",
633
- "loc_8": "<loc_8>",
634
- "loc_80": "<loc_80>",
635
- "loc_81": "<loc_81>",
636
- "loc_82": "<loc_82>",
637
- "loc_83": "<loc_83>",
638
- "loc_84": "<loc_84>",
639
- "loc_85": "<loc_85>",
640
- "loc_86": "<loc_86>",
641
- "loc_87": "<loc_87>",
642
- "loc_88": "<loc_88>",
643
- "loc_89": "<loc_89>",
644
- "loc_9": "<loc_9>",
645
- "loc_90": "<loc_90>",
646
- "loc_91": "<loc_91>",
647
- "loc_92": "<loc_92>",
648
- "loc_93": "<loc_93>",
649
- "loc_94": "<loc_94>",
650
- "loc_95": "<loc_95>",
651
- "loc_96": "<loc_96>",
652
- "loc_97": "<loc_97>",
653
- "loc_98": "<loc_98>",
654
- "loc_99": "<loc_99>",
655
  "pad_token": "<|end_of_text|>",
656
- "screentag_close": "</screentag>",
657
- "screentag_open": "<screentag>",
658
- "start_of_role": "<|start_of_role|>",
659
  "unk_token": {
660
  "content": "<|unk|>",
661
  "lstrip": false,
 
1
  {
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
  "additional_special_tokens": [
3
  {
4
  "content": "<fake_token_around_image>",
 
29
  "rstrip": false,
30
  "single_word": false
31
  },
 
 
32
  "eos_token": {
33
  "content": "<|end_of_text|>",
34
  "lstrip": false,
 
36
  "rstrip": false,
37
  "single_word": false
38
  },
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
39
  "pad_token": "<|end_of_text|>",
 
 
 
40
  "unk_token": {
41
  "content": "<|unk|>",
42
  "lstrip": false,
tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json CHANGED
The diff for this file is too large to render. See raw diff
 
vocab.json CHANGED
The diff for this file is too large to render. See raw diff