Hugging Face's logo

objectai
/
obj_v1

Image-Text-to-Text
Transformers
Safetensors
English
qwen3_vl
document-understanding
information-extraction
vision-language
qwen3-vl
structured-data-extraction
multimodal
document-ai
conversational
Model card Files
xet
 Community
1
objectai commited on
Commit
b1abaaa
·
verified ·
1 Parent(s): 78362e3

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +17 -26
README.md CHANGED
@@ -9,18 +9,30 @@ tags:
9
  - qwen3-vl
10
  library_name: transformers
11
  ---
12
-
13
  # obj_v1
14
-
15
  Vision-language model fine-tuned for structured data extraction from Indian
16
  financial documents. Give it a page image and a JSON schema; it returns the
17
  schema filled in from what is on the page.
18
-
19
  A 4B vision-language model, LoRA fine-tuned and merged. Nothing extra is needed
20
  at load time -- it is a plain bf16 checkpoint.
21
 
22
- ## Serving with vLLM
 
 
 
 
 
 
 
 
 
 
 
 
 
 
23
 
 
24
  ```bash
25
  vllm serve objectai/obj_v1 \
26
  --served-model-name obj_v1 \
@@ -29,23 +41,16 @@ vllm serve objectai/obj_v1 \
29
  --mm-processor-kwargs '{"max_pixels":1003520}' \
30
  --trust-remote-code
31
  ```
32
-
33
  `max_pixels` is 1280x28x28, the resolution the model was trained at. Raising it
34
  wastes KV cache; lowering it makes small print unreadable.
35
-
36
  ## Calling it
37
-
38
  The server is OpenAI-compatible, so an ordinary chat completion works:
39
-
40
  ```python
41
  import base64, json, openai
42
-
43
  client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
44
-
45
  image = base64.b64encode(open("cheque.jpg", "rb").read()).decode()
46
  schema = {"cheque_details": {"amount": "number", "payee": "string",
47
  "date": "string", "cheque_number": "string"}}
48
-
49
  response = client.chat.completions.create(
50
  model="obj_v1",
51
  temperature=0.0,
@@ -67,46 +72,32 @@ response = client.chat.completions.create(
67
  )
68
  print(response.choices[0].message.content)
69
  ```
70
-
71
  ## Prompt format
72
-
73
  Match training or accuracy drops. The system prompt above is verbatim, and the
74
  user turn is the image followed by exactly two lines:
75
-
76
  ```
77
  document_type: <type>
78
  schema: <compact json>
79
  ```
80
-
81
  Set `temperature=0.0` so the same page yields the same answer.
82
-
83
  ## Requirements
84
-
85
  | | |
86
  | --- | --- |
87
  | Weights | 8.9 GB (bf16) |
88
  | VRAM | 16 GB minimum, 24 GB comfortable |
89
  | Precision | bf16 (Ampere or newer; use fp16 below that) |
90
  | Context | 16384 covers the longest documents |
91
-
92
  Runs on an L4, A10G, L40S, A100 or RTX 4090. On a T4 add `--dtype float16`.
93
-
94
  Long documents matter: `bank_statement` and `form16` answers run to ~2500
95
  tokens, so `max_tokens` below 4096 truncates them mid-JSON.
96
-
97
  ## Output
98
-
99
  Compact JSON matching the requested schema. Fields absent from the page come
100
  back `null` rather than guessed. Values found on the page that the schema did
101
  not ask for are placed under `extras` when that key is included in the schema.
102
-
103
  ## Limitations
104
-
105
  - Trained on Indian financial documents; other domains and layouts are untested.
106
  - Handwriting is the weakest case, particularly digits at low resolution.
107
  - The model does not verify its own arithmetic. Totals that must reconcile
108
  should be checked by the caller.
109
-
110
  ## License
111
-
112
- Apache 2.0. Fine-tuned from Qwen3-VL-4B-Instruct, which is Apache 2.0.
 
9
  - qwen3-vl
10
  library_name: transformers
11
  ---
 
12
  # obj_v1
 
13
  Vision-language model fine-tuned for structured data extraction from Indian
14
  financial documents. Give it a page image and a JSON schema; it returns the
15
  schema filled in from what is on the page.
 
16
  A 4B vision-language model, LoRA fine-tuned and merged. Nothing extra is needed
17
  at load time -- it is a plain bf16 checkpoint.
18
 
19
+ ## Authors
20
+
21
+ <p align="left">
22
+ <!-- <a href="https://www.linkedin.com/in/ahmedzaweel/">
23
+ <img src="https://img.shields.io/badge/LinkedIn-Ahmed%20Zaweel-0A66C2?style=for-the-badge&logo=linkedin&logoColor=white" alt="Ahmed Zaweel on LinkedIn" />
24
+ </a> -->
25
+ &nbsp;&nbsp;
26
+ <a href="https://www.linkedin.com/in/rachit-kumar-b41299228/">
27
+ <img src="https://img.shields.io/badge/LinkedIn-Rachit%20Kumar-0A66C2?style=for-the-badge&logo=linkedin&logoColor=white" alt="Rachit Kumar on LinkedIn" />
28
+ </a>
29
+ &nbsp;&nbsp;
30
+ <a href="https://www.linkedin.com/in/ahmedzaweel/">
31
+ <img src="https://img.shields.io/badge/LinkedIn-Ahmed%20Zaweel-0A66C2?style=for-the-badge&logo=linkedin&logoColor=white" alt="Ahmed Zaweel on LinkedIn" />
32
+ </a>
33
+ </p>
34
 
35
+ ## Serving with vLLM
36
  ```bash
37
  vllm serve objectai/obj_v1 \
38
  --served-model-name obj_v1 \
 
41
  --mm-processor-kwargs '{"max_pixels":1003520}' \
42
  --trust-remote-code
43
  ```
 
44
  `max_pixels` is 1280x28x28, the resolution the model was trained at. Raising it
45
  wastes KV cache; lowering it makes small print unreadable.
 
46
  ## Calling it
 
47
  The server is OpenAI-compatible, so an ordinary chat completion works:
 
48
  ```python
49
  import base64, json, openai
 
50
  client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
 
51
  image = base64.b64encode(open("cheque.jpg", "rb").read()).decode()
52
  schema = {"cheque_details": {"amount": "number", "payee": "string",
53
  "date": "string", "cheque_number": "string"}}
 
54
  response = client.chat.completions.create(
55
  model="obj_v1",
56
  temperature=0.0,
 
72
  )
73
  print(response.choices[0].message.content)
74
  ```
 
75
  ## Prompt format
 
76
  Match training or accuracy drops. The system prompt above is verbatim, and the
77
  user turn is the image followed by exactly two lines:
 
78
  ```
79
  document_type: <type>
80
  schema: <compact json>
81
  ```
 
82
  Set `temperature=0.0` so the same page yields the same answer.
 
83
  ## Requirements
 
84
  | | |
85
  | --- | --- |
86
  | Weights | 8.9 GB (bf16) |
87
  | VRAM | 16 GB minimum, 24 GB comfortable |
88
  | Precision | bf16 (Ampere or newer; use fp16 below that) |
89
  | Context | 16384 covers the longest documents |
 
90
  Runs on an L4, A10G, L40S, A100 or RTX 4090. On a T4 add `--dtype float16`.
 
91
  Long documents matter: `bank_statement` and `form16` answers run to ~2500
92
  tokens, so `max_tokens` below 4096 truncates them mid-JSON.
 
93
  ## Output
 
94
  Compact JSON matching the requested schema. Fields absent from the page come
95
  back `null` rather than guessed. Values found on the page that the schema did
96
  not ask for are placed under `extras` when that key is included in the schema.
 
97
  ## Limitations
 
98
  - Trained on Indian financial documents; other domains and layouts are untested.
99
  - Handwriting is the weakest case, particularly digits at low resolution.
100
  - The model does not verify its own arithmetic. Totals that must reconcile
101
  should be checked by the caller.
 
102
  ## License
103
+ Apache 2.0. Fine-tuned from Qwen3-VL-4B-Instruct, which is Apache 2.0.