Image-Text-to-Text
Transformers
Safetensors
qwen3_vl
multimodal
scientific
protein
rna
dna
molecule
weather
medical-imaging
conversational
Instructions to use sais-org/MKB with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use sais-org/MKB with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="sais-org/MKB") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("sais-org/MKB") model = AutoModelForMultimodalLM.from_pretrained("sais-org/MKB", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use sais-org/MKB with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "sais-org/MKB" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sais-org/MKB", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/sais-org/MKB
- SGLang
How to use sais-org/MKB with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "sais-org/MKB" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sais-org/MKB", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "sais-org/MKB" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sais-org/MKB", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use sais-org/MKB with Docker Model Runner:
docker model run hf.co/sais-org/MKB
Rename model to ShenZhen (神珍)
Browse files
README.md
CHANGED
|
@@ -25,13 +25,13 @@ extra_gated_description: >-
|
|
| 25 |
|
| 26 |
<div align="center">
|
| 27 |
|
| 28 |
-
[🤗 Model](https://huggingface.co/sais-org/
|
| 29 |
|
| 30 |
</div>
|
| 31 |
|
| 32 |
-
#
|
| 33 |
|
| 34 |
-
**
|
| 35 |
supports scientific **understanding and generation** across Earth science,
|
| 36 |
proteins, RNA, DNA, and small molecules within a single **8B** model. Native
|
| 37 |
scientific encoders/decoders wrap a shared **Qwen3-VL-8B-Instruct** backbone, so
|
|
@@ -67,13 +67,13 @@ space — natural language in and out, no per-task fine-tuning.
|
|
| 67 |
|
| 68 |
## Benchmarks
|
| 69 |
|
| 70 |
-
**
|
| 71 |
no scientific encoders) and **Intern-S1-Pro** (**~1T** MoE scientific model).
|
| 72 |
**Bold** = best; <u>underline</u> = second-best.
|
| 73 |
|
| 74 |
### Biological sequence understanding
|
| 75 |
|
| 76 |
-
| Task | Metric |
|
| 77 |
|:-----|:------:|:----------------:|:-------------------------:|:-------------------:|
|
| 78 |
| DNA · Epigenetic marks (EMP) | MCC | **71.99** | 3.64 | <u>14.02</u> |
|
| 79 |
| DNA · Promoter det. 300bp (PD300) | MCC | **91.17** | 58.18 | <u>82.65</u> |
|
|
@@ -91,11 +91,11 @@ no scientific encoders) and **Intern-S1-Pro** (**~1T** MoE scientific model).
|
|
| 91 |
| Cross-modal · AAN (antibody–antigen) | MCC | <u>42.96</u> | 1.06 | **44.76** |
|
| 92 |
| Cross-modal · EPI (enhancer–promoter) | MCC | <u>-0.03</u> | **3.37** | -1.30 |
|
| 93 |
|
| 94 |
-
<sub>Aggregate over 20 biological-understanding benchmarks:
|
| 95 |
|
| 96 |
### Molecule understanding (SMolInstruct)
|
| 97 |
|
| 98 |
-
| Task | Metric |
|
| 99 |
|:-----|:------:|:----------------:|:-------:|
|
| 100 |
| BBBP | Acc | **96.95** | 74.60 |
|
| 101 |
| HIV | Acc | **97.00** | 96.70 |
|
|
@@ -106,20 +106,20 @@ no scientific encoders) and **Intern-S1-Pro** (**~1T** MoE scientific model).
|
|
| 106 |
|
| 107 |
### Earth-science forecasting — vs ECMWF HRES (day-10, global ERA5 0.25°)
|
| 108 |
|
| 109 |
-
| Variable | Metric |
|
| 110 |
|:---------|:------:|:----------------:|:----------------:|
|
| 111 |
| Z500 | RMSE ↓ | **≈740** | ≈810 |
|
| 112 |
| T2M | RMSE ↓ (K) | **≈2.65** | ≈2.90 |
|
| 113 |
| MSL | RMSE ↓ (Pa) | **≈680** | ≈745 |
|
| 114 |
|
| 115 |
-
<sub>
|
| 116 |
|
| 117 |
### Medical-image segmentation
|
| 118 |
|
| 119 |
Mean Dice (%) on the BiomedParse test splits, 102,855 image–prompt pairs across
|
| 120 |
nine imaging modalities, versus six modality-native segmentation specialists.
|
| 121 |
|
| 122 |
-
| Modality | # Samples |
|
| 123 |
|:---------|----------:|:-----------:|:-----------:|:------:|:---:|:----:|:-----------:|:--------:|
|
| 124 |
| **All** | 102,855 | **91.20** | <u>90.73</u> | 83.55 | 71.29 | 35.40 | 15.37 | 15.10 |
|
| 125 |
| CT | 45,306 | **93.36** | <u>92.25</u> | 83.87 | 74.10 | 28.93 | 9.59 | 10.34 |
|
|
@@ -139,9 +139,9 @@ nine imaging modalities, versus six modality-native segmentation specialists.
|
|
| 139 |
Runs via the accompanying code repository (custom multimodal architecture).
|
| 140 |
|
| 141 |
```bash
|
| 142 |
-
git clone https://github.com/Shanghai-Academy-of-AI-For-Science/
|
| 143 |
pip install -r requirements.txt # Python 3.10; transformers==5.0.0
|
| 144 |
-
hf download sais-org/
|
| 145 |
|
| 146 |
export PYTHONPATH=$PWD/code
|
| 147 |
python code/inference.py --model_path model --greedy --max_new_tokens 64 \
|
|
@@ -160,7 +160,7 @@ Each task has a specific `--system` prompt that fixes the output format; see
|
|
| 160 |
|
| 161 |
## License
|
| 162 |
|
| 163 |
-
**Composite license.**
|
| 164 |
except the SAM 3 branch — are **Apache-2.0**, built on Qwen3-VL (Apache-2.0) and
|
| 165 |
including merged ESM-2 (MIT) and Polaris/Suiren-derived encoders.
|
| 166 |
|
|
@@ -173,10 +173,10 @@ See `THIRD_PARTY_LICENSES.md` / `NOTICE` for the full third-party breakdown.
|
|
| 173 |
## Citation
|
| 174 |
|
| 175 |
```bibtex
|
| 176 |
-
@misc{
|
| 177 |
-
title = {
|
| 178 |
author = {Hesen Chen and Xinyu Su and Xiaomeng Yang and Yuetan Lin and Zixiong Yang and Zhiyu Tan and Hao Li},
|
| 179 |
year = {2026},
|
| 180 |
-
note = {https://huggingface.co/sais-org/
|
| 181 |
}
|
| 182 |
```
|
|
|
|
| 25 |
|
| 26 |
<div align="center">
|
| 27 |
|
| 28 |
+
[🤗 Model](https://huggingface.co/sais-org/ShenZhen) • [💻 GitHub](https://github.com/Shanghai-Academy-of-AI-For-Science/ShenZhen) • [📜 Technical Report (coming soon)](#) • [⚖️ License: Apache-2.0 + SAM License](https://github.com/Shanghai-Academy-of-AI-For-Science/ShenZhen/blob/main/LICENSE)
|
| 29 |
|
| 30 |
</div>
|
| 31 |
|
| 32 |
+
# 神珍 (ShenZhen)
|
| 33 |
|
| 34 |
+
**神珍 is a unified scientific multimodal foundation model** that
|
| 35 |
supports scientific **understanding and generation** across Earth science,
|
| 36 |
proteins, RNA, DNA, and small molecules within a single **8B** model. Native
|
| 37 |
scientific encoders/decoders wrap a shared **Qwen3-VL-8B-Instruct** backbone, so
|
|
|
|
| 67 |
|
| 68 |
## Benchmarks
|
| 69 |
|
| 70 |
+
**神珍** (**8B**) vs **Biology-Instructions** (Llama-3.1-**8B**, text-token,
|
| 71 |
no scientific encoders) and **Intern-S1-Pro** (**~1T** MoE scientific model).
|
| 72 |
**Bold** = best; <u>underline</u> = second-best.
|
| 73 |
|
| 74 |
### Biological sequence understanding
|
| 75 |
|
| 76 |
+
| Task | Metric | 神珍 (8B) | Biology-Instructions (8B) | Intern-S1-Pro (~1T) |
|
| 77 |
|:-----|:------:|:----------------:|:-------------------------:|:-------------------:|
|
| 78 |
| DNA · Epigenetic marks (EMP) | MCC | **71.99** | 3.64 | <u>14.02</u> |
|
| 79 |
| DNA · Promoter det. 300bp (PD300) | MCC | **91.17** | 58.18 | <u>82.65</u> |
|
|
|
|
| 91 |
| Cross-modal · AAN (antibody–antigen) | MCC | <u>42.96</u> | 1.06 | **44.76** |
|
| 92 |
| Cross-modal · EPI (enhancer–promoter) | MCC | <u>-0.03</u> | **3.37** | -1.30 |
|
| 93 |
|
| 94 |
+
<sub>Aggregate over 20 biological-understanding benchmarks: 神珍 matches or beats the ~1T Intern-S1-Pro on 10/20 and the same-scale 8B text-token baseline on 16/20.</sub>
|
| 95 |
|
| 96 |
### Molecule understanding (SMolInstruct)
|
| 97 |
|
| 98 |
+
| Task | Metric | 神珍 (8B) | LlaSMol |
|
| 99 |
|:-----|:------:|:----------------:|:-------:|
|
| 100 |
| BBBP | Acc | **96.95** | 74.60 |
|
| 101 |
| HIV | Acc | **97.00** | 96.70 |
|
|
|
|
| 106 |
|
| 107 |
### Earth-science forecasting — vs ECMWF HRES (day-10, global ERA5 0.25°)
|
| 108 |
|
| 109 |
+
| Variable | Metric | 神珍 (8B) | ECMWF HRES (NWP) |
|
| 110 |
|:---------|:------:|:----------------:|:----------------:|
|
| 111 |
| Z500 | RMSE ↓ | **≈740** | ≈810 |
|
| 112 |
| T2M | RMSE ↓ (K) | **≈2.65** | ≈2.90 |
|
| 113 |
| MSL | RMSE ↓ (Pa) | **≈680** | ≈745 |
|
| 114 |
|
| 115 |
+
<sub>神珍 tracks or beats the operational physics-based HRES system, with the advantage growing at longer lead times.</sub>
|
| 116 |
|
| 117 |
### Medical-image segmentation
|
| 118 |
|
| 119 |
Mean Dice (%) on the BiomedParse test splits, 102,855 image–prompt pairs across
|
| 120 |
nine imaging modalities, versus six modality-native segmentation specialists.
|
| 121 |
|
| 122 |
+
| Modality | # Samples | 神珍 | BiomedParse | MedSAM | SAM | SAM3 | DINO+MedSAM | DINO+SAM |
|
| 123 |
|:---------|----------:|:-----------:|:-----------:|:------:|:---:|:----:|:-----------:|:--------:|
|
| 124 |
| **All** | 102,855 | **91.20** | <u>90.73</u> | 83.55 | 71.29 | 35.40 | 15.37 | 15.10 |
|
| 125 |
| CT | 45,306 | **93.36** | <u>92.25</u> | 83.87 | 74.10 | 28.93 | 9.59 | 10.34 |
|
|
|
|
| 139 |
Runs via the accompanying code repository (custom multimodal architecture).
|
| 140 |
|
| 141 |
```bash
|
| 142 |
+
git clone https://github.com/Shanghai-Academy-of-AI-For-Science/ShenZhen && cd ShenZhen
|
| 143 |
pip install -r requirements.txt # Python 3.10; transformers==5.0.0
|
| 144 |
+
hf download sais-org/ShenZhen --local-dir ./model
|
| 145 |
|
| 146 |
export PYTHONPATH=$PWD/code
|
| 147 |
python code/inference.py --model_path model --greedy --max_new_tokens 64 \
|
|
|
|
| 160 |
|
| 161 |
## License
|
| 162 |
|
| 163 |
+
**Composite license.** 神珍's own components — the code, and all weights
|
| 164 |
except the SAM 3 branch — are **Apache-2.0**, built on Qwen3-VL (Apache-2.0) and
|
| 165 |
including merged ESM-2 (MIT) and Polaris/Suiren-derived encoders.
|
| 166 |
|
|
|
|
| 173 |
## Citation
|
| 174 |
|
| 175 |
```bibtex
|
| 176 |
+
@misc{shenzhen2026,
|
| 177 |
+
title = {ShenZhen (神珍): A Unified Scientific Multimodal Foundation Model},
|
| 178 |
author = {Hesen Chen and Xinyu Su and Xiaomeng Yang and Yuetan Lin and Zixiong Yang and Zhiyu Tan and Hao Li},
|
| 179 |
year = {2026},
|
| 180 |
+
note = {https://huggingface.co/sais-org/ShenZhen}
|
| 181 |
}
|
| 182 |
```
|