| | --- |
| | license: apache-2.0 |
| | language: |
| | - en |
| | base_model: |
| | - openmmlab/mask-rcnn |
| | - microsoft/swin-base-patch4-window7-224-in22k |
| | pipeline_tag: image-segmentation |
| | --- |
| | |
| | # Model Card for ChartPointNet-InstanceSeg |
| |
|
| | ChartPointNet-InstanceSeg is a high-precision data point instance segmentation model for scientific charts. It uses Mask R-CNN with a Swin Transformer backbone to detect and segment individual data points, especially in dense and small-object scenarios common in scientific figures. |
| |
|
| | ## Model Details |
| |
|
| | ### Model Description |
| |
|
| | ChartPointNet-InstanceSeg is designed for pixel-precise instance segmentation of data points in scientific charts (e.g., scatter plots). It leverages Mask R-CNN with a Swin Transformer backbone, trained on enhanced COCO-style datasets with instance masks for data points. The model is ideal for extracting quantitative data from scientific figures and for downstream chart analysis. |
| |
|
| | - **Developed by:** Hansheng Zhu |
| | - **Model type:** Instance Segmentation |
| | - **License:** Apache-2.0 |
| | - **Finetuned from model:** openmmlab/mask-rcnn |
| |
|
| | ### Model Sources |
| |
|
| | - **Repository:** [https://github.com/hanszhu/ChartSense](https://github.com/hanszhu/ChartSense) |
| | - **Paper:** https://arxiv.org/abs/2106.01841 |
| |
|
| | ## Uses |
| |
|
| | ### Direct Use |
| |
|
| | - Instance segmentation of data points in scientific charts |
| | - Automated extraction of quantitative data from figures |
| | - Preprocessing for downstream chart understanding and data mining |
| |
|
| | ### Downstream Use |
| |
|
| | - As a preprocessing step for chart structure parsing or data extraction |
| | - Integration into document parsing, digital library, or accessibility systems |
| |
|
| | ### Out-of-Scope Use |
| |
|
| | - Segmentation of non-data-point elements |
| | - Use on figures outside the supported chart types |
| | - Medical or legal decision making |
| |
|
| | ## Bias, Risks, and Limitations |
| |
|
| | - The model is limited to data point segmentation in scientific charts. |
| | - May not generalize to figures with highly unusual styles or poor image quality. |
| | - Potential dataset bias: Training data is sourced from scientific literature. |
| |
|
| | ### Recommendations |
| |
|
| | Users should verify predictions on out-of-domain data and be aware of the model’s limitations regarding chart style and domain. |
| |
|
| | ## How to Get Started with the Model |
| |
|
| | ```python |
| | import torch |
| | from mmdet.apis import inference_detector, init_detector |
| | |
| | config_file = 'legend_match_swin/mask_rcnn_swin_datapoint.py' |
| | checkpoint_file = 'chart_datapoint.pth' |
| | model = init_detector(config_file, checkpoint_file, device='cuda:0') |
| | |
| | result = inference_detector(model, 'example_chart.png') |
| | # result: list of detected masks and class labels |
| | ``` |
| |
|
| | ## Training Details |
| |
|
| | ### Training Data |
| |
|
| | - **Dataset:** Enhanced COCO-style scientific chart dataset with instance masks |
| | - Data point class with pixel-precise segmentation masks |
| | - Images and annotations filtered and preprocessed for optimal Swin Transformer performance |
| |
|
| | ### Training Procedure |
| |
|
| | - Images resized to 1120x672 |
| | - Mask R-CNN with Swin Transformer backbone |
| | - **Training regime:** fp32 |
| | - **Optimizer:** AdamW |
| | - **Batch size:** 8 |
| | - **Epochs:** 36 |
| | - **Learning rate:** 1e-4 |
| |
|
| | ## Evaluation |
| |
|
| | ### Testing Data, Factors & Metrics |
| |
|
| | - **Testing Data:** Held-out split from enhanced COCO-style dataset |
| | - **Factors:** Data point density, image quality |
| | - **Metrics:** mAP (mean Average Precision), AP50, AP75, per-class AP |
| |
|
| | ### Results |
| |
|
| | | Category | mAP | mAP_50 | mAP_75 | mAP_s | mAP_m | mAP_l | |
| | |-----------------|-------|--------|--------|-------|-------|-------| |
| | | data-point | 0.485 | 0.687 | 0.581 | 0.487 | 0.05 | nan | |
| | |
| | #### Summary |
| | |
| | The model achieves strong mAP for data point segmentation, excelling in dense and small-object scenarios. It is highly effective for scientific figures requiring pixel-level accuracy. |
| | |
| | ## Environmental Impact |
| | |
| | - **Hardware Type:** NVIDIA V100 GPU |
| | - **Hours used:** 10 |
| | - **Cloud Provider:** Google Cloud |
| | - **Compute Region:** us-central1 |
| | - **Carbon Emitted:** ~15 kg CO2eq (estimated) |
| | |
| | ## Technical Specifications |
| | |
| | ### Model Architecture and Objective |
| | |
| | - Mask R-CNN with Swin Transformer backbone |
| | - Instance segmentation head for data point class |
| | |
| | ### Compute Infrastructure |
| | |
| | - **Hardware:** NVIDIA V100 GPU |
| | - **Software:** PyTorch 1.13, MMDetection 2.x, Python 3.9 |
| | |
| | ## Citation |
| | |
| | **BibTeX:** |
| | |
| | ```bibtex |
| | @article{DocFigure2021, |
| | title={DocFigure: A Dataset for Scientific Figure Classification}, |
| | author={S. Afzal, et al.}, |
| | journal={arXiv preprint arXiv:2106.01841}, |
| | year={2021} |
| | } |
| | ``` |
| | |
| | **APA:** |
| | |
| | Afzal, S., et al. (2021). DocFigure: A Dataset for Scientific Figure Classification. arXiv preprint arXiv:2106.01841. |
| | |
| | ## Glossary |
| | |
| | - **Data Point:** An individual visual marker representing a value in a scientific chart (e.g., a dot in a scatter plot) |
| | |
| | ## More Information |
| | |
| | - [DocFigure Paper](https://arxiv.org/abs/2106.01841) |
| | |
| | ## Model Card Authors |
| | |
| | Hansheng Zhu |
| | |
| | ## Model Card Contact |
| | |
| | hanszhu05@gmail.com |
