| --- |
| title: NanoVLM |
| emoji: 🤖 |
| colorFrom: blue |
| colorTo: indigo |
| sdk: gradio |
| sdk_version: "5.0.0" |
| app_file: app.py |
| pinned: false |
| license: mit |
| --- |
| |
| # NanoVLM |
|
|
| NanoVLM is a lightweight Vision-Language Model (VLM) designed for efficient multimodal understanding. It combines image encoding and language generation to answer questions about images while remaining suitable for resource-constrained environments. |
|
|
| ## Features |
|
|
| - 🖼️ Image understanding |
| - 💬 Visual Question Answering (VQA) |
| - ⚡ Lightweight architecture |
| - 🤗 Hugging Face compatible |
| - 📓 Training and inference notebook included |
|
|
| ## Model Architecture |
|
|
| The model consists of: |
|
|
| - Vision Encoder |
| - Projection Layer |
| - Language Model |
| - Cross-modal fusion between visual and textual representations |
|
|
| ``` |
| Image |
| │ |
| Vision Encoder |
| │ |
| Projection Layer |
| │ |
| Language Model |
| │ |
| Generated Answer |
| ``` |
|
|
| ## Installation |
|
|
| Clone the repository |
|
|
| ```bash |
| git clone https://github.com/your-username/nanovlm.git |
| cd nanovlm |
| ``` |
|
|
| Install dependencies |
|
|
| ```bash |
| pip install -r requirements.txt |
| ``` |
|
|
| ## Usage |
|
|
| ### Inference |
|
|
| ```python |
| from PIL import Image |
| |
| image = Image.open("example.jpg") |
| |
| question = "What is in this image?" |
| |
| answer = model.chat(image, question) |
| |
| print(answer) |
| ``` |
|
|
| ### Notebook |
|
|
| Open the notebook: |
|
|
| ``` |
| NanoVlm_fixed.ipynb |
| ``` |
|
|
| and execute all cells. |
|
|
| ## Dataset |
|
|
| The model can be trained on datasets such as: |
|
|
| - COCO Captions |
| - VQAv2 |
| - Flickr30k |
| - Custom datasets |
|
|
| ## Training |
|
|
| Example: |
|
|
| ```bash |
| python train.py |
| ``` |
|
|
| Adjust hyperparameters such as: |
|
|
| - Learning rate |
| - Batch size |
| - Number of epochs |
| - Image resolution |
|
|
| ## Results |
|
|
| | Metric | Value | |
| |---------|------:| |
| | Accuracy | -- | |
| | BLEU | -- | |
| | CIDEr | -- | |
|
|
| *(Replace with your experimental results.)* |
|
|
| ## Repository Structure |
|
|
| ``` |
| . |
| ├── NanoVlm_fixed.ipynb |
| ├── app.py |
| ├── requirements.txt |
| ├── README.md |
| ├── images/ |
| └── model/ |
| ``` |
|
|
| ## Requirements |
|
|
| - Python 3.10+ |
| - PyTorch |
| - Transformers |
| - Pillow |
| - Torchvision |
| - Gradio (optional) |
|
|
| Install: |
|
|
| ```bash |
| pip install torch torchvision transformers pillow gradio |
| ``` |
|
|
| ## Citation |
|
|
| If you use this project, please cite: |
|
|
| ```bibtex |
| @misc{nanovlm2026, |
| title={NanoVLM: A Lightweight Vision-Language Model}, |
| author={Your Name}, |
| year={2026}, |
| publisher={Hugging Face} |
| } |
| ``` |
|
|
| ## License |
|
|
| MIT License |
|
|
| ## Acknowledgements |
|
|
| This project builds upon the excellent work of: |
|
|
| - Hugging Face Transformers |
| - PyTorch |
| - Vision Transformer (ViT) |
| - Large Language Models research |
|
|
| --- |
|
|
| ⭐ If you find this project useful, consider giving it a star on GitHub or liking it on Hugging Face. |
