README.md

---
title: MMDiff
emoji: 🎨
colorFrom: blue
colorTo: yellow
sdk: gradio
sdk_version: "5.23.0"
app_file: app.py
short_description: Multi-modal generation with diffusion transformers
python_version: "3.10"
startup_duration_timeout: "10m"
---

# MMDiff: Extending Diffusion Transformers for Multi-Modal Generation

This Space demonstrates MMDiff, a method that extends frozen diffusion transformers (FLUX.1-dev) to generate images alongside dense predictions (saliency maps, segmentation maps, depth maps) in a single forward pass.

## How it works

1. A text prompt is used to generate an image with FLUX.1-dev
2. During denoising, intermediate transformer features and concept attention maps are captured
3. Lightweight trained decoder heads (DPT, DeepLabV3+) decode these features into dense predictions:
   - **Saliency** (DUTS): Binary foreground/background segmentation
   - **Segmentation** (Pascal VOC): 21-class semantic segmentation
   - **Depth** (NYU Depth V2): Monocular depth estimation

## Model

- **Backbone**: [black-forest-labs/FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) (frozen)
- **Decoder weights**: [yagmurakarken/mmdiff](https://huggingface.co/yagmurakarken/mmdiff)
- **Paper**: [MMDiff: Extending Diffusion Transformers for Multi-Modal Generation](https://huggingface.co/papers/2606.16673)