README.md

---
title: ControlFoley Video to Audio
emoji: 🎬
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
python_version: 3.10.13
license: apache-2.0
---

# ControlFoley Video → Foley Audio — ZeroGPU Space

A Gradio/Hugging Face ZeroGPU wrapper around the official
[Xiaomi Research ControlFoley](https://github.com/xiaomi-research/controlfoley)
inference code.

## What this Space does

- Upload a video.
- Generate synchronized Foley/audio from the video.
- Optionally guide generation with a text prompt.
- Preview and download the generated FLAC.
- Preview and download the video with the generated soundtrack.
- Includes the official `assets/001.mp4` skateboard example.

## Hardware

After creating the Space, open **Settings → Hardware** and select **ZeroGPU**.

The app uses:

```python
@spaces.GPU(size="xlarge", duration=...)
```

`xlarge` is intentionally selected because ControlFoley loads a large main
checkpoint plus multiple visual/audio feature models. If your testing confirms
that the complete model stack fits comfortably in 48 GB VRAM, change
`size="xlarge"` to `size="large"` to reduce ZeroGPU quota consumption.

## Model downloads

On first startup the app downloads:

- Official ControlFoley source from GitHub.
- ControlFoley `weights/*`.
- ControlFoley `ext_weights/*`.
- Additional upstream pretrained components requested by ControlFoley
  (for example CLIP/MusicGen) through their normal loaders.

The official ControlFoley model repository is large, so the first cold start
will take significantly longer than later cached starts.

## Test

Use the bundled sample and this prompt:

```text
the skateboard wheels scraping and grinding on the ground.
```

Recommended first test:

- Duration: `8 s`
- CFG: `4.5`
- Steps: `25`
- Seed: `42`

For a quicker smoke test, use `10–15` steps.

## Licensing

The upstream **source code** repository is Apache-2.0.

The **ControlFoley model weights** on Hugging Face are published under
**CC BY-NC 4.0**. That means the model weights are intended for non-commercial
use unless you obtain different permission from the model owner.

Review the upstream licenses before publishing or using this Space in a
commercial context.

## Upstream

- GitHub: `xiaomi-research/controlfoley`
- Model: `YJX-Xiaomi/ControlFoley`
- Paper: *ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling*