README.md

metadata

title: ControlFoley Video to Audio
emoji: 🎬
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
python_version: 3.10.13
license: apache-2.0

ControlFoley Video → Foley Audio — ZeroGPU Space

A Gradio/Hugging Face ZeroGPU wrapper around the official Xiaomi Research ControlFoley inference code.

What this Space does

• Upload a video.
• Generate synchronized Foley/audio from the video.
• Optionally guide generation with a text prompt.
• Preview and download the generated FLAC.
• Preview and download the video with the generated soundtrack.
• Includes the official assets/001.mp4 skateboard example.

Hardware

After creating the Space, open Settings → Hardware and select ZeroGPU.

The app uses:

@spaces.GPU(size="xlarge", duration=...)

xlarge is intentionally selected because ControlFoley loads a large main checkpoint plus multiple visual/audio feature models. If your testing confirms that the complete model stack fits comfortably in 48 GB VRAM, change size="xlarge" to size="large" to reduce ZeroGPU quota consumption.

Model downloads

On first startup the app downloads:

• Official ControlFoley source from GitHub.
• ControlFoley weights/*.
• ControlFoley ext_weights/*.
• Additional upstream pretrained components requested by ControlFoley (for example CLIP/MusicGen) through their normal loaders.

The official ControlFoley model repository is large, so the first cold start will take significantly longer than later cached starts.

Test

Use the bundled sample and this prompt:

the skateboard wheels scraping and grinding on the ground.

Recommended first test:

• Duration: 8 s
• CFG: 4.5
• Steps: 25
• Seed: 42

For a quicker smoke test, use 10–15 steps.

Licensing

The upstream source code repository is Apache-2.0.

The ControlFoley model weights on Hugging Face are published under CC BY-NC 4.0. That means the model weights are intended for non-commercial use unless you obtain different permission from the model owner.

Review the upstream licenses before publishing or using this Space in a commercial context.

Upstream

• GitHub: xiaomi-research/controlfoley
• Model: YJX-Xiaomi/ControlFoley
• Paper: ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling