Hugging Face's logo

Spaces:
yasserrmd
/
ControlFoley
Running on Zero

App Files Community
ControlFoley
File size: 2,443 Bytes
51b6cc2
0526b4c
 
 
 
51b6cc2
1a00154
51b6cc2
 
0526b4c
 
51b6cc2
 
0526b4c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
---
title: ControlFoley Video to Audio
emoji: 🎬
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
python_version: 3.10.13
license: apache-2.0
---

# ControlFoley Video → Foley Audio — ZeroGPU Space

A Gradio/Hugging Face ZeroGPU wrapper around the official
[Xiaomi Research ControlFoley](https://github.com/xiaomi-research/controlfoley)
inference code.

## What this Space does

- Upload a video.
- Generate synchronized Foley/audio from the video.
- Optionally guide generation with a text prompt.
- Preview and download the generated FLAC.
- Preview and download the video with the generated soundtrack.
- Includes the official `assets/001.mp4` skateboard example.

## Hardware

After creating the Space, open **Settings → Hardware** and select **ZeroGPU**.

The app uses:

```python
@spaces.GPU(size="xlarge", duration=...)
```

`xlarge` is intentionally selected because ControlFoley loads a large main
checkpoint plus multiple visual/audio feature models. If your testing confirms
that the complete model stack fits comfortably in 48 GB VRAM, change
`size="xlarge"` to `size="large"` to reduce ZeroGPU quota consumption.

## Model downloads

On first startup the app downloads:

- Official ControlFoley source from GitHub.
- ControlFoley `weights/*`.
- ControlFoley `ext_weights/*`.
- Additional upstream pretrained components requested by ControlFoley
  (for example CLIP/MusicGen) through their normal loaders.

The official ControlFoley model repository is large, so the first cold start
will take significantly longer than later cached starts.

## Test

Use the bundled sample and this prompt:

```text
the skateboard wheels scraping and grinding on the ground.
```

Recommended first test:

- Duration: `8 s`
- CFG: `4.5`
- Steps: `25`
- Seed: `42`

For a quicker smoke test, use `10–15` steps.

## Licensing

The upstream **source code** repository is Apache-2.0.

The **ControlFoley model weights** on Hugging Face are published under
**CC BY-NC 4.0**. That means the model weights are intended for non-commercial
use unless you obtain different permission from the model owner.

Review the upstream licenses before publishing or using this Space in a
commercial context.

## Upstream

- GitHub: `xiaomi-research/controlfoley`
- Model: `YJX-Xiaomi/ControlFoley`
- Paper: *ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling*