MiniMax-AI commited on
Commit
80d66b1
·
1 Parent(s): d7047d5

Add diffusers usage section to the README (#3)

Browse files

- Add diffusers usage section to the README (640470d00e8e75ade0b64b149a38eef61efd5cc7)

Files changed (1) hide show
  1. README.md +64 -0
README.md CHANGED
@@ -143,6 +143,70 @@ The following end-to-end example contains the complete lyrics, music description
143
  |---|---|---|
144
  | Text-to-music | [View script](https://huggingface.co/MiniMaxAI/MiniMax-Music3/blob/main/scripts/end_to_end/minimax_ttm_test.py) | [minimax_ttm.wav](https://huggingface.co/MiniMaxAI/MiniMax-Music3/blob/main/assets/minimax_ttm.wav) |
145
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
146
  ## Prompt Enhancement
147
 
148
  A concise natural-language description can be used directly. For richer prompts and more precise control, use the provided [`music-caption-rewriter`](skills/music-caption-rewriter/SKILL.md) skill to expand it into a Structured Caption containing `Global Metadata`, `Vocal Details`, and `Arrangement`. The skill preserves musical instructions attached to lyric section tags in the arrangement description while keeping the lyric text in the lyrics input.
 
143
  |---|---|---|
144
  | Text-to-music | [View script](https://huggingface.co/MiniMaxAI/MiniMax-Music3/blob/main/scripts/end_to_end/minimax_ttm_test.py) | [minimax_ttm.wav](https://huggingface.co/MiniMaxAI/MiniMax-Music3/blob/main/assets/minimax_ttm.wav) |
145
 
146
+ ## 🧨 Diffusers
147
+
148
+ MiniMax Music 3 is available as a [diffusers](https://github.com/huggingface/diffusers) modular pipeline. Until [huggingface/diffusers#14456](https://github.com/huggingface/diffusers/pull/14456) is merged, install diffusers from the PR commit:
149
+
150
+ The snippet below fits 24GB+ VRAM GPUs
151
+
152
+ ```bash
153
+ pip install git+https://github.com/huggingface/diffusers@dafe3733fcfdbf3c48915fe77be3aef65b5d6a2d transformers accelerate soundfile
154
+ ```
155
+
156
+ ```python
157
+ import soundfile as sf
158
+ import torch
159
+ from diffusers import ModularPipeline
160
+
161
+ pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-Music3")
162
+ pipe.load_components(dtype=torch.bfloat16)
163
+ pipe.to("cuda")
164
+
165
+ lyrics = """[verse]
166
+ Morning light filtering through the pine
167
+ Every quiet street is yours and mine
168
+ [chorus]
169
+ Softly the world begins to breathe"""
170
+
171
+ prompt = (
172
+ "Genre: acoustic pop. BPM: 96. Key: C major. Warm and intimate, building gently into the chorus. "
173
+ "Vocals: soft female lead, close and breathy, light stacked harmonies in the chorus. "
174
+ "Arrangement: fingerpicked guitar and soft piano; brushed drums and upright bass enter in the chorus."
175
+ )
176
+
177
+ audio = pipe(
178
+ prompt=prompt,
179
+ lyrics=lyrics,
180
+ audio_duration=60.0,
181
+ generator=torch.Generator("cuda").manual_seed(7),
182
+ output="audios",
183
+ )[0]
184
+
185
+ sf.write("song.wav", audio.T.float().cpu().numpy(), pipe.sampling_rate)
186
+ ```
187
+
188
+ ### Low VRAM
189
+
190
+ The full precision fits under 24GB of VRAM. With automatic CPU offloading, generation takes in ~22 GB; additionally streaming the language model layer by layer makes it fit even 8 GB video cards:
191
+
192
+ ```python
193
+ import torch
194
+ from diffusers import ComponentsManager, ModularPipeline
195
+ from diffusers.hooks import apply_group_offloading
196
+
197
+ manager = ComponentsManager()
198
+ manager.enable_auto_cpu_offload(device="cuda")
199
+ pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-Music3", components_manager=manager)
200
+ pipe.load_components(dtype=torch.bfloat16)
201
+
202
+ # Only needed below ~22 GB of VRAM — slower, but fits in 8 GB.
203
+ apply_group_offloading(
204
+ pipe.language_model, onload_device=torch.device("cuda"), offload_type="leaf_level", use_stream=True
205
+ )
206
+
207
+
208
+ ```
209
+
210
  ## Prompt Enhancement
211
 
212
  A concise natural-language description can be used directly. For richer prompts and more precise control, use the provided [`music-caption-rewriter`](skills/music-caption-rewriter/SKILL.md) skill to expand it into a Structured Caption containing `Global Metadata`, `Vocal Details`, and `Arrangement`. The skill preserves musical instructions attached to lyric section tags in the arrangement description while keeping the lyric text in the lyrics input.