Stable Audio 3
English
music
audio
autoencoder
lora
thecollabagepatch commited on
Commit
6156258
·
verified ·
1 Parent(s): 5005213

Point links at the fork branch, add origin note, add base-model ladder

Browse files
Files changed (1) hide show
  1. README.md +26 -12
README.md CHANGED
@@ -36,7 +36,7 @@ transform re-encodes its input, so a chained session compounds them.
36
  | **neurofunk**, after 3 round trips | <audio controls src="https://huggingface.co/thepatch/same-l-decoder-lora/resolve/main/samples/neurofunk_stock_d3.wav"></audio> | <audio controls src="https://huggingface.co/thepatch/same-l-decoder-lora/resolve/main/samples/neurofunk_v3_d3.wav"></audio> |
37
 
38
  15 s clips, same latent and seed, decoder adapter the only variable, peak-matched
39
- to −1 dBFS. Generated with the `koan` DiT LoRA on `medium`. These three prompts
40
  were not picked for artifact content, and the fresh-generation rows are there to
41
  show the progression rather than because much is audible in them.
42
 
@@ -51,12 +51,21 @@ pair where it carries. What changes here is air rather than squeak count: on thi
51
  pair tonality moves 0.08 dB, which is nothing, while 16-22 kHz comes back
52
  +2.4 dB.
53
 
 
 
 
 
 
 
 
 
54
  ## Usage
55
 
56
- Requires `target: "decoder"` support in
57
- [Stability-AI/stable-audio-3](https://github.com/Stability-AI/stable-audio-3).
58
- The `target` field in the checkpoint config routes it; there is no special
59
- loading path. Ship at strength **1.0**, where it was trained and auditioned.
 
60
 
61
  ```bash
62
  python run_gradio.py --model medium-base --lora-ckpt-path squeakfix_v3.safetensors
@@ -80,8 +89,8 @@ audio = ae.decode(ae.encode(waveform))
80
 
81
  ## Numbers
82
 
83
- Four `koan` generations of 2 minutes, screened from 12 candidates by tonality
84
- runaway. Each latent is decoded twice, adapter off and on, and each chain
85
  re-encodes in place, so dN is N round trips of that decoder config.
86
 
87
  | depth | tonality p95 | tonal frames | 12-16 kHz | SI-SDR vs own d0 |
@@ -107,8 +116,11 @@ with re-encoding (+0.70, +1.31, +1.76, +2.30 dB at d0 to d3), while SI-SDR
107
  against its own d0 runs 3.6 to 3.9 dB better. It drifts less rather than erasing
108
  what drifts.
109
 
110
- This table is `koan` at 120 s and fp16. An earlier unscreened run covering `kev`
111
- as well, at 30 s and 120 s, showed the same direction at depth.
 
 
 
112
 
113
  ## Limits
114
 
@@ -128,8 +140,10 @@ as well, at 30 s and 120 s, showed the same direction at depth.
128
  wrong.
129
  - **The gain scales with re-encode depth**, so it is small for one-shot
130
  generation and compounds across chained continuations.
131
- - **The squeaks are partly a prompt-distribution problem.** Out-of-distribution
132
- prompts produce far more of the offending frames.
 
 
133
  - **One checkpoint, one dataset**, step 2000, chosen on re-encode behaviour and
134
  listening. Your material may want a different one.
135
  - **Neither half of the autoencoder is bit-deterministic** (encoder 1.5 %
@@ -158,7 +172,7 @@ drift), and latents sampled from the base DiT. No discriminator, since
158
  adversarial training makes a decoder invent *more* detail.
159
 
160
  Full method, losses, ablations and traps: [Decoder LoRA
161
- workflow](https://github.com/Stability-AI/stable-audio-3/blob/main/docs/workflows/decoder-lora.md).
162
  Provenance is embedded in the file's `lora_config` metadata. `v1` (identical
163
  recipe, `--lambda_patch 0`) is an internal control, not published.
164
 
 
36
  | **neurofunk**, after 3 round trips | <audio controls src="https://huggingface.co/thepatch/same-l-decoder-lora/resolve/main/samples/neurofunk_stock_d3.wav"></audio> | <audio controls src="https://huggingface.co/thepatch/same-l-decoder-lora/resolve/main/samples/neurofunk_v3_d3.wav"></audio> |
37
 
38
  15 s clips, same latent and seed, decoder adapter the only variable, peak-matched
39
+ to −1 dBFS, generated on `medium` with a DiT LoRA attached. These three prompts
40
  were not picked for artifact content, and the fresh-generation rows are there to
41
  show the progression rather than because much is audible in them.
42
 
 
51
  pair tonality moves 0.08 dB, which is nothing, while 16-22 kHz comes back
52
  +2.4 dB.
53
 
54
+ ## How this came about
55
+
56
+ I do a lot of transformations and continuations using my guitar as input audio.
57
+ When I first noticed these squeaks in the outputs, I mentioned that my trick was
58
+ to stem separate in Ableton and then apply the erosion effect to the percussion
59
+ layer. Claude used that as a guide for refining the methodology. The technical
60
+ version is below.
61
+
62
  ## Usage
63
 
64
+ Requires the `target: "decoder"` support on the
65
+ [`decoder-lora` branch of betweentwomidnights/stable-audio-3](https://github.com/betweentwomidnights/stable-audio-3/tree/decoder-lora).
66
+ It is not upstream yet, so that branch is currently the only place a decoder LoRA
67
+ will load. The `target` field in the checkpoint config routes it; there is no
68
+ special loading path. Ship at strength **1.0**, where it was trained and auditioned.
69
 
70
  ```bash
71
  python run_gradio.py --model medium-base --lora-ckpt-path squeakfix_v3.safetensors
 
89
 
90
  ## Numbers
91
 
92
+ Four generations of 2 minutes, made with a DiT LoRA on `medium` and screened
93
+ from 12 candidates by tonality runaway. Each latent is decoded twice, adapter off and on, and each chain
94
  re-encodes in place, so dN is N round trips of that decoder config.
95
 
96
  | depth | tonality p95 | tonal frames | 12-16 kHz | SI-SDR vs own d0 |
 
116
  against its own d0 runs 3.6 to 3.9 dB better. It drifts less rather than erasing
117
  what drifts.
118
 
119
+ Run again on the base model with no DiT LoRA, the same ladder gives a stock climb
120
+ of +6.73 dB against the adapter's +1.81 dB, so this is the autoencoder's
121
+ behaviour rather than something a LoRA introduces. An earlier unscreened run
122
+ covering a second, quite different LoRA, at 30 s and 120 s, showed the same
123
+ direction at depth. All of these are fp16 at 120 s.
124
 
125
  ## Limits
126
 
 
140
  wrong.
141
  - **The gain scales with re-encode depth**, so it is small for one-shot
142
  generation and compounds across chained continuations.
143
+ - **The squeaks are worst on out-of-distribution prompts**, which is easy to hit
144
+ when a DiT LoRA has pulled the model toward a narrow style. We have not
145
+ measured with-LoRA against without-LoRA on matched content, and on its own the
146
+ base model runs away slightly more rather than less.
147
  - **One checkpoint, one dataset**, step 2000, chosen on re-encode behaviour and
148
  listening. Your material may want a different one.
149
  - **Neither half of the autoencoder is bit-deterministic** (encoder 1.5 %
 
172
  adversarial training makes a decoder invent *more* detail.
173
 
174
  Full method, losses, ablations and traps: [Decoder LoRA
175
+ workflow](https://github.com/betweentwomidnights/stable-audio-3/blob/decoder-lora/docs/workflows/decoder-lora.md).
176
  Provenance is embedded in the file's `lora_config` metadata. `v1` (identical
177
  recipe, `--lambda_patch 0`) is an internal control, not published.
178