How to run Stable Audio Open with 8GB of RAM using Audiyo.

If you’ve ever tried to generate high-quality audio locally using stabilityai/stable-audio-open-1.0, you’ve probably hit the exact same wall we did: VRAM.

Out of the box, running a standard StableAudioPipeline from Hugging Face Diffusers eats roughly 13.8 GB of GPU memory just to render 10 seconds of 44.1kHz stereo audio. If you don't own an enterprise card or an RTX 3090/4090, your reward is usually a instant CUDA out of memory exception.

We built Audiyo to fix this. It’s an open-source Python library and CLI wrapper that drops peak VRAM usage down to 5.86 GB on the default setting—a 57% reduction—without sacrificing audio quality or requiring a multi-thousand-dollar GPU.

Here is the story of how we built it, how we tested it locally before ever touching a GPU, and the exact benchmark numbers from our runs.

Why Audio Diffusion Kills VRAM

Audio models aren't just text models with a different output head. Running Stable Audio Open means keeping three distinct heavyweights in memory simultaneously:

1. A 1B-parameter Diffusion Transformer (DiT)
2. A T5 / CLAP Text Encoder
3. An Oobleck VAE Decoder (which handles 44.1kHz stereo audio)

When you generate audio, the DiT operates in latent space across dozens of denoising steps. Once latents are computed, passing those high-dimensional tensors into the VAE decoder to construct raw 44.1kHz audio waveforms creates massive temporary memory spikes.

If you keep everything resident on the GPU in fp32 or unoptimized fp16, an 8GB or 12GB consumer GPU doesn't stand a chance.

Step 1: Building a "Testkit" Before Touching a GPU

Renting GPUs or downloading 4GB model weights every time you fix a typo is slow and expensive. To build Audiyo efficiently, we started by creating a local stand-in model: testkit.

The testkit is a faithful 5.38M parameter proxy built with the exact architectural layout of Stable Audio Open:

• A mini VAE decoder (585K params)
• A mini text encoder & projection layer (350K params)
• A 6-layer DiT at width 256 (4.44M params with to_q, to_k, to_v, and to_out attention projections)
• Cosine schedule v-prediction and timing conditioning

Because testkit used the exact method names and tensor flows as the real backend, we could run our full test suite locally on a standard CPU in milliseconds.

Bugs We Caught Before Ever Firing Up CUDA

Building testkit first paid off immediately. It caught several subtle, real-world bugs before we ever touched a real GPU:

• Tiled VAE Frame Dropping: During tiled decoding (chunking audio to prevent VRAM spikes), our initial boundary trimming math dropped frames at chunk joins (outputting 86,016 samples instead of 90,112). We fixed the slice alignment math locally before it could ruin real audio outputs.
• Adapter Stacking in PEFT: Reloading a LoRA adapter in the same Python session was nesting PEFT wrappers inside each other like Russian dolls, degrading performance and throwing warnings.
• Eval-Mode Ordering Bug: We noticed that dropout was still active during text encoding because .eval() was being called after encoding instead of before. Fixing this gave us bit-exact, seed-matched determinism.
• Silent Adapter Overwrites: We added a safeguard preventing training scripts from silently overwriting a completed adapter directory with fresh random weights if rerun accidentally.

Once testkit passed 47 unit tests locally, we knew our pipeline orchestration was solid.

Step 2: The Real Numbers (Tesla T4 GPU Benchmarks)

With the codebase validated, we brought Audiyo to a real NVIDIA Tesla T4 GPU (16GB VRAM) and loaded the real stabilityai/stable-audio-open-1.0 weights.

We designed four memory presets that trade a fraction of speed for massive memory savings:

Key Takeaway

In our default balanced mode (which uses model CPU offloading and bfloat16), peak VRAM dropped from 13.8 GB to 5.86 GB—a 57.5% memory reduction.

When the model is done generating, resting VRAM drops down to just 0.32 GB, releasing GPU memory back to your system.

How to Use Audiyo

We wanted Audiyo to be as simple as Unsloth or vLLM: no complex configuration files, just 3 lines of Python.

1. Installation

pip install audiyo

(Dependencies like peft, accelerate, einops, and safetensors are handled automatically.)

2. Python API

from audiyo import AudioModel

# Load Stable Audio Open in balanced memory mode
model = AudioModel.from_pretrained(
    "stabilityai/stable-audio-open-1.0",
    memory_mode="balanced"
)

# Generate 10 seconds of 44.1kHz stereo audio
result = model.generate(
    prompt="Rain falling against a window with soft distant thunder",
    duration_seconds=10,
    seed=42
)

# Save output
result.save("rain.wav")

3. LoRA Fine-Tuning

Audiyo also supports fine-tuning custom audio LoRAs directly on consumer GPUs without running out of memory:

# Fine-tune a LoRA adapter on a custom audio directory
report = model.finetune(
    dataset="./my_rain_samples",
    output_dir="./my_rain_adapter",
    max_steps=100
)

4. Terminal CLI

audiyo generate "Cinematic synth drone with heavy sub bass" -o drone.wav --memory-mode balanced

The Future

We hope to scale our repository to thousands of stars and many more model architectures. If you would like to contribute look at the CONTRIBUTING.md on our repo. Please star it if you like it!

What We Learned

Audiyo isn't a new AI model, and we aren't claiming to have invented new samplers. It’s an engineering layer built to solve a practical problem: making high-quality, open audio models accessible to developers who don't have access to cloud datacenters.

By focusing on clean systems work—CPU offloading, tiled decoding, explicit memory presets, and thorough unit testing—we were able to bring 44.1kHz stereo audio generation down to standard 8GB GPUs.

Try it out, run the benchmarks on your own GPU, and let us know what you build!

• PyPI: pypi.org/project/audiyo
• GitHub: github.com/TeamAudiyo/Audiyo

— TeamAudiyo