DEMON

Diffusion Engine for Musical Orchestrated Noise

DEMON is a real-time streaming diffusion engine for music generation and transformation on top of ACE-Step v1.5.

It is not a standalone base model. DEMON is an engine/runtime layer that makes diffusion-based music generation interactive: source audio, prompts, LoRAs, denoise strength, guidance, per-frame curves, and other controls can be changed while generation is already running.

Instead of waiting for one full diffusion generation to finish, DEMON keeps several generations in flight at once using a ring buffer. After warmup, finished latents stream out continuously, making diffusion feel playable.

What is DEMON?

DEMON turns ACE-Step v1.5 into a live, controllable streaming system.

A standard diffusion pipeline usually works like this:

submit request → wait for all denoising steps → decode → hear result

DEMON instead works like this:

keep multiple generations in flight → advance them every tick → stream finished latents continuously

That means controls can become musical performance parameters instead of offline generation settings.

Why this matters

Diffusion models are powerful, but they are usually slow, one-shot systems. DEMON makes diffusion-based music generation responsive enough for live control.

You can change parameters while the system is running, including prompt blends, LoRA strength, source preservation, denoise behavior, guidance, velocity scaling, channel guidance, and per-frame modulation curves.

Highlights

Streaming diffusion for ACE-Step v1.5
Ring-buffer scheduling with multiple in-flight generations
TensorRT acceleration for low-latency ticks
Hot-mutable controls during generation
Hot-resizable ring buffer depth
Per-frame modulation curves at latent-frame resolution
Heterogeneous slots, where each in-flight generation can carry its own seed, denoise schedule, conditioning, CFG mode, curves, and masks
Multi-condition compositing
LoRA hot-swapping without rebuilding the TensorRT decoder engine
Windowed VAE decoding for low-latency audio updates
Streaming output is bit-identical to batch output

What is in this Hugging Face repo?

This Hugging Face page is the release/model card for DEMON.

DEMON itself is an engine built around ACE-Step v1.5. The full source code, examples, demos, TensorRT build scripts, and documentation are available here:

https://github.com/daydreamlive/DEMON

If this Hugging Face repository is used as a release mirror, source paths mentioned below refer to the GitHub repository.

If you are looking for the underlying ACE-Step v1.5 base model, see:

https://huggingface.co/ACE-Step/Ace-Step1.5

Paper and technical notes

DEMON paper: https://arxiv.org/abs/2605.28657
FastOobleckDecoder / VAE distillation: coming soon
Latent Channel Semantics / 64-channel VAE characterization: coming soon

Links will be updated as companion artifacts are released.

Tested hardware

DEMON has been tested on:

NVIDIA RTX 3090
NVIDIA RTX 4090
NVIDIA RTX 5090

The headline performance numbers below are from an RTX 5090.

Performance

RTX 5090, ACE-Step v1.5 turbo, all-TensorRT, depth=4, steps=8, vae_window=3s, 60-second source.

Metric	Value
Tick latency, decoder forward, depth=4	~43 ms
Windowed VAE decode, 3 s	4.5 ms
Production throughput, depth=4	11.3 generations/s
Per-frame control resolution	25 Hz
Streaming vs. batch quality	bit-identical output

The paper reports up to 12.3 decoder completions per second on a single RTX 5090, and 11.3 generations per second at production ring depth 4.

Requirements

Python 3.11
NVIDIA GPU
ACE-Step v1.5 checkpoints in checkpoints/
Node.js 20+ if running the bundled web demo
TensorRT 10.16.x for TensorRT acceleration

ACE-Step checkpoints are auto-downloaded on first run where supported.

LoRAs are not auto-downloaded. Drop .safetensors LoRA files into:

$ACESTEP_MODELS_DIR/loras/

By default this resolves to:

~/.daydream-scope/models/demon/loras/

Setup

Clone the source repository:

git clone https://github.com/daydreamlive/DEMON.git
cd DEMON

Install Python dependencies:

uv sync

Run the main local demo:

uv run python -u -m demos.realtime_motion_graph_web.run

Then open:

http://localhost:6660

The launcher starts:

backend: http://localhost:1318
frontend: http://localhost:6660

Hosted demo

If you do not have a local GPU, or just want to try DEMON first, use the hosted instance:

https://music.daydream.live

Running with acceleration backends

The DiT decoder and VAE can choose backends independently.

Supported backend values:

tensorrt
compile
eager

Use TensorRT for the fastest path:

uv run python -u -m demos.realtime_motion_graph_web.run -- --accel tensorrt

Use TensorRT for the decoder and eager mode for the VAE:

uv run python -u -m demos.realtime_motion_graph_web.run -- \
  --accel tensorrt \
  --vae-accel eager

Use PyTorch compile mode if you do not want to build TensorRT engines yet:

uv run python -u -m demos.realtime_motion_graph_web.run -- --accel compile

Recommended starting point:

TRT windowed VAE decoder + compile decoder

The windowed VAE decoder is the cheapest TensorRT engine to build, is checkpoint- and duration-agnostic, and unlocks low-latency streaming decode.

Building TensorRT engines

DEMON targets TensorRT 10.16.x.

TensorRT plans are version- and GPU-architecture-specific by default, so rebuild after changing TensorRT, CUDA, driver, or the GPU used for inference.

Build the full matrix:

uv run python -m acestep.engine.trt.build --all

Build 60-second engines only:

uv run python -m acestep.engine.trt.build --all --duration 60

Build only the windowed VAE decoder:

uv run python -m acestep.engine.trt.build --vae-only --duration 60

Preview what would be built:

uv run python -m acestep.engine.trt.build --all --dry-run

Force rebuild:

uv run python -m acestep.engine.trt.build --all --force-rebuild

Force rebuild and ONNX re-export:

uv run python -m acestep.engine.trt.build --all --duration 60 --force-rebuild --force-onnx

TensorRT engine layout:

trt_engines/
  _onnx/
    vae_encode/vae_encode.onnx
    vae_decode/vae_decode.onnx
    decoder/decoder.onnx
    decoder_refit/decoder_refit.onnx

  decoder_mixed_refit_b8_60s/
    decoder_mixed_refit_b8_60s.engine

  vae_decode_fp16_3to30s/
    vae_decode_fp16_3to30s.engine

Pass engine paths to Session directly:

from acestep.engine.session import Session

session = Session(
    decoder_backend="tensorrt",
    vae_backend="tensorrt",
    vae_window=3.0,
    trt_engines={
        "decoder": "trt_engines/decoder_mixed_refit_b8_60s/decoder_mixed_refit_b8_60s.engine",
        "vae_encode": "trt_engines/vae_encode_fp16_60s/vae_encode_fp16_60s.engine",
        "vae_decode": "trt_engines/vae_decode_fp16_3to30s/vae_decode_fp16_3to30s.engine",
    },
)

Core idea: ring-buffer streaming diffusion

DEMON keeps several generations in flight at different denoising stages.

Each tick advances active slots by one denoising step. After warmup, the system continuously emits completed latents.

Throughput scales with:

depth / steps

For example:

depth = 4
steps = 8

means the engine can emit completed results at a steady streaming rate once the ring buffer is warm.

Ring buffer depth

pipeline_depth controls how many generations are in flight.

Higher depth:

smoother parameter sweeps
more slots in different denoising phases
higher throughput
more VRAM and per-tick compute
slower convergence for some submission-time changes

Lower depth:

snappier control response
lower VRAM
lower per-tick compute
more discrete changes

Depth can be changed while the system is running:

pipeline.set_depth(n)

Active slots drain naturally.

Song duration and TensorRT profiles

TensorRT engines are profile-specific.

A 240-second engine reserves more workspace than a 60-second engine, even if the current workload is only 60 seconds. Build only the durations you need.

Per-engine peak workspace measured in isolation on RTX 5090:

Component	60s engine	240s engine	Difference
Decoder, refit	13,511 MB	15,911 MB	+2,400 MB
VAE decode	10,547 MB	10,814 MB	+267 MB
VAE encode	4,178 MB	10,614 MB	+6,436 MB

These are per-engine peaks captured in separate subprocesses, not a live-runtime sum. At inference time, the decoder peak dominates and the VAE workspaces do not peak alongside it, which is why the live demo fits on a 24 GB card.

The comparison is what matters: switching the three engines from 240 seconds to 60 seconds frees about 9 GB.

VAE windowing

When vae_window > 0, decode happens in overlapped time windows instead of full-length decode.

This is what unlocks low-latency streaming updates: only the requested window is decoded per call rather than the full latent.

Set:

vae_window = 0

to use full-length decode.

Set:

vae_window = 3.0

to decode three-second windows.

Programmatic use: Session API

The Session API is the main programmatic surface.

It loads the model once and exposes the core primitives:

prepare_source
encode_text
generate
decode
stream
apply_lora

Minimal skeleton:

from acestep.engine.session import Session
from acestep.constants import TASK_INSTRUCTIONS

session = Session(
    decoder_backend="compile",  # "tensorrt", "compile", or "eager"
    vae_backend="compile",      # "tensorrt", "compile", or "eager"
    vae_window=3.0,             # 0 = full decode; >0 = windowed decode
)

# Replace this with your own audio loading.
audio = load_audio("source.wav")

# Load audio, encode it, and extract semantic context.
source = session.prepare_source(audio)

# Encode text conditioning once and reuse it.
cond = session.encode_text(
    tags="deathstep death",
    instruction=TASK_INSTRUCTIONS["cover"],
    refer_latent=source.latent,
    bpm=136,
    duration=60.0,
    key="G# minor",
)

# Generate multiple variants cheaply after warmup.
for seed in [1528, 9999, 42]:
    latent = session.generate(
        conditioning=cond,
        context_latent=source.context_latent,
        source_latent=source.latent,
        seed=seed,
    )

    audio_out = session.decode(latent)

    # Replace this with your own audio saving.
    save_audio(audio_out, f"out_{seed}.wav")

Streaming use

Streaming wraps the same primitives in a StreamHandle:

handle = session.stream(
    source=source,
    conditioning=cond,
    pipeline_depth=4,
)

for tick in range(128):
    latent = handle.tick()

    if latent is not None:
        audio = handle.decode(
            latent,
            t_start=0.0,
        )

Shared curve overrides bypass normal ring-buffer drain and take effect on the next tick:

handle.pipeline.set_shared_curve("velocity_scale", 1.2)
handle.pipeline.set_shared_curve("sde_denoise_curve", torch.tensor([...]))

Revert a shared override:

handle.pipeline.set_shared_curve("velocity_scale", None)

Typed node graph

DEMON exposes a typed node graph for building higher-level applications.

The node graph contains composable operations for:

latent operations
audio operations
conditioning
curves
masks
solver controls
config
DCW correction
channel guidance

The graph is wired through:

NodeDefinition
NodePort
NodeParam

Registration validates keyword arguments so applications can safely build on top of the same engine primitives.

This means a CLI, notebook, VST, web demo, MCP tool, or custom protocol can drive the same underlying system.

Engine features

Streaming diffusion

StreamPipeline maintains a ring buffer of in-flight generations. Each tick runs a batched decoder forward pass that advances active slots by one denoising step.

When CFG is active, the engine runs positive and negative branches as needed.

The decoder dispatches to TensorRT or PyTorch through the same code path.

Heterogeneous slots

Every in-flight slot carries its own SlotRequest.

A slot can have its own:

seed
denoise strength
cached timestep schedule
source latent
per-frame curves
conditioning
CFG mode
x0 target
latent-noise mask

A single ring buffer can mix different request types at the same time.

For example, the same active buffer can contain:

denoise = 1.0 regeneration
denoise = 0.5 style transfer
RCFG-self request

and still batch them together in one forward pass.

Scalar-or-curve modulation

Many controls accept either a scalar or a [T] tensor.

Supported scalar-or-curve controls include:

velocity scale
SDE re-noise
ODE noise injection
guidance scale
x0 target strength
x0 target curve
initial noise mix
APG momentum
CFG rescale
DCW scalers
condition temporal weights

Values are canonicalized at the boundary so kernels see one consistent shape.

Channel guidance

Channel guidance applies a [1, T, 64] per-channel gain to xt before each forward pass.

This has its own surface:

pipeline.set_channel_gain_tensor(...)

It is separate from normal [T] curve controls because it is both per-channel and per-frame.

Shared mutable curves

Shared mutable curves override selected curve-shaped fields on every in-flight slot at once.

Supported shared curve names include:

velocity_scale
sde_denoise_curve
ode_noise_curve
apg_momentum
x0_target_strength
cfg_rescale_curve

Set a shared curve:

pipeline.set_shared_curve("velocity_scale", 1.2)

Revert to per-slot behavior:

pipeline.set_shared_curve("velocity_scale", None)

Shared mutable curves take effect on the next tick instead of waiting for new submissions to drain through the ring buffer.

Multi-condition compositing

Within a single slot, the decoder can run once per active condition and blend velocities per frame using temporal_weight.

Conditions can be gated by step range.

Typed entry points include:

ConditioningBlend
ConditioningCombine

CFG modes

DEMON supports three CFG modes:

Standard CFG
Runs an unconditional forward pass every step.
RCFG-initialize
Runs one unconditional forward pass per slot, then caches it for the rest of the schedule.
RCFG-self
Runs zero unconditional forwards. The slot's initial noise stands in as the virtual unconditional velocity.

All three modes support APG momentum and optional per-frame CFG rescale curves.

Latent-noise-mask inpainting

DEMON supports latent-noise-mask inpainting with two-sided x0 blending:

pre-blend on xt
post-blend on predicted x0

This matches ComfyUI-style semantics and lets the decoder see correctly noised context in preserved regions.

A per-step strength function can be used for progressive masking.

DCW post-step correction

DEMON includes wavelet-domain sampler-side correction from Yu et al., ported from upstream ACE-Step v0.1.7.

Supported modes:

low
high
double
pix

Advanced controls include:

mult_blend
mag_phase
soft_thresh

At zero, the advanced surface is byte-identical to the upstream reference.

Update DCW live:

pipeline.set_dcw(...)

Hot LoRA

When the decoder runs in TensorRT mode, LoRA updates are applied through a refitter against the live engine.

Supported LoRA lifecycle operations include:

register
enable
set_strength
remove

TensorRT acceleration

DEMON can accelerate the DiT decoder, VAE encode, and VAE decode independently.

Component	Backend	Notes
Decoder	`tensorrt`	Fastest path. Requires a built decoder engine for the target duration and checkpoint. Refit-enabled engines support LoRA swaps.
Decoder	`compile`	Uses `torch.compile`. Long warmup, no TensorRT engine required.
Decoder	`eager`	Plain PyTorch. Useful for debugging.
VAE encode/decode	`tensorrt`	Fastest VAE path. Windowed decode engine is reused across durations.
VAE encode/decode	`compile`	Uses `torch.compile`.
VAE encode/decode	`eager`	Plain PyTorch. Useful for debugging.

Demo applications

DEMON ships a flagship reference application plus focused examples.

Realtime motion graph web demo

Run:

uv run python -u -m demos.realtime_motion_graph_web.run

Then open:

http://localhost:6660

The web demo lets you:

feed source audio
write prompts
blend two prompts live
change denoise strength
hot-swap LoRAs
blend timbre and structure references
draw automation curves
map MIDI controls
record output
drive the system through the onboard MCP server

Forward backend flags after --:

uv run python -u -m demos.realtime_motion_graph_web.run -- --accel tensorrt
uv run python -u -m demos.realtime_motion_graph_web.run -- --checkpoint xl

Other examples

examples/session_demo.py
examples/realtime_cover.py
examples/covers/
demos/test_stream_cover_graph.py

Feature examples include:

Script	Feature
`cover_basic.py`	Standard cover pipeline
`prompt_blend.py`	Two prompts blended with a temporal curve
`sde_denoise_curve.py`	Per-frame SDE re-noise modulation
`velocity_scaling.py`	Per-frame transformation rate control
`lora_generation.py`	LoRA-conditioned generation
`x0_target_blend.py`	Two-pass morphing toward a target latent
`conditioning_average.py`	Fuse two conditionings
`guidance_curve.py`	Per-frame CFG scale
`latent_noise_mask.py`	Latent-space inpainting
`initial_noise_curve.py`	Per-frame noise/source init mix
`ode_noise_injection.py`	Stochastic ODE step
`cover_semantic_blend.py`	Blend semantic hints from two sources
`x0_target_from_reference.py`	Pre-generate a target latent, then morph toward it

Tests

Run:

uv run pytest tests/ -v

Limitations

DEMON requires an NVIDIA GPU for practical real-time use.
TensorRT engines are GPU-, CUDA-, driver-, and TensorRT-version specific.
TensorRT engines may need to be rebuilt locally.
Real-time performance depends on GPU, song duration, ring-buffer depth, denoising steps, VAE window size, and selected backend.
DEMON is an engine around ACE-Step v1.5, not a replacement for the ACE-Step base model.
Users should review the ACE-Step model card, license, and usage notes before deploying systems built on DEMON.
Generated audio quality depends on the source audio, prompts, LoRAs, schedule settings, and backend configuration.

Responsible use

DEMON is intended for creative music generation, live performance, research, prototyping, and audio experimentation.

Users are responsible for ensuring they have the rights to any source audio, prompts, LoRAs, datasets, or generated outputs they use, especially for commercial use.

Do not use DEMON to imitate or misuse the identity, voice, likeness, or creative work of others without appropriate rights or consent.

Citation

If you use DEMON in your work, please cite the DEMON paper:

@misc{fosdick2026demon,
  title={DEMON: Diffusion Engine for Musical Orchestrated Noise},
  author={Fosdick, Ryan},
  year={2026},
  eprint={2605.28657},
  archivePrefix={arXiv},
  primaryClass={cs.SD},
  url={https://arxiv.org/abs/2605.28657}
}

Please also cite ACE-Step when appropriate, since DEMON is built on top of ACE-Step v1.5.

Acknowledgments

DEMON is built on top of ACE-Step.

The base diffusion model, VAE, text encoder, and 5 Hz language model are ACE-Step's work. Without the ACE-Step team releasing the v1.5 weights and code under MIT, DEMON would not exist.

Thank you to the ACE-Step team for making this work possible.

Authors

DEMON was originally created by Ryan Fosdick.

Maintained by Daydream Live and contributors.

Ryan Fosdick: https://ryanontheinside.com
Daydream Live: https://daydream.live

Downloads last month: -

Model tree for daydreamlive/DEMON

Base model

ACE-Step/Ace-Step1.5

Adapter

(14)

this model

Paper for daydreamlive/DEMON

DEMON: Diffusion Engine for Musical Orchestrated Noise

Paper • 2605.28657 • Published 6 days ago • 4