Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Echo","local":"echo","sections":[{"title":"Convert the checkpoint","local":"convert-the-checkpoint","sections":[],"depth":2},{"title":"Inference","local":"inference","sections":[],"depth":2},{"title":"EchoModularPipeline","local":"diffusers.EchoModularPipeline","sections":[],"depth":2},{"title":"EchoBlocks","local":"diffusers.EchoBlocks","sections":[],"depth":2}],"depth":1}"/> | |
| <link href="/docs/diffusers/pr_14696/en/_app/immutable/entry/start.ChQYU6cA.js" rel="modulepreload"> | |
| <link href="/docs/diffusers/pr_14696/en/_app/immutable/chunks/QGNXUmTZ.js" rel="modulepreload"> | |
| <link href="/docs/diffusers/pr_14696/en/_app/immutable/chunks/jDjavuwI.js" rel="modulepreload"> | |
| <link href="/docs/diffusers/pr_14696/en/_app/immutable/entry/app.Y-N5bBSC.js" rel="modulepreload"> | |
| <link href="/docs/diffusers/pr_14696/en/_app/immutable/chunks/CMF6C6Mt.js" rel="modulepreload"> | |
| <link href="/docs/diffusers/pr_14696/en/_app/immutable/chunks/DRZ5Z616.js" rel="modulepreload"> | |
| <link href="/docs/diffusers/pr_14696/en/_app/immutable/chunks/DsnmJJEf.js" rel="modulepreload"> | |
| <link href="/docs/diffusers/pr_14696/en/_app/immutable/chunks/i0LL_sTA.js" rel="modulepreload"> | |
| <link href="/docs/diffusers/pr_14696/en/_app/immutable/nodes/0.jmrvKR2S.js" rel="modulepreload"> | |
| <link href="/docs/diffusers/pr_14696/en/_app/immutable/chunks/9Uo8d8lw.js" rel="modulepreload"> | |
| <link href="/docs/diffusers/pr_14696/en/_app/immutable/nodes/160.Do6c3RN_.js" rel="modulepreload"> | |
| <link href="/docs/diffusers/pr_14696/en/_app/immutable/chunks/BLbO27j4.js" rel="modulepreload"> | |
| <!--rbrdnd--><meta name="hf:doc:metadata" content="{"title":"Echo","local":"echo","sections":[{"title":"Convert the checkpoint","local":"convert-the-checkpoint","sections":[],"depth":2},{"title":"Inference","local":"inference","sections":[],"depth":2},{"title":"EchoModularPipeline","local":"diffusers.EchoModularPipeline","sections":[],"depth":2},{"title":"EchoBlocks","local":"diffusers.EchoBlocks","sections":[],"depth":2}],"depth":1}"/><!----> | |
| <link href="/docs/diffusers/pr_14696/en/_app/immutable/assets/0.tn0RQdqM.css" rel="modulepreload"> <!--[--><!--[0--><!--[--><!--[0--><!--[--><p></p> <!--[0--><h1 class="relative group"><a id="echo" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#echo"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Echo</span></h1><!--]--><!----> <p><a href="https://github.com/jd-opensource/JoyAI-Echo" rel="nofollow">Echo</a> is a long-video generation model. It adds an optional clean first | |
| frame, ordered image/audio memory slots, and a stochastic few-step Distribution Matching Distillation (DMD) sampler. | |
| The pipeline generates synchronized video and audio.</p> <p>Echo is implemented as a Modular Pipeline so its text encoding, memory conditioning, stochastic DMD denoising, and | |
| decoding blocks can be run as a complete workflow or composed independently.</p> <!--[1--><h2 class="relative group"><a id="convert-the-checkpoint" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#convert-the-checkpoint"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Convert the checkpoint</span></h2><!--]--><!----> <p>Convert the BF16 Echo release checkpoint before loading it. The converter reuses the Gemma text encoder and | |
| tokenizer directly from <code>google/gemma-3-12b-it</code>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-bash "><!---->python scripts/convert_echo_to_diffusers.py \ | |
| --checkpoint /path/to/echo15_full_dmd \ | |
| --output-path /path/to/Echo-Diffusers \ | |
| --repo-id jdopensource/JoyAI-Echo<!----></pre></div><!----> <p>The Gemma repository is gated, so users must accept its license and authenticate with Hugging Face before loading the | |
| pipeline. Pass a different <code>--base-model</code> only when the compatible Gemma model and tokenizer are stored together at | |
| that repository or path root. <code>--repo-id</code> records portable Hub references for the converted Echo components; without | |
| it, the index targets the local output path.</p> <!--[1--><h2 class="relative group"><a id="inference" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#inference"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Inference</span></h2><!--]--><!----> <p>The released model uses 241 frames in its long-video example. The video RoPE coordinates remain at the training rate | |
| of 24 fps, independently of the output container rate.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">import</span> torchaudio | |
| <span class="hljs-keyword">from</span> PIL <span class="hljs-keyword">import</span> Image | |
| <span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> ComponentsManager, ModularPipeline | |
| <span class="hljs-keyword">from</span> diffusers.utils <span class="hljs-keyword">import</span> encode_video | |
| model_path = <span class="hljs-string">"/path/to/Echo-Diffusers"</span> | |
| manager = ComponentsManager() | |
| pipe = ModularPipeline.from_pretrained(model_path, components_manager=manager) | |
| pipe.load_components(dtype={<span class="hljs-string">"default"</span>: torch.bfloat16, <span class="hljs-string">"audio_vae"</span>: torch.float32}) | |
| manager.enable_auto_cpu_offload(device=<span class="hljs-string">"cuda"</span>) | |
| pipe.vae.enable_tiling() | |
| first_frame = Image.<span class="hljs-built_in">open</span>(<span class="hljs-string">"first_frame.png"</span>).convert(<span class="hljs-string">"RGB"</span>) | |
| memory_images = [Image.<span class="hljs-built_in">open</span>(path).convert(<span class="hljs-string">"RGB"</span>) <span class="hljs-keyword">for</span> path <span class="hljs-keyword">in</span> [<span class="hljs-string">"memory_0.png"</span>, <span class="hljs-string">"memory_1.png"</span>]] | |
| memory_audio_with_rates = [torchaudio.load(path) <span class="hljs-keyword">for</span> path <span class="hljs-keyword">in</span> [<span class="hljs-string">"memory_0.wav"</span>, <span class="hljs-string">"memory_1.wav"</span>]] | |
| memory_audio = [waveform <span class="hljs-keyword">for</span> waveform, _ <span class="hljs-keyword">in</span> memory_audio_with_rates] | |
| memory_audio_rates = [sample_rate <span class="hljs-keyword">for</span> _, sample_rate <span class="hljs-keyword">in</span> memory_audio_with_rates] | |
| output = pipe( | |
| prompt=<span class="hljs-string">"A cinematic dialogue scene in a quiet cafe."</span>, | |
| image=first_frame, | |
| memory_images=memory_images, | |
| memory_audio_waveforms=memory_audio, | |
| memory_audio_sample_rates=memory_audio_rates, | |
| width=<span class="hljs-number">1280</span>, | |
| height=<span class="hljs-number">736</span>, | |
| num_frames=<span class="hljs-number">241</span>, | |
| frame_rate=<span class="hljs-number">25.0</span>, | |
| model_frame_rate=<span class="hljs-number">24.0</span>, | |
| generator=torch.Generator(device=<span class="hljs-string">"cuda"</span>).manual_seed(<span class="hljs-number">42</span>), | |
| output_type=<span class="hljs-string">"np"</span>, | |
| output=[<span class="hljs-string">"videos"</span>, <span class="hljs-string">"audio"</span>], | |
| ) | |
| encode_video( | |
| output[<span class="hljs-string">"videos"</span>][<span class="hljs-number">0</span>], | |
| fps=<span class="hljs-number">25</span>, | |
| audio=output[<span class="hljs-string">"audio"</span>][<span class="hljs-number">0</span>].<span class="hljs-built_in">float</span>().cpu(), | |
| audio_sample_rate=pipe.vocoder.config.output_sampling_rate, | |
| output_path=<span class="hljs-string">"echo.mp4"</span>, | |
| )<!----></pre></div><!----> <p>The default DMD sigma schedule is the released eight-step schedule. It predicts x0 at every step and re-noises with | |
| fresh Gaussian noise at the next sigma, so a seeded <code>torch.Generator</code> controls both the initial noise and all | |
| intermediate re-noising.</p> <p>Raw audio-memory encoding requires <code>torchaudio</code>. For reference parity, keep <code>audio_vae</code> in FP32 as shown above. | |
| Modular workflows can cache and reuse the condition encoder’s packed token outputs by running that block separately.</p> <!--[1--><h2 class="relative group"><a id="diffusers.EchoModularPipeline" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#diffusers.EchoModularPipeline"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>EchoModularPipeline</span></h2><!--]--><!----> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><div><span class="group flex space-x-1.5 items-center text-gray-800 bg-gradient-to-r rounded-tr-lg -mt-4 -ml-4 pt-3 px-2.5" id="diffusers.EchoModularPipeline"><!----><h3 class="!m-0"><span class="flex-1 break-all md:text-lg bg-gradient-to-r px-2.5 py-1.5 rounded-xl from-indigo-50/70 to-white dark:from-gray-900 dark:to-gray-950 dark:text-indigo-300 text-indigo-700"><svg class="mr-1.5 text-indigo-500 inline-block -mt-0.5" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" focusable="false" role="img" width=".8em" height=".8em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 24 24"><path class="uim-quaternary" d="M20.23 7.24L12 12L3.77 7.24a1.98 1.98 0 0 1 .7-.71L11 2.76c.62-.35 1.38-.35 2 0l6.53 3.77c.29.173.531.418.7.71z" opacity=".25" fill="currentColor"></path><path class="uim-tertiary" d="M12 12v9.5a2.09 2.09 0 0 1-.91-.21L4.5 17.48a2.003 2.003 0 0 1-1-1.73v-7.5a2.06 2.06 0 0 1 .27-1.01L12 12z" opacity=".5" fill="currentColor"></path><path class="uim-primary" d="M20.5 8.25v7.5a2.003 2.003 0 0 1-1 1.73l-6.62 3.82c-.275.13-.576.198-.88.2V12l8.23-4.76c.175.308.268.656.27 1.01z" fill="currentColor"></path></svg><span class="font-light">class</span> <span class="font-medium">diffusers.</span><span class="font-semibold">EchoModularPipeline</span></span></h3><!----> <a id="diffusers.EchoModularPipeline" class="header-link invisible with-hover:group-hover:visible pr-2" href="#diffusers.EchoModularPipeline"><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></a> <!--[0--><a class="!ml-auto !text-gray-400 !no-underline text-sm flex items-center" href="https://github.com/huggingface/diffusers/blob/vr_14696/src/diffusers/modular_pipelines/echo/modular_pipeline.py#L27" target="_blank"><span><</span> <span class="hidden md:block mx-0.5 hover:!underline">source</span> <span>></span></a><!--]--></span> <!--[0--><p class="font-mono text-xs md:text-sm !leading-relaxed !my-6"><span>(</span> <!--[--><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">blocks<span class="opacity-60">: diffusers.modular_pipelines.modular_pipeline.ModularPipelineBlocks | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">pretrained_model_name_or_path<span class="opacity-60">: str | os.PathLike | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">components_manager<span class="opacity-60">: diffusers.modular_pipelines.components_manager.ComponentsManager | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">collection<span class="opacity-60">: str | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">workflow<span class="opacity-60">: str | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">modular_config_dict<span class="opacity-60">: dict[str, typing.Any] | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">config_dict<span class="opacity-60">: dict[str, typing.Any] | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">**kwargs<span class="opacity-60"></span></span></span><!--]--> <span>)</span> <!--[-1--><!--]--></p><!--]--> <div class="!mb-10 relative docstring-details "><!--[-1--><!--]--> <!--[-1--><!--]--> <!--[-1--><!--]--> <!--[-1--><!--]--> <!--[-1--><!--]--></div></div><!----> <p>A ModularPipeline for the Echo long-video DMD checkpoint.</p></div> <!--[1--><h2 class="relative group"><a id="diffusers.EchoBlocks" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#diffusers.EchoBlocks"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>EchoBlocks</span></h2><!--]--><!----> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><div><span class="group flex space-x-1.5 items-center text-gray-800 bg-gradient-to-r rounded-tr-lg -mt-4 -ml-4 pt-3 px-2.5" id="diffusers.EchoBlocks"><!----><h3 class="!m-0"><span class="flex-1 break-all md:text-lg bg-gradient-to-r px-2.5 py-1.5 rounded-xl from-indigo-50/70 to-white dark:from-gray-900 dark:to-gray-950 dark:text-indigo-300 text-indigo-700"><svg class="mr-1.5 text-indigo-500 inline-block -mt-0.5" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" focusable="false" role="img" width=".8em" height=".8em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 24 24"><path class="uim-quaternary" d="M20.23 7.24L12 12L3.77 7.24a1.98 1.98 0 0 1 .7-.71L11 2.76c.62-.35 1.38-.35 2 0l6.53 3.77c.29.173.531.418.7.71z" opacity=".25" fill="currentColor"></path><path class="uim-tertiary" d="M12 12v9.5a2.09 2.09 0 0 1-.91-.21L4.5 17.48a2.003 2.003 0 0 1-1-1.73v-7.5a2.06 2.06 0 0 1 .27-1.01L12 12z" opacity=".5" fill="currentColor"></path><path class="uim-primary" d="M20.5 8.25v7.5a2.003 2.003 0 0 1-1 1.73l-6.62 3.82c-.275.13-.576.198-.88.2V12l8.23-4.76c.175.308.268.656.27 1.01z" fill="currentColor"></path></svg><span class="font-light">class</span> <span class="font-medium">diffusers.</span><span class="font-semibold">EchoBlocks</span></span></h3><!----> <a id="diffusers.EchoBlocks" class="header-link invisible with-hover:group-hover:visible pr-2" href="#diffusers.EchoBlocks"><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></a> <!--[0--><a class="!ml-auto !text-gray-400 !no-underline text-sm flex items-center" href="https://github.com/huggingface/diffusers/blob/vr_14696/src/diffusers/modular_pipelines/echo/modular_blocks_echo.py#L26" target="_blank"><span><</span> <span class="hidden md:block mx-0.5 hover:!underline">source</span> <span>></span></a><!--]--></span> <!--[0--><p class="font-mono text-xs md:text-sm !leading-relaxed !my-6"><span>(</span> <!--[--><!--]--> <span>)</span> <!--[-1--><!--]--></p><!--]--> <div class="!mb-10 relative docstring-details "><!--[-1--><!--]--> <!--[-1--><!--]--> <!--[-1--><!--]--> <!--[-1--><!--]--> <!--[-1--><!--]--></div></div><!----> <p>Echo reference-to-video generation with clean first-frame conditioning, ordered image/audio memory slots, and | |
| stochastic DMD denoising.</p> <p>Components: | |
| text_encoder (<code>PreTrainedModel</code>) tokenizer (<code>PreTrainedTokenizerBase</code>) connectors (<code>LTX2TextConnectors</code>) vae | |
| (<code>AutoencoderKLLTX2Video</code>) audio_vae (<code>AutoencoderKLLTX2Audio</code>) transformer (<code>LTX2VideoTransformer3DModel</code>) | |
| video_processor (<code>VideoProcessor</code>) vocoder (<code>LTX2Vocoder</code>)</p> <p>Inputs: | |
| prompt (<code>str</code>): | |
| The prompt or prompts to guide image generation. | |
| max_sequence_length (<code>int</code>, <em>optional</em>, defaults to 1024): | |
| Maximum sequence length for prompt encoding. | |
| image (<code>Image | Tensor</code>, <em>optional</em>): | |
| Optional single first frame used as a clean reference condition. | |
| memory_images (<code>list</code>, <em>optional</em>): | |
| Ordered reference images, one per Echo memory slot. | |
| memory_audio_waveforms (<code>list</code>, <em>optional</em>): | |
| Ordered memory waveforms as <code>(channels, samples)</code> tensors. Inputs longer than 9.62 seconds are cropped to | |
| their highest-response window. Use <code>None</code> for a silent slot. | |
| memory_audio_sample_rates (<code>int | list</code>, <em>optional</em>): | |
| Sampling rate shared by all memory waveforms, or one rate per slot. | |
| height (<code>int</code>, <em>optional</em>, defaults to 512): | |
| The height in pixels of the generated image. | |
| width (<code>int</code>, <em>optional</em>, defaults to 704): | |
| The width in pixels of the generated image. | |
| model_frame_rate (<code>float</code>, <em>optional</em>, defaults to 24.0): | |
| Training-time frame rate used for Echo video RoPE coordinates. | |
| memory_position_offset (<code>float</code>, <em>optional</em>, defaults to 500.0): | |
| Temporal center assigned to the first memory slot. | |
| memory_position_slot_stride (<code>float</code>, <em>optional</em>, defaults to 50.0): | |
| Temporal distance between consecutive memory-slot centers. | |
| num_frames (<code>int</code>, <em>optional</em>, defaults to 241): | |
| Number of generated pixel frames; must be <code>1 + k * vae_temporal_compression_ratio</code>. | |
| frame_rate (<code>float</code>, <em>optional</em>, defaults to 25.0): | |
| Frame rate of the generated video. | |
| latents (<code>Tensor</code>, <em>optional</em>): | |
| Pre-generated noisy latents for image generation. | |
| audio_latents (<code>Tensor</code>, <em>optional</em>): | |
| Optional packed initial audio noise latents. | |
| generator (<code>Generator</code>, <em>optional</em>): | |
| Torch generator for deterministic generation. | |
| num_videos_per_prompt (<code>int</code>, <em>optional</em>, defaults to 1): | |
| The number of images to generate per prompt. | |
| sigmas (<code>list | tuple</code>): | |
| DMD sigma schedule, including the terminal zero. | |
| attention_kwargs (<code>dict</code>, <em>optional</em>): | |
| Additional kwargs for attention processors. | |
| output_type (<code>str</code>, <em>optional</em>, defaults to pil): | |
| Output format: ‘pil’, ‘np’, ‘pt’. | |
| decode_timestep (<code>None</code>, <em>optional</em>, defaults to 0.0): | |
| The timestep at which the VAE decodes the final latents. | |
| decode_noise_scale (<code>None</code>, <em>optional</em>): | |
| Noise interpolation factor applied to the latents at the decode timestep.</p> <p>Outputs: | |
| videos (<code>list</code>): | |
| The generated videos. | |
| audio (<code>Tensor</code>): | |
| The generated audio waveform.</p></div> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/pipelines/echo.md" target="_blank"><svg class="mr-1" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M31,16l-7,7l-1.41-1.41L28.17,16l-5.58-5.59L24,9l7,7z"></path><path d="M1,16l7-7l1.41,1.41L3.83,16l5.58,5.59L8,23l-7-7z"></path><path d="M12.419,25.484L17.639,6.552l1.932,0.518L14.351,26.002z"></path></svg><!----> <span><span class="underline">Update</span> on GitHub</span></a><!----> <p></p><!--]--><!----><!--]--><!--]--><!--]--> <!--[-1--><!--]--><!--]--> | |
| <script> | |
| { | |
| __sveltekit_bp7dlj = { | |
| base: "/docs/diffusers/pr_14696/en", | |
| assets: "/docs/diffusers/pr_14696/en" | |
| }; | |
| const element = document.currentScript.parentElement; | |
| Promise.all([ | |
| import("/docs/diffusers/pr_14696/en/_app/immutable/entry/start.ChQYU6cA.js"), | |
| import("/docs/diffusers/pr_14696/en/_app/immutable/entry/app.Y-N5bBSC.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 160], | |
| data: [null,null], | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 28.5 kB
- Xet hash:
- 54969c11c17c902acd52c82f4710d2e7ca72c79fff6311ff97ebbaa71937c08a
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.