Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Scheduler features","local":"scheduler-features","sections":[{"title":"Timestep schedules","local":"timestep-schedules","sections":[],"depth":2},{"title":"Timestep spacing","local":"timestep-spacing","sections":[],"depth":2},{"title":"Sigmas","local":"sigmas","sections":[{"title":"Karras sigmas","local":"karras-sigmas","sections":[],"depth":3}],"depth":2},{"title":"Rescale noise schedule","local":"rescale-noise-schedule","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/diffusers/pr_11686/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/diffusers/pr_11686/en/_app/immutable/entry/start.2b9667fb.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/pr_11686/en/_app/immutable/chunks/scheduler.8c3d61f6.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/pr_11686/en/_app/immutable/chunks/singletons.756349ae.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/pr_11686/en/_app/immutable/chunks/index.0997d446.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/pr_11686/en/_app/immutable/chunks/paths.8d5937da.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/pr_11686/en/_app/immutable/entry/app.a2a6117e.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/pr_11686/en/_app/immutable/chunks/index.da70eac4.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/pr_11686/en/_app/immutable/nodes/0.a31d0923.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/pr_11686/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/pr_11686/en/_app/immutable/nodes/304.da7f1b0c.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/pr_11686/en/_app/immutable/chunks/Tip.1d9b8c37.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/pr_11686/en/_app/immutable/chunks/CodeBlock.a9c4becf.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/pr_11686/en/_app/immutable/chunks/getInferenceSnippets.d00e08ac.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Scheduler features","local":"scheduler-features","sections":[{"title":"Timestep schedules","local":"timestep-schedules","sections":[],"depth":2},{"title":"Timestep spacing","local":"timestep-spacing","sections":[],"depth":2},{"title":"Sigmas","local":"sigmas","sections":[{"title":"Karras sigmas","local":"karras-sigmas","sections":[],"depth":3}],"depth":2},{"title":"Rescale noise schedule","local":"rescale-noise-schedule","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="scheduler-features" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#scheduler-features"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Scheduler features</span></h1> <p data-svelte-h="svelte-1isuw1i">The scheduler is an important component of any diffusion model because it controls the entire denoising (or sampling) process. There are many types of schedulers, some are optimized for speed and some for quality. With Diffusers, you can modify the scheduler configuration to use custom noise schedules, sigmas, and rescale the noise schedule. Changing these parameters can have profound effects on inference quality and speed.</p> <p data-svelte-h="svelte-1l3yyn3">This guide will demonstrate how to use these features to improve inference quality.</p> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-1qkepy0">Diffusers currently only supports the <code>timesteps</code> and <code>sigmas</code> parameters for a select list of schedulers and pipelines. Feel free to open a <a href="https://github.com/huggingface/diffusers/issues/new/choose" rel="nofollow">feature request</a> if you want to extend these parameters to a scheduler and pipeline that does not currently support it!</p></div> <h2 class="relative group"><a id="timestep-schedules" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#timestep-schedules"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Timestep schedules</span></h2> <p data-svelte-h="svelte-dn9g4h">The timestep or noise schedule determines the amount of noise at each sampling step. The scheduler uses this to generate an image with the corresponding amount of noise at each step. The timestep schedule is generated from the scheduler’s default configuration, but you can customize the scheduler to use new and optimized sampling schedules that aren’t in Diffusers yet.</p> <p data-svelte-h="svelte-1d6178p">For example, <a href="https://research.nvidia.com/labs/toronto-ai/AlignYourSteps/" rel="nofollow">Align Your Steps (AYS)</a> is a method for optimizing a sampling schedule to generate a high-quality image in as little as 10 steps. The optimal <a href="https://github.com/huggingface/diffusers/blob/a7bf77fc284810483f1e60afe34d1d27ad91ce2e/src/diffusers/schedulers/scheduling_utils.py#L51" rel="nofollow">10-step schedule</a> for Stable Diffusion XL is:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> diffusers.schedulers <span class="hljs-keyword">import</span> AysSchedules | |
| sampling_schedule = AysSchedules[<span class="hljs-string">"StableDiffusionXLTimesteps"</span>] | |
| <span class="hljs-built_in">print</span>(sampling_schedule) | |
| <span class="hljs-string">"[999, 845, 730, 587, 443, 310, 193, 116, 53, 13]"</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-17et4f1">You can use the AYS sampling schedule in a pipeline by passing it to the <code>timesteps</code> parameter.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->pipeline = StableDiffusionXLPipeline.from_pretrained( | |
| <span class="hljs-string">"SG161222/RealVisXL_V4.0"</span>, | |
| torch_dtype=torch.float16, | |
| variant=<span class="hljs-string">"fp16"</span>, | |
| ).to(<span class="hljs-string">"cuda"</span>) | |
| pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config, algorithm_type=<span class="hljs-string">"sde-dpmsolver++"</span>) | |
| prompt = <span class="hljs-string">"A cinematic shot of a cute little rabbit wearing a jacket and doing a thumbs up"</span> | |
| generator = torch.Generator(device=<span class="hljs-string">"cpu"</span>).manual_seed(<span class="hljs-number">2487854446</span>) | |
| image = pipeline( | |
| prompt=prompt, | |
| negative_prompt=<span class="hljs-string">""</span>, | |
| generator=generator, | |
| timesteps=sampling_schedule, | |
| ).images[<span class="hljs-number">0</span>]<!-- HTML_TAG_END --></pre></div> <div class="flex gap-4" data-svelte-h="svelte-1wwk2vr"><div><img class="rounded-xl" src="https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/ays.png"> <figcaption class="mt-2 text-center text-sm text-gray-500">AYS timestep schedule 10 steps</figcaption></div> <div><img class="rounded-xl" src="https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/10.png"> <figcaption class="mt-2 text-center text-sm text-gray-500">Linearly-spaced timestep schedule 10 steps</figcaption></div> <div><img class="rounded-xl" src="https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/25.png"> <figcaption class="mt-2 text-center text-sm text-gray-500">Linearly-spaced timestep schedule 25 steps</figcaption></div></div> <h2 class="relative group"><a id="timestep-spacing" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#timestep-spacing"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Timestep spacing</span></h2> <p data-svelte-h="svelte-qlgx89">The way sample steps are selected in the schedule can affect the quality of the generated image, especially with respect to <a href="#rescale-noise-schedule">rescaling the noise schedule</a>, which can enable a model to generate much brighter or darker images. Diffusers provides three timestep spacing methods:</p> <ul data-svelte-h="svelte-iba8ru"><li><code>leading</code> creates evenly spaced steps</li> <li><code>linspace</code> includes the first and last steps and evenly selects the remaining intermediate steps</li> <li><code>trailing</code> only includes the last step and evenly selects the remaining intermediate steps starting from the end</li></ul> <p data-svelte-h="svelte-xqwul8">It is recommended to use the <code>trailing</code> spacing method because it generates higher quality images with more details when there are fewer sample steps. But the difference in quality is not as obvious for more standard sample step values.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> StableDiffusionXLPipeline, DPMSolverMultistepScheduler | |
| pipeline = StableDiffusionXLPipeline.from_pretrained( | |
| <span class="hljs-string">"SG161222/RealVisXL_V4.0"</span>, | |
| torch_dtype=torch.float16, | |
| variant=<span class="hljs-string">"fp16"</span>, | |
| ).to(<span class="hljs-string">"cuda"</span>) | |
| pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config, timestep_spacing=<span class="hljs-string">"trailing"</span>) | |
| prompt = <span class="hljs-string">"A cinematic shot of a cute little black cat sitting on a pumpkin at night"</span> | |
| generator = torch.Generator(device=<span class="hljs-string">"cpu"</span>).manual_seed(<span class="hljs-number">2487854446</span>) | |
| image = pipeline( | |
| prompt=prompt, | |
| negative_prompt=<span class="hljs-string">""</span>, | |
| generator=generator, | |
| num_inference_steps=<span class="hljs-number">5</span>, | |
| ).images[<span class="hljs-number">0</span>] | |
| image<!-- HTML_TAG_END --></pre></div> <div class="flex gap-4" data-svelte-h="svelte-106qvjp"><div><img class="rounded-xl" src="https://huggingface.co/datasets/stevhliu/testing-images/resolve/main/trailing_spacing.png"> <figcaption class="mt-2 text-center text-sm text-gray-500">trailing spacing after 5 steps</figcaption></div> <div><img class="rounded-xl" src="https://huggingface.co/datasets/stevhliu/testing-images/resolve/main/leading_spacing.png"> <figcaption class="mt-2 text-center text-sm text-gray-500">leading spacing after 5 steps</figcaption></div></div> <h2 class="relative group"><a id="sigmas" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#sigmas"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Sigmas</span></h2> <p data-svelte-h="svelte-1v7gnu4">The <code>sigmas</code> parameter is the amount of noise added at each timestep according to the timestep schedule. Like the <code>timesteps</code> parameter, you can customize the <code>sigmas</code> parameter to control how much noise is added at each step. When you use a custom <code>sigmas</code> value, the <code>timesteps</code> are calculated from the custom <code>sigmas</code> value and the default scheduler configuration is ignored.</p> <p data-svelte-h="svelte-1qygm3g">For example, you can manually pass the <a href="https://github.com/huggingface/diffusers/blob/6529ee67ec02fcf58d2fd9242164ea002b351d75/src/diffusers/schedulers/scheduling_utils.py#L55" rel="nofollow">sigmas</a> for something like the 10-step AYS schedule from before to the pipeline.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> DiffusionPipeline, EulerDiscreteScheduler | |
| model_id = <span class="hljs-string">"stabilityai/stable-diffusion-xl-base-1.0"</span> | |
| pipeline = DiffusionPipeline.from_pretrained( | |
| <span class="hljs-string">"stabilityai/stable-diffusion-xl-base-1.0"</span>, | |
| torch_dtype=torch.float16, | |
| variant=<span class="hljs-string">"fp16"</span>, | |
| ).to(<span class="hljs-string">"cuda"</span>) | |
| pipeline.scheduler = EulerDiscreteScheduler.from_config(pipeline.scheduler.config) | |
| sigmas = [<span class="hljs-number">14.615</span>, <span class="hljs-number">6.315</span>, <span class="hljs-number">3.771</span>, <span class="hljs-number">2.181</span>, <span class="hljs-number">1.342</span>, <span class="hljs-number">0.862</span>, <span class="hljs-number">0.555</span>, <span class="hljs-number">0.380</span>, <span class="hljs-number">0.234</span>, <span class="hljs-number">0.113</span>, <span class="hljs-number">0.0</span>] | |
| prompt = <span class="hljs-string">"anthropomorphic capybara wearing a suit and working with a computer"</span> | |
| generator = torch.Generator(device=<span class="hljs-string">'cuda'</span>).manual_seed(<span class="hljs-number">123</span>) | |
| image = pipeline( | |
| prompt=prompt, | |
| num_inference_steps=<span class="hljs-number">10</span>, | |
| sigmas=sigmas, | |
| generator=generator | |
| ).images[<span class="hljs-number">0</span>]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-wa4ie3">When you take a look at the scheduler’s <code>timesteps</code> parameter, you’ll see that it is the same as the AYS timestep schedule because the <code>timestep</code> schedule is calculated from the <code>sigmas</code>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-built_in">print</span>(<span class="hljs-string">f" timesteps: <span class="hljs-subst">{pipe.scheduler.timesteps}</span>"</span>) | |
| <span class="hljs-string">"timesteps: tensor([999., 845., 730., 587., 443., 310., 193., 116., 53., 13.], device='cuda:0')"</span><!-- HTML_TAG_END --></pre></div> <h3 class="relative group"><a id="karras-sigmas" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#karras-sigmas"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Karras sigmas</span></h3> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-adlqce">Refer to the scheduler API <a href="../api/schedulers/overview">overview</a> for a list of schedulers that support Karras sigmas.</p> <p data-svelte-h="svelte-1uk1mrr">Karras sigmas should not be used for models that weren’t trained with them. For example, the base Stable Diffusion XL model shouldn’t use Karras sigmas but the <a href="https://hf.co/Lykon/dreamshaper-xl-1-0" rel="nofollow">DreamShaperXL</a> model can since they are trained with Karras sigmas.</p></div> <p data-svelte-h="svelte-r7q280">Karras scheduler’s use the timestep schedule and sigmas from the <a href="https://hf.co/papers/2206.00364" rel="nofollow">Elucidating the Design Space of Diffusion-Based Generative Models</a> paper. This scheduler variant applies a smaller amount of noise per step as it approaches the end of the sampling process compared to other schedulers, and can increase the level of details in the generated image.</p> <p data-svelte-h="svelte-1x9rkdg">Enable Karras sigmas by setting <code>use_karras_sigmas=True</code> in the scheduler.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> StableDiffusionXLPipeline, DPMSolverMultistepScheduler | |
| pipeline = StableDiffusionXLPipeline.from_pretrained( | |
| <span class="hljs-string">"SG161222/RealVisXL_V4.0"</span>, | |
| torch_dtype=torch.float16, | |
| variant=<span class="hljs-string">"fp16"</span>, | |
| ).to(<span class="hljs-string">"cuda"</span>) | |
| pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config, algorithm_type=<span class="hljs-string">"sde-dpmsolver++"</span>, use_karras_sigmas=<span class="hljs-literal">True</span>) | |
| prompt = <span class="hljs-string">"A cinematic shot of a cute little rabbit wearing a jacket and doing a thumbs up"</span> | |
| generator = torch.Generator(device=<span class="hljs-string">"cpu"</span>).manual_seed(<span class="hljs-number">2487854446</span>) | |
| image = pipeline( | |
| prompt=prompt, | |
| negative_prompt=<span class="hljs-string">""</span>, | |
| generator=generator, | |
| ).images[<span class="hljs-number">0</span>]<!-- HTML_TAG_END --></pre></div> <div class="flex gap-4" data-svelte-h="svelte-gruv4r"><div><img class="rounded-xl" src="https://huggingface.co/datasets/stevhliu/testing-images/resolve/main/karras_sigmas_true.png"> <figcaption class="mt-2 text-center text-sm text-gray-500">Karras sigmas enabled</figcaption></div> <div><img class="rounded-xl" src="https://huggingface.co/datasets/stevhliu/testing-images/resolve/main/karras_sigmas_false.png"> <figcaption class="mt-2 text-center text-sm text-gray-500">Karras sigmas disabled</figcaption></div></div> <h2 class="relative group"><a id="rescale-noise-schedule" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#rescale-noise-schedule"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Rescale noise schedule</span></h2> <p data-svelte-h="svelte-ufxjw">In the <a href="https://hf.co/papers/2305.08891" rel="nofollow">Common Diffusion Noise Schedules and Sample Steps are Flawed</a> paper, the authors discovered that common noise schedules allowed some signal to leak into the last timestep. This signal leakage at inference can cause models to only generate images with medium brightness. By enforcing a zero signal-to-noise ratio (SNR) for the timstep schedule and sampling from the last timestep, the model can be improved to generate very bright or dark images.</p> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-1u8ajsg">For inference, you need a model that has been trained with <em>v_prediction</em>. To train your own model with <em>v_prediction</em>, add the following flag to the <a href="https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image.py" rel="nofollow">train_text_to_image.py</a> or <a href="https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py" rel="nofollow">train_text_to_image_lora.py</a> scripts.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->--prediction_type=<span class="hljs-string">"v_prediction"</span><!-- HTML_TAG_END --></pre></div></div> <p data-svelte-h="svelte-v3q1b3">For example, load the <a href="https://hf.co/ptx0/pseudo-journey-v2" rel="nofollow">ptx0/pseudo-journey-v2</a> checkpoint which was trained with <code>v_prediction</code> and the <a href="/docs/diffusers/pr_11686/en/api/schedulers/ddim#diffusers.DDIMScheduler">DDIMScheduler</a>. Configure the following parameters in the <a href="/docs/diffusers/pr_11686/en/api/schedulers/ddim#diffusers.DDIMScheduler">DDIMScheduler</a>:</p> <ul data-svelte-h="svelte-1dozvyr"><li><code>rescale_betas_zero_snr=True</code> to rescale the noise schedule to zero SNR</li> <li><code>timestep_spacing="trailing"</code> to start sampling from the last timestep</li></ul> <p data-svelte-h="svelte-158nvmg">Set <code>guidance_rescale</code> in the pipeline to prevent over-exposure. A lower value increases brightness but some of the details may appear washed out.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> DiffusionPipeline, DDIMScheduler | |
| pipeline = DiffusionPipeline.from_pretrained(<span class="hljs-string">"ptx0/pseudo-journey-v2"</span>, use_safetensors=<span class="hljs-literal">True</span>) | |
| pipeline.scheduler = DDIMScheduler.from_config( | |
| pipeline.scheduler.config, rescale_betas_zero_snr=<span class="hljs-literal">True</span>, timestep_spacing=<span class="hljs-string">"trailing"</span> | |
| ) | |
| pipeline.to(<span class="hljs-string">"cuda"</span>) | |
| prompt = <span class="hljs-string">"cinematic photo of a snowy mountain at night with the northern lights aurora borealis overhead, 35mm photograph, film, professional, 4k, highly detailed"</span> | |
| generator = torch.Generator(device=<span class="hljs-string">"cpu"</span>).manual_seed(<span class="hljs-number">23</span>) | |
| image = pipeline(prompt, guidance_rescale=<span class="hljs-number">0.7</span>, generator=generator).images[<span class="hljs-number">0</span>] | |
| image<!-- HTML_TAG_END --></pre></div> <div class="flex gap-4" data-svelte-h="svelte-2pxhst"><div><img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/no-zero-snr.png"> <figcaption class="mt-2 text-center text-sm text-gray-500">default Stable Diffusion v2-1 image</figcaption></div> <div><img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/zero-snr.png"> <figcaption class="mt-2 text-center text-sm text-gray-500">image with zero SNR and trailing timestep spacing enabled</figcaption></div></div> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/diffusers/blob/main/docs/source/en/using-diffusers/scheduler_features.md" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_o82a48 = { | |
| assets: "/docs/diffusers/pr_11686/en", | |
| base: "/docs/diffusers/pr_11686/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/diffusers/pr_11686/en/_app/immutable/entry/start.2b9667fb.js"), | |
| import("/docs/diffusers/pr_11686/en/_app/immutable/entry/app.a2a6117e.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 304], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 37.4 kB
- Xet hash:
- eb7089ab988cbd8d46bc482444692c8f6eb4b26e100357d415cd58305ccabfa1
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.