README.md

metadata

library_name: diffusers
tags:
  - modular-diffusers
  - diffusers
  - qwenimage-layered
  - text-to-image

This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.

Pipeline Type: QwenImageLayeredAutoBlocks

Description: Auto Modular pipeline for layered denoising tasks using QwenImage-Layered.

This pipeline uses a 4-block architecture that can be customized and extended.

Example Usage

[TODO]

Pipeline Architecture

This modular pipeline is composed of the following blocks:

1. text_encoder (QwenImageLayeredTextEncoderStep)
   • QwenImage-Layered Text encoder step that encode the text prompt, will generate a prompt based on image if not provided.
   • resize: QwenImageLayeredResizeStep
     • Image Resize step that resize the image to a target area (defined by the resolution parameter from user) while maintaining the aspect ratio.
   • get_image_prompt: QwenImageLayeredGetImagePromptStep
     • Auto-caption step that generates a text prompt from the input image if none is provided.
   • encode: QwenImageTextEncoderStep
     • Text Encoder step that generates text embeddings to guide the image generation.
2. vae_encoder (QwenImageLayeredVaeEncoderStep)
   • Vae encoder step that encode the image inputs into their latent representations.
   • resize: QwenImageLayeredResizeStep
     • Image Resize step that resize the image to a target area (defined by the resolution parameter from user) while maintaining the aspect ratio.
   • preprocess: QwenImageEditProcessImagesInputStep
     • Image Preprocess step. Images needs to be resized first.
   • encode: QwenImageVaeEncoderStep
     • VAE Encoder step that converts processed_image into latent representations image_latents.
   • permute: QwenImageLayeredPermuteLatentsStep
     • Permute image latents from (B, C, 1, H, W) to (B, 1, C, H, W) for Layered packing.
3. denoise (QwenImageLayeredCoreDenoiseStep)
   • Core denoising workflow for QwenImage-Layered img2img task.
   • input: QwenImageLayeredInputStep
     • Input step that prepares the inputs for the layered denoising step. It:
   • prepare_latents: QwenImageLayeredPrepareLatentsStep
     • Prepare initial random noise (B, layers+1, C, H, W) for the generation process
   • set_timesteps: QwenImageLayeredSetTimestepsStep
     • Set timesteps step for QwenImage Layered with custom mu calculation based on image_latents.
   • prepare_rope_inputs: QwenImageLayeredRoPEInputsStep
     • Step that prepares the RoPE inputs for the denoising process. Should be place after prepare_latents step
   • denoise: QwenImageLayeredDenoiseStep
     • Denoise step that iteratively denoise the latents.
   • after_denoise: QwenImageLayeredAfterDenoiseStep
     • Unpack latents from (B, seq, C*4) to (B, C, layers+1, H, W) after denoising.
4. decode (QwenImageLayeredDecoderStep)
   • Decode unpacked latents (B, C, layers+1, H, W) into layer images.

Model Components

1. image_resize_processor (VaeImageProcessor)
2. text_encoder (Qwen2_5_VLForConditionalGeneration)
3. processor (Qwen2VLProcessor)
4. tokenizer (Qwen2Tokenizer): The tokenizer to use
5. guider (ClassifierFreeGuidance)
6. image_processor (VaeImageProcessor)
7. vae (AutoencoderKLQwenImage)
8. pachifier (QwenImageLayeredPachifier)
9. scheduler (FlowMatchEulerDiscreteScheduler)
10. transformer (QwenImageTransformer2DModel) ## Input/Output Specification

Inputs:

• image (Image | list): Reference image(s) for denoising. Can be a single image or list of images.
• resolution (int, optional, defaults to 640): The target area to resize the image to, can be 1024 or 640
• prompt (str, optional): The prompt or prompts to guide image generation.
• use_en_prompt (bool, optional, defaults to False): Whether to use English prompt template
• negative_prompt (str, optional): The prompt or prompts not to guide the image generation.
• max_sequence_length (int, optional, defaults to 1024): Maximum sequence length for prompt encoding.
• generator (Generator, optional): Torch generator for deterministic generation.
• num_images_per_prompt (int, optional, defaults to 1): The number of images to generate per prompt.
• latents (Tensor, optional): Pre-generated noisy latents for image generation.
• layers (int, optional, defaults to 4): Number of layers to extract from the image
• num_inference_steps (int, optional, defaults to 50): The number of denoising steps.
• sigmas (list, optional): Custom sigmas for the denoising process.
• attention_kwargs (dict, optional): Additional kwargs for attention processors.
• **denoiser_input_fields (None, optional): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc.
• output_type (str, optional, defaults to pil): Output format: 'pil', 'np', 'pt'.

Outputs:

• images (list): Generated images.