V-Express: Conditional Dropout for Progressive Training of Portrait Video Generation
Paper • 2406.02511 • Published • 11
import torch
from diffusers import DiffusionPipeline
# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("tk93/V-Express", dtype=torch.bfloat16, device_map="cuda")
prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt).images[0]V-Express Model Card
Project Page | Paper | Code
Introduction
Models
Audio Encoder
- model_ckpts/wav2vec2-base-960h. (It is also available from the original model card facebook/wav2vec2-base-960h)
Face Analysis
- model_ckpts/insightface_models/models/buffalo_l. (It is also available from the original repository insightface/buffalo_l)
V-Express
- model_ckpts/sd-vae-ft-mse. VAE encoder. (original model card stabilityai/sd-vae-ft-mse)
- model_ckpts/stable-diffusion-v1-5. Only the model configuration file for unet is needed here. (original model card runwayml/stable-diffusion-v1-5)
- model_ckpts/v-express. The video generation model conditional on audio and V-kps we call V-Express.
- You should download and put all
.binmodel tomodel_ckpts/v-expressdirectory, which includesaudio_projection.bin,denoising_unet.bin,motion_module.bin,reference_net.bin, andv_kps_guider.bin.
licence
see acknowledgements for more information.
- Downloads last month
- 707