import torch
from diffusers import DiffusionPipeline
from diffusers.utils import load_image
# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("lavinal712/controlnet-segmentation-sam-llava", dtype=torch.bfloat16, device_map="cuda")
prompt = "Turn this cat into a dog"
input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png")
image = pipe(image=input_image, prompt=prompt).images[0]Model Card for lavinal712/controlnet-segmentation-sam-llava
Model Description
This is controlnet weight trained on runwayml/stable-diffusion-v1-5 with segmentaion.
Training
This model was trained using a Segmented dataset based on the SAM-LLaVA-Captions10M Dataset. Stable Diffusion v1.5 checkpoint was used as the base model for the controlnet.
Training Method
- Train on SAM-LLAVA-55k for 60000 steps with batch size of 4.
- Downloads last month
- 4