| | --- |
| | license: creativeml-openrail-m |
| | library_name: diffusers |
| | tags: |
| | - stable-diffusion |
| | - stable-diffusion-diffusers |
| | - text-to-image |
| | - diffusers |
| | - diffusers-training |
| | - lora |
| | - stable-diffusion |
| | - stable-diffusion-diffusers |
| | - text-to-image |
| | - diffusers |
| | - diffusers-training |
| | - lora |
| | base_model: stabilityai/stable-diffusion-2-1 |
| | inference: true |
| | --- |
| | |
| |
|
| |
|
| | # LoRA text2image fine-tuning - remi349/sd_trained_3D_lora |
| | |
| | These are LoRA adaption weights are for stabilityai/stable-diffusion-2-1. The weights were fine-tuned on the remi349/finetuning_dataset_for_3D_training dataset thanks to the library [diffusers](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py). |
| | |
| | ## Intended uses & limitations |
| | |
| | This model aims at generating images of isolated objects, compatible with 2D_to_3D models like [Triposr](https://github.com/VAST-AI-Research/TripoSR) or [CRM](https://huggingface.co/Zhengyi/CRM). |
| | It was finetuned in order to create after a pipeline of prompt-to-3D model. |
| | |
| | #### How to use |
| | |
| | ```python |
| | # First load the basic architecture and everything |
| | import torch |
| | from diffusers import StableDiffusionPipeline |
| | pipe = StableDiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-1", torch_dtype=torch.float16) |
| | |
| | # Then add the lora weights to the model stable diffusion 2 |
| | pipe.unet.load_attn_procs('ACROSS-Lab/PromptTo3D_sd_finetuned') |
| | pipe.to("cuda") |
| | |
| | # Then you can begin the inference process on a prompt and save the image generated |
| | prompt = 'a rabbit with a yellow jacket' |
| | image = pipe(prompt, num_inference_steps=30, guidance_scale=7.5).images[0] |
| | image.save("my_image.png") |
| | ``` |
| | |
| | #### Limitations and bias |
| | |
| | This model is a first try some hyperparameters tuning should be done, but for that we would need a solid automated benchmark. |
| | |
| | ## Training details |
| | The model finetuned model is [Stable Diffusion 2](https://huggingface.co/stabilityai/stable-diffusion-2). |
| | The data used to train this model is the dataset available on uggingface at 'remi349/finetuning_dataset_for_3D_training'. |
| | you can download it thanks to the command |
| | ```python |
| | from datasets import load_dataset |
| | dataset = load_dataset("ACROSS-Lab/PromptTo3D_sd_dataset", split = 'train') |
| | ``` |
| | |
| | This dataset is a subset of the dataset [Objaverse](https://objaverse.allenai.org/). |
| | |
| | ## Collaboration |
| | This model and dataset has been made in collaboration by [Josué ADOSSEHOUN](https://huggingface.co/josh007) and [Rémi DUCOTTET](https://huggingface.co/remi349) |
