README.md

---
datasets:
- DaveLoay/NSynth_Bass_Captions
language:
- en
---
# Riffsuion Fine-Tune

This is a Fine-Tuned version of **Rifussion**, trained on **bass** samples extracted from the **NSynth** dataset.
The porpuse of this work is to evaluate the performance of the model to generate bass audio samples.

## Notes

* This is the way I found to achieve this goal, if you have a better idea for doing this, please share it with me.

## Quickstart Guide

Clone the **Riffusion** repository and install the requirements.txt file from: [Riffusion Github](https://github.com/riffusion/riffusion) 
```python
import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained("DaveLoay/Riffusion_FT_Bass_512_4000", torch_dtype=torch.float16).to(device)
prompt = "Your desired prompt"
image = pipe(prompt).images[0]
```

After that, you would have been generated an spectrogram saved on image. So if you want to convert this image into an audio file, you could use the **spectrogram_image_converter** mehtod contained in the **Rifussion** repo.

```python
from riffusion.spectrogram_image_converter import SpectrogramImageConverter
from riffusion.spectrogram_params import SpectrogramParams

params = SpectrogramParams()
converter = SpectrogramImageConverter(params)
audio = converter.audio_from_spectrogram_image(image)
```

## Fine Tuning

For the Fine-Tuning process, I used the bass samples from the test split in the NSynth dataset, which you can check out here: [NSynth Dataset](https://magenta.tensorflow.org/nsynth) 

You can find the pre-processed files in my repo, here: [DaveLoay/NSynth_Bass_Captions](DaveLoay/NSynth_Bass_Captions)

And as mention in the official **Rifussion** HF repo, I used the **train_text_to_image** script contained in the **Diffusers** repo, which you can check out here: [Diffusers Repo](https://github.com/huggingface/diffusers/tree/main/examples/text_to_image)

After configuring all dependencies, I used the following code to train the model:

```bash
  accelerate launch --mixed_precision="fp16"  train_text_to_image.py \
    --pretrained_model_name_or_path=riffusion/riffusion-model-v1 \
    --dataset_name=DaveLoay/NSynth_Bass_Captions \
    --resolution=512 \
    --use_ema \
    --train_batch_size=3 \
    --gradient_accumulation_steps=4 \
    --gradient_checkpointing \
    --max_train_steps=4000 \
    --learning_rate=1e-05 \
    --max_grad_norm=1 \
    --lr_scheduler="constant" --lr_warmup_steps=0 \
    --output_dir="Riffusion_FT_Bass_512_4000" \
    --push_to_hub
```

## Hardware

The hardware I used to fine-tune this model is:

* NVIDIA A100 40 GB vRAM hosted in Google Colab Pro

It took about 3 hours to complete the training process, and used about ~26 GB of vRAM.

## Credits

You can check the original repositories here:

[Riffusion](https://www.riffusion.com/)

[NSynth Dataset](https://magenta.tensorflow.org/nsynth)

[Diffusers](https://huggingface.co/docs/diffusers/index)