Update README.md
Browse files
README.md
CHANGED
|
@@ -1,9 +1,6 @@
|
|
| 1 |
# Brain-Diffuser
|
| 2 |
Official repository for the paper ["**Brain-Diffuser: Natural scene reconstruction from fMRI signals using generative latent diffusion**"](https://arxiv.org/abs/2303.05334) by Furkan Ozcelik and Rufin VanRullen.
|
| 3 |
|
| 4 |
-
## Results
|
| 5 |
-
The following are a few reconstructions obtained :
|
| 6 |
-
<p align="center"><img src="./figures/Reconstructions.png" width="600" ></p>
|
| 7 |
|
| 8 |
## Instructions
|
| 9 |
|
|
@@ -43,27 +40,12 @@ wget https://openaipublic.blob.core.windows.net/very-deep-vaes-assets/vdvae-asse
|
|
| 43 |
### Second Stage Reconstruction with Versatile Diffusion
|
| 44 |
|
| 45 |
1. Download pretrained Versatile Diffusion model "vd-four-flow-v1-0-fp16-deprecated.pth", "kl-f8.pth" and "optimus-vae.pth" from [HuggingFace](https://huggingface.co/shi-labs/versatile-diffusion/tree/main/pretrained_pth) and put them in `versatile_diffusion/pretrained/` folder
|
| 46 |
-
2. Extract CLIP-Text features of captions for any subject 'x' using `python scripts/cliptext_extract_features.py -sub x`
|
| 47 |
3. Extract CLIP-Vision features of stimuli images for any subject 'x' using `python scripts/clipvision_extract_features.py -sub x`
|
| 48 |
-
4. Train regression models from fMRI to CLIP-Text features and save test predictions using `python scripts/cliptext_regression.py -sub x`
|
| 49 |
5. Train regression models from fMRI to CLIP-Vision features and save test predictions using `python scripts/clipvision_regression.py -sub x`
|
| 50 |
6. Reconstruct images from predicted test features using `python scripts/versatilediffusion_reconstruct_images.py -sub x` . This code is written as you are using two 12GB GPUs but you may edit according to your setup.
|
| 51 |
|
| 52 |
-
|
| 53 |
-
### Quantitative Evaluation
|
| 54 |
-
Although results are expected to be similar, it may vary because of variations at reconstruction
|
| 55 |
-
1. Save test images to directory `python scripts/save_test_images.py`
|
| 56 |
-
2. Extract evaluation features for test images using `python scripts/eval_extract_features.py -sub 0`
|
| 57 |
-
3. Extract evaluation features for reconstructed images of any subject using `python scripts/eval_extract_features.py -sub x`
|
| 58 |
-
4. Obtain quantitative metric results for each subject using`python scripts/evaluate_reconstruction.py -sub x`
|
| 59 |
-
|
| 60 |
-
### ROI Analysis
|
| 61 |
-
It has a bug that prevents to get the exact results but provides an approximation for most of ROIs, hopefully will be fixed soon.
|
| 62 |
-
1. Extract ROI fMRI activations for any subject 'x' using `python scripts/roi_extract.py -sub x`
|
| 63 |
-
2. Generate VDVAE, CLIP-Text, CLIP-Vision features forom synthetic fMRI using `python scripts/roi_generate_features.py -sub x`
|
| 64 |
-
3. Generate VDVAE reconstructions for ROIs using `python scripts/roi_vdvae_reconstruct.py -sub x`
|
| 65 |
-
4. Generate Versatile Diffusion reconstructions for ROIs using `python scripts/roi_versatilediffusion_reconstruct.py -sub x`
|
| 66 |
-
|
| 67 |
## References
|
| 68 |
- Codes in vdvae directory are derived from [openai/vdvae](https://github.com/openai/vdvae)
|
| 69 |
- Codes in versatile_diffusion directory are derived from earlier version of [SHI-Labs/Versatile-Diffusion](https://github.com/SHI-Labs/Versatile-Diffusion)
|
|
|
|
| 1 |
# Brain-Diffuser
|
| 2 |
Official repository for the paper ["**Brain-Diffuser: Natural scene reconstruction from fMRI signals using generative latent diffusion**"](https://arxiv.org/abs/2303.05334) by Furkan Ozcelik and Rufin VanRullen.
|
| 3 |
|
|
|
|
|
|
|
|
|
|
| 4 |
|
| 5 |
## Instructions
|
| 6 |
|
|
|
|
| 40 |
### Second Stage Reconstruction with Versatile Diffusion
|
| 41 |
|
| 42 |
1. Download pretrained Versatile Diffusion model "vd-four-flow-v1-0-fp16-deprecated.pth", "kl-f8.pth" and "optimus-vae.pth" from [HuggingFace](https://huggingface.co/shi-labs/versatile-diffusion/tree/main/pretrained_pth) and put them in `versatile_diffusion/pretrained/` folder
|
| 43 |
+
<!-- 2. Extract CLIP-Text features of captions for any subject 'x' using `python scripts/cliptext_extract_features.py -sub x` -->
|
| 44 |
3. Extract CLIP-Vision features of stimuli images for any subject 'x' using `python scripts/clipvision_extract_features.py -sub x`
|
| 45 |
+
<!-- 4. Train regression models from fMRI to CLIP-Text features and save test predictions using `python scripts/cliptext_regression.py -sub x` --> -->
|
| 46 |
5. Train regression models from fMRI to CLIP-Vision features and save test predictions using `python scripts/clipvision_regression.py -sub x`
|
| 47 |
6. Reconstruct images from predicted test features using `python scripts/versatilediffusion_reconstruct_images.py -sub x` . This code is written as you are using two 12GB GPUs but you may edit according to your setup.
|
| 48 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
## References
|
| 50 |
- Codes in vdvae directory are derived from [openai/vdvae](https://github.com/openai/vdvae)
|
| 51 |
- Codes in versatile_diffusion directory are derived from earlier version of [SHI-Labs/Versatile-Diffusion](https://github.com/SHI-Labs/Versatile-Diffusion)
|