dineshsai07 commited on
Commit
e0dd259
·
verified ·
1 Parent(s): d7e0ac0

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -20
README.md CHANGED
@@ -1,9 +1,6 @@
1
  # Brain-Diffuser
2
  Official repository for the paper ["**Brain-Diffuser: Natural scene reconstruction from fMRI signals using generative latent diffusion**"](https://arxiv.org/abs/2303.05334) by Furkan Ozcelik and Rufin VanRullen.
3
 
4
- ## Results
5
- The following are a few reconstructions obtained :
6
- <p align="center"><img src="./figures/Reconstructions.png" width="600" ></p>
7
 
8
  ## Instructions
9
 
@@ -43,27 +40,12 @@ wget https://openaipublic.blob.core.windows.net/very-deep-vaes-assets/vdvae-asse
43
  ### Second Stage Reconstruction with Versatile Diffusion
44
 
45
  1. Download pretrained Versatile Diffusion model "vd-four-flow-v1-0-fp16-deprecated.pth", "kl-f8.pth" and "optimus-vae.pth" from [HuggingFace](https://huggingface.co/shi-labs/versatile-diffusion/tree/main/pretrained_pth) and put them in `versatile_diffusion/pretrained/` folder
46
- 2. Extract CLIP-Text features of captions for any subject 'x' using `python scripts/cliptext_extract_features.py -sub x`
47
  3. Extract CLIP-Vision features of stimuli images for any subject 'x' using `python scripts/clipvision_extract_features.py -sub x`
48
- 4. Train regression models from fMRI to CLIP-Text features and save test predictions using `python scripts/cliptext_regression.py -sub x`
49
  5. Train regression models from fMRI to CLIP-Vision features and save test predictions using `python scripts/clipvision_regression.py -sub x`
50
  6. Reconstruct images from predicted test features using `python scripts/versatilediffusion_reconstruct_images.py -sub x` . This code is written as you are using two 12GB GPUs but you may edit according to your setup.
51
 
52
-
53
- ### Quantitative Evaluation
54
- Although results are expected to be similar, it may vary because of variations at reconstruction
55
- 1. Save test images to directory `python scripts/save_test_images.py`
56
- 2. Extract evaluation features for test images using `python scripts/eval_extract_features.py -sub 0`
57
- 3. Extract evaluation features for reconstructed images of any subject using `python scripts/eval_extract_features.py -sub x`
58
- 4. Obtain quantitative metric results for each subject using`python scripts/evaluate_reconstruction.py -sub x`
59
-
60
- ### ROI Analysis
61
- It has a bug that prevents to get the exact results but provides an approximation for most of ROIs, hopefully will be fixed soon.
62
- 1. Extract ROI fMRI activations for any subject 'x' using `python scripts/roi_extract.py -sub x`
63
- 2. Generate VDVAE, CLIP-Text, CLIP-Vision features forom synthetic fMRI using `python scripts/roi_generate_features.py -sub x`
64
- 3. Generate VDVAE reconstructions for ROIs using `python scripts/roi_vdvae_reconstruct.py -sub x`
65
- 4. Generate Versatile Diffusion reconstructions for ROIs using `python scripts/roi_versatilediffusion_reconstruct.py -sub x`
66
-
67
  ## References
68
  - Codes in vdvae directory are derived from [openai/vdvae](https://github.com/openai/vdvae)
69
  - Codes in versatile_diffusion directory are derived from earlier version of [SHI-Labs/Versatile-Diffusion](https://github.com/SHI-Labs/Versatile-Diffusion)
 
1
  # Brain-Diffuser
2
  Official repository for the paper ["**Brain-Diffuser: Natural scene reconstruction from fMRI signals using generative latent diffusion**"](https://arxiv.org/abs/2303.05334) by Furkan Ozcelik and Rufin VanRullen.
3
 
 
 
 
4
 
5
  ## Instructions
6
 
 
40
  ### Second Stage Reconstruction with Versatile Diffusion
41
 
42
  1. Download pretrained Versatile Diffusion model "vd-four-flow-v1-0-fp16-deprecated.pth", "kl-f8.pth" and "optimus-vae.pth" from [HuggingFace](https://huggingface.co/shi-labs/versatile-diffusion/tree/main/pretrained_pth) and put them in `versatile_diffusion/pretrained/` folder
43
+ <!-- 2. Extract CLIP-Text features of captions for any subject 'x' using `python scripts/cliptext_extract_features.py -sub x` -->
44
  3. Extract CLIP-Vision features of stimuli images for any subject 'x' using `python scripts/clipvision_extract_features.py -sub x`
45
+ <!-- 4. Train regression models from fMRI to CLIP-Text features and save test predictions using `python scripts/cliptext_regression.py -sub x` --> -->
46
  5. Train regression models from fMRI to CLIP-Vision features and save test predictions using `python scripts/clipvision_regression.py -sub x`
47
  6. Reconstruct images from predicted test features using `python scripts/versatilediffusion_reconstruct_images.py -sub x` . This code is written as you are using two 12GB GPUs but you may edit according to your setup.
48
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49
  ## References
50
  - Codes in vdvae directory are derived from [openai/vdvae](https://github.com/openai/vdvae)
51
  - Codes in versatile_diffusion directory are derived from earlier version of [SHI-Labs/Versatile-Diffusion](https://github.com/SHI-Labs/Versatile-Diffusion)