Upload 7 files
Browse files- README.md +211 -0
- base_model_conditioned_orientation_epoch100.pth +3 -0
- base_model_vca_IAPS_quadrant.pth +3 -0
- config.json +118 -0
- inference_example.py +152 -0
- vca_IAPS_batch10_lr2e-4_epoch23.pth +3 -0
- vca_ckvideo_batch128_lr2e-5_epoch20.pth +3 -0
README.md
CHANGED
|
@@ -1,3 +1,214 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
tags:
|
| 5 |
+
- computer-vision
|
| 6 |
+
- affective-computing
|
| 7 |
+
- neuroscience
|
| 8 |
+
- image-regression
|
| 9 |
+
- vgg16
|
| 10 |
+
- pavlovian-conditioning
|
| 11 |
+
- valence-prediction
|
| 12 |
---
|
| 13 |
+
|
| 14 |
+
# Visual-Valence Model (VCA)
|
| 15 |
+
|
| 16 |
+
> A deep neural network model of associative emotional (Pavlovian fear) learning.
|
| 17 |
+
|
| 18 |
+
Companion model repository for **["Associative Emotional Learning in Convolutional Neural Networks"](https://arxiv.org/abs/2607.19327)**
|
| 19 |
+
(Leem, Keil, Ding & Fang; *Neural Computation*, in press).
|
| 20 |
+
|
| 21 |
+
- 📄 Paper: https://arxiv.org/abs/2607.19327
|
| 22 |
+
- 💻 Code: https://github.com/lab-smile/FearConditioningAI
|
| 23 |
+
|
| 24 |
+
> **Note:** This model is a research artifact for computational/cognitive neuroscience, released to
|
| 25 |
+
> reproduce and extend the paper's findings. It is **not** a general-purpose sentiment/emotion API and
|
| 26 |
+
> is not validated for clinical, diagnostic, or production affective-computing use.
|
| 27 |
+
|
| 28 |
+
---
|
| 29 |
+
|
| 30 |
+
## Model description
|
| 31 |
+
|
| 32 |
+
The Visual-Valence Model predicts the affective **valence** of a visual scene (1 = extreme displeasure,
|
| 33 |
+
9 = extreme pleasure) and reproduces hallmarks of Pavlovian (fear) conditioning when a novel, initially
|
| 34 |
+
neutral stimulus is repeatedly paired with an emotionally-charged one.
|
| 35 |
+
|
| 36 |
+
The architecture (`Visual_Cortex_Amygdala` in [`models/VGG_Model.py`](https://github.com/lab-smile/FearConditioningAI/blob/main/models/VGG_Model.py))
|
| 37 |
+
has three components, loosely modeling the primate visual/affective pathway:
|
| 38 |
+
|
| 39 |
+
| Component | Role | Implementation |
|
| 40 |
+
|---|---|---|
|
| 41 |
+
| **Visual Cortex module** ("High Road") | Ventral visual stream | VGG-16 (Simonyan & Zisserman, 2015), ImageNet-pretrained, **frozen** |
|
| 42 |
+
| **Shortcut Pathway** ("Middle Road") | Fast, low-resolution route from early vision to affect circuitry | Multi-scale pooling of VGG-16's early layers (layer index 10), combined via an Efficient Channel Attention (ECA) module (Wang et al., 2020) |
|
| 43 |
+
| **Valence Module** | Combines both pathways into a scalar valence judgment | Two fully-connected layers (amygdala LA/CE-nuclei- and OFC-inspired) + one sigmoid output unit |
|
| 44 |
+
|
| 45 |
+
The High Road output (4096-d) and Middle Road output (512-d) are concatenated (4608-d) and passed through
|
| 46 |
+
the Valence Module's fully connected layers to a single sigmoid unit, which is linearly rescaled from
|
| 47 |
+
`[0, 1]` to the `[1, 9]` IAPS valence scale at evaluation time.
|
| 48 |
+
|
| 49 |
+
### Five checkpoints, one training pipeline
|
| 50 |
+
|
| 51 |
+
This repository hosts every checkpoint along the training pipeline described in the
|
| 52 |
+
[GitHub README](https://github.com/lab-smile/FearConditioningAI#training) — same architecture
|
| 53 |
+
(`Visual_Cortex_Amygdala`) throughout, so any of them can be loaded with the same code:
|
| 54 |
+
|
| 55 |
+
| File | Stage | Description | Val. R | Val. MSE |
|
| 56 |
+
|---|---|---|---|---|
|
| 57 |
+
| `vca_ckvideo_batch128_lr2e-5_epoch20.pth` | 0 | Trained from scratch on the Cowen & Keltner Videoframe dataset | 0.386 | 0.043 |
|
| 58 |
+
| `vca_IAPS_batch10_lr2e-4_epoch23.pth` | 1 | Fine-tuned on full-size IAPS images | 0.538 | 0.192 |
|
| 59 |
+
| `base_model_vca_IAPS_quadrant.pth` | 2 | Fine-tuned to the quadrant-cropped input layout — **pre-conditioning**: has learned to decode valence from natural scenes (the US) but has never seen the conditioned stimulus (CS, a Gabor patch) | see paper | see paper |
|
| 60 |
+
| `base_model_conditioned_orientation_epoch1.pth` | 3, epoch 1 | First epoch of Pavlovian conditioning (CS+ Gabor patch paired with pleasant/unpleasant IAPS US); early/under-trained, kept for provenance | 0.660 | 0.466 |
|
| 61 |
+
| `base_model_conditioned_orientation_epoch100.pth` | 3, epoch 100 | **Post-conditioning (final)** — used throughout the paper's conditioning/generalization/representational-alignment analyses | see paper | see paper |
|
| 62 |
+
|
| 63 |
+
Stage 0/1 metrics are Pearson R / MSE on their respective held-out validation sets (read from each
|
| 64 |
+
checkpoint's own logged `best_per` / `best_loss`). Stage 2 and the final Stage 3 (epoch 100) numbers are
|
| 65 |
+
intentionally not restated here — see the paper for the definitive, fully-trained results.
|
| 66 |
+
|
| 67 |
+
Comparing the pre- (Stage 2) and post-conditioning (Stage 3, epoch 100) checkpoints' responses to the CS
|
| 68 |
+
alone is what reveals the learned CS→valence association (and, at the representation level, the
|
| 69 |
+
increasing CS/US alignment reported in the paper).
|
| 70 |
+
|
| 71 |
+
---
|
| 72 |
+
|
| 73 |
+
## Intended use
|
| 74 |
+
|
| 75 |
+
- Reproducing the paper's Pavlovian-conditioning experiments and figures.
|
| 76 |
+
- Extending the model to new conditioning paradigms, stimulus sets, or ablations (e.g. the
|
| 77 |
+
attention-free variant, `Visual_Cortex_Amygdala_wo_Attention`) for **research purposes**.
|
| 78 |
+
- Studying how a two-pathway (ventral-stream + shortcut) architecture with attention reproduces
|
| 79 |
+
behavioral/representational signatures of associative emotional learning.
|
| 80 |
+
- Feature extraction / representational analysis (see `Channel_Activity_Extraction.py`,
|
| 81 |
+
`Manifold_Visualization.py`, `SVM_Analysis_*.py` in the GitHub repo) for downstream neuroscience analyses.
|
| 82 |
+
|
| 83 |
+
**Out of scope:** general-purpose sentiment analysis, affect recognition on non-IAPS-like natural images,
|
| 84 |
+
clinical/diagnostic use, or any decision-making about real individuals' emotional states.
|
| 85 |
+
|
| 86 |
+
---
|
| 87 |
+
|
| 88 |
+
## Limitations
|
| 89 |
+
|
| 90 |
+
- **Frozen ImageNet backbone.** The High Road (VGG-16) is never fine-tuned, so the model inherits
|
| 91 |
+
ImageNet's visual biases; only the shortcut pathway and valence module are trained on affective data.
|
| 92 |
+
- **Narrow, licensed training data.** IAPS is a relatively small, curated stimulus set under a
|
| 93 |
+
data-use/confidentiality agreement (not redistributed with the code or this checkpoint); the model's
|
| 94 |
+
generalization to naturalistic, in-the-wild imagery is untested.
|
| 95 |
+
- **Two-alternative conditioning paradigm.** The conditioning stage pairs exactly two Gabor orientations
|
| 96 |
+
(45°, 135°) with unpleasant/pleasant IAPS images; behavior on other CS types has not been evaluated.
|
| 97 |
+
- **Not validated against individual human subjects.** Comparisons to human data in the paper are at the
|
| 98 |
+
group/aggregate level; the model is not a predictive model of any specific person's affective response.
|
| 99 |
+
- **Single scalar output.** The model predicts only valence (pleasant ↔ unpleasant), not arousal or
|
| 100 |
+
discrete emotion categories.
|
| 101 |
+
|
| 102 |
+
---
|
| 103 |
+
|
| 104 |
+
## Evaluation metrics
|
| 105 |
+
|
| 106 |
+
Model quality is reported using:
|
| 107 |
+
|
| 108 |
+
- **Pearson correlation (R / R²)** between predicted and ground-truth (SAM-rated) valence, computed by
|
| 109 |
+
`reg_eval_model` / `cond_eval_model` in [`utils.py`](https://github.com/lab-smile/FearConditioningAI/blob/main/utils.py).
|
| 110 |
+
- **Mean-squared error (MSE)** between predicted and ground-truth valence (rescaled to the 1–9 scale).
|
| 111 |
+
- Post-conditioning, generalization and CS/US **representational alignment** (single-unit tuning and
|
| 112 |
+
population-level RSA/t-SNE, via `Channel_Activity_Extraction.py`, `Manifold_Visualization.py`,
|
| 113 |
+
`SVM_Analysis_Emotion.py`, `SVM_Analysis_Before_After.py`) are used to assess whether conditioning
|
| 114 |
+
reproduces human associative-learning signatures.
|
| 115 |
+
|
| 116 |
+
The full quantitative results (per-stage R/MSE, generalization curves, and alignment statistics) are
|
| 117 |
+
reported in the paper's tables/figures — see https://arxiv.org/abs/2607.19327. The Stage 0/1 R and MSE
|
| 118 |
+
values in the checkpoint table above are real, logged validation numbers for those checkpoints; the
|
| 119 |
+
Stage 3/epoch-1 numbers are a very early, under-trained snapshot and are **not** representative of the
|
| 120 |
+
final, fully-conditioned (epoch 100) model — included only for provenance, not as a reported result.
|
| 121 |
+
|
| 122 |
+
---
|
| 123 |
+
|
| 124 |
+
## Training dataset
|
| 125 |
+
|
| 126 |
+
Training proceeds in stages, each building on the previous stage's checkpoint (see the GitHub README's
|
| 127 |
+
[Training](https://github.com/lab-smile/FearConditioningAI#training) section for exact commands):
|
| 128 |
+
|
| 129 |
+
| Stage | Dataset | Purpose |
|
| 130 |
+
|---|---|---|
|
| 131 |
+
| 0 | Cowen & Keltner (2017) Videoframe dataset (2,185 emotion-eliciting video clips, one frame sampled every 10th frame) | Pretrain valence regression from scratch on natural scenes |
|
| 132 |
+
| 1 | International Affective Picture System (IAPS; Bradley & Lang, 1994/2007), full-size images, 8:1:1 train/val/test split | Adapt to the US stimuli used in conditioning |
|
| 133 |
+
| 2 | IAPS, quadrant-cropped layout | Fine-tune to the spatial layout used during conditioning |
|
| 134 |
+
| 3 | IAPS (US, in the 4th quadrant) × Gabor patches (CS, in the 2nd quadrant; 45°/135° orientation, varying spatial frequency/contrast, generated via `Gabor4Seowung.m`) | Pavlovian conditioning: pair CS+ with pleasant/unpleasant US |
|
| 135 |
+
|
| 136 |
+
Labels are human valence ratings on a 1–9 scale (Self-Assessment Manikin; Bradley & Lang, 1994).
|
| 137 |
+
**The IAPS images themselves are not redistributed** with the code or this model repository due to a
|
| 138 |
+
data-use/confidentiality agreement — obtain access to IAPS independently to reproduce training from
|
| 139 |
+
scratch. The Gabor-patch CS stimuli are procedurally generated and have no such restriction.
|
| 140 |
+
|
| 141 |
+
Note the input layout differs by stage: Stage 0/1 checkpoints expect a full-frame natural image
|
| 142 |
+
(resize + normalize only), while Stage 2/3 checkpoints expect the quadrant-cropped layout (see
|
| 143 |
+
[Data preprocessing](https://github.com/lab-smile/FearConditioningAI#data-preprocessing) in the GitHub
|
| 144 |
+
README) — feeding a full-frame image to a Stage 2/3 checkpoint (or vice versa) will not reproduce the
|
| 145 |
+
reported behavior.
|
| 146 |
+
|
| 147 |
+
---
|
| 148 |
+
|
| 149 |
+
## How to use
|
| 150 |
+
|
| 151 |
+
This is a plain PyTorch checkpoint (not a `transformers` model), so inference requires the model class
|
| 152 |
+
definition from the companion GitHub repository. See [`inference_example.py`](inference_example.py) in
|
| 153 |
+
this repository for a complete, runnable example. In short:
|
| 154 |
+
|
| 155 |
+
```bash
|
| 156 |
+
git clone https://github.com/lab-smile/FearConditioningAI.git
|
| 157 |
+
cd FearConditioningAI
|
| 158 |
+
pip install -r requirements.txt # or: conda env create -f environment-<platform>.yml
|
| 159 |
+
```
|
| 160 |
+
|
| 161 |
+
```python
|
| 162 |
+
import torch
|
| 163 |
+
from huggingface_hub import hf_hub_download
|
| 164 |
+
from models.VGG_Model import Visual_Cortex_Amygdala
|
| 165 |
+
|
| 166 |
+
repo_id = "smilelab/visual-valence-model"
|
| 167 |
+
# swap in any filename from the checkpoint table above, e.g. "vca_ckvideo_batch128_lr2e-5_epoch20.pth"
|
| 168 |
+
ckpt_path = hf_hub_download(repo_id=repo_id, filename="base_model_conditioned_orientation_epoch100.pth")
|
| 169 |
+
|
| 170 |
+
model = Visual_Cortex_Amygdala()
|
| 171 |
+
checkpoint = torch.load(ckpt_path, map_location="cpu", weights_only=False)
|
| 172 |
+
model.load_state_dict(checkpoint["state_dict"], strict=False)
|
| 173 |
+
model.eval()
|
| 174 |
+
```
|
| 175 |
+
|
| 176 |
+
See `inference_example.py` for image preprocessing (resize/normalize + quadrant placement of the CS/US)
|
| 177 |
+
and how to rescale the model's sigmoid output back to the 1–9 valence scale.
|
| 178 |
+
|
| 179 |
+
---
|
| 180 |
+
|
| 181 |
+
## Citation
|
| 182 |
+
|
| 183 |
+
If you use this model, please cite the paper:
|
| 184 |
+
|
| 185 |
+
```bibtex
|
| 186 |
+
@article{leem2026associative,
|
| 187 |
+
title = {Associative Emotional Learning in Convolutional Neural Networks},
|
| 188 |
+
author = {Leem, Seowung and Keil, Andreas and Ding, Mingzhou and Fang, Ruogu},
|
| 189 |
+
journal = {Neural Computation},
|
| 190 |
+
year = {2026},
|
| 191 |
+
note = {in press},
|
| 192 |
+
eprint = {2607.19327},
|
| 193 |
+
archivePrefix = {arXiv},
|
| 194 |
+
url = {https://arxiv.org/abs/2607.19327}
|
| 195 |
+
}
|
| 196 |
+
```
|
| 197 |
+
|
| 198 |
+
Please also cite the datasets and methods this model builds on (IAPS, Cowen & Keltner Videoframe, SAM,
|
| 199 |
+
VGG-16, ImageNet, ECA-Net, Rescorla-Wagner) — full references in the
|
| 200 |
+
[GitHub README's Citations section](https://github.com/lab-smile/FearConditioningAI#citations).
|
| 201 |
+
|
| 202 |
+
---
|
| 203 |
+
|
| 204 |
+
## License
|
| 205 |
+
|
| 206 |
+
This model is released under the **MIT License**, matching the
|
| 207 |
+
[GitHub repository](https://github.com/lab-smile/FearConditioningAI/blob/main/LICENSE).
|
| 208 |
+
|
| 209 |
+
## Contact
|
| 210 |
+
|
| 211 |
+
| Name | Email |
|
| 212 |
+
|---|---|
|
| 213 |
+
| Seowung Leem | leem.s@ufl.edu |
|
| 214 |
+
| Dr. Ruogu Fang | ruogu.fang@bme.ufl.edu |
|
base_model_conditioned_orientation_epoch100.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9eba6472d0c9c3d0eb2eaefa0a4e9e4326921130002e2dc9c6f4dd315ea1468b
|
| 3 |
+
size 135
|
base_model_vca_IAPS_quadrant.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0e3d288ef2c6f8c98ea447308a275d5005f52ad1eaa19d6060ef821703ab12cf
|
| 3 |
+
size 135
|
config.json
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": ["Visual_Cortex_Amygdala"],
|
| 3 |
+
"model_type": "visual-valence-model",
|
| 4 |
+
"task": "image-valence-regression",
|
| 5 |
+
"source_repo": "https://github.com/lab-smile/FearConditioningAI",
|
| 6 |
+
"source_module": "models.VGG_Model",
|
| 7 |
+
"paper": "https://arxiv.org/abs/2607.19327",
|
| 8 |
+
|
| 9 |
+
"backbone": {
|
| 10 |
+
"name": "vgg16",
|
| 11 |
+
"pretrained_on": "imagenet-1k",
|
| 12 |
+
"batch_norm": false,
|
| 13 |
+
"frozen": true
|
| 14 |
+
},
|
| 15 |
+
|
| 16 |
+
"shortcut_pathway": {
|
| 17 |
+
"name": "middleroad",
|
| 18 |
+
"source_vgg_layer_index": 10,
|
| 19 |
+
"pooling": {
|
| 20 |
+
"middleroad_maxpool": {"kernel_size": 29, "stride": 14},
|
| 21 |
+
"global_maxpool_output_size": 2,
|
| 22 |
+
"maxpool1": {"kernel_size": 5, "stride": 3},
|
| 23 |
+
"maxpool2": {"kernel_size": 9, "stride": 5},
|
| 24 |
+
"maxpool3": {"kernel_size": 13, "stride": 7},
|
| 25 |
+
"adaptive_avgpool_output_sizes": [1, 2]
|
| 26 |
+
},
|
| 27 |
+
"attention": {
|
| 28 |
+
"type": "efficient_channel_attention",
|
| 29 |
+
"reference": "https://doi.org/10.1109/CVPR42600.2020.01155",
|
| 30 |
+
"conv1d_kernel_size": 3
|
| 31 |
+
},
|
| 32 |
+
"fc_layers": {
|
| 33 |
+
"input_size": 1024,
|
| 34 |
+
"hidden_sizes": [1024, 512],
|
| 35 |
+
"dropout": 0.5
|
| 36 |
+
}
|
| 37 |
+
},
|
| 38 |
+
|
| 39 |
+
"valence_module": {
|
| 40 |
+
"name": "VCA_FC",
|
| 41 |
+
"input_size": 4608,
|
| 42 |
+
"input_composition": {"highroad_features": 4096, "middleroad_features": 512},
|
| 43 |
+
"hidden_sizes": [1024, 1024],
|
| 44 |
+
"dropout": 0.5,
|
| 45 |
+
"output_size": 1,
|
| 46 |
+
"output_activation": "sigmoid",
|
| 47 |
+
"output_rescale_range": [1, 9],
|
| 48 |
+
"output_semantics": "valence rating (1 = extreme displeasure, 9 = extreme pleasure)"
|
| 49 |
+
},
|
| 50 |
+
|
| 51 |
+
"input": {
|
| 52 |
+
"image_size": 224,
|
| 53 |
+
"channels": 3,
|
| 54 |
+
"normalize_mean": [0.485, 0.456, 0.406],
|
| 55 |
+
"normalize_std": [0.229, 0.224, 0.225]
|
| 56 |
+
},
|
| 57 |
+
|
| 58 |
+
"checkpoints": {
|
| 59 |
+
"stage0_videoframe_pretrain": {
|
| 60 |
+
"filename": "vca_ckvideo_batch128_lr2e-5_epoch20.pth",
|
| 61 |
+
"stage": "Stage 0 (trained from scratch on Videoframe)",
|
| 62 |
+
"trained_on": ["Cowen & Keltner (2017) Videoframe"],
|
| 63 |
+
"input_layout": "full-frame (no quadrant cropping)",
|
| 64 |
+
"val_pearson_r": 0.386,
|
| 65 |
+
"val_mse": 0.043
|
| 66 |
+
},
|
| 67 |
+
"stage1_iaps_finetune": {
|
| 68 |
+
"filename": "vca_IAPS_batch10_lr2e-4_epoch23.pth",
|
| 69 |
+
"stage": "Stage 1 (fine-tuned on full-size IAPS)",
|
| 70 |
+
"trained_on": ["Cowen & Keltner (2017) Videoframe", "IAPS full-size"],
|
| 71 |
+
"input_layout": "full-frame (no quadrant cropping)",
|
| 72 |
+
"val_pearson_r": 0.538,
|
| 73 |
+
"val_mse": 0.192
|
| 74 |
+
},
|
| 75 |
+
"pre_conditioning": {
|
| 76 |
+
"filename": "base_model_vca_IAPS_quadrant.pth",
|
| 77 |
+
"stage": "Stage 2 (quadrant fine-tuning), before Pavlovian conditioning",
|
| 78 |
+
"trained_on": ["Cowen & Keltner (2017) Videoframe", "IAPS full-size", "IAPS quadrant-cropped"],
|
| 79 |
+
"input_layout": "quadrant-cropped (US in quadrant 4)",
|
| 80 |
+
"val_pearson_r": null,
|
| 81 |
+
"val_mse": null,
|
| 82 |
+
"note": "See paper for definitive evaluation numbers."
|
| 83 |
+
},
|
| 84 |
+
"post_conditioning_epoch1": {
|
| 85 |
+
"filename": "base_model_conditioned_orientation_epoch1.pth",
|
| 86 |
+
"stage": "Stage 3 (Pavlovian conditioning), epoch 1 of 100",
|
| 87 |
+
"trained_on": ["Cowen & Keltner (2017) Videoframe", "IAPS full-size", "IAPS quadrant-cropped", "IAPS Conditioning (US) x Gabor patch (CS)"],
|
| 88 |
+
"input_layout": "quadrant-cropped (CS in quadrant 2, US in quadrant 4)",
|
| 89 |
+
"val_pearson_r": 0.660,
|
| 90 |
+
"val_mse": 0.466,
|
| 91 |
+
"note": "Early/under-trained snapshot, kept for provenance; not representative of the final model."
|
| 92 |
+
},
|
| 93 |
+
"post_conditioning": {
|
| 94 |
+
"filename": "base_model_conditioned_orientation_epoch100.pth",
|
| 95 |
+
"stage": "Stage 3 (Pavlovian conditioning), epoch 100 (final, used in the paper)",
|
| 96 |
+
"trained_on": ["Cowen & Keltner (2017) Videoframe", "IAPS full-size", "IAPS quadrant-cropped", "IAPS Conditioning (US) x Gabor patch (CS)"],
|
| 97 |
+
"input_layout": "quadrant-cropped (CS in quadrant 2, US in quadrant 4)",
|
| 98 |
+
"val_pearson_r": null,
|
| 99 |
+
"val_mse": null,
|
| 100 |
+
"note": "See paper for definitive evaluation numbers.",
|
| 101 |
+
"conditioning_paradigm": {
|
| 102 |
+
"cs_stimulus": "Gabor patch (45 deg or 135 deg orientation)",
|
| 103 |
+
"us_stimulus": "IAPS image (unpleasant paired with 45 deg CS, pleasant paired with 135 deg CS)",
|
| 104 |
+
"cs_quadrant": 2,
|
| 105 |
+
"us_quadrant": 4
|
| 106 |
+
}
|
| 107 |
+
}
|
| 108 |
+
},
|
| 109 |
+
|
| 110 |
+
"checkpoint_format": {
|
| 111 |
+
"type": "torch.save dict",
|
| 112 |
+
"keys": ["model", "epoch", "best_per", "best_loss", "state_dict", "optimizer"],
|
| 113 |
+
"state_dict_key": "state_dict",
|
| 114 |
+
"note": "Load with model.load_state_dict(checkpoint['state_dict'], strict=False); the 'model' key is a pickled model object retained for reproducibility but should not be trusted/unpickled directly."
|
| 115 |
+
},
|
| 116 |
+
|
| 117 |
+
"license": "mit"
|
| 118 |
+
}
|
inference_example.py
ADDED
|
@@ -0,0 +1,152 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Example inference script for the Visual-Valence Model (VCA).
|
| 2 |
+
|
| 3 |
+
Requirements:
|
| 4 |
+
- Clone https://github.com/lab-smile/FearConditioningAI and run this script from
|
| 5 |
+
inside that repo (or add it to PYTHONPATH), so `models.VGG_Model` / `utils` are
|
| 6 |
+
importable. Install its dependencies (environment-<platform>.yml or requirements.txt).
|
| 7 |
+
- pip install huggingface_hub pillow torch torchvision
|
| 8 |
+
|
| 9 |
+
Usage:
|
| 10 |
+
python inference_example.py --image path/to/scene.jpg --checkpoint post
|
| 11 |
+
python inference_example.py --gabor path/to/gabor_patch.png --checkpoint post
|
| 12 |
+
python inference_example.py --image path/to/scene.jpg --checkpoint stage1 # full-frame checkpoint
|
| 13 |
+
|
| 14 |
+
Checkpoints (see CHECKPOINT_FILENAMES): stage0, stage1 (full-frame, Stages 0-1),
|
| 15 |
+
pre, post_epoch1, post (quadrant-cropped, Stages 2-3).
|
| 16 |
+
"""
|
| 17 |
+
|
| 18 |
+
import argparse
|
| 19 |
+
|
| 20 |
+
import torch
|
| 21 |
+
from huggingface_hub import hf_hub_download
|
| 22 |
+
from PIL import Image
|
| 23 |
+
from torchvision import transforms
|
| 24 |
+
|
| 25 |
+
from models.VGG_Model import Visual_Cortex_Amygdala
|
| 26 |
+
|
| 27 |
+
# Placeholder — update to the actual Hugging Face repo id once uploaded.
|
| 28 |
+
REPO_ID = "smilelab/visual-valence-model"
|
| 29 |
+
|
| 30 |
+
CHECKPOINT_FILENAMES = {
|
| 31 |
+
"stage0": "vca_ckvideo_batch128_lr2e-5_epoch20.pth", # trained from scratch on Videoframe
|
| 32 |
+
"stage1": "vca_IAPS_batch10_lr2e-4_epoch23.pth", # fine-tuned on full-size IAPS
|
| 33 |
+
"pre": "base_model_vca_IAPS_quadrant.pth", # Stage 2: quadrant fine-tune, before conditioning
|
| 34 |
+
"post_epoch1": "base_model_conditioned_orientation_epoch1.pth", # Stage 3, epoch 1 (early snapshot)
|
| 35 |
+
"post": "base_model_conditioned_orientation_epoch100.pth", # Stage 3, epoch 100 (final, after conditioning)
|
| 36 |
+
}
|
| 37 |
+
|
| 38 |
+
# Stage 0/1 checkpoints were trained on full-frame images; Stage 2/3 checkpoints expect the
|
| 39 |
+
# quadrant-cropped layout (see preprocess_natural_image vs. preprocess_cs_only below).
|
| 40 |
+
FULL_FRAME_CHECKPOINTS = {"stage0", "stage1"}
|
| 41 |
+
|
| 42 |
+
IMAGE_SIZE = 224
|
| 43 |
+
NORMALIZE_MEAN = [0.485, 0.456, 0.406]
|
| 44 |
+
NORMALIZE_STD = [0.229, 0.224, 0.225]
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def load_model(checkpoint: str = "post", device: str = "cpu") -> torch.nn.Module:
|
| 48 |
+
"""Download a checkpoint from the Hub and load it into a Visual_Cortex_Amygdala model."""
|
| 49 |
+
filename = CHECKPOINT_FILENAMES[checkpoint]
|
| 50 |
+
ckpt_path = hf_hub_download(repo_id=REPO_ID, filename=filename)
|
| 51 |
+
|
| 52 |
+
model = Visual_Cortex_Amygdala()
|
| 53 |
+
checkpoint_dict = torch.load(ckpt_path, map_location=device, weights_only=False)
|
| 54 |
+
model.load_state_dict(checkpoint_dict["state_dict"], strict=False)
|
| 55 |
+
model.to(device)
|
| 56 |
+
model.eval()
|
| 57 |
+
return model
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def preprocess_natural_image(image: Image.Image) -> torch.Tensor:
|
| 61 |
+
"""Full-frame preprocessing for the Stage 0/1 checkpoints (no quadrant cropping)."""
|
| 62 |
+
transform = transforms.Compose([
|
| 63 |
+
transforms.Resize((IMAGE_SIZE, IMAGE_SIZE)),
|
| 64 |
+
transforms.ToTensor(),
|
| 65 |
+
transforms.Normalize(mean=NORMALIZE_MEAN, std=NORMALIZE_STD),
|
| 66 |
+
])
|
| 67 |
+
return transform(image.convert("RGB")).unsqueeze(0)
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def _place_in_quadrant(patch: Image.Image, quadrant: int, canvas: Image.Image = None) -> Image.Image:
|
| 71 |
+
"""Resize `patch` to quarter size and paste it into one quadrant of `canvas` (new blank one if None).
|
| 72 |
+
|
| 73 |
+
quadrant: 1 = top-right, 2 = top-left, 3 = bottom-left, 4 = bottom-right.
|
| 74 |
+
"""
|
| 75 |
+
if canvas is None:
|
| 76 |
+
canvas = Image.new("RGB", (IMAGE_SIZE, IMAGE_SIZE))
|
| 77 |
+
patch = patch.convert("RGB").resize((IMAGE_SIZE // 2, IMAGE_SIZE // 2))
|
| 78 |
+
positions = {
|
| 79 |
+
1: (IMAGE_SIZE // 2, 0),
|
| 80 |
+
2: (0, 0),
|
| 81 |
+
3: (0, IMAGE_SIZE // 2),
|
| 82 |
+
4: (IMAGE_SIZE // 2, IMAGE_SIZE // 2),
|
| 83 |
+
}
|
| 84 |
+
canvas.paste(patch, positions[quadrant])
|
| 85 |
+
return canvas
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
def preprocess_quadrant(image: Image.Image, quadrant: int = 4) -> torch.Tensor:
|
| 89 |
+
"""Place a natural (US) scene alone into one quadrant of an otherwise-blank canvas.
|
| 90 |
+
|
| 91 |
+
Used for Stage 2/3 checkpoints, which were trained on quadrant-cropped US images
|
| 92 |
+
(see utils.Quadrant_Processing). Default quadrant 4 (bottom-right) matches training.
|
| 93 |
+
"""
|
| 94 |
+
canvas = _place_in_quadrant(image, quadrant)
|
| 95 |
+
transform = transforms.Compose([
|
| 96 |
+
transforms.ToTensor(),
|
| 97 |
+
transforms.Normalize(mean=NORMALIZE_MEAN, std=NORMALIZE_STD),
|
| 98 |
+
])
|
| 99 |
+
return transform(canvas).unsqueeze(0)
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
def preprocess_cs_only(gabor_patch: Image.Image, quadrant: int = 2) -> torch.Tensor:
|
| 103 |
+
"""Place a CS Gabor patch alone into one quadrant of an otherwise-blank canvas.
|
| 104 |
+
|
| 105 |
+
Mirrors the conditioning-stage input layout (see utils.Quadrant_Processing_Conditioning /
|
| 106 |
+
test_gaborpatches.py): the US quadrant is left blank so the response reflects only what
|
| 107 |
+
the model has learned to associate with the CS. Default quadrant 2 (top-left) matches
|
| 108 |
+
the CS placement used during Stage 3 training; only meaningful for Stage 2/3 checkpoints.
|
| 109 |
+
"""
|
| 110 |
+
return preprocess_quadrant(gabor_patch, quadrant)
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
@torch.no_grad()
|
| 114 |
+
def predict_valence(model: torch.nn.Module, input_tensor: torch.Tensor, device: str = "cpu") -> float:
|
| 115 |
+
"""Run the model and rescale its sigmoid output from [0, 1] to the [1, 9] IAPS valence scale."""
|
| 116 |
+
output = model(input_tensor.to(device))
|
| 117 |
+
valence = 1 + output.item() * (9 - 1)
|
| 118 |
+
return valence
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
def main():
|
| 122 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 123 |
+
parser.add_argument("--image", type=str, default=None, help="Path to a natural scene image (US-style input).")
|
| 124 |
+
parser.add_argument("--gabor", type=str, default=None, help="Path to a Gabor patch image (CS-only input).")
|
| 125 |
+
parser.add_argument("--checkpoint", choices=list(CHECKPOINT_FILENAMES), default="post",
|
| 126 |
+
help="Which checkpoint to load (see CHECKPOINT_FILENAMES). Default: 'post' "
|
| 127 |
+
"(final, post-conditioning model).")
|
| 128 |
+
parser.add_argument("--device", type=str, default="cpu")
|
| 129 |
+
args = parser.parse_args()
|
| 130 |
+
|
| 131 |
+
if not args.image and not args.gabor:
|
| 132 |
+
parser.error("Provide --image or --gabor.")
|
| 133 |
+
if args.gabor and args.checkpoint in FULL_FRAME_CHECKPOINTS:
|
| 134 |
+
parser.error(f"--gabor (CS-only input) isn't meaningful for checkpoint '{args.checkpoint}', "
|
| 135 |
+
f"which was trained on full-frame images without a CS. Use --image instead, or "
|
| 136 |
+
f"pick a Stage 2/3 checkpoint (pre, post_epoch1, post).")
|
| 137 |
+
|
| 138 |
+
model = load_model(checkpoint=args.checkpoint, device=args.device)
|
| 139 |
+
|
| 140 |
+
if args.image and args.checkpoint in FULL_FRAME_CHECKPOINTS:
|
| 141 |
+
input_tensor = preprocess_natural_image(Image.open(args.image))
|
| 142 |
+
elif args.image:
|
| 143 |
+
input_tensor = preprocess_quadrant(Image.open(args.image))
|
| 144 |
+
else:
|
| 145 |
+
input_tensor = preprocess_cs_only(Image.open(args.gabor))
|
| 146 |
+
|
| 147 |
+
valence = predict_valence(model, input_tensor, device=args.device)
|
| 148 |
+
print(f"Predicted valence (1=extreme displeasure, 9=extreme pleasure): {valence:.2f}")
|
| 149 |
+
|
| 150 |
+
|
| 151 |
+
if __name__ == "__main__":
|
| 152 |
+
main()
|
vca_IAPS_batch10_lr2e-4_epoch23.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a09e005aeccac3e5c3d279b451af26d1a4b4bde05009e8994b5f2160950953ef
|
| 3 |
+
size 1162295022
|
vca_ckvideo_batch128_lr2e-5_epoch20.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:15ab5c5a46f32cb6515419b516d6f6ab1ae9186a3e38d9e6f81435f93467d5b2
|
| 3 |
+
size 1162295201
|