lkid7715 commited on
Commit
ddedd58
·
verified ·
1 Parent(s): 90480f5

Upload 7 files

Browse files
README.md CHANGED
@@ -1,3 +1,214 @@
1
  ---
2
  license: mit
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
+ library_name: pytorch
4
+ tags:
5
+ - computer-vision
6
+ - affective-computing
7
+ - neuroscience
8
+ - image-regression
9
+ - vgg16
10
+ - pavlovian-conditioning
11
+ - valence-prediction
12
  ---
13
+
14
+ # Visual-Valence Model (VCA)
15
+
16
+ > A deep neural network model of associative emotional (Pavlovian fear) learning.
17
+
18
+ Companion model repository for **["Associative Emotional Learning in Convolutional Neural Networks"](https://arxiv.org/abs/2607.19327)**
19
+ (Leem, Keil, Ding & Fang; *Neural Computation*, in press).
20
+
21
+ - 📄 Paper: https://arxiv.org/abs/2607.19327
22
+ - 💻 Code: https://github.com/lab-smile/FearConditioningAI
23
+
24
+ > **Note:** This model is a research artifact for computational/cognitive neuroscience, released to
25
+ > reproduce and extend the paper's findings. It is **not** a general-purpose sentiment/emotion API and
26
+ > is not validated for clinical, diagnostic, or production affective-computing use.
27
+
28
+ ---
29
+
30
+ ## Model description
31
+
32
+ The Visual-Valence Model predicts the affective **valence** of a visual scene (1 = extreme displeasure,
33
+ 9 = extreme pleasure) and reproduces hallmarks of Pavlovian (fear) conditioning when a novel, initially
34
+ neutral stimulus is repeatedly paired with an emotionally-charged one.
35
+
36
+ The architecture (`Visual_Cortex_Amygdala` in [`models/VGG_Model.py`](https://github.com/lab-smile/FearConditioningAI/blob/main/models/VGG_Model.py))
37
+ has three components, loosely modeling the primate visual/affective pathway:
38
+
39
+ | Component | Role | Implementation |
40
+ |---|---|---|
41
+ | **Visual Cortex module** ("High Road") | Ventral visual stream | VGG-16 (Simonyan & Zisserman, 2015), ImageNet-pretrained, **frozen** |
42
+ | **Shortcut Pathway** ("Middle Road") | Fast, low-resolution route from early vision to affect circuitry | Multi-scale pooling of VGG-16's early layers (layer index 10), combined via an Efficient Channel Attention (ECA) module (Wang et al., 2020) |
43
+ | **Valence Module** | Combines both pathways into a scalar valence judgment | Two fully-connected layers (amygdala LA/CE-nuclei- and OFC-inspired) + one sigmoid output unit |
44
+
45
+ The High Road output (4096-d) and Middle Road output (512-d) are concatenated (4608-d) and passed through
46
+ the Valence Module's fully connected layers to a single sigmoid unit, which is linearly rescaled from
47
+ `[0, 1]` to the `[1, 9]` IAPS valence scale at evaluation time.
48
+
49
+ ### Five checkpoints, one training pipeline
50
+
51
+ This repository hosts every checkpoint along the training pipeline described in the
52
+ [GitHub README](https://github.com/lab-smile/FearConditioningAI#training) — same architecture
53
+ (`Visual_Cortex_Amygdala`) throughout, so any of them can be loaded with the same code:
54
+
55
+ | File | Stage | Description | Val. R | Val. MSE |
56
+ |---|---|---|---|---|
57
+ | `vca_ckvideo_batch128_lr2e-5_epoch20.pth` | 0 | Trained from scratch on the Cowen & Keltner Videoframe dataset | 0.386 | 0.043 |
58
+ | `vca_IAPS_batch10_lr2e-4_epoch23.pth` | 1 | Fine-tuned on full-size IAPS images | 0.538 | 0.192 |
59
+ | `base_model_vca_IAPS_quadrant.pth` | 2 | Fine-tuned to the quadrant-cropped input layout — **pre-conditioning**: has learned to decode valence from natural scenes (the US) but has never seen the conditioned stimulus (CS, a Gabor patch) | see paper | see paper |
60
+ | `base_model_conditioned_orientation_epoch1.pth` | 3, epoch 1 | First epoch of Pavlovian conditioning (CS+ Gabor patch paired with pleasant/unpleasant IAPS US); early/under-trained, kept for provenance | 0.660 | 0.466 |
61
+ | `base_model_conditioned_orientation_epoch100.pth` | 3, epoch 100 | **Post-conditioning (final)** — used throughout the paper's conditioning/generalization/representational-alignment analyses | see paper | see paper |
62
+
63
+ Stage 0/1 metrics are Pearson R / MSE on their respective held-out validation sets (read from each
64
+ checkpoint's own logged `best_per` / `best_loss`). Stage 2 and the final Stage 3 (epoch 100) numbers are
65
+ intentionally not restated here — see the paper for the definitive, fully-trained results.
66
+
67
+ Comparing the pre- (Stage 2) and post-conditioning (Stage 3, epoch 100) checkpoints' responses to the CS
68
+ alone is what reveals the learned CS→valence association (and, at the representation level, the
69
+ increasing CS/US alignment reported in the paper).
70
+
71
+ ---
72
+
73
+ ## Intended use
74
+
75
+ - Reproducing the paper's Pavlovian-conditioning experiments and figures.
76
+ - Extending the model to new conditioning paradigms, stimulus sets, or ablations (e.g. the
77
+ attention-free variant, `Visual_Cortex_Amygdala_wo_Attention`) for **research purposes**.
78
+ - Studying how a two-pathway (ventral-stream + shortcut) architecture with attention reproduces
79
+ behavioral/representational signatures of associative emotional learning.
80
+ - Feature extraction / representational analysis (see `Channel_Activity_Extraction.py`,
81
+ `Manifold_Visualization.py`, `SVM_Analysis_*.py` in the GitHub repo) for downstream neuroscience analyses.
82
+
83
+ **Out of scope:** general-purpose sentiment analysis, affect recognition on non-IAPS-like natural images,
84
+ clinical/diagnostic use, or any decision-making about real individuals' emotional states.
85
+
86
+ ---
87
+
88
+ ## Limitations
89
+
90
+ - **Frozen ImageNet backbone.** The High Road (VGG-16) is never fine-tuned, so the model inherits
91
+ ImageNet's visual biases; only the shortcut pathway and valence module are trained on affective data.
92
+ - **Narrow, licensed training data.** IAPS is a relatively small, curated stimulus set under a
93
+ data-use/confidentiality agreement (not redistributed with the code or this checkpoint); the model's
94
+ generalization to naturalistic, in-the-wild imagery is untested.
95
+ - **Two-alternative conditioning paradigm.** The conditioning stage pairs exactly two Gabor orientations
96
+ (45°, 135°) with unpleasant/pleasant IAPS images; behavior on other CS types has not been evaluated.
97
+ - **Not validated against individual human subjects.** Comparisons to human data in the paper are at the
98
+ group/aggregate level; the model is not a predictive model of any specific person's affective response.
99
+ - **Single scalar output.** The model predicts only valence (pleasant ↔ unpleasant), not arousal or
100
+ discrete emotion categories.
101
+
102
+ ---
103
+
104
+ ## Evaluation metrics
105
+
106
+ Model quality is reported using:
107
+
108
+ - **Pearson correlation (R / R²)** between predicted and ground-truth (SAM-rated) valence, computed by
109
+ `reg_eval_model` / `cond_eval_model` in [`utils.py`](https://github.com/lab-smile/FearConditioningAI/blob/main/utils.py).
110
+ - **Mean-squared error (MSE)** between predicted and ground-truth valence (rescaled to the 1–9 scale).
111
+ - Post-conditioning, generalization and CS/US **representational alignment** (single-unit tuning and
112
+ population-level RSA/t-SNE, via `Channel_Activity_Extraction.py`, `Manifold_Visualization.py`,
113
+ `SVM_Analysis_Emotion.py`, `SVM_Analysis_Before_After.py`) are used to assess whether conditioning
114
+ reproduces human associative-learning signatures.
115
+
116
+ The full quantitative results (per-stage R/MSE, generalization curves, and alignment statistics) are
117
+ reported in the paper's tables/figures — see https://arxiv.org/abs/2607.19327. The Stage 0/1 R and MSE
118
+ values in the checkpoint table above are real, logged validation numbers for those checkpoints; the
119
+ Stage 3/epoch-1 numbers are a very early, under-trained snapshot and are **not** representative of the
120
+ final, fully-conditioned (epoch 100) model — included only for provenance, not as a reported result.
121
+
122
+ ---
123
+
124
+ ## Training dataset
125
+
126
+ Training proceeds in stages, each building on the previous stage's checkpoint (see the GitHub README's
127
+ [Training](https://github.com/lab-smile/FearConditioningAI#training) section for exact commands):
128
+
129
+ | Stage | Dataset | Purpose |
130
+ |---|---|---|
131
+ | 0 | Cowen & Keltner (2017) Videoframe dataset (2,185 emotion-eliciting video clips, one frame sampled every 10th frame) | Pretrain valence regression from scratch on natural scenes |
132
+ | 1 | International Affective Picture System (IAPS; Bradley & Lang, 1994/2007), full-size images, 8:1:1 train/val/test split | Adapt to the US stimuli used in conditioning |
133
+ | 2 | IAPS, quadrant-cropped layout | Fine-tune to the spatial layout used during conditioning |
134
+ | 3 | IAPS (US, in the 4th quadrant) × Gabor patches (CS, in the 2nd quadrant; 45°/135° orientation, varying spatial frequency/contrast, generated via `Gabor4Seowung.m`) | Pavlovian conditioning: pair CS+ with pleasant/unpleasant US |
135
+
136
+ Labels are human valence ratings on a 1–9 scale (Self-Assessment Manikin; Bradley & Lang, 1994).
137
+ **The IAPS images themselves are not redistributed** with the code or this model repository due to a
138
+ data-use/confidentiality agreement — obtain access to IAPS independently to reproduce training from
139
+ scratch. The Gabor-patch CS stimuli are procedurally generated and have no such restriction.
140
+
141
+ Note the input layout differs by stage: Stage 0/1 checkpoints expect a full-frame natural image
142
+ (resize + normalize only), while Stage 2/3 checkpoints expect the quadrant-cropped layout (see
143
+ [Data preprocessing](https://github.com/lab-smile/FearConditioningAI#data-preprocessing) in the GitHub
144
+ README) — feeding a full-frame image to a Stage 2/3 checkpoint (or vice versa) will not reproduce the
145
+ reported behavior.
146
+
147
+ ---
148
+
149
+ ## How to use
150
+
151
+ This is a plain PyTorch checkpoint (not a `transformers` model), so inference requires the model class
152
+ definition from the companion GitHub repository. See [`inference_example.py`](inference_example.py) in
153
+ this repository for a complete, runnable example. In short:
154
+
155
+ ```bash
156
+ git clone https://github.com/lab-smile/FearConditioningAI.git
157
+ cd FearConditioningAI
158
+ pip install -r requirements.txt # or: conda env create -f environment-<platform>.yml
159
+ ```
160
+
161
+ ```python
162
+ import torch
163
+ from huggingface_hub import hf_hub_download
164
+ from models.VGG_Model import Visual_Cortex_Amygdala
165
+
166
+ repo_id = "smilelab/visual-valence-model"
167
+ # swap in any filename from the checkpoint table above, e.g. "vca_ckvideo_batch128_lr2e-5_epoch20.pth"
168
+ ckpt_path = hf_hub_download(repo_id=repo_id, filename="base_model_conditioned_orientation_epoch100.pth")
169
+
170
+ model = Visual_Cortex_Amygdala()
171
+ checkpoint = torch.load(ckpt_path, map_location="cpu", weights_only=False)
172
+ model.load_state_dict(checkpoint["state_dict"], strict=False)
173
+ model.eval()
174
+ ```
175
+
176
+ See `inference_example.py` for image preprocessing (resize/normalize + quadrant placement of the CS/US)
177
+ and how to rescale the model's sigmoid output back to the 1–9 valence scale.
178
+
179
+ ---
180
+
181
+ ## Citation
182
+
183
+ If you use this model, please cite the paper:
184
+
185
+ ```bibtex
186
+ @article{leem2026associative,
187
+ title = {Associative Emotional Learning in Convolutional Neural Networks},
188
+ author = {Leem, Seowung and Keil, Andreas and Ding, Mingzhou and Fang, Ruogu},
189
+ journal = {Neural Computation},
190
+ year = {2026},
191
+ note = {in press},
192
+ eprint = {2607.19327},
193
+ archivePrefix = {arXiv},
194
+ url = {https://arxiv.org/abs/2607.19327}
195
+ }
196
+ ```
197
+
198
+ Please also cite the datasets and methods this model builds on (IAPS, Cowen & Keltner Videoframe, SAM,
199
+ VGG-16, ImageNet, ECA-Net, Rescorla-Wagner) — full references in the
200
+ [GitHub README's Citations section](https://github.com/lab-smile/FearConditioningAI#citations).
201
+
202
+ ---
203
+
204
+ ## License
205
+
206
+ This model is released under the **MIT License**, matching the
207
+ [GitHub repository](https://github.com/lab-smile/FearConditioningAI/blob/main/LICENSE).
208
+
209
+ ## Contact
210
+
211
+ | Name | Email |
212
+ |---|---|
213
+ | Seowung Leem | leem.s@ufl.edu |
214
+ | Dr. Ruogu Fang | ruogu.fang@bme.ufl.edu |
base_model_conditioned_orientation_epoch100.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9eba6472d0c9c3d0eb2eaefa0a4e9e4326921130002e2dc9c6f4dd315ea1468b
3
+ size 135
base_model_vca_IAPS_quadrant.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0e3d288ef2c6f8c98ea447308a275d5005f52ad1eaa19d6060ef821703ab12cf
3
+ size 135
config.json ADDED
@@ -0,0 +1,118 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": ["Visual_Cortex_Amygdala"],
3
+ "model_type": "visual-valence-model",
4
+ "task": "image-valence-regression",
5
+ "source_repo": "https://github.com/lab-smile/FearConditioningAI",
6
+ "source_module": "models.VGG_Model",
7
+ "paper": "https://arxiv.org/abs/2607.19327",
8
+
9
+ "backbone": {
10
+ "name": "vgg16",
11
+ "pretrained_on": "imagenet-1k",
12
+ "batch_norm": false,
13
+ "frozen": true
14
+ },
15
+
16
+ "shortcut_pathway": {
17
+ "name": "middleroad",
18
+ "source_vgg_layer_index": 10,
19
+ "pooling": {
20
+ "middleroad_maxpool": {"kernel_size": 29, "stride": 14},
21
+ "global_maxpool_output_size": 2,
22
+ "maxpool1": {"kernel_size": 5, "stride": 3},
23
+ "maxpool2": {"kernel_size": 9, "stride": 5},
24
+ "maxpool3": {"kernel_size": 13, "stride": 7},
25
+ "adaptive_avgpool_output_sizes": [1, 2]
26
+ },
27
+ "attention": {
28
+ "type": "efficient_channel_attention",
29
+ "reference": "https://doi.org/10.1109/CVPR42600.2020.01155",
30
+ "conv1d_kernel_size": 3
31
+ },
32
+ "fc_layers": {
33
+ "input_size": 1024,
34
+ "hidden_sizes": [1024, 512],
35
+ "dropout": 0.5
36
+ }
37
+ },
38
+
39
+ "valence_module": {
40
+ "name": "VCA_FC",
41
+ "input_size": 4608,
42
+ "input_composition": {"highroad_features": 4096, "middleroad_features": 512},
43
+ "hidden_sizes": [1024, 1024],
44
+ "dropout": 0.5,
45
+ "output_size": 1,
46
+ "output_activation": "sigmoid",
47
+ "output_rescale_range": [1, 9],
48
+ "output_semantics": "valence rating (1 = extreme displeasure, 9 = extreme pleasure)"
49
+ },
50
+
51
+ "input": {
52
+ "image_size": 224,
53
+ "channels": 3,
54
+ "normalize_mean": [0.485, 0.456, 0.406],
55
+ "normalize_std": [0.229, 0.224, 0.225]
56
+ },
57
+
58
+ "checkpoints": {
59
+ "stage0_videoframe_pretrain": {
60
+ "filename": "vca_ckvideo_batch128_lr2e-5_epoch20.pth",
61
+ "stage": "Stage 0 (trained from scratch on Videoframe)",
62
+ "trained_on": ["Cowen & Keltner (2017) Videoframe"],
63
+ "input_layout": "full-frame (no quadrant cropping)",
64
+ "val_pearson_r": 0.386,
65
+ "val_mse": 0.043
66
+ },
67
+ "stage1_iaps_finetune": {
68
+ "filename": "vca_IAPS_batch10_lr2e-4_epoch23.pth",
69
+ "stage": "Stage 1 (fine-tuned on full-size IAPS)",
70
+ "trained_on": ["Cowen & Keltner (2017) Videoframe", "IAPS full-size"],
71
+ "input_layout": "full-frame (no quadrant cropping)",
72
+ "val_pearson_r": 0.538,
73
+ "val_mse": 0.192
74
+ },
75
+ "pre_conditioning": {
76
+ "filename": "base_model_vca_IAPS_quadrant.pth",
77
+ "stage": "Stage 2 (quadrant fine-tuning), before Pavlovian conditioning",
78
+ "trained_on": ["Cowen & Keltner (2017) Videoframe", "IAPS full-size", "IAPS quadrant-cropped"],
79
+ "input_layout": "quadrant-cropped (US in quadrant 4)",
80
+ "val_pearson_r": null,
81
+ "val_mse": null,
82
+ "note": "See paper for definitive evaluation numbers."
83
+ },
84
+ "post_conditioning_epoch1": {
85
+ "filename": "base_model_conditioned_orientation_epoch1.pth",
86
+ "stage": "Stage 3 (Pavlovian conditioning), epoch 1 of 100",
87
+ "trained_on": ["Cowen & Keltner (2017) Videoframe", "IAPS full-size", "IAPS quadrant-cropped", "IAPS Conditioning (US) x Gabor patch (CS)"],
88
+ "input_layout": "quadrant-cropped (CS in quadrant 2, US in quadrant 4)",
89
+ "val_pearson_r": 0.660,
90
+ "val_mse": 0.466,
91
+ "note": "Early/under-trained snapshot, kept for provenance; not representative of the final model."
92
+ },
93
+ "post_conditioning": {
94
+ "filename": "base_model_conditioned_orientation_epoch100.pth",
95
+ "stage": "Stage 3 (Pavlovian conditioning), epoch 100 (final, used in the paper)",
96
+ "trained_on": ["Cowen & Keltner (2017) Videoframe", "IAPS full-size", "IAPS quadrant-cropped", "IAPS Conditioning (US) x Gabor patch (CS)"],
97
+ "input_layout": "quadrant-cropped (CS in quadrant 2, US in quadrant 4)",
98
+ "val_pearson_r": null,
99
+ "val_mse": null,
100
+ "note": "See paper for definitive evaluation numbers.",
101
+ "conditioning_paradigm": {
102
+ "cs_stimulus": "Gabor patch (45 deg or 135 deg orientation)",
103
+ "us_stimulus": "IAPS image (unpleasant paired with 45 deg CS, pleasant paired with 135 deg CS)",
104
+ "cs_quadrant": 2,
105
+ "us_quadrant": 4
106
+ }
107
+ }
108
+ },
109
+
110
+ "checkpoint_format": {
111
+ "type": "torch.save dict",
112
+ "keys": ["model", "epoch", "best_per", "best_loss", "state_dict", "optimizer"],
113
+ "state_dict_key": "state_dict",
114
+ "note": "Load with model.load_state_dict(checkpoint['state_dict'], strict=False); the 'model' key is a pickled model object retained for reproducibility but should not be trusted/unpickled directly."
115
+ },
116
+
117
+ "license": "mit"
118
+ }
inference_example.py ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Example inference script for the Visual-Valence Model (VCA).
2
+
3
+ Requirements:
4
+ - Clone https://github.com/lab-smile/FearConditioningAI and run this script from
5
+ inside that repo (or add it to PYTHONPATH), so `models.VGG_Model` / `utils` are
6
+ importable. Install its dependencies (environment-<platform>.yml or requirements.txt).
7
+ - pip install huggingface_hub pillow torch torchvision
8
+
9
+ Usage:
10
+ python inference_example.py --image path/to/scene.jpg --checkpoint post
11
+ python inference_example.py --gabor path/to/gabor_patch.png --checkpoint post
12
+ python inference_example.py --image path/to/scene.jpg --checkpoint stage1 # full-frame checkpoint
13
+
14
+ Checkpoints (see CHECKPOINT_FILENAMES): stage0, stage1 (full-frame, Stages 0-1),
15
+ pre, post_epoch1, post (quadrant-cropped, Stages 2-3).
16
+ """
17
+
18
+ import argparse
19
+
20
+ import torch
21
+ from huggingface_hub import hf_hub_download
22
+ from PIL import Image
23
+ from torchvision import transforms
24
+
25
+ from models.VGG_Model import Visual_Cortex_Amygdala
26
+
27
+ # Placeholder — update to the actual Hugging Face repo id once uploaded.
28
+ REPO_ID = "smilelab/visual-valence-model"
29
+
30
+ CHECKPOINT_FILENAMES = {
31
+ "stage0": "vca_ckvideo_batch128_lr2e-5_epoch20.pth", # trained from scratch on Videoframe
32
+ "stage1": "vca_IAPS_batch10_lr2e-4_epoch23.pth", # fine-tuned on full-size IAPS
33
+ "pre": "base_model_vca_IAPS_quadrant.pth", # Stage 2: quadrant fine-tune, before conditioning
34
+ "post_epoch1": "base_model_conditioned_orientation_epoch1.pth", # Stage 3, epoch 1 (early snapshot)
35
+ "post": "base_model_conditioned_orientation_epoch100.pth", # Stage 3, epoch 100 (final, after conditioning)
36
+ }
37
+
38
+ # Stage 0/1 checkpoints were trained on full-frame images; Stage 2/3 checkpoints expect the
39
+ # quadrant-cropped layout (see preprocess_natural_image vs. preprocess_cs_only below).
40
+ FULL_FRAME_CHECKPOINTS = {"stage0", "stage1"}
41
+
42
+ IMAGE_SIZE = 224
43
+ NORMALIZE_MEAN = [0.485, 0.456, 0.406]
44
+ NORMALIZE_STD = [0.229, 0.224, 0.225]
45
+
46
+
47
+ def load_model(checkpoint: str = "post", device: str = "cpu") -> torch.nn.Module:
48
+ """Download a checkpoint from the Hub and load it into a Visual_Cortex_Amygdala model."""
49
+ filename = CHECKPOINT_FILENAMES[checkpoint]
50
+ ckpt_path = hf_hub_download(repo_id=REPO_ID, filename=filename)
51
+
52
+ model = Visual_Cortex_Amygdala()
53
+ checkpoint_dict = torch.load(ckpt_path, map_location=device, weights_only=False)
54
+ model.load_state_dict(checkpoint_dict["state_dict"], strict=False)
55
+ model.to(device)
56
+ model.eval()
57
+ return model
58
+
59
+
60
+ def preprocess_natural_image(image: Image.Image) -> torch.Tensor:
61
+ """Full-frame preprocessing for the Stage 0/1 checkpoints (no quadrant cropping)."""
62
+ transform = transforms.Compose([
63
+ transforms.Resize((IMAGE_SIZE, IMAGE_SIZE)),
64
+ transforms.ToTensor(),
65
+ transforms.Normalize(mean=NORMALIZE_MEAN, std=NORMALIZE_STD),
66
+ ])
67
+ return transform(image.convert("RGB")).unsqueeze(0)
68
+
69
+
70
+ def _place_in_quadrant(patch: Image.Image, quadrant: int, canvas: Image.Image = None) -> Image.Image:
71
+ """Resize `patch` to quarter size and paste it into one quadrant of `canvas` (new blank one if None).
72
+
73
+ quadrant: 1 = top-right, 2 = top-left, 3 = bottom-left, 4 = bottom-right.
74
+ """
75
+ if canvas is None:
76
+ canvas = Image.new("RGB", (IMAGE_SIZE, IMAGE_SIZE))
77
+ patch = patch.convert("RGB").resize((IMAGE_SIZE // 2, IMAGE_SIZE // 2))
78
+ positions = {
79
+ 1: (IMAGE_SIZE // 2, 0),
80
+ 2: (0, 0),
81
+ 3: (0, IMAGE_SIZE // 2),
82
+ 4: (IMAGE_SIZE // 2, IMAGE_SIZE // 2),
83
+ }
84
+ canvas.paste(patch, positions[quadrant])
85
+ return canvas
86
+
87
+
88
+ def preprocess_quadrant(image: Image.Image, quadrant: int = 4) -> torch.Tensor:
89
+ """Place a natural (US) scene alone into one quadrant of an otherwise-blank canvas.
90
+
91
+ Used for Stage 2/3 checkpoints, which were trained on quadrant-cropped US images
92
+ (see utils.Quadrant_Processing). Default quadrant 4 (bottom-right) matches training.
93
+ """
94
+ canvas = _place_in_quadrant(image, quadrant)
95
+ transform = transforms.Compose([
96
+ transforms.ToTensor(),
97
+ transforms.Normalize(mean=NORMALIZE_MEAN, std=NORMALIZE_STD),
98
+ ])
99
+ return transform(canvas).unsqueeze(0)
100
+
101
+
102
+ def preprocess_cs_only(gabor_patch: Image.Image, quadrant: int = 2) -> torch.Tensor:
103
+ """Place a CS Gabor patch alone into one quadrant of an otherwise-blank canvas.
104
+
105
+ Mirrors the conditioning-stage input layout (see utils.Quadrant_Processing_Conditioning /
106
+ test_gaborpatches.py): the US quadrant is left blank so the response reflects only what
107
+ the model has learned to associate with the CS. Default quadrant 2 (top-left) matches
108
+ the CS placement used during Stage 3 training; only meaningful for Stage 2/3 checkpoints.
109
+ """
110
+ return preprocess_quadrant(gabor_patch, quadrant)
111
+
112
+
113
+ @torch.no_grad()
114
+ def predict_valence(model: torch.nn.Module, input_tensor: torch.Tensor, device: str = "cpu") -> float:
115
+ """Run the model and rescale its sigmoid output from [0, 1] to the [1, 9] IAPS valence scale."""
116
+ output = model(input_tensor.to(device))
117
+ valence = 1 + output.item() * (9 - 1)
118
+ return valence
119
+
120
+
121
+ def main():
122
+ parser = argparse.ArgumentParser(description=__doc__)
123
+ parser.add_argument("--image", type=str, default=None, help="Path to a natural scene image (US-style input).")
124
+ parser.add_argument("--gabor", type=str, default=None, help="Path to a Gabor patch image (CS-only input).")
125
+ parser.add_argument("--checkpoint", choices=list(CHECKPOINT_FILENAMES), default="post",
126
+ help="Which checkpoint to load (see CHECKPOINT_FILENAMES). Default: 'post' "
127
+ "(final, post-conditioning model).")
128
+ parser.add_argument("--device", type=str, default="cpu")
129
+ args = parser.parse_args()
130
+
131
+ if not args.image and not args.gabor:
132
+ parser.error("Provide --image or --gabor.")
133
+ if args.gabor and args.checkpoint in FULL_FRAME_CHECKPOINTS:
134
+ parser.error(f"--gabor (CS-only input) isn't meaningful for checkpoint '{args.checkpoint}', "
135
+ f"which was trained on full-frame images without a CS. Use --image instead, or "
136
+ f"pick a Stage 2/3 checkpoint (pre, post_epoch1, post).")
137
+
138
+ model = load_model(checkpoint=args.checkpoint, device=args.device)
139
+
140
+ if args.image and args.checkpoint in FULL_FRAME_CHECKPOINTS:
141
+ input_tensor = preprocess_natural_image(Image.open(args.image))
142
+ elif args.image:
143
+ input_tensor = preprocess_quadrant(Image.open(args.image))
144
+ else:
145
+ input_tensor = preprocess_cs_only(Image.open(args.gabor))
146
+
147
+ valence = predict_valence(model, input_tensor, device=args.device)
148
+ print(f"Predicted valence (1=extreme displeasure, 9=extreme pleasure): {valence:.2f}")
149
+
150
+
151
+ if __name__ == "__main__":
152
+ main()
vca_IAPS_batch10_lr2e-4_epoch23.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a09e005aeccac3e5c3d279b451af26d1a4b4bde05009e8994b5f2160950953ef
3
+ size 1162295022
vca_ckvideo_batch128_lr2e-5_epoch20.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:15ab5c5a46f32cb6515419b516d6f6ab1ae9186a3e38d9e6f81435f93467d5b2
3
+ size 1162295201