Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,183 @@
+---
+license: mit
+tags:
+  - facial-reaction-generation
+  - dyadic-interaction
+  - diffusion-model
+  - affective-computing
+  - react2026
+  - mafrg
+language:
+  - en
+pipeline_tag: other
+---
+
+# CaReDiff — Causal Reaction Diffusion (REACT 2026)
+
+Model checkpoints for **CaReDiff**, a submission to the [REACT 2026 Challenge](https://sites.google.com/view/react2026/home) (ACM MM 2026, Multiple Appropriate Facial Reaction Generation).
+
+Given speaker behaviour (audio, video, 3DMM coefficients, facial attributes), CaReDiff generates multiple appropriate listener facial reactions (25-d: 15 AUs + valence/arousal + 8 expressions) with an auxiliary EEG prediction head.
+
+- **Code:** https://github.com/smu-ivpl/CaReDiff
+- **Checkpoints (this repo):** https://huggingface.co/IVPL/CaReDiff
+
+## Tracks
+
+| Track | Architecture | Description |
+|-------|-------------|-------------|
+| Generic Online | PerFRDiff + EEG | Diffusion-based generation over autoregressive windows |
+| Generic Offline | PerFRDiff + EEG | Diffusion-based full-sequence generation |
+| Personalised Online | PerFRDiff + PRA + EEG | Frozen generic backbone + Personalised Residual Adapter (autoregressive windows) |
+| Personalised Offline | PerFRDiff + PRA + EEG | Frozen generic backbone + Personalised Residual Adapter (full-sequence) |
+
+## Repository Layout
+
+```
+CaReDiff/
+├── generic/
+│   ├── online/          prior + denoiser + EEG head (checkpoint_120.pth)
+│   └── offline/         prior + denoiser + EEG head (checkpoint_120.pth)
+└── personalised/
+    ├── online/          shared backbone + 3 adapters (personality / lhfb / both)
+    └── offline/         shared backbone + 3 adapters (personality / lhfb / both)
+```
+
+Each generic track folder contains:
+
+```
+<track>/
+├── DiffusionPriorNetwork/checkpoint_120.pth
+├── CausalTransformerDenoiser/checkpoint_120.pth
+└── EEGPredictionHead/checkpoint_120.pth
+```
+
+## Usage
+
+1. Clone the code repository:
+   ```bash
+   git clone https://github.com/smu-ivpl/CaReDiff
+   ```
+2. Set up the environment (see the [code README](https://github.com/smu-ivpl/CaReDiff)):
+   ```bash
+   conda create -n react python=3.10 && conda activate react
+   conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia
+   conda install -c fvcore -c iopath -c conda-forge fvcore iopath
+   pip install -r generic/code/requirements.txt
+   ```
+3. Download the checkpoints from this repo and place them under `save/`. For the generic tracks:
+   ```
+   save/motion_diffusion/react_2025/
+   ├── online/checkpoints/pretrained/
+   │   ├── DiffusionPriorNetwork/checkpoint_120.pth
+   │   ├── CausalTransformerDenoiser/checkpoint_120.pth
+   │   └── EEGPredictionHead/checkpoint_120.pth
+   └── offline/checkpoints/pretrained/
+       └── ... (same structure)
+   ```
+4. Run evaluation (see below).
+
+Full placement instructions, SHA-256 checksums, and per-track details are in each variant's `checkpoints/README.md` in the code repository.
+
+### Generic Evaluation
+
+```bash
+# Generic Online
+python main.py --config-name generic_online/motion_diffusion \
+    stage=test data_dir=./datasets/REACT2026/ \
+    trainer.batch_size=1 resume_id=pretrained
+
+# Generic Offline
+python main.py --config-name generic_offline/motion_diffusion \
+    stage=test data_dir=./datasets/REACT2026/ \
+    trainer.batch_size=1 resume_id=pretrained
+```
+
+### Personalised Evaluation
+
+Each personalised track uses a frozen generic backbone plus one of three
+condition adapters (`personality` / `lhfb` / `both`). Download the
+`personalised/<track>/` folder and set `PKG` to its parent absolute path; the
+commands reference every weight explicitly through `PKG`.
+
+```bash
+PKG=<absolute path of the downloaded personalised folder>
+
+# Personalised Offline (personality condition)
+python main.py --config-name g2p_delta stage=test task=offline \
+  data_dir=<MARS_ROOT> run_id=eval_offline_personality \
+  trainer.batch_size=4 num_gts=10 \
+  trainer.generic.eval_condition_mode=matched \
+  trainer.generic.eval_eeg=false \
+  trainer.main_model.args.personal_condition_mode=personality_only \
+  resume_id=personality \
+  trainer.ckpt_dir=$PKG/offline/adapters \
+  trainer.pretrained.diffusion_decoder=$PKG/offline/backbone/CausalTransformerDenoiser/checkpoint_120.pth \
+  trainer.pretrained.diffusion_prior=$PKG/offline/backbone/DiffusionPriorNetwork/checkpoint_120.pth \
+  trainer.pretrained.eeg_head_checkpoint=$PKG/offline/backbone/EEGPredictionHead/checkpoint_120.pth
+
+# Personalised Online (personality condition)
+python main.py --config-name g2p_delta_online stage=test task=online \
+  data_dir=<MARS_ROOT> run_id=eval_online_personality \
+  trainer.batch_size=4 num_gts=10 \
+  trainer.generic.eval_condition_mode=matched \
+  trainer.generic.eval_eeg=false \
+  trainer.main_model.args.personal_condition_mode=personality_only \
+  trainer.model.diff_model.diffusion_decoder.args.past_l_emotion_drop_prob=0.2 \
+  resume_id=personality \
+  trainer.ckpt_dir=$PKG/online/adapters \
+  trainer.pretrained.diffusion_decoder=$PKG/online/backbone/CausalTransformerDenoiser/checkpoint_120.pth \
+  trainer.pretrained.diffusion_prior=$PKG/online/backbone/DiffusionPriorNetwork/checkpoint_120.pth \
+  trainer.pretrained.eeg_head_checkpoint=$PKG/online/backbone/EEGPredictionHead/checkpoint_120.pth
+```
+
+For the `lhfb` / `both` conditions, change `resume_id` (`lhfb` or `both`) and
+`personal_condition_mode` (`3dmm_only` or `3dmm_personality`) accordingly. For
+the online track, keep `past_l_emotion_drop_prob=0.2` — it enables the
+past-listener conditioning pathway and is required to reproduce the reported
+numbers.
+
+## Additional Requirements
+
+- **Post-processor (EmotionVAE) checkpoint** — required for evaluation on every `stage=test` run. Obtain from the [official REACT 2026 baseline](https://github.com/reactmultimodalchallenge/baseline_react2026) and place at `pretrained_models/post_processor/checkpoint.pth`.
+- **PIRender + FaceVerse** — needed only for FRREa (rendered-frame FID) evaluation.
+- **MARS dataset** — obtain through the challenge organisers.
+
+## Metrics
+
+| Metric | Description |
+|--------|-------------|
+| FRC ↑ | Facial Reaction Consistency |
+| FRD ↓ | Facial Reaction Distance (FID-based) |
+| TLCC ↑ | Time-Lagged Cross-Correlation |
+| S_MSE ↓ | Smoothness (MSE of consecutive frames) |
+| FRVar ↑ | Facial Reaction Variance |
+| FRREa ↓ | Facial Reaction Realism (FID on rendered frames) |
+
+## Citation
+
+```bibtex
+@article{song2023multiple,
+  title={Multiple Appropriate Facial Reaction Generation in Dyadic Interaction Settings: What, Why and How?},
+  author={Song, Siyang and Spitale, Micol and Luo, Yiming and Bal, Batuhan and Gunes, Hatice},
+  journal={arXiv preprint arXiv:2302.06514},
+  year={2023}
+}
+
+@inproceedings{song2025react,
+  title={React 2025: the third multiple appropriate facial reaction generation challenge},
+  author={Song, Siyang and Spitale, Micol and Kong, Xiangyu and Zhu, Hengde and Luo, Cheng and Palmero, Cristina and Barquero, German and others},
+  booktitle={Proceedings of the 33rd ACM International Conference on Multimedia},
+  pages={13979--13984},
+  year={2025}
+}
+```
+
+## Acknowledgement
+
+- [FaceVerse](https://github.com/LizhenWangT/FaceVerse)
+- [PIRender](https://github.com/RenYurui/PIRender)
+- [REACT 2026 Baseline](https://github.com/reactmultimodalchallenge/baseline_react2026)
+
+## License
+
+MIT