AffectFlow-DINO

Uncertainty-aware multi-task facial affect estimation for the 11th ABAW Multi-Task Learning challenge: DINOv3 backbone + deterministic VA/expression/AU heads + a conditional rectified-flow head over the joint 22-dimensional affect vector.

Code, training scripts, and the full 28-experiment ablation study: github.com/Bekhouche/AffectFlow-DINO.

Paper: AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow — Salah Eddine Bekhouche, Abdellah Zakaria Sellam, Fadi Dornaika, Abdenour Hadid. arXiv:2607.13250, 2026.

Quickstart

pip install torch torchvision transformers huggingface_hub pillow
git clone https://github.com/Bekhouche/AffectFlow-DINO.git && cd AffectFlow-DINO
python inference.py --model finetune-flow-retune-b10 --image face.jpg

Models in this repo

Each subfolder is one checkpoint: model.pt (weights + architecture config) plus, where applicable, au_thresholds.json and/or expr_weights.json post-hoc calibration files that inference.py applies automatically.

Folder	Backbone	Best decode	P_MTL (calibrated)	P_VA	P_EXPR	P_AU
finetune-flow-retune-b10	ViT-S/16, fine-tuned	Det	1.177	0.325	0.350	0.502
vitb-finetune	ViT-B/16, fine-tuned	Det	1.116	0.354	0.255	0.507
finetune-flow-retune-b05	ViT-S/16, fine-tuned	Det / Flow	1.062 / 0.956	0.291	0.303	0.469
finetune	ViT-S/16, fine-tuned	Det	1.101	0.318	0.285	0.497
au-posw-cw	ViT-S/16, frozen	Det	0.890 (no cal.)	0.211	0.240	0.439
psp	ViT-S/16, frozen (patch soft-pool)	Det	0.859 (no cal.)	0.235	0.236	0.389
affectflow-base	ViT-S/16, frozen	Flow	0.888	0.237	0.210	0.441
det-baseline	ViT-S/16, frozen	Det	0.793 (no cal.)	0.199	0.216	0.378

P_MTL = P_VA + P_EXPR + P_AU is the official ABAW MTL composite metric (P_VA = mean CCC of valence/arousal, P_EXPR = expression macro-F1, P_AU = mean AU F1), evaluated on the s-Aff-Wild2 validation split. Official challenge baseline: P_MTL = 0.450.

These 8 checkpoints are a curated subset of the ~30 ablation runs behind the paper — one per architecturally distinct configuration. The full ablation table, including sweeps that reused one of these architectures with different loss weights/regularization, is in the GitHub repo's EXPERIMENTS.md.

Input / output

• Input: a cropped, roughly frontal face image, resized to 224x224 and ImageNet-normalized (handled automatically by inference.py).
• Output: valence, arousal in [-1, 1]; one of 8 expressions (Neutral, Anger, Disgust, Fear, Happiness, Sadness, Surprise, Other); a subset of 12 active Action Units (AU1, AU2, AU4, AU6, AU7, AU10, AU12, AU15, AU23, AU24, AU25, AU26).
• Two decode modes: deterministic (task heads directly) or flow (average of N sampled rectified-flow trajectories — enables uncertainty estimation via trajectory spread).

Limitations

Trained and validated only on s-Aff-Wild2 (in-the-wild but frame-level, no temporal context). Expression prediction remains the weakest task (P_EXPR <= 0.35): Fear and Sadness are rare classes that only partially recover under calibration. Not evaluated on demographic subgroups; treat outputs as research signals, not clinical or high-stakes decisions.

Citation

@article{bekhouche2026affectflowdino,
  title         = {AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow},
  author        = {Bekhouche, Salah Eddine and Sellam, Abdellah Zakaria and Dornaika, Fadi and Hadid, Abdenour},
  journal       = {arXiv preprint arXiv:2607.13250},
  year          = {2026}
}

License

MIT.