---
license: cc-by-nc-4.0
datasets:
- nkp37/OpenVid-1M
tags:
- motion
- planning
---
Schrödinger's Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics
Timy Phan* · Jannik Wiese* · Björn Ommer
CompVis Group @ LMU Munich, Munich Center for Machine Learning (MCML)
ECCV 2026
*Equal contribution
[](https://compvis.github.io/schroedingers_cat/)
[](https://arxiv.org/abs/2607.25984)
[](https://github.com/CompVis/schroedingers_cat)
Official Code for the paper "Schrödinger's Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics" accepted at ECCV 2026.
## 💡 TL;DR
GARFIELD learns a structured latent representation of possible scene kinematics from an image and optional sparse spatio-temporal constraints. The representation supports both joint trajectory sampling and direct point-wise density estimation.
## 📝 Overview
Goal Aware Representations of Future kInEmatic Latent Distributions (GARFIELD) represents future scene kinematics as localized motion distributions. Given an initial image and optional sparse constraints, a joint encoder produces spatio-temporal latents for scene elements over time. These latents encode uncertainty about where each element may move, and separate decoders expose that uncertainty as point samples, coherent trajectory rollouts, or probability heatmaps.
The full pipeline consists of four components:
- **Encoder:** The joint encoder combines image features and sparse kinematic constraints into structured latents, each tied to one scene element and timestep.
- **Pointwise Decoder:** The pointwise decoder trains the latents to represent localized distributions by sampling individual future track positions from each latent component.
- **Full Decoder:** The full decoder samples complete trajectories jointly, preserving dependencies across points, timesteps, and scene elements for coherent motion realizations.
- **Density Decoder:** The density decoder deterministically maps localized latents to probability heatmaps, enabling fast uncertainty inspection without Monte-Carlo sampling.
## 📊 Results
### Motion Planning
GARFIELD infers accurate motion from very sparse information and, with only four conditioning points, outperforms Motion-I2V using sixteen.
### Direct Density Decoding
The density decoder achieves superior discrete energy scores while estimating motion densities orders of magnitude faster than Monte-Carlo sampling.
### Entropy-informed Conditioning
Selecting additional constraints by entropy collapses uncertainty without ground-truth errors and reaches comparable performance with fewer conditioning points.
## 🛠️ Usage
### Weights and Inference Setup
For inference, you can clone the GitHub repository by running:
```bash
git clone https://github.com/CompVis/schroedingers_cat
cd schroedingers_cat
```
Then, download pretrained model weights from 🤗 huggingface running:
```bash
hf download CompVis/schroedingers_cat --include "*.pt" --local-dir ckpts
```
Finally, create an environment to run the repository:
Conda (recommended):
```bash
conda create -n garfield python=3.12 -y
conda activate garfield
python -m pip install --upgrade pip
pip install -r requirements.txt
```
Virtual environment:
```bash
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt
```
## Citation
```bibtex
@inproceedings{phan2026schrodingerscat,
title = {Schrödinger's Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics},
author = {Phan, Timy and Wiese, Jannik and Ommer, Björn},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026}
}
```