TempoPFN: Synthetic Pre-Training of Linear RNNs for Zero-Shot Time Series Forecasting

---

TempoPFN introduced in TempoPFN: Synthetic Pre-Training of Linear RNNs for Zero-Shot Time Series Forecasting, is a univariate time series foundation model pretrained entirely on synthetic data. It delivers top-tier zero-shot forecasting accuracy while remaining fully reproducible and free from real-data leakage.

Built on a Linear RNN (GatedDeltaProduct) backbone, TempoPFN performs end-to-end forecasting without patching or windowing. Its design enables fully parallelizable training and inference while maintaining stable temporal state-tracking across long sequences. The GatedDeltaProduct architecture is based on DeltaProduct, extended with state-weaving for time series forecasting. For detailed information about the architecture and custom modifications, see src/models/gated_deltaproduct/README.md.

This repository includes the pretrained 38M parameter model (models/checkpoint_38M.pth), all training and inference code, and the complete synthetic data generation pipeline used for pretraining.

📋 Model Details

Parameters	38M
Architecture	Linear RNN — GatedDeltaProduct with state-weaving
Pretraining data	100% synthetic (no real-world data, no benchmark leakage)
Task	Univariate zero-shot time series forecasting
Checkpoint	models/checkpoint_38M.pth
Paper	arXiv:2510.25502
License	Apache 2.0 (academic and commercial use)

✨ Why TempoPFN?

• High Performance, No Real Data: Achieves top-tier competitive results on GIFT-Eval, outperforming all existing synthetic-only approaches and surpassing the vast majority of models trained on real-world data. This ensures full reproducibility and eliminates benchmark leakage.
• Parallel and Efficient: The linear recurrence design enables full-sequence parallelization. This gives us the best of both worlds: the linear efficiency of an RNN, but with the training parallelism of a Transformer.
• Open and Reproducible: Includes the full synthetic data pipeline, configurations, and scripts to reproduce training from scratch.
• State-Tracking Stability: The GatedDeltaProduct recurrence and state-weaving mechanism preserve temporal continuity and information flow across long horizons, improving robustness without non-linear recurrence.

⚙️ Installation

Requires Git LFS. The checkpoint (models/checkpoint_38M.pth) is stored with Git LFS. Without it, git clone gives you a small pointer file instead of the weights.

# Ubuntu / Debian
sudo apt-get install git-lfs
# macOS
brew install git-lfs

git lfs install

# 1. Clone the repository (weights included via Git LFS)
git clone https://huggingface.co/AutoML-org/TempoPFN
cd TempoPFN

# 2. Create and activate a virtual environment (Python 3.12 recommended)
python3 -m venv venv && source venv/bin/activate
export PYTHONPATH=$PWD

# 3. Install PyTorch matching your CUDA version (example: CUDA 12.8)
pip install torch --index-url https://download.pytorch.org/whl/cu128

# 4. Install TempoPFN
pip install .
# Optional extras for training and synthetic data generation.
# Note: quote the brackets — zsh (default on macOS) treats them as globs.
pip install '.[dev]'

# 5. Run the quick-start script
python examples/quick_start_tempo_pfn.py

Prefer a notebook? Run jupyter notebook examples/quick_start_tempo_pfn.ipynb instead of step 5.

Hardware & Performance Tips

GPU required: Inference needs a CUDA-capable GPU with a matching PyTorch build. Tested on NVIDIA A100/H100.

First run: The first inference at a new sequence length is slow while Triton compiles its kernels. Subsequent runs are fast.

Cache Tip: If using a network filesystem, prevent slowdowns by routing caches to a local directory (like /tmp) before running:

LOCAL_CACHE_BASE="${TMPDIR:-/tmp}/tsf-$(date +%s)"
mkdir -p "${LOCAL_CACHE_BASE}/triton" "${LOCAL_CACHE_BASE}/torchinductor"
export TRITON_CACHE_DIR="${LOCAL_CACHE_BASE}/triton"
export TORCHINDUCTOR_CACHE_DIR="${LOCAL_CACHE_BASE}/torchinductor"

python examples/quick_start_tempo_pfn.py

🚂 Training

All training and model parameters are controlled via YAML files in configs/.

# Single-GPU (Debug)
torchrun --standalone --nproc_per_node=1 src/training/trainer_dist.py --config ./configs/train.yaml

# Multi-GPU (e.g., 8 GPUs)
torchrun --standalone --nproc_per_node=8 src/training/trainer_dist.py --config ./configs/train.yaml

💾 Synthetic Data Generation

A core contribution of this work is our open-source synthetic data pipeline, located in src/synthetic_generation/. It combines diverse generators with a powerful augmentation cascade.

Generators Used:

• Adapted Priors: ForecastPFN, KernelSynth, GaussianProcess (GP), and CauKer (Structural Causal Models).
• Novel Priors: SDE (a flexible regime-switching Ornstein-Uhlenbeck process), Sawtooth, StepFunction, Anomaly, Spikes, SineWave, and Audio-Inspired generators (Stochastic Rhythms, Financial Volatility, Network Topology, Multi-Scale Fractals).

You can easily generate your own data by installing the development dependencies and instantiating a generator wrapper. See examples/generate_synthetic_data.py for a minimal script, or inspect the generator code in src/synthetic_generation/.

🤝 License

This project is licensed under the Apache 2.0 License. See the LICENSE file for details. This permissive license allows for both academic and commercial use.

📚 Citation

If you find TempoPFN useful in your research, please consider citing our paper:

@misc{moroshan2025tempopfn,
  title={TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-Shot Time Series Forecasting}, 
  author={Vladyslav Moroshan and Julien Siems and Arber Zela and Timur Carstensen and Frank Hutter},
  year={2025},
  eprint={2510.25502},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}