---
license: apache-2.0
tags:
  - video
  - relighting
  - inverse-rendering
  - diffusion
  - cosmos
pipeline_tag: image-to-image
---

# World Inverse Renderer

Video inverse rendering model based on NVIDIA Cosmos 7B video diffusion transformer, fine-tuned on [custom dataset](https://github.com/ShandaAI/AlayaRenderer?tab=readme-ov-file).

## Model Description

This model performs **inverse rendering** on images and videos: given an input RGB frame, it estimates physically-based G-buffer maps:

- **Basecolor** (albedo)
- **Normal** (surface normals)
- **Depth**
- **Roughness**
- **Metallic**

These G-buffers can then be used with a forward renderer to relight the scene under arbitrary environment lighting (HDRI maps).

## Architecture

- Based on NVIDIA Cosmos 7B video diffusion transformer
- Fine-tuned on [custom dataset](https://github.com/ShandaAI/AlayaRenderer?tab=readme-ov-file)
- Supports both single-image and multi-frame video inverse rendering

## Usage

```bash
# Inverse rendering on images
CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) python cosmos_predict1/diffusion/inference/inference_inverse_renderer.py \
    --checkpoint_dir checkpoints --diffusion_transformer_dir Diffusion_Renderer_Inverse_Cosmos_7B \
    --dataset_path=your_input_images/ --num_video_frames 1 --group_mode webdataset \
    --video_save_folder=output/ --save_video=False

# Inverse rendering on video frames
CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) python cosmos_predict1/diffusion/inference/inference_inverse_renderer.py \
    --checkpoint_dir checkpoints --diffusion_transformer_dir Diffusion_Renderer_Inverse_Cosmos_7B \
    --dataset_path=your_video_frames/ --num_video_frames 57 \
    --video_save_folder=output/
```

## Requirements

- Python 3.10
- NVIDIA GPU with >= 16GB VRAM (48GB+ recommended)
- CUDA 12.0+