douyeszn/sparserenderformer-gp-sparse

SparseRenderFormer (GP-Sparse)

A retrofit of RenderFormer that replaces the dense view-independent self-attention with a block-sparse pattern derived from the scene's bounding volume hierarchy.

The idea

Sparse attention normally gets its block structure from sequence position: nearby tokens are likely to interact. RenderFormer's view-independent stage is permutation invariant, so index adjacency says nothing about whether two triangles exchange light. The block structure is therefore built from geometry instead: a median-split BVH over triangle centroids permutes triangles so each leaf is a contiguous run of slots, which lets leaf index serve as kernel block index.

Three branches are unioned into one mask and evaluated by a single fused kernel:

• coarse — each leaf pooled by triangle area into one token, visible to every query. Not optional: without it a triangle cannot see a light source outside its own neighbourhood, and training cannot recover a token it never attends to.
• selected — a learned scorer retrieves additional leaves per query block.
• local — the nearest leaves by block-centroid distance.

The differential form factor A_j cos(t_i) cos(t_j) / r^2 is added to the logits of retained pairs as a score modifier: no parameters, under 0.5% of runtime.

Speed

View-independent stage, A100 80GB, bfloat16, batch 1, 12 layers:

N	dense	this model	speed-up
8,192	14.2 ms / 1.47 GB	25.5 ms / 0.37 GB	0.6x
16,384	56.5 ms / 1.77 GB	44.4 ms / 0.58 GB	1.3x
32,768	220.9 ms / 2.38 GB	87.4 ms / 0.98 GB	2.5x

Crossover is near N = 14,000. Below it this model is slower than dense.

Note these cover the attention stage only. The feed-forward network is O(N) and does not shrink, so the model-level speed-up is lower and is not yet measured.

Quality, and its limits

Fine-tuned 3,000 steps against Blender Cycles references. 19.78 -> 27.35 dB, where dense RenderFormer scores 31.23 dB on the same references: a deficit of about 3.3 dB.

Two caveats that bound this number, both important:

1. Quality was measured where the method is slower. Every scene is at most 3,841 triangles, below the N = 14,000 crossover. Quality at 16k and 32k, where the speed-up applies, is not yet measured.
2. The deficit is dominated by training-set size, not by sparsity. A controlled sweep varying only the number of distinct training scenes moved PSNR by 4.53 dB: 58 scenes plateaus within 250 steps, while 250 and 1,000 scenes were still improving when the budget ended. This checkpoint was trained on 58 scenes.

Warm start

GP-Sparse keeps RenderFormer's query, key, value and output projections and changes only which keys each query may see. Splitting the pretrained fused in_proj into separate Q, K and V transfers 99.27% of the checkpoint (205.2M of 206.7M parameters); only the block scorer and branch gates are new.

This bounds the claim: it shows sparse attention can recover dense behaviour cheaply, not that it is natively trainable to that quality from scratch.

Files

• student_bf16.pt — bfloat16 weights, optimiser state stripped. Inference here runs under bfloat16 autocast, so this loses nothing that is used.

Loading

import torch
from huggingface_hub import hf_hub_download
from renderformer.models.config import RenderFormerConfig
from renderformer.models.renderformer import RenderFormer

path = hf_hub_download("douyeszn/sparserenderformer-gp-sparse", "student_bf16.pt")
ckpt = torch.load(path, map_location="cpu", weights_only=False)
model = RenderFormer(RenderFormerConfig(**ckpt["config"]))
model.load_state_dict(ckpt["model_state_dict"])

Code: the scene generator, Cycles pipeline and every experiment script are released alongside this model.

Data

Trained against douyeszn/sparserenderformer-cycles, path-traced references in RenderFormer's exact HDF5 schema. RenderFormer's own 2M-scene training set was never published.

Citing

This is work in progress; a preprint is not yet posted. Please cite RenderFormer (Zeng et al., SIGGRAPH 2025) for the base model.