VoxStruct3D: Structure First, Image Follows for High-Fidelity 3D MRI Generation
Keywords: 3D MRI Generation, Voxel-Space Generation, Flow Matching, Structure Guidance
Project Page: VoxStruct3D
Code: GitHub
High-fidelity 3D MRI generation requires two properties at the same time: globally coherent anatomy and sharp, fine-grained voxel details.
Most existing methods perform diffusion or flow matching in a compressed latent space. This substantially reduces the computational cost of volumetric generation, but it also introduces an image-autoencoder bottleneck. Once subtle anatomical boundaries are weakened during encoding and decoding, the generative model may not be able to recover them.
We introduce VoxStruct3D, a flow-matching framework that generates full-resolution 3D MRI volumes directly in voxel space, without an image autoencoder. To address the main challenges of direct voxel-space generation, we develop two complementary components: the Volumetric Voxel Generator (VVG) and Structure First, Image Follows (SFIF).
Two challenges in voxel-space 3D generation
Local continuity: neighboring patches can create block artifacts
A straightforward voxel-space design divides a 3D volume into patches, models their interactions with a Transformer, and independently maps each output token back to a voxel block.
This independent reconstruction is particularly problematic in 3D. Neighboring 2D patches meet along an edge, while neighboring 3D patches meet across an entire 2D face. Even when the Transformer captures global token interactions, independently decoded blocks can still produce visible grid-aligned seams.
Global coherence: recovering anatomy directly from noise is difficult
Latent representations do more than compress an image: they also provide a compact structural abstraction. Once the image autoencoder is removed, the model must recover global brain anatomy directly from a high-dimensional noisy volume.
At high noise levels, the image stream contains little reliable anatomical information. If sampling enters an invalid global configuration early, later local refinement may be unable to correct it, leading to severely distorted anatomy.
The core design of VoxStruct3D
We address these two challenges with:
- Volumetric Voxel Generator (VVG): an overlapping voxel decoder that allows neighboring tokens to jointly reconstruct shared regions and improves local continuity.
- Structure First, Image Follows (SFIF): a compact structure stream that evolves ahead of the image stream and provides progressive, one-way anatomical guidance.
VVG focuses on how local boundaries are reconstructed, while SFIF focuses on how global anatomy is established.
VVG: jointly reconstructing neighboring voxel regions
Conventional unpatchify operations independently map each output token to a non-overlapping 3D block. VVG replaces this operation with an overlapping volumetric decoder built from multiple 3D transposed-convolution stages. Each kernel is larger than its stride, so neighboring tokens contribute to shared voxel regions.
As a result, a patch boundary is no longer determined by a single token. It is jointly reconstructed and locally fused from multiple neighboring predictions.
VVG further combines:
- time-modulated residual refinement to adapt reconstruction to different noise levels;
- U-Net-style skip fusion to preserve fine-grained features from the patch encoder;
- a DiT backbone to model global interactions between image and structure tokens.
In our BraTS 2021 ablation, overlapping upsampling reduces 3DINO FID from 43.41 to 19.48, and the complete VVG further improves it to 12.89. Without VVG, the generated volumes exhibit visible artifacts aligned with the token grid.
SFIF: structure first, image follows
VVG improves local reconstruction, but it does not by itself guarantee globally valid anatomy. We therefore introduce Structure First, Image Follows (SFIF).
We first extract features with a frozen pretrained 3DINO medical encoder. A StructVAE then compresses these features into compact structure tokens. StructVAE acts as a learned low-pass filter: it removes unnecessary appearance variation while retaining the dominant anatomical information needed to guide generation.
The image and structure streams are generated jointly by a shared DiT, but they follow different clocks. The structure stream always stays ahead of the image stream, allowing a progressively clarified anatomical state to guide voxel formation.
SFIF also includes two important designs:
- Patch-Aligned RoPE (PA-RoPE) maps image and structure tokens from unequal grids into a shared 3D coordinate system.
- Asymmetric attention allows image tokens to attend to structure tokens, while preventing the less advanced image state from contaminating the structure stream.
The structure stream serves only as an internal anatomical guide. The final MRI is still generated entirely by the image branch in the original voxel domain. Both streams evolve in a single coupled trajectory, so SFIF does not require a separate structure-generation pass.
What is new?
1. Full-resolution 3D MRI generation without an image autoencoder
VoxStruct3D applies flow matching directly in voxel space, avoiding the irreversible loss of subtle boundaries and local details that can occur during image compression.
2. Overlapping decoding for 3D patch boundaries
VVG does more than increase decoder capacity. It changes the mapping between tokens and voxel regions so that neighboring tokens jointly reconstruct patch interfaces.
3. A structure-leading joint trajectory
SFIF does not use anatomy as a static condition. Instead, the image and structure states evolve together while the structure stream remains temporally ahead, providing increasingly clear guidance throughout sampling.
4. Spatially aligned, one-way structure guidance
PA-RoPE aligns unequal 3D token grids, while asymmetric attention controls the direction of information flow. Together, they allow the image stream to use structural evidence without injecting unstable appearance information back into the compact structure representation.
Results
We evaluate VoxStruct3D on pathological and healthy T1-weighted brain MRI and compare it with HA-GAN, 3D-LDM, 3D MedDiffusion, WDM, and MOTFM.
VoxStruct3D achieves a strong overall balance of feature-distribution alignment, sample diversity, and perceptual quality:
- Pathological MRI: 3DINO FID of 12.89 and MS-SSIM of 0.7413.
- Healthy MRI: 3DINO FID of 8.476 and MS-SSIM of 0.7320.
We also study whether the generated data retain information useful for downstream classification. Across three classifiers, synthetic-only training trails real-only training by just 1.97 accuracy points on average. Combining real and synthetic data improves average accuracy and balanced accuracy by 2.09 and 2.19 points, respectively.
These results suggest that the generated MRIs are not only visually realistic but also preserve condition-discriminative information.
Conclusion
The main idea behind VoxStruct3D is simple: preserve detail by generating in the original voxel space, while allowing anatomy to form ahead of the image.
VVG improves local continuity through overlapping volumetric decoding, and SFIF promotes global anatomical coherence through a structure-leading joint trajectory. Together, they address the local block artifacts and global structural failures that arise in direct voxel-space generation.
Inference code, checkpoints, and generated samples are available in our repository. We hope VoxStruct3D will be useful to researchers working on 3D medical-image generation, flow matching, and synthetic data for medical AI.
Paper: https://arxiv.org/abs/2608.04557
Project Page: https://neesky.github.io/VoxStruct3D/
Code: https://github.com/Neesky/VoxStruct3D