Add model-card YAML metadata (license/pipeline_tag/base_model) + fix links
Browse files
README.md
CHANGED
|
@@ -1,3 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# TripoSplat → Core AI (zoo's first 3D)
|
| 2 |
|
| 3 |
[VAST-AI/TripoSplat](https://github.com/VAST-AI-Research/TripoSplat) — **single image → 3D Gaussian
|
|
@@ -7,24 +21,27 @@ viewer (e.g. Apple RealityKit on visionOS, or MetalSplatter on iOS/macOS).
|
|
| 7 |
Pure-PyTorch pipeline (no diffusers/CUDA kernels): bg-removal → DINOv3 ViT-H encode + Flux2-VAE encode
|
| 8 |
→ 20-step flow-matching DiT denoiser → octree probability sampler → Gaussian decoder → splats.
|
| 9 |
|
|
|
|
|
|
|
|
|
|
| 10 |
## What runs on Core AI
|
| 11 |
|
| 12 |
-
5 neural nets converted (each gated converted-vs-eager **cos = 1.000000**
|
| 13 |
|
| 14 |
-
| net | shape | bundle |
|
| 15 |
-
|---|---|---|
|
| 16 |
-
| DINOv3 ViT-H encoder | (1,3,1024,1024)→(1,4101,1280) | `
|
| 17 |
-
| Flux2-VAE encoder | (1,3,1024,1024)→(1,4096,128) | `
|
| 18 |
-
| DiT denoiser (one step) | latent(1,8192,16)+cam(1,1,5)+t+feat1(1,4101,1280)+feat2(1,4101,128)→latent,cam | `
|
| 19 |
-
| Octree probability decoder | x(1,8192,3)+l(1,)+cond(1,8192,16)→logits(1,8192,8) | `
|
| 20 |
-
| Decode (gs + build_gaussians + .ply activations, baked) | points(1,8192,3)+cond(1,8192,16)→(262144,14) | `
|
| 21 |
|
| 22 |
The flow-matching sampler (`FlowEulerCfgSampler`) and the octree `sample_probs` systematic resampling
|
| 23 |
stay **host-side** (data-dependent control flow). Scripts: `_conv_*.py` convert+gate each net;
|
| 24 |
`_conv_fp16.py` makes the half-size fp16 bundles; `_conv_decode.py` bakes build_gaussians + the
|
| 25 |
Gaussian `.ply`-activation math into one net so the runner just writes raw floats.
|
| 26 |
|
| 27 |
-
## model.py patches (the reusable contribution — see
|
| 28 |
|
| 29 |
coreai-torch 0.4.0 needed six edits to VAST's `model.py`; all are general gotchas:
|
| 30 |
|
|
@@ -46,9 +63,9 @@ artifact). Octree decoder: int64 `l` (resolution) input → CoreAIError 3 at run
|
|
| 46 |
## Running it
|
| 47 |
|
| 48 |
- **Mac**: `_run_coreai.py` (or `app_backend.py --input <img>`) loads the bundles via coreai.runtime
|
| 49 |
-
(`SpecializationOptions.default()` = GPU; ~
|
| 50 |
-
gate vs torch-DiT: **cos 0.999999**.
|
| 51 |
-
- **Mac app / iPhone client**: `
|
| 52 |
(capture on iPhone → Mac server `server.py` → view splats in MetalSplatter / RealityKit).
|
| 53 |
|
| 54 |
## On-device note
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
pipeline_tag: image-to-3d
|
| 4 |
+
tags:
|
| 5 |
+
- coreai
|
| 6 |
+
- core-ml
|
| 7 |
+
- apple-silicon
|
| 8 |
+
- gaussian-splatting
|
| 9 |
+
- image-to-3d
|
| 10 |
+
- 3d-generation
|
| 11 |
+
base_model:
|
| 12 |
+
- VAST-AI/TripoSplat
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
# TripoSplat → Core AI (zoo's first 3D)
|
| 16 |
|
| 17 |
[VAST-AI/TripoSplat](https://github.com/VAST-AI-Research/TripoSplat) — **single image → 3D Gaussian
|
|
|
|
| 21 |
Pure-PyTorch pipeline (no diffusers/CUDA kernels): bg-removal → DINOv3 ViT-H encode + Flux2-VAE encode
|
| 22 |
→ 20-step flow-matching DiT denoiser → octree probability sampler → Gaussian decoder → splats.
|
| 23 |
|
| 24 |
+
**This repo holds the Core AI `.aimodel` bundles** (each is a directory). Conversion + runner scripts
|
| 25 |
+
live in the [coreai-models-community](https://huggingface.co/mlboydaisuke) zoo (`conversion/triposplat/`).
|
| 26 |
+
|
| 27 |
## What runs on Core AI
|
| 28 |
|
| 29 |
+
5 neural nets converted (each gated converted-vs-eager **cos = 1.000000**):
|
| 30 |
|
| 31 |
+
| net | shape | bundle | dtype |
|
| 32 |
+
|---|---|---|---|
|
| 33 |
+
| DINOv3 ViT-H encoder | (1,3,1024,1024)→(1,4101,1280) | `dinov3_fp16.aimodel` | fp16 |
|
| 34 |
+
| Flux2-VAE encoder | (1,3,1024,1024)→(1,4096,128) | `vae_fp16.aimodel` | fp16 |
|
| 35 |
+
| DiT denoiser (one step) | latent(1,8192,16)+cam(1,1,5)+t+feat1(1,4101,1280)+feat2(1,4101,128)→latent,cam | `dit_fp16.aimodel` | fp16 |
|
| 36 |
+
| Octree probability decoder | x(1,8192,3)+l(1,)+cond(1,8192,16)→logits(1,8192,8) | `octree_fp32.aimodel` | fp32 |
|
| 37 |
+
| Decode (gs + build_gaussians + .ply activations, baked) | points(1,8192,3)+cond(1,8192,16)→(262144,14) | `decode_fp32.aimodel` | fp32 |
|
| 38 |
|
| 39 |
The flow-matching sampler (`FlowEulerCfgSampler`) and the octree `sample_probs` systematic resampling
|
| 40 |
stay **host-side** (data-dependent control flow). Scripts: `_conv_*.py` convert+gate each net;
|
| 41 |
`_conv_fp16.py` makes the half-size fp16 bundles; `_conv_decode.py` bakes build_gaussians + the
|
| 42 |
Gaussian `.ply`-activation math into one net so the runner just writes raw floats.
|
| 43 |
|
| 44 |
+
## model.py patches (the reusable contribution — see the zoo's conversion guide)
|
| 45 |
|
| 46 |
coreai-torch 0.4.0 needed six edits to VAST's `model.py`; all are general gotchas:
|
| 47 |
|
|
|
|
| 63 |
## Running it
|
| 64 |
|
| 65 |
- **Mac**: `_run_coreai.py` (or `app_backend.py --input <img>`) loads the bundles via coreai.runtime
|
| 66 |
+
(`SpecializationOptions.default()` = GPU; ~2 min/gen at 20 steps on Apple silicon, full quality).
|
| 67 |
+
End-to-end latent gate vs torch-DiT: **cos 0.999999**.
|
| 68 |
+
- **Mac app / iPhone client**: `TripoSplatMac` (standalone) and `TripoSplatPhone`
|
| 69 |
(capture on iPhone → Mac server `server.py` → view splats in MetalSplatter / RealityKit).
|
| 70 |
|
| 71 |
## On-device note
|