mlboydaisuke commited on
Commit
b228317
·
verified ·
1 Parent(s): 3e1f4f7

Add model-card YAML metadata (license/pipeline_tag/base_model) + fix links

Browse files
Files changed (1) hide show
  1. README.md +29 -12
README.md CHANGED
@@ -1,3 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  # TripoSplat → Core AI (zoo's first 3D)
2
 
3
  [VAST-AI/TripoSplat](https://github.com/VAST-AI-Research/TripoSplat) — **single image → 3D Gaussian
@@ -7,24 +21,27 @@ viewer (e.g. Apple RealityKit on visionOS, or MetalSplatter on iOS/macOS).
7
  Pure-PyTorch pipeline (no diffusers/CUDA kernels): bg-removal → DINOv3 ViT-H encode + Flux2-VAE encode
8
  → 20-step flow-matching DiT denoiser → octree probability sampler → Gaussian decoder → splats.
9
 
 
 
 
10
  ## What runs on Core AI
11
 
12
- 5 neural nets converted (each gated converted-vs-eager **cos = 1.000000**, fp16):
13
 
14
- | net | shape | bundle |
15
- |---|---|---|
16
- | DINOv3 ViT-H encoder | (1,3,1024,1024)→(1,4101,1280) | `dinov3` |
17
- | Flux2-VAE encoder | (1,3,1024,1024)→(1,4096,128) | `vae` |
18
- | DiT denoiser (one step) | latent(1,8192,16)+cam(1,1,5)+t+feat1(1,4101,1280)+feat2(1,4101,128)→latent,cam | `dit` |
19
- | Octree probability decoder | x(1,8192,3)+l(1,)+cond(1,8192,16)→logits(1,8192,8) | `octree` |
20
- | Decode (gs + build_gaussians + .ply activations, baked) | points(1,8192,3)+cond(1,8192,16)→(262144,14) | `decode` |
21
 
22
  The flow-matching sampler (`FlowEulerCfgSampler`) and the octree `sample_probs` systematic resampling
23
  stay **host-side** (data-dependent control flow). Scripts: `_conv_*.py` convert+gate each net;
24
  `_conv_fp16.py` makes the half-size fp16 bundles; `_conv_decode.py` bakes build_gaussians + the
25
  Gaussian `.ply`-activation math into one net so the runner just writes raw floats.
26
 
27
- ## model.py patches (the reusable contribution — see ../../knowledge/conversion-guide.md)
28
 
29
  coreai-torch 0.4.0 needed six edits to VAST's `model.py`; all are general gotchas:
30
 
@@ -46,9 +63,9 @@ artifact). Octree decoder: int64 `l` (resolution) input → CoreAIError 3 at run
46
  ## Running it
47
 
48
  - **Mac**: `_run_coreai.py` (or `app_backend.py --input <img>`) loads the bundles via coreai.runtime
49
- (`SpecializationOptions.default()` = GPU; ~1 min/gen at 20 steps, full quality). End-to-end latent
50
- gate vs torch-DiT: **cos 0.999999**.
51
- - **Mac app / iPhone client**: `apps/` — `TripoSplatMac` (standalone) and `TripoSplatPhone`
52
  (capture on iPhone → Mac server `server.py` → view splats in MetalSplatter / RealityKit).
53
 
54
  ## On-device note
 
1
+ ---
2
+ license: mit
3
+ pipeline_tag: image-to-3d
4
+ tags:
5
+ - coreai
6
+ - core-ml
7
+ - apple-silicon
8
+ - gaussian-splatting
9
+ - image-to-3d
10
+ - 3d-generation
11
+ base_model:
12
+ - VAST-AI/TripoSplat
13
+ ---
14
+
15
  # TripoSplat → Core AI (zoo's first 3D)
16
 
17
  [VAST-AI/TripoSplat](https://github.com/VAST-AI-Research/TripoSplat) — **single image → 3D Gaussian
 
21
  Pure-PyTorch pipeline (no diffusers/CUDA kernels): bg-removal → DINOv3 ViT-H encode + Flux2-VAE encode
22
  → 20-step flow-matching DiT denoiser → octree probability sampler → Gaussian decoder → splats.
23
 
24
+ **This repo holds the Core AI `.aimodel` bundles** (each is a directory). Conversion + runner scripts
25
+ live in the [coreai-models-community](https://huggingface.co/mlboydaisuke) zoo (`conversion/triposplat/`).
26
+
27
  ## What runs on Core AI
28
 
29
+ 5 neural nets converted (each gated converted-vs-eager **cos = 1.000000**):
30
 
31
+ | net | shape | bundle | dtype |
32
+ |---|---|---|---|
33
+ | DINOv3 ViT-H encoder | (1,3,1024,1024)→(1,4101,1280) | `dinov3_fp16.aimodel` | fp16 |
34
+ | Flux2-VAE encoder | (1,3,1024,1024)→(1,4096,128) | `vae_fp16.aimodel` | fp16 |
35
+ | DiT denoiser (one step) | latent(1,8192,16)+cam(1,1,5)+t+feat1(1,4101,1280)+feat2(1,4101,128)→latent,cam | `dit_fp16.aimodel` | fp16 |
36
+ | Octree probability decoder | x(1,8192,3)+l(1,)+cond(1,8192,16)→logits(1,8192,8) | `octree_fp32.aimodel` | fp32 |
37
+ | Decode (gs + build_gaussians + .ply activations, baked) | points(1,8192,3)+cond(1,8192,16)→(262144,14) | `decode_fp32.aimodel` | fp32 |
38
 
39
  The flow-matching sampler (`FlowEulerCfgSampler`) and the octree `sample_probs` systematic resampling
40
  stay **host-side** (data-dependent control flow). Scripts: `_conv_*.py` convert+gate each net;
41
  `_conv_fp16.py` makes the half-size fp16 bundles; `_conv_decode.py` bakes build_gaussians + the
42
  Gaussian `.ply`-activation math into one net so the runner just writes raw floats.
43
 
44
+ ## model.py patches (the reusable contribution — see the zoo's conversion guide)
45
 
46
  coreai-torch 0.4.0 needed six edits to VAST's `model.py`; all are general gotchas:
47
 
 
63
  ## Running it
64
 
65
  - **Mac**: `_run_coreai.py` (or `app_backend.py --input <img>`) loads the bundles via coreai.runtime
66
+ (`SpecializationOptions.default()` = GPU; ~2 min/gen at 20 steps on Apple silicon, full quality).
67
+ End-to-end latent gate vs torch-DiT: **cos 0.999999**.
68
+ - **Mac app / iPhone client**: `TripoSplatMac` (standalone) and `TripoSplatPhone`
69
  (capture on iPhone → Mac server `server.py` → view splats in MetalSplatter / RealityKit).
70
 
71
  ## On-device note