Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,92 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
+
tags:
|
| 4 |
+
- robotics
|
| 5 |
+
- tactile
|
| 6 |
+
- visuo-tactile
|
| 7 |
+
- representation-learning
|
| 8 |
+
library_name: pytorch
|
| 9 |
---
|
| 10 |
+
|
| 11 |
+
# NeoForce: A Unified Force Tactile Representation Model
|
| 12 |
+
|
| 13 |
+
[Technical Report](https://research.neoteai.com/assets/n0-foundation-paper.pdf) 路 [Project Website](http://research.neoteai.com/n0-foundation) 路 [Code](https://github.com/SparkleXFantasy/N0-Foundation-Preview) 路 [OpenNeoData](https://huggingface.co/datasets/NeoteAIEmbodied/OpenNeoData)
|
| 14 |
+
|
| 15 |
+
Tactile hardware is fragmented, so a model built on one signal format is bound to the device that produced it. NeoForce instead describes every tactile observation as a **dense three-axis force field** over the sensing surface, capturing shear and pressure in a form that is physically grounded and shared across sensors, and learns a temporally structured representation on top of it.
|
| 16 |
+
|
| 17 |
+
The model is a joint visual-tactile ViT: RGB frames and dual-sensor force fields are patched onto one shared token grid and run through a single trunk, so the modalities attend to each other directly instead of being fused after the fact. A temporal transformer over per-frame CLS pairs supplies context, and a progressive decoder reconstructs the force field and contact mask at the full input resolution, supervised at every scale it passes through.
|
| 18 |
+
|
| 19 |
+
This repository holds the weights. The code, together with the data preparation, training and evaluation guides, lives in the [code repository](https://github.com/SparkleXFantasy/N0-Foundation-Preview).
|
| 20 |
+
|
| 21 |
+
## Files
|
| 22 |
+
|
| 23 |
+
| File | Size | What it is |
|
| 24 |
+
|---|---|---|
|
| 25 |
+
| `neoforce/neoforce.pt` | 1.25 GB | the NeoForce model |
|
| 26 |
+
| `visuo_tactile_conversion/visuo_tactile_conversion.pt` | 49 MB | the tactile conversion model |
|
| 27 |
+
|
| 28 |
+
### `neoforce/neoforce.pt`
|
| 29 |
+
|
| 30 |
+
The trained NeoForce model at step 100,000, kept as a full training state rather than inference weights alone:
|
| 31 |
+
|
| 32 |
+
| Key | Contents |
|
| 33 |
+
|---|---|
|
| 34 |
+
| `model` | the student: the visual-tactile encoder plus the JEPA predictor, 156.14 M parameters |
|
| 35 |
+
| `teacher` | the EMA copy the JEPA and DINO targets are drawn from |
|
| 36 |
+
| `model_cfg` | the model configuration, so nothing has to be restated to rebuild it |
|
| 37 |
+
| `data_cfg`, `force_norm` | the per-channel scale the force targets were divided by |
|
| 38 |
+
| `step` | the training step the weights come from |
|
| 39 |
+
|
| 40 |
+
### `visuo_tactile_conversion/visuo_tactile_conversion.pt`
|
| 41 |
+
|
| 42 |
+
The frozen tactile conversion model, which turns raw tactile camera frames into the force field NeoForce consumes:
|
| 43 |
+
|
| 44 |
+
```
|
| 45 |
+
gel image -> bird-view correction -> optical flow based on the reference frame
|
| 46 |
+
-> conversion network -> masked, scaled force field
|
| 47 |
+
```
|
| 48 |
+
|
| 49 |
+
## Setup
|
| 50 |
+
|
| 51 |
+
Each file has a fixed path the code looks in:
|
| 52 |
+
|
| 53 |
+
```bash
|
| 54 |
+
git clone https://github.com/SparkleXFantasy/N0-Foundation-Preview
|
| 55 |
+
cd N0-Foundation-Preview/neoforce
|
| 56 |
+
|
| 57 |
+
mkdir -p weights neoforce/visuo_tactile_conversion/model
|
| 58 |
+
curl -L -o weights/neoforce.pt \
|
| 59 |
+
https://huggingface.co/NeoteAI/NeoForce/resolve/main/neoforce/neoforce.pt
|
| 60 |
+
curl -L -o neoforce/visuo_tactile_conversion/model/visuo_tactile_conversion.pt \
|
| 61 |
+
https://huggingface.co/NeoteAI/NeoForce/resolve/main/visuo_tactile_conversion/visuo_tactile_conversion.pt
|
| 62 |
+
```
|
| 63 |
+
|
| 64 |
+
The [Installation Guide](https://github.com/SparkleXFantasy/N0-Foundation-Preview/blob/main/neoforce/docs/Installation.md) covers the rest of the setup.
|
| 65 |
+
|
| 66 |
+
## Model
|
| 67 |
+
|
| 68 |
+
| | |
|
| 69 |
+
|---|---|
|
| 70 |
+
| Backbone | ViT-B, 12 layers, 768-dim, patch 20 |
|
| 71 |
+
| Input | 4 frames @ 30 fps, RGB 360脳640 + tactile representation `(6, 360, 640)` |
|
| 72 |
+
| Output | tactile representation `(6, 360, 640)`, contact mask `(6, 360, 640)`, global force `(6,)` |
|
| 73 |
+
|
| 74 |
+
The six force channels are `[left fx, fy, fz, right fx, fy, fz]`, one triplet per tactile sensor.
|
| 75 |
+
|
| 76 |
+
Trained on [OpenNeoData](https://huggingface.co/datasets/NeoteAIEmbodied/OpenNeoData), the 5,000-hour open-source subset of NeoData.
|
| 77 |
+
|
| 78 |
+
## Citation
|
| 79 |
+
|
| 80 |
+
```bibtex
|
| 81 |
+
@misc{n0foundation,
|
| 82 |
+
title={N0-Foundation: Towards the Age of Tactile Intelligence},
|
| 83 |
+
author={NeoteAI Team and TEAI Team},
|
| 84 |
+
year={2026},
|
| 85 |
+
url={https://research.neoteai.com/assets/n0-foundation-paper.pdf},
|
| 86 |
+
note={Technical Report}
|
| 87 |
+
}
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
## License
|
| 91 |
+
|
| 92 |
+
[MIT](https://opensource.org/licenses/MIT)
|