Molmo2Fish

Molmo2Fish is Molmo2-8B fine-tuned to track fish in sonar video, and to edit those tracks in response to natural-language feedback.

Track correction is treated as a conversation: the model is shown a set of existing tracks — its own earlier predictions, another tracker's output, or corrupted ground truth — is told in words what is wrong with them, and returns a repaired set of tracks.

Code	github.com/tidalove/molmo2fish
Data	tidalove/cfc-track-instruction
Source video	Caltech Fish Counting

Files

file	what it is
*.safetensors, config.json, …	HuggingFace-format weights at the repo root — what vLLM and launch_scripts/hf_eval.py consume
Molmo2Fish-step420-raw.tar	the raw training checkpoint (sharded model + optimizer state), for resuming fine-tuning

Training

Rank 64 LoRA fine-tuning of Molmo2-8B on the full CFC mixture, step 420. Adapters on all three components — LLM, ViT, and connector — with the base weights frozen:

torchrun --nproc-per-node=8 launch_scripts/sft.py /path/to/Molmo2-8B cfc_correction \
  --lora_llm --lora_vit --lora_connector --lora_rank 64 \
  --save_folder=/path/to/save/folder

The cfc_correction mixture combines pure tracking, targeted correction, synthetically corrupted correction, correction of real model predictions at two quality levels (molmo_high / molmo_low), and text-only correction. See the dataset card for what each config contains.

Usage

git clone https://github.com/tidalove/molmo2fish.git && cd molmo2fish
pip install torchcodec && pip install -e .[all]

export MOLMO_DATA_DIR=./data
python -m scripts.download_datasets cfc --n-procs 8

hf download tidalove/Molmo2Fish --exclude "*.tar" --local-dir Molmo2Fish-HF/step420-hf
python launch_scripts/hf_eval.py Molmo2Fish-HF/step420-hf \
  cfc_hf_correction_molmo_low_full_eval_2fps

Correction tasks report HOTA_before (the tracks the model was handed), HOTA_after (what it returned), and norm_delta_HOTA (the fraction of available headroom it closed), alongside a per-river breakdown and the directional net-count error nMAE.

Evaluation runs on 6 fps clips with tracks annotated at 2 fps, sliced from the original CFC videos — metrics are not comparable to those computed on the original CFC release.

Citation

@article{molmo2fish,
    title={Teach a Molmo2Fish: Towards interactive fish tracking with natural language guidance},
    author={Kai van Brunt and Justin Kay and Sara Beery},
    year={2026}
}