Add canonical HetMoE checkpoints (ConvNet experts, hyperparams, ConvNet MoE, genomic zoo seeds 0/1/42) + model card

README.md

@@ -0,0 +1,85 @@
+---
+license: cc-by-nd-4.0
+tags:
+- TFBS
+- transcription-factor
+- genomics
+- mixture-of-experts
+- DNA
+library_name: pytorch
+pretty_name: ENCODE-TFBS Heterogeneous Mixture-of-Experts checkpoints
+datasets:
+- Lab-Rasool/ENCODE-TFBS
+---
+
+# ENCODE-TFBS — Heterogeneous Mixture-of-Experts checkpoints
+
+Trained model checkpoints for **robust transcription-factor binding-site (TFBS) prediction with a
+heterogeneous Mixture-of-Experts (MoE)**. A dense, soft MoE gates over per-expert *embeddings* from a
+heterogeneous expert zoo (modified-DeepBIND **ConvNet** + **DeepSEA** + **DanQ**, each probed to a
+common 32-dim embedding), which improves **out-of-distribution (OOD)** generalization to unseen
+transcription factors.
+
+These weights back the paper *"Robust Transcription Factor Binding Site Prediction and Explainability
+Using a Heterogeneous Mixture of Experts Architecture."* Code, training and evaluation pipeline:
+**https://github.com/lab-rasool/TFBS**. Training/eval data: **[Lab-Rasool/ENCODE-TFBS](https://huggingface.co/datasets/Lab-Rasool/ENCODE-TFBS)**.
+
+## Headline result (genomic, fair-negative protocol, 7 training factors)
+
+Feeding the unchanged embedding-gating MoE a heterogeneous expert pool beats a fine-tuned DNABERT-6
+baseline on the motif-bearing OOD strata, averaged over seeds 0/1/42:
+
+| Model | OOD AUC (mean ± std) |
+|---|---|
+| **HetMoE (this work)** | **0.821 ± 0.005** |
+| DNABERT-6 | 0.799 ± 0.008 |
+
+Margin **+0.022**. Per-seed: seed 42 → 0.827, seed 0 → 0.818, seed 1 → 0.819.
+
+## Repository contents
+
+```
+experts/<TF>.pth            7 ConvNet experts (modified DeepBIND), one per training TF
+hyperparams/<TF>.pth        per-expert hyperparameters (reproduce training without Optuna)
+moe/moe_model.pth           homogeneous ConvNet-only MoE gate (+ moe_model_config.pth)
+zoo/seed{0,1,42}/           heterogeneous zoo probes — DeepSEA_<TF>.pth, DanQ_<TF>.pth
+                            (E=32 FeatureProbeExpert heads over frozen DeepSEA/DanQ trunks)
+```
+
+The genomic HetMoE for a given seed is the **21-expert** pool: the 7 `experts/` ConvNets plus the
+14 `zoo/seed<N>/` DeepSEA + DanQ probes, with the `MixtureOfExperts` gate applied unchanged over the
+concatenated 32-dim embeddings. Only the three paper seeds (0, 1, 42) are published here.
+
+**Transcription factors.** Training: `ARID3A, FOXM1, GATA3, JUND, MAX, GABPA, SP1`. OOD evaluation is
+stratified into within-family, cross-family (e.g. CTCF, STAT3), cell-line-transfer, and a separately
+reported non-motif appendix — see `tfbs/constants.py` in the code repo.
+
+## Usage
+
+Install the `tfbs` package and load with the provided classes (`map_location` handles CPU-only nodes):
+
+```python
+from huggingface_hub import hf_hub_download
+import torch
+from tfbs.models import ConvNet, MixtureOfExperts  # pip install -e . from github.com/lab-rasool/TFBS
+
+ckpt = hf_hub_download("Lab-Rasool/ENCODE-TFBS", "experts/GATA3.pth")
+state = torch.load(ckpt, map_location="cpu", weights_only=True)
+```
+
+See the GitHub repo's README and `experiments/hetmoe/` for the full caching + gating pipeline that
+rebuilds the heterogeneous MoE from these checkpoints. DNABERT-6 features are derived on the fly from
+`zhihan1996/DNA_bert_6`; no BERT weights are stored here.
+
+## Reproducibility
+
+The ConvNet conv bias (`wRect`) is a saved `nn.Parameter` and expert order is pinned to
+`tfbs.constants.TRAIN_TFS`, so re-running evaluation from these checkpoints is byte-identical on a
+given machine (minor device-numerics differences may remain across machines).
+
+## License
+
+`cc-by-nd-4.0`, matching the [ENCODE-TFBS dataset](https://huggingface.co/datasets/Lab-Rasool/ENCODE-TFBS).
+The underlying ENCODE data are from the ENCODE Project.
+</content>
+</invoke>

experts/ARID3A.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4efd60a57c44b1e64843e95edb2e11cc165cde205130e8e9c6cf39161ccf8055
+size 14015

experts/FOXM1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:975a35fc997e82f28407239fcc6ea1e28204d5a62cc42d78314d6c86d56e3560
+size 14001

experts/GABPA.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe321624827d16046fca976896b9f750116bfc82d5fb429b97f62b08c4064d13
+size 14001

experts/GATA3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47b1fa9e8a5999fca72764f90a7f2f3463ff4e93824d91033605a5f88511f207
+size 11953

experts/JUND.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b3d7658b13d5ebd027e7c9e690bc0d12fd7124d06552650c17cee67e19b9b15
+size 13987

experts/MAX.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11699dfb462e55aeea972260b58cef0f343382af25cddd9fe2819f90e88a238b
+size 13973

experts/SP1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e9e8000fe574fb7abb0a33e803d33786df00dcbe7e6ea7d303d239bd24c28726
+size 13973

hyperparams/ARID3A.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69723db55b671abb810ed9eb412e6fff650afd4a68f563133af65a958ebb8b1e
+size 1116

hyperparams/FOXM1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:357f80ec774546c72f295f123578fe17bd233078c14eb647a726209e82349c21
+size 1112

hyperparams/GABPA.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8282368fec800af39cf6b73b7d1ac755e5fccb04ae3fe5c5f2752aadda9e753
+size 1505

hyperparams/GATA3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0e18f0451abd8e2d68183bc9baf8a25d3fbc12f2eb22fb51c1fcd601e541cbf
+size 1112

hyperparams/JUND.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5cf6e6b693552296e5e2bb1bd5210bc44eff72f82106f5e144efe5a8fe44478
+size 1499

hyperparams/MAX.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb7f8e8b2410d954e45f78ac676c69f74ed86c564caa89b8620001c17818ea3b
+size 1493

hyperparams/SP1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:43449c01491c33b733a7d343b4269694ccb0b4eddc50248ea28b38081903d35a
+size 1493

moe/moe_model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2691d32a2657f81569a77e330570a58f992b263dd03797bc32fa808aa33400fb
+size 42589

moe/moe_model_config.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28d41ddf4c274ee30910ea84469df1018981392270ce43daea53ab1836c615ae
+size 42821

zoo/seed0/DanQ_ARID3A.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:836851e6e6e6e95bd1fbf317143d339c866999d71f37792aac5e65107e18393e
+size 13355333

zoo/seed0/DanQ_FOXM1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a75d2cf059acf233e671754ef731dc12de7e4890fccede1f98ef7c856d9d9a1a
+size 13355320

zoo/seed0/DanQ_GABPA.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:550b88212e928b18d7ba99a709f8d0269176c183a0cdd9dee55a949165b0c438
+size 13355320

zoo/seed0/DanQ_GATA3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfc174a6a51a4c707ec1e28b7ab1514e5264356703f566ed03e30fca5590c0c0
+size 13355320

zoo/seed0/DanQ_JUND.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:deedea72c7cf2b1146dbbf393809064214a90b97ad196fb178ee8e58023cf2ea
+size 13355307

zoo/seed0/DanQ_MAX.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23035168217e783151a4b47e4f1a5af9da5d73c78762cd5ae83e1aabe0422830
+size 13355294

zoo/seed0/DanQ_SP1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38c8b420ac16a5da3c18319dede337595e7b943045aecb07024c7a8992621064
+size 13355294

zoo/seed0/DeepSEA_ARID3A.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6f8011ad7aac80c16cc74b66fd4b50c7b36c89ddd655be15b36c58e0b312269
+size 15932465

zoo/seed0/DeepSEA_FOXM1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60a093719479ea752d408ca8305b8e437ebacd72b299509291e4ef3bf400a802
+size 15932434

zoo/seed0/DeepSEA_GABPA.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98e1c50a2ba518d6ca582fd77d2f1862f77fa007d6dc08a3da6a59b9e545aa3e
+size 15932434

zoo/seed0/DeepSEA_GATA3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca0d962d5bd7075390b60cb4002137550750daa70aacb89d51244676d39ac286
+size 15932434

zoo/seed0/DeepSEA_JUND.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7246e7b43ec0aee87e7c47dd6e33399d5d5b844fe9827e352f1e249dd9a9c6d3
+size 15932403

zoo/seed0/DeepSEA_MAX.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84c060396c1029ff4853ba759ee86959adf36789624fd7d984a020a398c8634b
+size 15932372

zoo/seed0/DeepSEA_SP1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84117bd087d7fcf65223dc8ffd92ce8f333a28bbd7b04956e611e23484b19689
+size 15932372

zoo/seed1/DanQ_ARID3A.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e363f1f78c6df17d3ad11c2ad7dacb47eff1d498e44337a2e0be8867209a2cc4
+size 13355333

zoo/seed1/DanQ_FOXM1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84efef5b802666c87e85d96b652c3bc79e75c592e415499ee79593243d67180f
+size 13355320

zoo/seed1/DanQ_GABPA.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b2d16c13c6dec5d578c3d0cead1cfa70252ca699b1fcd1f3e1c814f8947eaf4
+size 13355320

zoo/seed1/DanQ_GATA3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aab399a4441d780f6aeaedc5ad0b13e4790e55fa6bdd60387e2996b73d90c4bc
+size 13355320

zoo/seed1/DanQ_JUND.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd5f0019e982edc4020e43e55d05a20256242d0e55342b1e9b8184ef2d599a21
+size 13355307

zoo/seed1/DanQ_MAX.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b10221d7f8fd2e2611eb1bcce622a3e74cd033ef4b98f94b5d8f071035f9a1a0
+size 13355294

zoo/seed1/DanQ_SP1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45c36276448914482d050d809e64602728cd6de3a30610dc628930df052c8bb0
+size 13355294

zoo/seed1/DeepSEA_ARID3A.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1d94c16657bcd63e899319837130fcfdf1f2e9669a4352b20462d759e5d8adc
+size 15932465

zoo/seed1/DeepSEA_FOXM1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46048196259b0522abfc08e55e47de1a82cafcc252db0d26e392156a07c731be
+size 15932434

zoo/seed1/DeepSEA_GABPA.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:713577da4ad4b4fec7129197e7c6e227355bb8c8b1435a42f58bec940dcfa676
+size 15932434

zoo/seed1/DeepSEA_GATA3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:730f1a0b2c69a18c43e22701b4987cea6f82270a707a2b575e1503138861efd6
+size 15932434

zoo/seed1/DeepSEA_JUND.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a2fd5fa816511b7e1ecb77674f6d1ba9ec28f07ea9ab00116cc17542fbe5699
+size 15932403

zoo/seed1/DeepSEA_MAX.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b163cb5bcdb6274ba9e2144e2a50b82edc5b6866ebce5dcffb20d8efc010132
+size 15932372

zoo/seed1/DeepSEA_SP1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0fd1f8c9f025011fda2a903cc5ce5a454bf6c2358fd461224b36e7d184c4e9c
+size 15932372

zoo/seed42/DanQ_ARID3A.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:313920f4420373a50de3784ae2db5dbf50bd709c00444ebeada5b759c2882f5d
+size 13355333

zoo/seed42/DanQ_FOXM1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd88aa0f23b54ea2fc66930c2364efcf70d34217b688f2df4c4359deffc76377
+size 13355320

zoo/seed42/DanQ_GABPA.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38c6c57a36abfc1a98f49141120f6e32f8e6cafcbd1dac7d67fd4008e53e8574
+size 13355320

zoo/seed42/DanQ_GATA3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c641c6bab925d9fde89cd3ebb1e0ea4b7fd48d51f39b9608afadfbc104c0fa3c
+size 13355320

zoo/seed42/DanQ_JUND.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5a3bd530ac4710514117f5ea6edc34cd4df60d65d3c3e05908109b55d89ecb3
+size 13355307