Hugging Face's logo
VIDRAFT_LAB's picture
🔄 In a Training Loop

VIDRAFT_LAB

SeaWolf-AI
115 32 234
openfree's profile picture mayafree's profile picture cutechicken's profile picture
·
https://www.vidraft.net

AI & ML interests

Contact: arxivgpt@gmail.com

Recent Activity

repliedto their post about 5 hours ago
We opened a benchmark for drug property prediction tools. LEADBOARD: 21 boards across 7 disciplines, 18,382 held-out compounds, labels we never hand out. Two numbers we hit while building it are the reason it exists. First. Split the hERG cardiotoxicity data at random and you get AUROC 0.818. Split it by first-report year instead and you get 0.606. Same molecules, same fingerprints, same learner, same hyperparameters. The only thing that changed was where the line went, and the score moved 0.211. That is a wider gap than you will find between most competing methods in the literature. Second. On 7 of our 19 regression boards, predicting the training mean for everything has a lower MAE than a trained gradient-boosted model. hERG is one of them, 0.599 against 0.589. The trained model loses. So every board publishes its homework before anyone submits. Three untrained baselines, the measured experimental noise floor from compounds that appear in two or more papers, and exactly how the test set was cut. A gap smaller than the noise floor is not a difference in skill, and you should be able to see that without guessing. Entering is simple. Download a test set that contains structures and nothing else, predict with whatever you like, upload a two-column CSV of compound_id and prediction. Trained model, physics engine, LLM, rule of thumb. We do not care what is inside. We measure the output. Post: https://huggingface.co/blog/FINAL-Bench/leadboard-drug Leaderboard: https://huggingface.co/spaces/FINAL-Bench/leadboard
repliedto their post about 8 hours ago
We opened a benchmark for drug property prediction tools. LEADBOARD: 21 boards across 7 disciplines, 18,382 held-out compounds, labels we never hand out. Two numbers we hit while building it are the reason it exists. First. Split the hERG cardiotoxicity data at random and you get AUROC 0.818. Split it by first-report year instead and you get 0.606. Same molecules, same fingerprints, same learner, same hyperparameters. The only thing that changed was where the line went, and the score moved 0.211. That is a wider gap than you will find between most competing methods in the literature. Second. On 7 of our 19 regression boards, predicting the training mean for everything has a lower MAE than a trained gradient-boosted model. hERG is one of them, 0.599 against 0.589. The trained model loses. So every board publishes its homework before anyone submits. Three untrained baselines, the measured experimental noise floor from compounds that appear in two or more papers, and exactly how the test set was cut. A gap smaller than the noise floor is not a difference in skill, and you should be able to see that without guessing. Entering is simple. Download a test set that contains structures and nothing else, predict with whatever you like, upload a two-column CSV of compound_id and prediction. Trained model, physics engine, LLM, rule of thumb. We do not care what is inside. We measure the output. Post: https://huggingface.co/blog/FINAL-Bench/leadboard-drug Leaderboard: https://huggingface.co/spaces/FINAL-Bench/leadboard
reacted to theirpost with ❤️ about 16 hours ago
We opened a benchmark for drug property prediction tools. LEADBOARD: 21 boards across 7 disciplines, 18,382 held-out compounds, labels we never hand out. Two numbers we hit while building it are the reason it exists. First. Split the hERG cardiotoxicity data at random and you get AUROC 0.818. Split it by first-report year instead and you get 0.606. Same molecules, same fingerprints, same learner, same hyperparameters. The only thing that changed was where the line went, and the score moved 0.211. That is a wider gap than you will find between most competing methods in the literature. Second. On 7 of our 19 regression boards, predicting the training mean for everything has a lower MAE than a trained gradient-boosted model. hERG is one of them, 0.599 against 0.589. The trained model loses. So every board publishes its homework before anyone submits. Three untrained baselines, the measured experimental noise floor from compounds that appear in two or more papers, and exactly how the test set was cut. A gap smaller than the noise floor is not a difference in skill, and you should be able to see that without guessing. Entering is simple. Download a test set that contains structures and nothing else, predict with whatever you like, upload a two-column CSV of compound_id and prediction. Trained model, physics engine, LLM, rule of thumb. We do not care what is inside. We measure the output. Post: https://huggingface.co/blog/FINAL-Bench/leadboard-drug Leaderboard: https://huggingface.co/spaces/FINAL-Bench/leadboard
View all activity

Organizations

FINAL_Bench's profile picture Gemma Challenge's profile picture PPALLI's profile picture