🤝 Open to Collab

Dipankar Sarkar PRO

dipankarsarkar

6 1441 286

https://www.dipankar.cc

AI & ML interests

Building the AI-native stack. Agents as infrastructure, safety as architecture, performance as plumbing. I publish the receipts: papers, datasets, demos.

Recent Activity

reacted to pollix's post with 🚀 about 2 hours ago

stuntd 0.1.2 is out 🎉 stuntd sits in front of your LLM, learns its typed decisions and answers the confident ones locally with a small head on the Laya encoder by @convaiinnovations. About 20ms on GPU and 60ms on CPU, and anything it isn't sure about still goes to the big model. New in 0.1.2: - decisions with several fields, like category + urgency + needs_human in one call, answered locally only when every field is sure - the Anthropic Messages API learns too, not only OpenAI - auto_retrain: the daemon retrains a site in the background once enough new traffic comes in, so collect, train, shadow and live run on their own - serve --lazy loads the checkpoint on the first request Try it in the browser: https://huggingface.co/spaces/pollix/stuntd Code: https://github.com/bladedevoff/stuntd ``` pip install -U stuntd ```

reacted to pierrexsq's post with 🔥 about 2 hours ago

⚡ Introducing ZooWork Instinct Tuned 4B — an open decision model Instead of generating text, Instinct scores candidate logits in one forward pass and returns temperature-scaled probabilities for choice, noul, and ordinal score questions. 🎯 4B parameters based on Qwen3.5-4B 📊 198/231 (85.71%) on the published JevBench tasks ⚡ Decode-free, single-order decision inference 📜 Apache-2.0 open weights 🤗 Model and weights: https://huggingface.co/srpone/instinct-tuned-4b 💻 Open-source runtime: https://github.com/SerendipityOneInc/instinct 🌐 Hosted API: https://instinct.zoowork.ai/ The published JevBench tasks were used during development, so the result is a transparent public-set reference—not a held-out claim. Feedback on domain transfer, calibration, and option-order sensitivity is very welcome. 🚀

upvoted a paper about 2 hours ago

Scheduling Recursive Reasoning in Looped Transformers

View all activity

Organizations

Posts 3

Post

1842

I audited one of my own evaluations. The ranking did not hold up the way I expected.

Eight open models, one task: infer the structure of a prompt. Then ask again with the identical call. Caching off.

- Agreement between repeated identical calls (mean Jaccard) ranged from 0.39 to 0.96 across models.
- Only 35 of 127 prompt-model cells were perfectly reproducible on every run.
- I bootstrapped the reproducibility ranking over prompts. The two least reproducible models kept their rank in 99% and 86% of resamples. The middle four kept theirs in 27% to 48%.

So the table reliably finds the worst model. It does not reliably find the best.

Reproducible is also not the same as correct. F1 against gold annotations ran from 0.56 to 0.99.

By the audit date, 4 of the 8 model variants had been retired (HTTP 410). The study as specified can no longer be re-run. The saved outputs are what survives, so I published all of them: every inferred structure, every run, the prompts and the annotations.

Paper: How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure (2609.30074)
Dataset: dipankarsarkar/llm-evaluation-self-audit
Code: https://github.com/sarkar-dipankar/llm-evaluation-self-audit

How many of the leaderboard rankings you rely on would survive re-running the same calls?