Hugging Face's logo
Activity Feed

AI & ML interests

None defined yet.
loubnabnlย 
posted an update 3 days ago
view post
Post
3,960
๐ˆ๐ง๐ญ๐ซ๐จ๐๐ฎ๐œ๐ข๐ง๐  ๐Ÿงฌ ๐…๐ข๐ง๐ž๐ƒ๐๐€: ๐ญ๐ก๐ž ๐›๐ž๐ฌ๐ญ ๐๐š๐ญ๐š๐ฌ๐ž๐ญ ๐ญ๐จ ๐ญ๐ž๐š๐œ๐ก ๐š๐ง๐ฒ ๐ฅ๐š๐ง๐ ๐ฎ๐š๐ ๐ž ๐ฆ๐จ๐๐ž๐ฅ ๐ƒ๐๐€.
Training Qwen3-14B on it produces Carbon-X, the strongest open model on DNA benchmarks, on par with GPT-6.1 Sol.

It contains:
- Midtraining: 300B+ tokens of genomic sequences, papers and web pages mixing DNA with text
- SFT: 200K samples, from variant effect prediction to sequence question answering
- RL: 98K prompts with verifiable rewards

Data, models and the full recipe are open.
HuggingFaceBio/FineDNA

Enjoy this late Halloween treat! ๐ŸŽƒ
  • 4 replies
ยท
lvwerraย 
authored a paper about 1 year ago
lvwerraย 
authored a paper over 1 year ago
loubnabnlย 
authored a paper over 1 year ago
loubnabnlย 
posted an update over 1 year ago
lvwerraย 
authored a paper over 1 year ago
loubnabnlย 
authored 2 papers over 1 year ago
lvwerraย 
authored a paper over 1 year ago
lvwerraย 
authored a paper almost 2 years ago
loubnabnlย 
posted an update almost 2 years ago
view post
Post
3,765
Making SmolLM2 reproducible: open-sourcing our training & evaluation toolkit ๐Ÿ› ๏ธ https://github.com/huggingface/smollm/

- Pre-training code with nanotron
- Evaluation suite with lighteval
- Synthetic data generation using distilabel (powers our new SFT dataset HuggingFaceTB/smoltalk)
- Post-training scripts with TRL & the alignment handbook
- On-device tools with llama.cpp for summarization, rewriting & agents

Apache 2.0 licensed. V2 pre-training data mix coming soon!

Which other tools should we add next?
lvwerraย 
authored a paper about 2 years ago
lvwerraย 
authored a paper over 2 years ago
loubnabnlย 
authored a paper over 2 years ago
loubnabnlย 
posted an update over 2 years ago
view post
Post
5,932
๐Ÿท FineWeb technical report is out and so is ๐Ÿ“š FineWeb-Edu, a 1.3 trillion tokens dataset that outperforms all other open web datasets, with remarkable improvements on educational benchmarksย such as MMLU, ARC, and OpenBookQA.

Technical report: HuggingFaceFW/blogpost-fineweb-v1
Dataset: HuggingFaceFW/fineweb-edu

We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens.

You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.

Enjoy!
lvwerraย 
authored a paper over 2 years ago
loubnabnlย 
authored a paper over 2 years ago
loubnabnlย 
posted an update over 2 years ago
view post
Post
6,556
We've just published a detailed blog post on the creation of Cosmopedia dataset. We hope this will provide insights about generating synthetic data at scale for pre-training.
https://huggingface.co/blog/cosmopedia

Here are some key takeaways:
๐ŸŽฏ Prompt curation is crucial: we want to cover many topics with few duplicates.
๐Ÿ“š You can leverage various resources for diversity: using different seed data, generation formats, and target audiences.
โš™๏ธ The importance of a good technical stack: for scalable generations with tools like llm-swarm and fast model training and evaluation.

Have a good read!
  • 1 reply
ยท
thhausย 
authored a paper over 2 years ago
RaymondLiย 
authored a paper over 2 years ago