google/reveal
Viewer
• Updated
• 6.1k • 48 • 34
Leo Innocenzi
leryud
·
AI & ML interests
None yet
Recent Activity
liked
a dataset about 1 month ago
google/reveal liked
a model about 1 month ago
unsloth/GLM-4.7-Flash-FP8-Dynamic liked
a model about 2 months ago
zilliz/semantic-highlight-bilingual-v1 Organizations
None yet
liked a dataset about 1 month ago
liked a model about 1 month ago
liked 2 models about 2 months ago
liked a dataset 3 months ago
reacted to hesamation's post with ❤️ 3 months ago
Post
3133
this is big... 50 AI researchers from Bytedance, Alibaba, Tencent, and other labs/universities just published a 300-page paper with surprising lessons about coding models and agents (data, pre and post-training, etc).
key highlights:
> small LLMs can beat proprietary giants
RL (RLVR specifically) gives small open-source models an edge over big models in reasoning. a 14B model trained with RLVR on high-quality verified problems can match the performance of OpenAI's o3.
> models have a hard time learning Python.
mixing language models during pre-training is good, but Python behaves different from statically typed languages. languages with similar syntax (Java and C#, or JavaScript and TypeScript) creates high positive synergy. mixing Python heavily into the training of statically typed languages can actually hurt because of Python's dynamic typing.
> not all languages are equal (coding scaling laws)
the amount of data required to specialize a model on a language drastically depends on the language. paper argues like C# and Java are easier to learn (less training data required). languages like Python and Javascript are actually more tricky to learn, ironically (you see AI most used for these languages :)
> MoE vs Dense (ability vs stability)
MoE models offer higher capacity, but are much more fragile during SFT than dense models. hyperparams in training have a more drastic effect in MoE models, while dense models are more stable. MoE models also require constant learning rate schedules to avoid routing instability.
> code models are "insecure" by default (duh)
training on public repos makes models learn years of accumulated insecure coding patterns. safety fine-tuning often fails to work much on code. a model might refuse to write a hate speech email but will happily generate a SQL-injection vulnerable function because it "works."
read the full paper:
From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence (2511.18538)
key highlights:
> small LLMs can beat proprietary giants
RL (RLVR specifically) gives small open-source models an edge over big models in reasoning. a 14B model trained with RLVR on high-quality verified problems can match the performance of OpenAI's o3.
> models have a hard time learning Python.
mixing language models during pre-training is good, but Python behaves different from statically typed languages. languages with similar syntax (Java and C#, or JavaScript and TypeScript) creates high positive synergy. mixing Python heavily into the training of statically typed languages can actually hurt because of Python's dynamic typing.
> not all languages are equal (coding scaling laws)
the amount of data required to specialize a model on a language drastically depends on the language. paper argues like C# and Java are easier to learn (less training data required). languages like Python and Javascript are actually more tricky to learn, ironically (you see AI most used for these languages :)
> MoE vs Dense (ability vs stability)
MoE models offer higher capacity, but are much more fragile during SFT than dense models. hyperparams in training have a more drastic effect in MoE models, while dense models are more stable. MoE models also require constant learning rate schedules to avoid routing instability.
> code models are "insecure" by default (duh)
training on public repos makes models learn years of accumulated insecure coding patterns. safety fine-tuning often fails to work much on code. a model might refuse to write a hate speech email but will happily generate a SQL-injection vulnerable function because it "works."
read the full paper:
From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence (2511.18538)
- 1 reply
liked 6 models 4 months ago
QuantTrio/GLM-4.6-GPTQ-Int4-Int8Mix
Text Generation • 69B • Updated
• 24 • 4
lukealonso/MiniMax-M2-NVFP4
115B • Updated
• 26 • 13
cerebras/GLM-4.6-REAP-268B-A32B-FP8
Text Generation • 269B • Updated
• 10 • 12
cerebras/GLM-4.6-REAP-218B-A32B-FP8
Text Generation • Updated
• 10 • 44
cerebras/GLM-4.6-REAP-218B-A32B
Text Generation • 218B • Updated
• 21 • 15
mradermacher/GLM-4.6-REAP-218B-A32B-GGUF
Updated
• 5
liked a Space 4 months ago
liked a Space over 3 years ago
