Pankaj Singh

why don't you guys try making a simplifer first? simplify pretraining data as much as possible then train the small models on it? Small models can't learn much from sophisticated datasets. Yes it's a extensive thing to do but that's what really can make a difference otherwise you guys will keep training small models and they won't improve a bit.