Running
gml
🎛
gpu scripting language for the web
🤏 quecto mode
appvoid
appvoid
AI & ML interests
singularity through byte-level tokens and small language models - creating applications and ideas out of the void
Recent Activity
posted an update about 2 hours ago
We trained a 10.9M byte-level recurrent Transformer on L3 and L6. (Loop 3 and Loop 6)
Yet L4/L5 improved too, L8 held up, and the L3→L6 gain grew during training.
Same weights. More compute. Better predictions.
This is a new architecture for effective compute after several steps beyond original training!
We mixed and matched components like time and mhc into an ouro-like byte-level language model and the result is BET, a byte-level step-elastic transformer that can run computation steps without significant degradation.
One of the coolest parts of this training was discovering how Gradient Descent decided to use the first layer as what we would consider a scratchpad! Totally destroyed for the decoder but somehow makes total sense for the next layer!
I believe looped-transformers are the future of edge computing and this is a first step towards it.
Blogpost: https://medium.com/@appvoidofficial/byte-level-elasticity-182fe2ed1d2f
https://huggingface.co/appvoid/bet-10m updated a collection about 3 hours ago
releases published a model about 3 hours ago
appvoid/bet-10mOrganizations
