Question: edge/mobile deployment viability?
#75
by 3morixd - opened
We evaluate every interesting model for mobile deployment on our 40-phone farm (Snapdragon 865).
Question: has anyone tested this model on mobile/edge devices? What are the key constraints (size, speed, memory)?
Our criteria: <2GB after Q4_K_M, >10 t/s generation, >2GB RAM free.
Would love to hear from anyone with mobile experience.
- Dispatch AI (FZE), Sharjah UAE