TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
Paper
• 2404.11912 • Published
• 17
we utilize a 4K retrieval cache as an intermediate draft cache in our hierarchical system, while leveraging the JackFram/Llama68M (JF68M) [28] model