HenryHHHH/DistilLlamaV1

Overview

This model is a distilled version of LLaMA 2, containing approximately 80 million parameters. It was trained using a mix of OpenWebText and WikiText Raw V1 datasets. Knowledge distillation was employed to transfer knowledge from a larger "teacher" model—Meta’s 7B LLaMA 2—to help this smaller model mimic the behavior of the teacher. This version is the latest version of DistilLlama, which has gone through 5 days of training using two Nvidia A100 80G GPU.

Update

30 out of 300 checkpoints were examined, and the one with the best performance in semantic and factual accuracy has now been updated in this repository.

Model Architecture

The architecture is based on LLaMA 2, with the following parameters:

Parameter	Value
Hidden Dimension	512
Intermediate Dimension	1536
Max Positional Embeddings	128
Attention Heads	8
Transformer Layers	16

Evaluation Metrics

1. Cosine Similarity using Word Embeddings

   • Description: Measures semantic similarity by mapping words/phrases to vectors.
   • Equation: Cosine Similarity = ( A • B ) / ( ||A|| ||B|| )
   • Example: "The dog chased the cat." vs. "A canine pursued a feline." (High similarity)

2. Exact Match (EM)

   • Description: Checks if critical keywords are present.
   • Example:
     • Expected: "Paris"
     • Response: "The capital of France is Paris." (EM = 1)

3. ROUGE Score

   • Description: Measures the overlap of the longest common subsequences between reference and response texts.
   • Equation:
     • Precision = Precision = LCS(R, C) / Length of C
     • Recall = Recall = LCS(R, C) / Length of R

Model Evaluation Summary

Model Name	Duration (s)	Emissions (kgCO₂e)	Avg. EM	Avg. Cosine Similarity	Avg. ROUGE Score
LLaMA-2-7B-HF	18215.61	1.84e-01	0.715	0.7257	0.0821
baby-llama-58m	57.20	2.73e-06	0.025	0.6556	0.0097
DistilLlama	77.12	7.79e-04	0.02	0.6623	0.0115
DistilLlamaV1	78.46	8.49e-04	0.065	0.6776	0.0135

Note: CodeCarbon was used to track carbon emission. Allocated 80GB memory, 32 cores, Intel(R) Xeon(R) Gold 6448H for the evaluation

GitHub Repositories

• Training Repo: DistilLlama Training Repository
• Evaluation Repo: Knowledge Distillation Evaluation Repository

Reference

@misc{timiryasov2023babyllamaknowledgedistillation, title={Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty}, author={Inar Timiryasov and Jean-Loup Tastet}, year={2023}, eprint={2308.02019}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2308.02019}, }

Note: The repository will be updated as training progresses. Last update 2024-11-06