| --- |
| base_model: microsoft/Phi-3-mini-4k-instruct |
| library_name: peft |
| pipeline_tag: text-generation |
| tags: |
| - base_model:adapter:microsoft/Phi-3-mini-4k-instruct |
| - lora |
| - transformers |
| license: apache-2.0 |
| datasets: |
| - teknium/OpenHermes-2.5 |
| - Magpie-Align/Magpie-Phi3-Pro-300K-Filtered |
| language: |
| - en |
| --- |
| |
| # Model Card for Model ID |
|
|
| MicroAtlas-V1 is a general purpose model trained on both the teknium/OpenHermes-2.5 dataset and the Magpie-Align/Phi3-Pro-300K-Filtered dataset |
| and designed to provide speed, efficiency, and intelligence while still being relatively small. |
|
|
| the .gguf version of this model has only 3.15 M parameters, making it extremely small. |
|
|
|
|
|
|
|
|
| ## Model Details |
| OpenHermes dataset: |
|
|
| 1 Epoch |
|
|
| 8 Batch Size |
|
|
| 1 Gradient Accumulation |
|
|
| 5e-5 LR |
|
|
| 16 LoRa r |
|
|
| 32 LoRa Alpha |
|
|
| 300 Warmup steps |
|
|
| 500 Eval steps |
|
|
| Trained only on Attention layers. |
|
|
| Magpie dataset: |
|
|
| 1 Epoch |
|
|
| 16 Batch Size |
|
|
| 1 Gradient Accumulation |
|
|
| 1e-4 LR |
|
|
| 16 LoRa r |
|
|
| 32 LoRa Alpha |
|
|
| 150 Warmup steps |
|
|
| 500 Eval steps |
|
|
| Trained with Gate,Up, and Down layers. |
|
|
| ### Model Description |
|
|
| This model excels at creating bullet point formatting. |
|
|
|
|
|
|
| - **Developed by:** Turtle170 (anonymous |
| - **Language(s) (NLP):** English |
| - **License:** apache-2.0 |
| - **Finetuned from model :** Phi-3-Mini-4k-Instruct with turtle170/Phi-3-Mini-OpenHermes-V1 adapters |
|
|
|
|
|
|
| ### Direct Use |
|
|
| For direct use, the easiest method is to just download the .gguf file from turtle170/Phi-3-Mini-OpenHermes-Magpie-V1-F16-GGUF and loading it into llama.cpp or Ollama. |
|
|
|
|
| ### Out-of-Scope Use |
|
|
| Users of this model need only adhere to the **Microsoft Phi-3** Terms of use, |
| and you are solely responsible for any misuse of this model, as according to Section 7 and 8 of |
| the apache-2.0 licence |
|
|
| ## Bias, Risks, and Limitations |
|
|
| As this model was trained on a small base model, and only exposed to 2 50k example datasets, |
| so you should not expect much from it. |
| However, this model is smart for its size. |
|
|
| ### Recommendations |
|
|
| This model |
|
|
| Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations. |
|
|
|
|
| ## Training Details |
| Stated above. |
|
|
| ### Training Data |
| teknium/OpenHermes-2.5 dataset and the Magpie-Align/Phi3-Pro-300K-Filtered dataset. |
|
|
|
|
| ### Training Procedure |
|
|
| <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> |
|
|
|
|
| #### Training Hyperparameters |
|
|
| - **Training regime:** The OpenHermes run was on fp16 mixed precision, while the Magpie run was on fp32 mixed precison. |
|
|
| #### Speeds, Sizes, Times |
| The Magpie adapter is about 100-200 Mb. |
|
|
|
|
| ## Evaluation |
|
|
| The evaluation strategy was epochs, and the results were 0.4203 loss. |
|
|
|
|
| #### Metrics |
| 1 Epoch --> fast while prevents overfitting. |
|
|
| 16 Batch Size --> Helps squeeze every bit of intelligence. |
|
|
| 1 Gradient Accumulation --> fast, while not crashing the model. |
|
|
| 1e-4 LR --> helps prevent breaking the intelligence stored on the Hermes run. |
|
|
| 16 LoRa r --> helps the model understand the harder examples in the Magpie run. |
|
|
| 32 LoRa Alpha --> self-explanatory. Alpha = LoRa r x 2 |
|
|
| 150 Warmup steps -->fast, and since the starting loss was already 0.4 |
|
|
| 1500 Eval steps --> the loss had fluctuated between 0.4 and 0.6, and eval wastes time, so i chose it to only be 2 per run. |
|
|
|
|
| ### Results |
|
|
| eval loss: 0.4 |
| Avg. train loss: 0.4 |
|
|
|
|
| ## Environmental Impact |
|
|
|
|
| - **Hardware Type:** 2x NVIDIA Tesla T4s |
| - **Hours used:** 12 |
| - **Cloud Provider:** Kaggle |
| - **Compute Region:** asia-east1 |
| - **Carbon Emitted:** 0.47 kg |
|
|
|
|
| ### Model Architecture and Objective |
| To provide a smart model while keeping the size small. |
|
|
| ### Framework versions |
|
|
| - PEFT 0.17.1 |
