| | --- |
| | license: mit |
| | datasets: |
| | - tiiuae/falcon-refinedweb |
| | - HuggingFaceFW/fineweb |
| | base_model: |
| | - cckm/tinymistral_950m |
| | language: |
| | - en |
| | pipeline_tag: text-generation |
| | library_name: PyTorch |
| | --- |
| | |
| | ## A deep and narrow Mistral model (950M params) |
| | This checkpoint is for a small (950M params), deep and narrow (40 layers, hidden size=1440) Mistral model, as described in this [[blog post]](https://epsilons.ai/blog.html#post1_3). It is meant for edge applications. |
| | |
| | It was trained with ~400B tokens from RefinedWeb, and ~400B tokens from FineWeb (up to epoch 202418). It is a base model, and has not gone through instruct or chat fine-tuning. |
| | |
| | LM Harness numbers: |
| | | Benchmark | Result | |
| | | ----- | ----- | |
| | | arc_c | 0.2884 | |
| | | arc_e | 0.5139 | |
| | | boolq | 0.6089 | |
| | | hellaswag | 0.5888 | |
| | | obqa | 0.3280 | |
| | | piqa | 0.7388 | |
| | | siqa | 0.4038 | |
| | | wino | 0.5627 | |
