| | --- |
| | base_model: |
| | - appvoid/arco |
| | - h2oai/h2o-danube3-500m-base |
| | library_name: transformers |
| | tags: |
| | - mergekit |
| | - merge |
| |
|
| | --- |
| | # arco+ |
| |
|
| | This is an untrained passthrough model based on arco and danube as a first effort to train a small enough reasoning language model that generalizes across all kind of reasoning tasks. |
| |
|
| | #### Benchmarks |
| |
|
| | | Parameters | Model | MMLU | ARC | HellaSwag | PIQA | Winogrande | Average | |
| | | -----------|--------------------------------|-------|-------|-----------|--------|------------|---------| |
| | | 488m | arco-lite | **23.22** | 33.45 | 56.55| 69.70 | **59.19**| 48.46 | |
| | | 773m | arco-plus | 23.06 | **36.43** | **60.09**|**72.36**| **60.46**| **50.48** | |
| |
|
| | #### Configuration |
| |
|
| | The following YAML configuration was used to produce this model: |
| |
|
| | ```yaml |
| | slices: |
| | - sources: |
| | - model: appvoid/arco |
| | layer_range: [0, 14] |
| | - sources: |
| | - model: h2oai/h2o-danube3-500m-base |
| | layer_range: [4, 16] |
| | |
| | merge_method: passthrough |
| | dtype: float16 |
| | |
| | ``` |
| |
|
