Update README.md

fd1081d verified 12 months ago

4.56 kB

	---
	license: llama3.2
	datasets:
	- CarrotAI/Magpie-Ko-Pro-AIR
	- CarrotAI/Carrot
	- CarrotAI/ko-instruction-dataset
	language:
	- ko
	- en
	base_model:
	- meta-llama/Llama-3.2-3B-Instruct
	pipeline_tag: text-generation
	new_version: CarrotAI/Llama-3.2-Rabbit-Ko-3B-Instruct-2412
	---

	![image/webp](https://cdn-uploads.huggingface.co/production/uploads/64633ebb39359568c63b52ad/VMYmYpoc0pQVSgZI0kme0.webp)

	## Model Description

	### Model Details
	- Name: Carrot Llama-3.2 Rabbit Ko
	- Version: 3B Instruct
	- Base Model: CarrotAI/Llama-3.2-Rabbit-Ko-3B-Instruct
	- Languages: Korean, English
	- Model Type: Large Language Model (Instruction-tuned)

	### Training Process
	본 모델은 다음과 같은 주요 훈련 단계를 거쳤습니다:

	1. SFT (Supervised Fine-Tuning)
	- 고품질 한국어 및 영어 데이터셋을 사용하여 기본 모델을 세부 조정

	### Limitations
	- 3B 파라미터 규모로 인한 복잡한 작업에서의 제한적 성능
	- 특정 도메인에 대한 깊이 있는 전문성 부족
	- 편향성 및 환각 가능성

	### Ethics Statement
	모델 개발 과정에서 윤리적 고려사항을 최대한 반영하였으나, 사용자는 항상 결과를 비판적으로 검토해야 합니다.

	### How to Use
	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer

	model = AutoModelForCausalLM.from_pretrained("CarrotAI/Llama-3.2-Rabbit-Ko-3B-Instruct")
	tokenizer = AutoTokenizer.from_pretrained("CarrotAI/Llama-3.2-Rabbit-Ko-3B-Instruct")
	```

	## Score

	\|Tasks\|Version\| Filter \|n-shot\| Metric \| \|Value \| \|Stderr\|
	\|-----\|------:\|----------------\|-----:\|-----------\|---\|-----:\|---\|-----:\|
	\|gsm8k\| 3\|flexible-extract\| 5\|exact_match\|↑ \|0.6490\|± \|0.0131\|
	\| \| \|strict-match \| 5\|exact_match\|↑ \|0.0023\|± \|0.0013\|
	\|gsm8k-ko\| 3\|flexible-extract\| 5\|exact_match\|↑ \|0.3275\|± \|0.0134\|
	\| \| \|strict-match \| 5\|exact_match\|↑ \|0.2737\|± \|0.0134\|
	\|ifeval\| 4\|none \| 5\|inst_level_loose_acc \|↑ \|0.8058\|± \| N/A\|
	\| \| \|none \| 5\|inst_level_strict_acc \|↑ \|0.7686\|± \| N/A\|
	\| \| \|none \| 5\|prompt_level_loose_acc \|↑ \|0.7320\|± \|0.0191\|
	\| \| \|none \| 5\|prompt_level_strict_acc\|↑ \|0.6858\|± \|0.0200\|

	\| Tasks \|Version\|Filter\|n-shot\| Metric \| \|Value \| \|Stderr\|
	\|-------------------------------\|------:\|------\|-----:\|--------\|---\|-----:\|---\|-----:\|
	\|haerae \| 1\|none \| \|acc \|↑ \|0.4180\|± \|0.0148\|
	\| \| \|none \| \|acc_norm\|↑ \|0.4180\|± \|0.0148\|
	\| - haerae_general_knowledge \| 1\|none \| 5\|acc \|↑ \|0.3125\|± \|0.0350\|
	\| \| \|none \| 5\|acc_norm\|↑ \|0.3125\|± \|0.0350\|
	\| - haerae_history \| 1\|none \| 5\|acc \|↑ \|0.3404\|± \|0.0347\|
	\| \| \|none \| 5\|acc_norm\|↑ \|0.3404\|± \|0.0347\|
	\| - haerae_loan_word \| 1\|none \| 5\|acc \|↑ \|0.4083\|± \|0.0379\|
	\| \| \|none \| 5\|acc_norm\|↑ \|0.4083\|± \|0.0379\|
	\| - haerae_rare_word \| 1\|none \| 5\|acc \|↑ \|0.4815\|± \|0.0249\|
	\| \| \|none \| 5\|acc_norm\|↑ \|0.4815\|± \|0.0249\|
	\| - haerae_standard_nomenclature\| 1\|none \| 5\|acc \|↑ \|0.4771\|± \|0.0405\|
	\| \| \|none \| 5\|acc_norm\|↑ \|0.4771\|± \|0.0405\|

	\| Tasks \|Version\|Filter\|n-shot\| Metric \| \|Value \| \|Stderr\|
	\|----------------\|------:\|------\|-----:\|--------\|---\|-----:\|---\|------\|
	\|kobest_boolq \| 1\|none \| 5\|acc \|↑ \|0.7664\|± \|0.0113\|
	\| \| \|none \| 5\|f1 \|↑ \|0.7662\|± \| N/A\|
	\|kobest_copa \| 1\|none \| 5\|acc \|↑ \|0.5620\|± \|0.0157\|
	\| \| \|none \| 5\|f1 \|↑ \|0.5612\|± \| N/A\|
	\|kobest_hellaswag\| 1\|none \| 5\|acc \|↑ \|0.3840\|± \|0.0218\|
	\| \| \|none \| 5\|acc_norm\|↑ \|0.4900\|± \|0.0224\|
	\| \| \|none \| 5\|f1 \|↑ \|0.3807\|± \| N/A\|
	\|kobest_sentineg \| 1\|none \| 5\|acc \|↑ \|0.5869\|± \|0.0247\|
	\| \| \|none \| 5\|f1 \|↑ \|0.5545\|± \| N/A\|
	\|kobest_wic \| 1\|none \| 5\|acc \|↑ \|0.4952\|± \|0.0141\|
	\| \| \|none \| 5\|f1 \|↑ \|0.4000\|± \| N/A\|

	---
	license: llama3.2
	datasets:
	- CarrotAI/Magpie-Ko-Pro-AIR
	- CarrotAI/Carrot
	- CarrotAI/ko-instruction-dataset
	language:
	- ko
	- en
	base_model:
	- meta-llama/Llama-3.2-3B-Instruct
	pipeline_tag: text-generation
	new_version: CarrotAI/Llama-3.2-Rabbit-Ko-3B-Instruct-2412
	---

	![image/webp](https://cdn-uploads.huggingface.co/production/uploads/64633ebb39359568c63b52ad/VMYmYpoc0pQVSgZI0kme0.webp)

	## Model Description

	### Model Details
	- Name: Carrot Llama-3.2 Rabbit Ko
	- Version: 3B Instruct
	- Base Model: CarrotAI/Llama-3.2-Rabbit-Ko-3B-Instruct
	- Languages: Korean, English
	- Model Type: Large Language Model (Instruction-tuned)

	### Training Process
	본 모델은 다음과 같은 주요 훈련 단계를 거쳤습니다:

	1. SFT (Supervised Fine-Tuning)
	- 고품질 한국어 및 영어 데이터셋을 사용하여 기본 모델을 세부 조정

	### Limitations
	- 3B 파라미터 규모로 인한 복잡한 작업에서의 제한적 성능
	- 특정 도메인에 대한 깊이 있는 전문성 부족
	- 편향성 및 환각 가능성

	### Ethics Statement
	모델 개발 과정에서 윤리적 고려사항을 최대한 반영하였으나, 사용자는 항상 결과를 비판적으로 검토해야 합니다.

	### How to Use
	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer

	model = AutoModelForCausalLM.from_pretrained("CarrotAI/Llama-3.2-Rabbit-Ko-3B-Instruct")
	tokenizer = AutoTokenizer.from_pretrained("CarrotAI/Llama-3.2-Rabbit-Ko-3B-Instruct")
	```

	## Score

	\|Tasks\|Version\| Filter \|n-shot\| Metric \| \|Value \| \|Stderr\|
	\|-----\|------:\|----------------\|-----:\|-----------\|---\|-----:\|---\|-----:\|
	\|gsm8k\| 3\|flexible-extract\| 5\|exact_match\|↑ \|0.6490\|± \|0.0131\|
	\| \| \|strict-match \| 5\|exact_match\|↑ \|0.0023\|± \|0.0013\|
	\|gsm8k-ko\| 3\|flexible-extract\| 5\|exact_match\|↑ \|0.3275\|± \|0.0134\|
	\| \| \|strict-match \| 5\|exact_match\|↑ \|0.2737\|± \|0.0134\|
	\|ifeval\| 4\|none \| 5\|inst_level_loose_acc \|↑ \|0.8058\|± \| N/A\|
	\| \| \|none \| 5\|inst_level_strict_acc \|↑ \|0.7686\|± \| N/A\|
	\| \| \|none \| 5\|prompt_level_loose_acc \|↑ \|0.7320\|± \|0.0191\|
	\| \| \|none \| 5\|prompt_level_strict_acc\|↑ \|0.6858\|± \|0.0200\|

	\| Tasks \|Version\|Filter\|n-shot\| Metric \| \|Value \| \|Stderr\|
	\|-------------------------------\|------:\|------\|-----:\|--------\|---\|-----:\|---\|-----:\|
	\|haerae \| 1\|none \| \|acc \|↑ \|0.4180\|± \|0.0148\|
	\| \| \|none \| \|acc_norm\|↑ \|0.4180\|± \|0.0148\|
	\| - haerae_general_knowledge \| 1\|none \| 5\|acc \|↑ \|0.3125\|± \|0.0350\|
	\| \| \|none \| 5\|acc_norm\|↑ \|0.3125\|± \|0.0350\|
	\| - haerae_history \| 1\|none \| 5\|acc \|↑ \|0.3404\|± \|0.0347\|
	\| \| \|none \| 5\|acc_norm\|↑ \|0.3404\|± \|0.0347\|
	\| - haerae_loan_word \| 1\|none \| 5\|acc \|↑ \|0.4083\|± \|0.0379\|
	\| \| \|none \| 5\|acc_norm\|↑ \|0.4083\|± \|0.0379\|
	\| - haerae_rare_word \| 1\|none \| 5\|acc \|↑ \|0.4815\|± \|0.0249\|
	\| \| \|none \| 5\|acc_norm\|↑ \|0.4815\|± \|0.0249\|
	\| - haerae_standard_nomenclature\| 1\|none \| 5\|acc \|↑ \|0.4771\|± \|0.0405\|
	\| \| \|none \| 5\|acc_norm\|↑ \|0.4771\|± \|0.0405\|

	\| Tasks \|Version\|Filter\|n-shot\| Metric \| \|Value \| \|Stderr\|
	\|----------------\|------:\|------\|-----:\|--------\|---\|-----:\|---\|------\|
	\|kobest_boolq \| 1\|none \| 5\|acc \|↑ \|0.7664\|± \|0.0113\|
	\| \| \|none \| 5\|f1 \|↑ \|0.7662\|± \| N/A\|
	\|kobest_copa \| 1\|none \| 5\|acc \|↑ \|0.5620\|± \|0.0157\|
	\| \| \|none \| 5\|f1 \|↑ \|0.5612\|± \| N/A\|
	\|kobest_hellaswag\| 1\|none \| 5\|acc \|↑ \|0.3840\|± \|0.0218\|
	\| \| \|none \| 5\|acc_norm\|↑ \|0.4900\|± \|0.0224\|
	\| \| \|none \| 5\|f1 \|↑ \|0.3807\|± \| N/A\|
	\|kobest_sentineg \| 1\|none \| 5\|acc \|↑ \|0.5869\|± \|0.0247\|
	\| \| \|none \| 5\|f1 \|↑ \|0.5545\|± \| N/A\|
	\|kobest_wic \| 1\|none \| 5\|acc \|↑ \|0.4952\|± \|0.0141\|
	\| \| \|none \| 5\|f1 \|↑ \|0.4000\|± \| N/A\|