README.md

---
license: llama3
base_model:
- meta-llama/Meta-Llama-3-8B-Instruct
language:
- en
- ko
tags:
- facebook
- meta
- llama
- llama-3
- llama-3-ko
---
<p align="left">
  <img src="https://cdn-uploads.huggingface.co/production/uploads/646484cfb90150b2706df03b/BEOyMpnnY9VY2KXlc3V2F.png" width="20%"/>
<p>

# Llama-3-MAAL-8B-Instruct-v0.1
we release MAAL, Multilingual Adaptive Augmentation Language-model which comprises a groundbreaking fusion of multilingual capabilities and adaptive augmentation techniques. 

- **Developed by:** [maum.ai Brain NLP](https://maum-ai.github.io). Jaeyoon Jung, Jinjoo Lee, Yongjae Lee, Dongjun Lee, Woosung Joo
- **Language(s) (NLP):** Korean, English (currently, bilingual)

## Model Description

Version 0.1 uses cross-lingual training to transfer instruction-following capabilities from English to Korean.

- We Trained this model on an 8 H100-80G for 1 day with cross-lingual training dataset
- we recommend using the fixed system prompt for the model unless you fine-tune it
```
너는 마음에이아이의 챗봇 MAAL이다. 고객의 질문에 친절하게 답하여라.
```

## sample inference code (GPU)

```
import transformers
import torch

model_id = "maum-ai/Llama-3-MAAL-8B-Instruct-v0.1"
model = transformers.AutoModelForCausalLM.from_pretrained(model_id).to("cuda")
tokenizer = transformers.AutoTokenizer.from_pretrained(model_id)
streamer = transformers.TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

# we recommend using the fixed prompt for the model unless you fine-tune it
prompt = "너는 마음에이아이의 챗봇 MAAL이다. 고객의 질문에 친절하게 답하여라."
instruction = "사과 한 박스에는 사과가 30개 들어있는데, 처음에는 사과 3박스가 있었고, 내가 사과 5개를 먹었어. 남은 사과는 총 몇개야?"

messages = [
    {"role": "system", "content": f"{prompt}"},
    {"role": "user", "content": f"{instruction}"}
    ]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors='pt').to("cuda")
outputs = model.generate(inputs, streamer=streamer, max_new_tokens=1024, pad_token_id=tokenizer.eos_token_id)
```

## Evaluation Results

As the main goal of version 0.1 is to **transfer instruction-following capabilities from English to Korean** without utilizing continuous pre-training, etc., we select [**LogicKor**](https://github.com/StableFluffy/LogicKor) as our evaluation method to assess the Korean instruction skills. 

We compare our model with a similar parameter model (less than 13B) that has been fine-tuned on the Korean dataset. \* denotes our self-report result.

|Model|single-turn(↑)|multi-turn(↑)|average(↑)|
|-|-|-|-|
|maum-ai/Llama-3-MAAL-8B-Instruct-v0.1*|**5.80**|4.66|**5.23**|
|maywell/Synatra-kiqu-10.7B|5.71|4.73|5.22|
|yanolja/EEVE-Korean-Instruct-10.8B-v1.0|5.78|3.92|4.85|
|nlpai-lab/KULLM3|4.61|**4.83**|4.72|
|MLP-KTLim/llama3-Bllossom*|2.11|1.57|1.84|

## Limitations
Due to this model being trained on a small dataset, it has several limitations.
- Hard to generate diverse Korean texts
- lack of Korean knowledge & Culture (localization)
- Not work with Image inputs and video inputs

## Todo
we will solve these limitations one by one by upgrading this model like as...
- Enhance the Korean generation through Vocabulary Expansion & Continuous pre-training. (more Korean corpus!)
- Localize with cultural adaptation method and additional Korean knowledge data. [*similar idea*](https://aclanthology.org/2023.emnlp-main.18/)
- Develop a Vision Language Model that can handle both video and image inputs. [*similar idea*](https://github.com/PKU-YuanGroup/Video-LLaVA)