Create README.md

README.md

@@ -0,0 +1,100 @@
+---
+license: apache-2.0
+language:
+- en
+- ko
+library_name: transformers
+tags:
+- KPMG
+- Lighthouse
+- legal
+- finance
+- account
+- DocAI
+- LLM
+- NLP
+- DPO
+---
+# Model Card for Mistral-7B-lighthouse-merge-v0.1
+
+This model is based on the ["mistralai/Mistral-7B-v0.1"](https://huggingface.co/mistralai/Mistral-7B-v0.1) model, fine-tuned through DPO training, and is a combination of that model and ["mistralai/Mistral-7B-instruct-v0.2"](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) using the ["mergekit"](https://github.com/cg123/mergekit) with the "slerp" method.
+
+### Jinja Prompt Template
+```
+{{ bos_token }}
+{% for message in messages %}
+  {% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}
+  {% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}
+  {% elif message['role']=='system' %}{{ message['content'] + '\n\n' }}
+  {% else %}{{ raise_exception('Only user and assistant roles are supported!') }}
+  {% endif %}
+{% endfor %}
+```
+
+This format is available as a [chat template](https://huggingface.co/docs/transformers/main/chat_templating) via the `apply_chat_template()` method:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+device = "cuda" # the device to load the model onto
+
+model = AutoModelForCausalLM.from_pretrained("lighthouse-kr/Mistral-7B-lighthouse-merge-v0.1")
+tokenizer = AutoTokenizer.from_pretrained("lighthouse-kr/Mistral-7B-lighthouse-merge-v0.1")
+
+messages = [
+    {"role": "system", "content": "You are the 'KPMG Lighthouse' assistant AI. Provide accurate information in response to your questions or commands. However, if any information is uncertain, either refrain from answering or specify the uncertain aspects."}
+    {"role": "user", "content": "What is the KPMG Lighthouse?"},
+    {"role": "assistant", "content": "KPMG Lighthouse is a global center of excellence within KPMG, focusing on data and analytics, artificial intelligence, and technological innovation. The primary objective of this organization is to provide better insights and solutions to KPMG's clients by leveraging data science, artificial intelligence, and big data analytics. The Lighthouse team assists companies in understanding complex data, improving their business strategies through it, and developing innovative technological solutions. They also offer a range of services including custom application development, advanced analytics, machine learning, and AI solution implementation."},
+    {"role": "user", "content": "Is there a center in South Korea?"}
+]
+
+
+"""
+# Chat template
+"<s>{System message}
+
+[INST] {Instruction} [/INST]"
+"{Assistant respond}</s> "
+"[INST] {Instruction} [/INST]"
+"""
+
+encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")
+
+"""
+# Converted messages by tokenizer
+"<s>You are the 'KPMG Lighthouse' assistant AI. Provide accurate information in response to your questions or commands. However, if any information is uncertain, either refrain from answering or specify the uncertain aspects.
+
+[INST] What is the KPMG Lighthouse? [/INST]"
+"KPMG Lighthouse is a global center of excellence within KPMG, focusing on data and analytics, artificial intelligence, and technological innovation. The primary objective of this organization is to provide better insights and solutions to KPMG's clients by leveraging data science, artificial intelligence, and big data analytics. The Lighthouse team assists companies in understanding complex data, improving their business strategies through it, and developing innovative technological solutions. They also offer a range of services including custom application development, advanced analytics, machine learning, and AI solution implementation.</s> "
+"[INST] Is there a center in South Korea? [/INST]"
+"""
+
+model_inputs = encodeds.to(device)
+model.to(device)
+
+generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
+decoded = tokenizer.batch_decode(generated_ids)
+print(decoded[0])
+```
+
+## Benchmark
+
+https://github.com/EleutherAI/lm-evaluation-harness
+
+| Benchmark       | Accuracy (acc, none) | Accuracy Std. Error (acc_stderr, none) | F1 Score (f1, none) | F1 Std. Error (f1_stderr, none) |
+|-----------------|----------------------|----------------------------------------|---------------------|---------------------------------|
+| kobest_boolq    | 0.8703703703703703   | 0.008967581939336385                   | 0.8702057584740511  | N/A                             |
+| kobest_copa     | 0.662                | 0.014965960710224485                   | 0.6613443626861604  | N/A                             |
+| kobest_hellaswag| 0.45                 | 0.022270877485360437                   | 0.44624027875520467 | N/A                             |
+| kobest_sentineg | 0.8942065491183879   | 0.015456128580187963                   | 0.894092987804878   | N/A                             |
+| kobest_wic      | 0.611904761904762    | 0.013734036852973102                   | 0.6118927832869621  | N/A                             |
+
+
+## Model Architecture
+This model is based on Mistral-7B-v0.1, a transformer model with the following architecture choices:
+- Grouped-Query Attention
+- Sliding-Window Attention
+- Byte-fallback BPE tokenizer
+
+## Requirements
+transformers >= 4.33.4