File size: 5,688 Bytes
34ef09c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
0875e77
34ef09c
0875e77
 
 
 
 
 
 
 
 
 
 
34ef09c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
---
license: apache-2.0
language:
- en
- ko
library_name: transformers
tags:
- KPMG
- Lighthouse
- legal
- finance
- account
- DocAI
- LLM
- NLP
- DPO
---
# Model Overview

This model is a result of a sophisticated merging process involving two distinct models, Model A and Model B. The merging methodology employed is the "slerp" technique, facilitated by the use of [mergekit](https://github.com/cg123/mergekit).

## Component Models

### Model A
- **Source**: [mistralai/Mistral-7B-instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)
- **Description**: This model is designed to provide instruction-based outputs, enhancing the clarity and precision in response generation.

### Model B
- **Source**: Based on [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
- **Enhancements**: Fine-tuned using DPO (Direct Preference Optimization) training, augmenting its capabilities for more adaptive and context-aware responses.

### Jinja Prompt Template
```
{{ bos_token }}
{% for message in messages %}
  {% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}
  {% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}
  {% elif message['role']=='system' %}{{ message['content'] + '\n\n' }}
  {% else %}{{ raise_exception('Only user and assistant roles are supported!') }}
  {% endif %}
{% endfor %}
```

This format is available as a [chat template](https://huggingface.co/docs/transformers/main/chat_templating) via the `apply_chat_template()` method:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained("lighthouse-kr/Mistral-7B-lighthouse-merge-v0.1")
tokenizer = AutoTokenizer.from_pretrained("lighthouse-kr/Mistral-7B-lighthouse-merge-v0.1")

messages = [
    {"role": "system", "content": "You are the 'KPMG Lighthouse' assistant AI. Provide accurate information in response to your questions or commands. However, if any information is uncertain, either refrain from answering or specify the uncertain aspects."}
    {"role": "user", "content": "What is the KPMG Lighthouse?"},
    {"role": "assistant", "content": "KPMG Lighthouse is a global center of excellence within KPMG, focusing on data and analytics, artificial intelligence, and technological innovation. The primary objective of this organization is to provide better insights and solutions to KPMG's clients by leveraging data science, artificial intelligence, and big data analytics. The Lighthouse team assists companies in understanding complex data, improving their business strategies through it, and developing innovative technological solutions. They also offer a range of services including custom application development, advanced analytics, machine learning, and AI solution implementation."},
    {"role": "user", "content": "Is there a center in South Korea?"}
]

"""
# Chat template
"<s>{System message}

[INST] {Instruction} [/INST]"
"{Assistant respond}</s> "
"[INST] {Instruction} [/INST]"
"""

encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")

"""
# Converted messages by tokenizer
"<s>You are the 'KPMG Lighthouse' assistant AI. Provide accurate information in response to your questions or commands. However, if any information is uncertain, either refrain from answering or specify the uncertain aspects.

[INST] What is the KPMG Lighthouse? [/INST]"
"KPMG Lighthouse is a global center of excellence within KPMG, focusing on data and analytics, artificial intelligence, and technological innovation. The primary objective of this organization is to provide better insights and solutions to KPMG's clients by leveraging data science, artificial intelligence, and big data analytics. The Lighthouse team assists companies in understanding complex data, improving their business strategies through it, and developing innovative technological solutions. They also offer a range of services including custom application development, advanced analytics, machine learning, and AI solution implementation.</s> "
"[INST] Is there a center in South Korea? [/INST]"
"""

model_inputs = encodeds.to(device)
model.to(device)

generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])
```

## Benchmark

https://github.com/EleutherAI/lm-evaluation-harness

| Benchmark       | Accuracy (acc, none) | Accuracy Std. Error (acc_stderr, none) | F1 Score (f1, none) | F1 Std. Error (f1_stderr, none) |
|-----------------|----------------------|----------------------------------------|---------------------|---------------------------------|
| kobest_boolq    | 0.8703703703703703   | 0.008967581939336385                   | 0.8702057584740511  | N/A                             |
| kobest_copa     | 0.662                | 0.014965960710224485                   | 0.6613443626861604  | N/A                             |
| kobest_hellaswag| 0.45                 | 0.022270877485360437                   | 0.44624027875520467 | N/A                             |
| kobest_sentineg | 0.8942065491183879   | 0.015456128580187963                   | 0.894092987804878   | N/A                             |
| kobest_wic      | 0.611904761904762    | 0.013734036852973102                   | 0.6118927832869621  | N/A                             |


## Model Architecture
This model is based on Mistral-7B-v0.1, a transformer model with the following architecture choices:
- Grouped-Query Attention
- Sliding-Window Attention
- Byte-fallback BPE tokenizer

## Requirements
transformers >= 4.33.4