README.md

metadata

license: apache-2.0
language:
  - en
  - ko
library_name: transformers
tags:
  - KPMG
  - Lighthouse
  - legal
  - finance
  - account
  - DocAI
  - LLM
  - NLP
  - DPO

Model Card for Mistral-7B-lighthouse-merge-v0.1

This model is a result of merging two models A and B.
The method used for merging is "slerp" with mergekit.

A: mistralai/Mistral-7B-instruct-v0.2
B: mistralai/Mistral-7B-v0.1 fine-tuned through DPO training.

Jinja Prompt Template

{{ bos_token }}
{% for message in messages %}
  {% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}
  {% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}
  {% elif message['role']=='system' %}{{ message['content'] + '\n\n' }}
  {% else %}{{ raise_exception('Only user and assistant roles are supported!') }}
  {% endif %}
{% endfor %}

This format is available as a chat template via the apply_chat_template() method:

from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained("lighthouse-kr/Mistral-7B-lighthouse-merge-v0.1")
tokenizer = AutoTokenizer.from_pretrained("lighthouse-kr/Mistral-7B-lighthouse-merge-v0.1")

messages = [
    {"role": "system", "content": "You are the 'KPMG Lighthouse' assistant AI. Provide accurate information in response to your questions or commands. However, if any information is uncertain, either refrain from answering or specify the uncertain aspects."}
    {"role": "user", "content": "What is the KPMG Lighthouse?"},
    {"role": "assistant", "content": "KPMG Lighthouse is a global center of excellence within KPMG, focusing on data and analytics, artificial intelligence, and technological innovation. The primary objective of this organization is to provide better insights and solutions to KPMG's clients by leveraging data science, artificial intelligence, and big data analytics. The Lighthouse team assists companies in understanding complex data, improving their business strategies through it, and developing innovative technological solutions. They also offer a range of services including custom application development, advanced analytics, machine learning, and AI solution implementation."},
    {"role": "user", "content": "Is there a center in South Korea?"}
]

"""
# Chat template
"<s>{System message}

[INST] {Instruction} [/INST]"
"{Assistant respond}</s> "
"[INST] {Instruction} [/INST]"
"""

encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")

"""
# Converted messages by tokenizer
"<s>You are the 'KPMG Lighthouse' assistant AI. Provide accurate information in response to your questions or commands. However, if any information is uncertain, either refrain from answering or specify the uncertain aspects.

[INST] What is the KPMG Lighthouse? [/INST]"
"KPMG Lighthouse is a global center of excellence within KPMG, focusing on data and analytics, artificial intelligence, and technological innovation. The primary objective of this organization is to provide better insights and solutions to KPMG's clients by leveraging data science, artificial intelligence, and big data analytics. The Lighthouse team assists companies in understanding complex data, improving their business strategies through it, and developing innovative technological solutions. They also offer a range of services including custom application development, advanced analytics, machine learning, and AI solution implementation.</s> "
"[INST] Is there a center in South Korea? [/INST]"
"""

model_inputs = encodeds.to(device)
model.to(device)

generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])

Benchmark

https://github.com/EleutherAI/lm-evaluation-harness

Benchmark	Accuracy (acc, none)	Accuracy Std. Error (acc_stderr, none)	F1 Score (f1, none)	F1 Std. Error (f1_stderr, none)
kobest_boolq	0.8703703703703703	0.008967581939336385	0.8702057584740511	N/A
kobest_copa	0.662	0.014965960710224485	0.6613443626861604	N/A
kobest_hellaswag	0.45	0.022270877485360437	0.44624027875520467	N/A
kobest_sentineg	0.8942065491183879	0.015456128580187963	0.894092987804878	N/A
kobest_wic	0.611904761904762	0.013734036852973102	0.6118927832869621	N/A

Model Architecture

This model is based on Mistral-7B-v0.1, a transformer model with the following architecture choices:

• Grouped-Query Attention
• Sliding-Window Attention
• Byte-fallback BPE tokenizer

Requirements

transformers >= 4.33.4