File size: 8,965 Bytes
c8004e2
f7d5e2b
c8004e2
46612b9
 
 
 
 
 
e180a4f
ccf61ca
bc4a72f
356227a
 
 
 
 
 
 
 
 
 
 
 
 
ae0428d
 
 
 
 
 
46612b9
 
 
 
bede256
c8004e2
 
 
 
 
46612b9
 
c8004e2
e1d5826
d9472c9
c8004e2
 
 
 
46612b9
c8004e2
46612b9
 
 
 
 
c8004e2
46612b9
 
 
c8004e2
46612b9
 
 
 
c8004e2
 
 
46612b9
 
c8004e2
5189f32
 
 
 
 
 
 
 
 
 
 
46612b9
c8004e2
5189f32
 
c8004e2
5189f32
c8004e2
5189f32
c8004e2
46612b9
5189f32
 
46612b9
c8004e2
46612b9
 
 
 
 
c8004e2
 
46612b9
 
 
 
c8004e2
46612b9
 
 
 
 
 
 
 
 
c8004e2
 
46612b9
c8004e2
46612b9
c8004e2
 
 
 
 
 
 
46612b9
 
 
c8004e2
 
 
 
 
46612b9
 
 
 
 
 
 
c8004e2
 
 
 
 
 
 
46612b9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c8004e2
 
 
ecdb82d
c8004e2
46612b9
 
78d6b41
c8004e2
46612b9
 
 
ef8bb4a
c8004e2
 
bc4a72f
46612b9
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
---
library_name: peft
tags:
- SFT
- PEFT
- Mental Health
- Psychotherapy
- Fine-tuning
- Text Generation
- Chatbot
license: mit
widget:
- example_title: Depersonalization
  messages:
  - role: user
    content: >-
      I feel like I don't exist and my body is not my own, like if I'm sombody
      else observin me, what could be this disorder?
- example_title: Confused
  messages:
  - role: user
    content: >-
      I'm always confused when making decisions, I cannot choose one option, I
      must have only one option so I can make a decision, what are the causes of
      being indecisive?
inference:
  parameters:
    max_new_tokens: 256
    stop:
    - <|end_of_text|>
    - <|eot_id|>
datasets:
- mpingale/mental-health-chat-dataset
- Amod/mental_health_counseling_conversations
- heliosbrahma/mental_health_chatbot_dataset
pipeline_tag: text-generation
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->
A fine-tuned Llama-3-8B-instruct trained to provide initial support and guidance in mental health by leveraging the power of large language models (LLMs).
It's a fine-tuned version of the LLaMA 3 model specifically trained on mental health counseling conversations.

<img src="https://cdn-uploads.huggingface.co/production/uploads/65f2cc4457fbd3c9fb65b34c/ujViqdS9zgW-IrEFbAKLw.jpeg" width="480"  style="width:400px;">



## Model Details

Llama-3-8B-chat-psychotherapist

Provides active listening: The model is trained to follow conversation flows and respond attentively to user expressions.
Offers empathetic support: The model's responses are designed to be supportive and understanding, fostering a safe space for users to share their concerns.
Guides users towards self-reflection: The model can ask open-ended questions and prompt users to explore their thoughts and feelings.
Offers resources and information: Depending on the conversation, the model can provide users with relevant mental health resources and information.
Important to Note:

**This model is intended for initial support and guidance, not a replacement for professional mental health care.
It's crucial to emphasize that users should seek professional help if needed.
The model is still under development, and its responses may require human oversight for accuracy and safety.**

Further Considerations:
You can specify the type of mental health conversations the model is trained on (e.g., anxiety, depression, general well-being).
Briefly mention the dataset used for fine-tuning to showcase the model's training grounds.
If applicable, mention any limitations of the model, such as inability to diagnose or provide specific treatment plans.



## How to Get Started with the Model
Use the code below to get started with the model.

### Read model

```
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM

model_id = "zementalist/llama-3-8B-chat-psychotherapist"
config = PeftConfig.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path)
model = PeftModel.from_pretrained(model, model_id)

```

### Inference


```

question = "I feel like I don't exist and my body is not my own, I'm sombody else observing me, what causes these ideas?"

messages = [
    {"role": "system", "content": "Answer the following inquiry:"},
    {"role": "user", "content": question}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)


terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.01
)
response = outputs[0][input_ids.shape[-1]:]
output = tokenizer.decode(response, skip_special_tokens=True)


print(output)

```



## Training Details

### Training Data

1. [Amod/mental_health_counseling_conversations](https://huggingface.co/datasets/Amod/mental_health_counseling_conversations)
2. [mpingale/mental-health-chat-dataset](https://huggingface.co/datasets/mpingale/mental-health-chat-dataset)
3. [heliosbrahma/mental_health_chatbot_dataset](https://huggingface.co/datasets/heliosbrahma/mental_health_chatbot_dataset)


#### Training Hyperparameters


* epochs: 2
* learning rate: 6e-5
* learning rate scheduler type: constant
* Warmup ratio: 0.03
* optimizer: paged_adamw_32bit
* weight_decay: 0.001
* fp16: True

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->



| **Step** | **Training Loss** |
|---------|-------------------|
| 100     | 2.637700          |
| 200     | 2.350900          |
| 300     | 2.234500          |
| 400     | 2.230800          |
| 500     | 2.194900          |
| 600     | 2.041100          |
| 700     | 2.033800          |
| 800     | 1.988600          |
| 900     | 1.898900          |
| 1000    | 1.823600          |
| 1100    | 1.943700          |
| 1200    | 1.789900          |
| 1300    | 1.840700          |
| 1400    | 1.760900          |
| 1500    | 1.718400          |
| 1600    | 1.725600          |
| 1700    | 1.717600          |
| 1800    | 1.628200          |
| 1900    | 1.693300          |
| 2000    | 1.535600          |
| 2100    | 1.630100          |
| 2200    | 1.592300          |
| 2300    | 1.628400          |
| 2400    | 1.462500          |
| 2500    | 1.504500          |
| 2600    | 1.517500          |
| 2700    | 1.375100          |
| 2800    | 1.452800          |
| 2900    | 1.527200          |
| 3000    | 1.418600          |
| 3100    | 1.437700          |
| 3200    | 1.247900          |
| 3300    | 1.134900          |
| 3400    | 1.283800          |
| 3500    | 1.378600          |
| 3600    | 1.114600          |
| 3700    | 1.300900          |
| 3800    | 1.225700          |
| 3900    | 1.190000          |
| 4000    | 1.260200          |
| 4100    | 1.053300          |
| 4200    | 1.173000          |
| 4300    | 1.275900          |
| 4400    | 1.079800          |
| 4500    | 1.121100          |
| 4600    | 1.043100          |
| 4700    | 1.121000          |
| 4800    | 1.087200          |
| 4900    | 1.102400          |
| 5000    | 1.126100          |
| 5100    | 1.182200          |
| 5200    | 1.156500          |
| 5300    | 0.979700          |
| 5400    | 1.145300          |
| 5500    | 1.018100          |
| 5600    | 1.070500          |
| 5700    | 0.954900          |
| 5800    | 1.074200          |
| 5900    | 1.022800          |
| 6000    | 1.037700          |
| 6100    | 0.913400          |
| 6200    | 0.929500          |



### Inference example

1. Question: I feel like I don't exist and my body is not my own,
like if I'm sombody else observin me, what could be this disorder?
* * Answer: You're describing a phenomenon called depersonalization. It's a common symptom of anxiety disorders, post-traumatic stress disorder (PTSD), and other mental health conditions. <br> Depersonalization can make you feel like you're observing yourself from the outside, like you're watching a movie or reading a book about yourself. You might also feel like you're not in control of your body or that you're just a puppet. <br> This feeling can be very distressing and can make you wonder if you're even alive. It's a very disorienting and unsettling experience. <br> If you're experiencing depersonalization, it's important to seek help from a mental health professional. They can help you identify the underlying causes of your feelings and develop coping strategies to manage them. <br>

2. Question: I'm always confused when making decisions, I cannot choose one option,
I must have only one option so I can make a decision, what are the causes of being
indecisive?
* * Answer: Indecisiveness is a common phenomenon that many people experience. It's not a mental health condition, but it can be frustrating and time-consuming. Here are some possible causes of indecisiveness: <ol> <li>Fear of making the wrong decision: Sometimes, people are afraid of making the wrong choice, which can lead to indecisiveness. This fear can be due to various factors such as fear of failure, fear of rejection, or fear of the unknown.</li> <li>Lack of clear goals: When people don't have clear goals, they can be indecisive. This is because they don't know what they want, which can lead to confusion and indecisiveness.</li> <li>Overthinking: Overthinking can be a major cause of indecisiveness. When people overthink, they can get caught up in their thoughts and be unable to make a decision.</li> <li>Lack of self-confidence: People who lack self-confidence can be indecisive. This is because they don't believe in themselves and are afraid of making the wrong decision.</li> <li>Avoidance: Avoiding a decision can be a cause of indecisiveness. When people avoid making a decision, they can be indecisive because they don't know</li> </ol>



- **Developed by:** [Zeyad Khalid]