license: apache-2.0
tags:
- generated_from_trainer
metrics:
- f1
- auc
model-index:
- name: pretrained_model
results:
- task:
name: Text Classification
type: text-classification
metrics:
- name: F1
type: f1
value: 0.6356
- name: AUC
type: auc
value: 0.7643
widget:
- text: >-
I have trouble understanding what other people think or feel. I also like
numbers, and finding patterns in numbers.
This model is a hybrid fine-tuned version of distilbert-base-uncased on Reddit dataset contains text related to mental health reports of users. it predicts mental health disorders from textual content.
It achieves the following results on the validation set:
- Loss: 0.1873
- F1: 0.6356
- AUC: 0.7643
- Precision: 0.7671
Description
This model is based on an existing lighter variation of BERT (distilBERT), in order to predict different mental disorders.
- It is using combinded features of sentiments and emotions (distilbert-base-uncased-finetuned-sst-2-english and roberta-base-go_emotions).
- It is trained on a costume dataset of texts or posts (from Reddit) about general experiences of users with mental health problems.
- All direct mentions of the disorder names in the texts were removed.
It includes the following classes:
- Borderline
- Anxiety
- Depression
- Bipolar
- OCD
- ADHD
- Schizophrenia
- Asperger
- PTSD
Training
Train size: 90%
Val size: 10%
Training set class counts (text samples) after balancing:
Borderline: 10398
Anxiety: 10393
Depression: 10400
Bipolar: 10359
OCD: 10413
ADHD: 10412
Schizophrenia: 10447
Asperger: 10470
PTSD: 10489
Validation set class counts after balancing:
Borderline: 1180
Anxiety: 1185
Depression: 1178
Bipolar: 1219
OCD: 1165
ADHD: 1166
Schizophrenia: 1131
Asperger: 1108
PTSD: 1089
model-finetuning: distilbert/distilbert-base-uncased
additional features (GoEmotions - SamLowe/roberta-base-go_emotions + SST2 - distilbert/distilbert-base-uncased-finetuned-sst-2-english):
negative, positive, admiration, amusement, anger, annoyance, approval, caring, confusion, curiosity,
desire, disappointment, disapproval, disgust, embarrassment, excitement, fear, gratitude, grief,
joy, love, nervousness, optimism, pride, realization, relief, remorse, sadness, surprise, neutral
The following hyperparameters were used during training:
learning_rate: 1e-5
train_batch_size: 64
val_batch_size: 64
weight_decay: 0.01
optimizer: AdamW
num_epochs: 2-3
Training results
Epoch | Training Loss | Validation Loss |
---|---|---|
1.0 | 0.2660 | 0.2031 |
2.0 | 0.1891 | 0.1872 |
F1 Score: 0.6355
AUC Score: 0.7642
Classification Report
Borderline:
Precision: 0.7606
Recall: 0.4525
F1-score: 0.5674
Anxiety:
Precision: 0.7063
Recall: 0.5459
F1-score: 0.6158
Depression:
Precision: 0.7286
Recall: 0.4626
F1-score: 0.5659
Bipolar:
Precision: 0.7997
Recall: 0.4487
F1-score: 0.5748
OCD:
Precision: 0.8222
Recall: 0.5957
F1-score: 0.6908
ADHD:
Precision: 0.8856
Recall: 0.5711
F1-score: 0.6944
Schizophrenia:
Precision: 0.7540
Recall: 0.6153
F1-score: 0.6777
Asperger:
Precision: 0.6743
Recall: 0.6335
F1-score: 0.6533
PTSD:
Precision: 0.7724
Recall: 0.6235
F1-score: 0.6900