Training set class counts after balancing: _Borderline 10398 _Anxiety 10393 _Depression 10400 _Bipolar 10359 _OCD 10413 _ADHD 10412 _Schizophrenia 10447 _Asperger 10470 _PTSD 10489 dtype: object Validation set class counts after balancing: _Borderline 1180 _Anxiety 1185 _Depression 1178 _Bipolar 1219 _OCD 1165 _ADHD 1166 _Schizophrenia 1131 _Asperger 1108 _PTSD 1089 dtype: object Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Epoch 1, Train Loss: 0.2660, Val Loss: 0.2032 Epoch 2, Train Loss: 0.1891, Val Loss: 0.1873 F1 Score on Validation Set: 0.6356 AUC Score on Validation Set: 0.7643 Classification Report, AUC Score, F1 Score, and Losses Train Losses: [0.2660091122228647, 0.18910447711386752] Validation Losses: [0.20317846728614503, 0.18728890073445678] F1 Score: [0.6355810277640191] AUC Score: [0.7642971226283637] Classification Report: _Borderline: Precision: 0.7606837606837606 Recall: 0.45254237288135596 F1-score: 0.5674814027630181 _Anxiety: Precision: 0.7063318777292577 Recall: 0.5459915611814345 F1-score: 0.6158971918134221 _Depression: Precision: 0.7286096256684492 Recall: 0.4626485568760611 F1-score: 0.5659397715472482 _Bipolar: Precision: 0.7997076023391813 Recall: 0.44872846595570137 F1-score: 0.5748817656332107 _OCD: Precision: 0.8222748815165877 Recall: 0.5957081545064378 F1-score: 0.6908909905425585 _ADHD: Precision: 0.8856382978723404 Recall: 0.5711835334476844 F1-score: 0.6944734098018769 _Schizophrenia: Precision: 0.7540628385698809 Recall: 0.6153846153846154 F1-score: 0.6777020447906524 _Asperger: Precision: 0.6743515850144092 Recall: 0.6335740072202166 F1-score: 0.6533271288971615 _PTSD: Precision: 0.7724687144482366 Recall: 0.6235078053259872 F1-score: 0.6900406504065041