bert-finetuned-arcchialogy-ner-hp-tunned

This model is a fine-tuned version of bert-base-cased on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.2776
• Precision: 0.5066
• Recall: 0.6780
• F1: 0.5799
• Accuracy: 0.9444
• Classification Report Details: {'B-ART': {'precision': 0.49854227405247814, 'recall': 0.6785714285714286, 'f1-score': 0.5747899159663865, 'support': 252.0}, 'B-CON': {'precision': 0.3862815884476534, 'recall': 0.6645962732919255, 'f1-score': 0.4885844748858447, 'support': 161.0}, 'B-LOC': {'precision': 0.8098591549295775, 'recall': 0.7718120805369127, 'f1-score': 0.7903780068728522, 'support': 149.0}, 'B-MAT': {'precision': 0.5185185185185185, 'recall': 0.35, 'f1-score': 0.417910447761194, 'support': 40.0}, 'B-PER': {'precision': 0.8026666666666666, 'recall': 0.8958333333333334, 'f1-score': 0.8466947960618847, 'support': 336.0}, 'B-SPE': {'precision': 0.4642857142857143, 'recall': 0.8387096774193549, 'f1-score': 0.5977011494252874, 'support': 31.0}, 'I-ART': {'precision': 0.5234899328859061, 'recall': 0.44192634560906513, 'f1-score': 0.4792626728110599, 'support': 353.0}, 'I-CON': {'precision': 0.42657342657342656, 'recall': 0.5169491525423728, 'f1-score': 0.4674329501915709, 'support': 118.0}, 'I-LOC': {'precision': 0.8677685950413223, 'recall': 0.8267716535433071, 'f1-score': 0.8467741935483871, 'support': 254.0}, 'I-MAT': {'precision': 0.36363636363636365, 'recall': 0.10810810810810811, 'f1-score': 0.16666666666666666, 'support': 37.0}, 'I-PER': {'precision': 0.8685567010309279, 'recall': 0.7539149888143176, 'f1-score': 0.807185628742515, 'support': 447.0}, 'I-SPE': {'precision': 0.8484848484848485, 'recall': 0.6666666666666666, 'f1-score': 0.7466666666666667, 'support': 42.0}, 'O': {'precision': 0.9772175264743029, 'recall': 0.9717888024733105, 'f1-score': 0.9744956039431298, 'support': 20701.0}, 'accuracy': 0.9444177828192487, 'macro avg': {'precision': 0.6427601008482852, 'recall': 0.652742193146931, 'f1-score': 0.6311187056571883, 'support': 22921.0}, 'weighted avg': {'precision': 0.9482823264495688, 'recall': 0.9444177828192487, 'f1-score': 0.9454997462611199, 'support': 22921.0}}
• Classfication Report Seqeval: {'ART': {'precision': 0.3918918918918919, 'recall': 0.5753968253968254, 'f1-score': 0.4662379421221865, 'support': 252}, 'CON': {'precision': 0.34146341463414637, 'recall': 0.6086956521739131, 'f1-score': 0.4375, 'support': 161}, 'LOC': {'precision': 0.6772151898734177, 'recall': 0.7181208053691275, 'f1-score': 0.6970684039087949, 'support': 149}, 'MAT': {'precision': 0.4, 'recall': 0.3, 'f1-score': 0.34285714285714286, 'support': 40}, 'PER': {'precision': 0.6972010178117048, 'recall': 0.8154761904761905, 'f1-score': 0.7517146776406035, 'support': 336}, 'SPE': {'precision': 0.3559322033898305, 'recall': 0.6774193548387096, 'f1-score': 0.4666666666666666, 'support': 31}, 'micro avg': {'precision': 0.5065535851966075, 'recall': 0.6780185758513931, 'f1-score': 0.5798764342453663, 'support': 969}, 'macro avg': {'precision': 0.4772839529334985, 'recall': 0.6158514713757943, 'f1-score': 0.5270074721992324, 'support': 969}, 'weighted avg': {'precision': 0.5324363984456255, 'recall': 0.6780185758513931, 'f1-score': 0.5908666023378706, 'support': 969}}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 7.73381107021748e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy	Classification Report Details	Classfication Report Seqeval
No log	1.0	125	0.2224	0.4311	0.6388	0.5148	0.9384	{'B-ART': {'precision': 0.4114942528735632, 'recall': 0.7103174603174603, 'f1-score': 0.5211062590975255, 'support': 252.0}, 'B-CON': {'precision': 0.3333333333333333, 'recall': 0.6832298136645962, 'f1-score': 0.4480651731160896, 'support': 161.0}, 'B-LOC': {'precision': 0.7906976744186046, 'recall': 0.6845637583892618, 'f1-score': 0.7338129496402878, 'support': 149.0}, 'B-MAT': {'precision': 0.41025641025641024, 'recall': 0.4, 'f1-score': 0.4050632911392405, 'support': 40.0}, 'B-PER': {'precision': 0.7877984084880637, 'recall': 0.8839285714285714, 'f1-score': 0.8330995792426368, 'support': 336.0}, 'B-SPE': {'precision': 0.43137254901960786, 'recall': 0.7096774193548387, 'f1-score': 0.5365853658536586, 'support': 31.0}, 'I-ART': {'precision': 0.548, 'recall': 0.3881019830028329, 'f1-score': 0.45439469320066334, 'support': 353.0}, 'I-CON': {'precision': 0.4864864864864865, 'recall': 0.4576271186440678, 'f1-score': 0.47161572052401746, 'support': 118.0}, 'I-LOC': {'precision': 0.9016393442622951, 'recall': 0.6496062992125984, 'f1-score': 0.7551487414187643, 'support': 254.0}, 'I-MAT': {'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 37.0}, 'I-PER': {'precision': 0.8980169971671388, 'recall': 0.70917225950783, 'f1-score': 0.7925, 'support': 447.0}, 'I-SPE': {'precision': 0.8888888888888888, 'recall': 0.7619047619047619, 'f1-score': 0.8205128205128205, 'support': 42.0}, 'O': {'precision': 0.9734328792359529, 'recall': 0.9699531423602724, 'f1-score': 0.9716898954703833, 'support': 20701.0}, 'accuracy': 0.93844073120719, 'macro avg': {'precision': 0.6047244018792575, 'recall': 0.6160063529066994, 'f1-score': 0.5956611145550837, 'support': 22921.0}, 'weighted avg': {'precision': 0.9440821519802611, 'recall': 0.93844073120719, 'f1-score': 0.939621956411553, 'support': 22921.0}}	{'ART': {'precision': 0.3130434782608696, 'recall': 0.5714285714285714, 'f1-score': 0.40449438202247195, 'support': 252}, 'CON': {'precision': 0.2865671641791045, 'recall': 0.5962732919254659, 'f1-score': 0.38709677419354843, 'support': 161}, 'LOC': {'precision': 0.5328947368421053, 'recall': 0.5436241610738255, 'f1-score': 0.5382059800664453, 'support': 149}, 'MAT': {'precision': 0.28205128205128205, 'recall': 0.275, 'f1-score': 0.27848101265822783, 'support': 40}, 'PER': {'precision': 0.6777493606138107, 'recall': 0.7886904761904762, 'f1-score': 0.7290233837689133, 'support': 336}, 'SPE': {'precision': 0.3728813559322034, 'recall': 0.7096774193548387, 'f1-score': 0.4888888888888889, 'support': 31}, 'micro avg': {'precision': 0.431058495821727, 'recall': 0.6388028895768834, 'f1-score': 0.5147609147609148, 'support': 969}, 'macro avg': {'precision': 0.41086456297989593, 'recall': 0.5807823199955297, 'f1-score': 0.4710317369330826, 'support': 969}, 'weighted avg': {'precision': 0.46954669166794494, 'recall': 0.6388028895768834, 'f1-score': 0.5321924756996534, 'support': 969}}
No log	2.0	250	0.2168	0.4916	0.6646	0.5652	0.9429	{'B-ART': {'precision': 0.53156146179402, 'recall': 0.6349206349206349, 'f1-score': 0.5786618444846293, 'support': 252.0}, 'B-CON': {'precision': 0.40441176470588236, 'recall': 0.6832298136645962, 'f1-score': 0.5080831408775982, 'support': 161.0}, 'B-LOC': {'precision': 0.7928571428571428, 'recall': 0.7449664429530202, 'f1-score': 0.7681660899653979, 'support': 149.0}, 'B-MAT': {'precision': 0.40625, 'recall': 0.325, 'f1-score': 0.3611111111111111, 'support': 40.0}, 'B-PER': {'precision': 0.7875647668393783, 'recall': 0.9047619047619048, 'f1-score': 0.8421052631578947, 'support': 336.0}, 'B-SPE': {'precision': 0.4375, 'recall': 0.9032258064516129, 'f1-score': 0.5894736842105263, 'support': 31.0}, 'I-ART': {'precision': 0.6050420168067226, 'recall': 0.40793201133144474, 'f1-score': 0.4873096446700508, 'support': 353.0}, 'I-CON': {'precision': 0.3413173652694611, 'recall': 0.4830508474576271, 'f1-score': 0.4, 'support': 118.0}, 'I-LOC': {'precision': 0.83203125, 'recall': 0.8385826771653543, 'f1-score': 0.8352941176470589, 'support': 254.0}, 'I-MAT': {'precision': 0.6666666666666666, 'recall': 0.05405405405405406, 'f1-score': 0.1, 'support': 37.0}, 'I-PER': {'precision': 0.8964497041420119, 'recall': 0.6778523489932886, 'f1-score': 0.7719745222929937, 'support': 447.0}, 'I-SPE': {'precision': 0.8055555555555556, 'recall': 0.6904761904761905, 'f1-score': 0.7435897435897436, 'support': 42.0}, 'O': {'precision': 0.9734628770301624, 'recall': 0.9728515530650693, 'f1-score': 0.9731571190412912, 'support': 20701.0}, 'accuracy': 0.9429344269447232, 'macro avg': {'precision': 0.6523592747436157, 'recall': 0.6400695604072921, 'f1-score': 0.6122250985421765, 'support': 22921.0}, 'weighted avg': {'precision': 0.9461931421892993, 'recall': 0.9429344269447232, 'f1-score': 0.9429981382746248, 'support': 22921.0}}	{'ART': {'precision': 0.4171597633136095, 'recall': 0.5595238095238095, 'f1-score': 0.47796610169491527, 'support': 252}, 'CON': {'precision': 0.31189710610932475, 'recall': 0.6024844720496895, 'f1-score': 0.4110169491525424, 'support': 161}, 'LOC': {'precision': 0.6363636363636364, 'recall': 0.7046979865771812, 'f1-score': 0.6687898089171975, 'support': 149}, 'MAT': {'precision': 0.35294117647058826, 'recall': 0.3, 'f1-score': 0.3243243243243243, 'support': 40}, 'PER': {'precision': 0.6759493670886076, 'recall': 0.7946428571428571, 'f1-score': 0.7305061559507523, 'support': 336}, 'SPE': {'precision': 0.3283582089552239, 'recall': 0.7096774193548387, 'f1-score': 0.4489795918367347, 'support': 31}, 'micro avg': {'precision': 0.4916030534351145, 'recall': 0.6646026831785345, 'f1-score': 0.5651601579640193, 'support': 969}, 'macro avg': {'precision': 0.45377820971683175, 'recall': 0.6118377574413959, 'f1-score': 0.5102638219794111, 'support': 969}, 'weighted avg': {'precision': 0.5176198298607021, 'recall': 0.6646026831785345, 'f1-score': 0.5764832576766821, 'support': 969}}
No log	3.0	375	0.2434	0.5098	0.6718	0.5797	0.9459	{'B-ART': {'precision': 0.48623853211009177, 'recall': 0.6309523809523809, 'f1-score': 0.5492227979274611, 'support': 252.0}, 'B-CON': {'precision': 0.45021645021645024, 'recall': 0.6459627329192547, 'f1-score': 0.5306122448979592, 'support': 161.0}, 'B-LOC': {'precision': 0.7702702702702703, 'recall': 0.7651006711409396, 'f1-score': 0.7676767676767676, 'support': 149.0}, 'B-MAT': {'precision': 0.49019607843137253, 'recall': 0.625, 'f1-score': 0.5494505494505495, 'support': 40.0}, 'B-PER': {'precision': 0.7952755905511811, 'recall': 0.9017857142857143, 'f1-score': 0.8451882845188284, 'support': 336.0}, 'B-SPE': {'precision': 0.4716981132075472, 'recall': 0.8064516129032258, 'f1-score': 0.5952380952380952, 'support': 31.0}, 'I-ART': {'precision': 0.5743801652892562, 'recall': 0.3937677053824363, 'f1-score': 0.4672268907563025, 'support': 353.0}, 'I-CON': {'precision': 0.4580152671755725, 'recall': 0.5084745762711864, 'f1-score': 0.4819277108433735, 'support': 118.0}, 'I-LOC': {'precision': 0.8859649122807017, 'recall': 0.7952755905511811, 'f1-score': 0.8381742738589212, 'support': 254.0}, 'I-MAT': {'precision': 0.2857142857142857, 'recall': 0.21621621621621623, 'f1-score': 0.24615384615384617, 'support': 37.0}, 'I-PER': {'precision': 0.8710526315789474, 'recall': 0.7404921700223713, 'f1-score': 0.8004836759371221, 'support': 447.0}, 'I-SPE': {'precision': 0.875, 'recall': 0.6666666666666666, 'f1-score': 0.7567567567567568, 'support': 42.0}, 'O': {'precision': 0.9755425588476968, 'recall': 0.974977054248587, 'f1-score': 0.9752597245711524, 'support': 20701.0}, 'accuracy': 0.9459011386937742, 'macro avg': {'precision': 0.6453511427441055, 'recall': 0.6670094685815509, 'f1-score': 0.6464132014297796, 'support': 22921.0}, 'weighted avg': {'precision': 0.9477970361286707, 'recall': 0.9459011386937742, 'f1-score': 0.9460706028043416, 'support': 22921.0}}	{'ART': {'precision': 0.38375350140056025, 'recall': 0.5436507936507936, 'f1-score': 0.44991789819376027, 'support': 252}, 'CON': {'precision': 0.4, 'recall': 0.5962732919254659, 'f1-score': 0.4788029925187033, 'support': 161}, 'LOC': {'precision': 0.6335403726708074, 'recall': 0.6845637583892618, 'f1-score': 0.6580645161290323, 'support': 149}, 'MAT': {'precision': 0.31746031746031744, 'recall': 0.5, 'f1-score': 0.3883495145631068, 'support': 40}, 'PER': {'precision': 0.6882793017456359, 'recall': 0.8214285714285714, 'f1-score': 0.7489823609226594, 'support': 336}, 'SPE': {'precision': 0.36363636363636365, 'recall': 0.6451612903225806, 'f1-score': 0.4651162790697675, 'support': 31}, 'micro avg': {'precision': 0.5097885669537979, 'recall': 0.6718266253869969, 'f1-score': 0.5796972395369546, 'support': 969}, 'macro avg': {'precision': 0.4644449761522808, 'recall': 0.6318462842861122, 'f1-score': 0.5315389268995049, 'support': 969}, 'weighted avg': {'precision': 0.5270757308963521, 'recall': 0.6718266253869969, 'f1-score': 0.5883682802345359, 'support': 969}}
0.0959	4.0	500	0.2776	0.5066	0.6780	0.5799	0.9444	{'B-ART': {'precision': 0.49854227405247814, 'recall': 0.6785714285714286, 'f1-score': 0.5747899159663865, 'support': 252.0}, 'B-CON': {'precision': 0.3862815884476534, 'recall': 0.6645962732919255, 'f1-score': 0.4885844748858447, 'support': 161.0}, 'B-LOC': {'precision': 0.8098591549295775, 'recall': 0.7718120805369127, 'f1-score': 0.7903780068728522, 'support': 149.0}, 'B-MAT': {'precision': 0.5185185185185185, 'recall': 0.35, 'f1-score': 0.417910447761194, 'support': 40.0}, 'B-PER': {'precision': 0.8026666666666666, 'recall': 0.8958333333333334, 'f1-score': 0.8466947960618847, 'support': 336.0}, 'B-SPE': {'precision': 0.4642857142857143, 'recall': 0.8387096774193549, 'f1-score': 0.5977011494252874, 'support': 31.0}, 'I-ART': {'precision': 0.5234899328859061, 'recall': 0.44192634560906513, 'f1-score': 0.4792626728110599, 'support': 353.0}, 'I-CON': {'precision': 0.42657342657342656, 'recall': 0.5169491525423728, 'f1-score': 0.4674329501915709, 'support': 118.0}, 'I-LOC': {'precision': 0.8677685950413223, 'recall': 0.8267716535433071, 'f1-score': 0.8467741935483871, 'support': 254.0}, 'I-MAT': {'precision': 0.36363636363636365, 'recall': 0.10810810810810811, 'f1-score': 0.16666666666666666, 'support': 37.0}, 'I-PER': {'precision': 0.8685567010309279, 'recall': 0.7539149888143176, 'f1-score': 0.807185628742515, 'support': 447.0}, 'I-SPE': {'precision': 0.8484848484848485, 'recall': 0.6666666666666666, 'f1-score': 0.7466666666666667, 'support': 42.0}, 'O': {'precision': 0.9772175264743029, 'recall': 0.9717888024733105, 'f1-score': 0.9744956039431298, 'support': 20701.0}, 'accuracy': 0.9444177828192487, 'macro avg': {'precision': 0.6427601008482852, 'recall': 0.652742193146931, 'f1-score': 0.6311187056571883, 'support': 22921.0}, 'weighted avg': {'precision': 0.9482823264495688, 'recall': 0.9444177828192487, 'f1-score': 0.9454997462611199, 'support': 22921.0}}	{'ART': {'precision': 0.3918918918918919, 'recall': 0.5753968253968254, 'f1-score': 0.4662379421221865, 'support': 252}, 'CON': {'precision': 0.34146341463414637, 'recall': 0.6086956521739131, 'f1-score': 0.4375, 'support': 161}, 'LOC': {'precision': 0.6772151898734177, 'recall': 0.7181208053691275, 'f1-score': 0.6970684039087949, 'support': 149}, 'MAT': {'precision': 0.4, 'recall': 0.3, 'f1-score': 0.34285714285714286, 'support': 40}, 'PER': {'precision': 0.6972010178117048, 'recall': 0.8154761904761905, 'f1-score': 0.7517146776406035, 'support': 336}, 'SPE': {'precision': 0.3559322033898305, 'recall': 0.6774193548387096, 'f1-score': 0.4666666666666666, 'support': 31}, 'micro avg': {'precision': 0.5065535851966075, 'recall': 0.6780185758513931, 'f1-score': 0.5798764342453663, 'support': 969}, 'macro avg': {'precision': 0.4772839529334985, 'recall': 0.6158514713757943, 'f1-score': 0.5270074721992324, 'support': 969}, 'weighted avg': {'precision': 0.5324363984456255, 'recall': 0.6780185758513931, 'f1-score': 0.5908666023378706, 'support': 969}}

Framework versions

• Transformers 4.45.2
• Pytorch 2.4.1
• Datasets 3.0.1
• Tokenizers 0.20.1