s-nlp/xlmr_formality_classifier

XLMRoberta-based classifier trained on XFORMAL.

all

	precision	recall	f1-score	support
0	0.744912	0.927790	0.826354	108019
1	0.889088	0.645630	0.748048	96845
accuracy			0.794405	204864
macro avg	0.817000	0.786710	0.787201	204864
weighted avg	0.813068	0.794405	0.789337	204864

en

	precision	recall	f1-score	support
0	0.800053	0.962981	0.873988	22151
1	0.945106	0.725899	0.821124	19449
accuracy			0.852139	41600
macro avg	0.872579	0.844440	0.847556	41600
weighted avg	0.867869	0.852139	0.849273	41600

fr

	precision	recall	f1-score	support
0	0.746709	0.925738	0.826641	21505
1	0.887305	0.650592	0.750731	19327
accuracy			0.795504	40832
macro avg	0.817007	0.788165	0.788686	40832
weighted avg	0.813257	0.795504	0.790711	40832

it

	precision	recall	f1-score	support
0	0.721282	0.914669	0.806545	21528
1	0.864887	0.607135	0.713445	19368
accuracy			0.769024	40896
macro avg	0.793084	0.760902	0.759995	40896
weighted avg	0.789292	0.769024	0.762454	40896

pt

	precision	recall	f1-score	support
0	0.717546	0.908167	0.801681	21637
1	0.853628	0.599700	0.704481	19323
accuracy			0.762646	40960
macro avg	0.785587	0.753933	0.753081	40960
weighted avg	0.781743	0.762646	0.755826	40960

How to use

from transformers import XLMRobertaTokenizerFast, XLMRobertaForSequenceClassification

# load tokenizer and model weights
tokenizer = XLMRobertaTokenizerFast.from_pretrained('SkolkovoInstitute/xlmr_formality_classifier')
model = XLMRobertaForSequenceClassification.from_pretrained('SkolkovoInstitute/xlmr_formality_classifier')

# prepare the input
batch = tokenizer.encode('ты супер', return_tensors='pt')

# inference
model(batch)

Citation

@inproceedings{dementieva-etal-2023-detecting,
    title = "Detecting Text Formality: A Study of Text Classification Approaches",
    author = "Dementieva, Daryna  and
      Babakov, Nikolay  and
      Panchenko, Alexander",
    editor = "Mitkov, Ruslan  and
      Angelova, Galia",
    booktitle = "Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing",
    month = sep,
    year = "2023",
    address = "Varna, Bulgaria",
    publisher = "INCOMA Ltd., Shoumen, Bulgaria",
    url = "https://aclanthology.org/2023.ranlp-1.31",
    pages = "274--284",
    abstract = "Formality is one of the important characteristics of text documents. The automatic detection of the formality level of a text is potentially beneficial for various natural language processing tasks. Before, two large-scale datasets were introduced for multiple languages featuring formality annotation{---}GYAFC and X-FORMAL. However, they were primarily used for the training of style transfer models. At the same time, the detection of text formality on its own may also be a useful application. This work proposes the first to our knowledge systematic study of formality detection methods based on statistical, neural-based, and Transformer-based machine learning methods and delivers the best-performing models for public usage. We conducted three types of experiments {--} monolingual, multilingual, and cross-lingual. The study shows the overcome of Char BiLSTM model over Transformer-based ones for the monolingual and multilingual formality classification task, while Transformer-based classifiers are more stable to cross-lingual knowledge transfer.",
}

Licensing Information

Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.