Personality Trait Identification Using the R ussian Feature Extraction Toolkit (2021)

Abstract: Feature engineering is an important step in classical NLP pipelines, but machine learning engineers may not be aware of the signals to look for when processing foreign language text. The Russian Feature Extraction Toolkit (RFET) is a collection of feature extraction libraries bundled for ease of use by engineers who do not speak Russian. RFET’s current feature set includes features applicable to social media genres of text and to computational social science tasks. We demonstrate the effectiveness of the tool by using it in a personality trait identification task. We compare the performance of Support Vector Machines (SVMs) trained with and without the features provided by RFET; we also compare it to a SVM with neural embedding features generated by Sentence-BERT.

## introduction data scientists and computer scientists working on natural language processing (nlp) problems are occasionally confronted with tasks in languages they do not know well. technical linguistic issues that may be well known to researchers and language practitioners familiar with the language can be quite important to effective language engineering, and can take a lengthy career to master. the russian feature extraction toolkit (rfet) unites a suite of tools for russian language processing that would allow a programmer unfamiliar with the language the ability to get started quickly in common text classification tasks such as sociolinguistic factor classification, sentiment classification, and emotion analysis. additionally, this toolkit provides those advanced in russian nlp convenient accessibility of features and feature combinations to quickly iterate through multiple experiment scenarios. deep learning approaches are showing great promise, and feature extraction is less commonly utilized in these approaches. still, there are existing systems in production which harness classical machine learning techniques. rfet, and tools like it for other languages, could improve performance of these systems without a complete redesign. further, there are many languages in which deep learning approaches still lack pre-trained models or even a sufficient quantity of text examples to create them. the methodology presented in this paper could be used to design feature extraction toolkits for these languages, in these cases. this toolkit does not replace feature extraction functions in libraries like nltk @xcite or scikit-learn @xcite . these systems can produce languageindependent statistical measurements on text that could be used as features. it also does not seek to replace toolkits for basic nlp pipeline elements, such natasha, foot_0 though there is some overlap between the tools. this toolkit focuses on uniting russian language resources that provide linguistically-informed features that are distinctive of the russian language or otherwise not adequately represented in features used in text analytics or corpus linguistics, with a focus on social media russian. the toolkit includes 70 features. in order to validate the toolkit's utility, we evaluate its efficacy in a challenging task with little prior work in russian: namely, the identification of personality traits from usergenerated social media text. this task is further described in section 2.2. section 3 describes rfet's collected features in more detail; section 4 describes the dataset and other methodological details of the task and our results. other sections outline larger context and limitations of the study. ## discussion for the personality trait identification task on this social media dataset, we see that a classic svm baseline using rfet features outperforms a similar svm without the rfet features. we likewise see that the svm with rfet features outperforms a model using a transformer model pre-trained on russian text. one trait where the advantage of the rfet features is particularly large is the agreeableness trait. one possible explanation for this may be differences in participants' use of emoji, emoticons, and/or emotional words correlating with their agreeableness trait values. rfet includes features specifically developed for emoji, emoticons, and emotional words, and even the svm bow model may be somewhat sensitive to them; the pretrained model, on the other hand, may be ignoring emoji and emoticons, since such "words" may not have appeared in its original training data. another advantage the standard machine learning methods have over neural models is interpretability. since a svm was used with a linear kernel, it is possible to extract feature importance from each model, and gain insights into where the strongest correlations lie. to do this, we employed the r package e1071 @xcite . every model is trained independently, and so produces different feature weights, but in general, these features appeared as the most important for our machine learning models: lastly, rubert was trained on wikipedia and books. our experiments were on social media text, which rfet was designed to address. this genre mismatch (and rubert's resulting limited coverage for wordpieces specific to social media) may have limited sentence rubert's effectiveness for this task. one possible weakness in the rfet model (as trained on these training data) is sparsity of certain features. certain rfet features may be overfitting to particular participants. idiosyncratic uses of words from a few people with unusual personality trait values or a large volume of posts may be inappropriately generalized as signals for those trait values. one possible mitigation, of course, is the collection of data from a much larger set of individuals. in these experiments, we apply a more local mitigation strategy: combining different kinds of laughter into a single feature. one strength of rfet is the flexibility of combining or splitting apart features according to the need of the task. for example, personality trait (or other attribute) detection may benefit from combining features; author identification or verification may benefit from keeping very specific features split apart. future feature extractors include frequency of russian diminutives through the usage of suffixes and infixes, ratio of other language scripts being used in the text and usage and frequency of filler words and phrases. additional information about sentiment and prevalence of emojis will be incorporated from the emoji sentiment ranking @xcite , foot_10 which provides usage statistics and associated sentiment for each emoji. an expanded list of emoticons will also be added to reflect the frequency and usage of more emoticons. future iterations of rfet may utilize rusentilex @xcite , a russian emotion lexicon with 16,057 entries. each entry includes a description of its syntactic category, a lemmatized version, the sentiment valence, and source of the valence (opinion, feeling, or fact). ambiguous entries in rusentilex (with more than one possible value for valence or source) are also elaborated with examples. we anticipate these features will provide better coverage of sentiment than the 590 russian translation of the emolex dictionary. ## conclusion this paper introduces the russian feature extraction toolkit, an api for feature extraction on the russian language. each feature in the toolkit utilizes linguistic knowledge of the russian language. it is designed to get a russian non-speaker up and running quickly on russian nlp tasks, and to speed up the workflow of russian speaking nlp programmers. we have shown that it improves performance on a big five personality trait inference task relative to a svm baseline with only languageindependent features and, more surprisingly, to a pre-trained transformer baseline using sentence rubert. this suggests that rfet's features specific to social media can be very useful for enhancing state-of-the-art methods for certain genres or domains. licensing: we plan to release rfet for noncommercial research and education. a public api will be made available for demonstration purposes. for commercial licenses, contact the university of maryland's office of technology commercialization. 13