Using Language Models for assessment of users’ satisfaction with their partner in P ersian (2025)

Abstract: Sentiment analysis, the process of gauging user attitudes and emotions through their textual data, including social media posts and other forms of communication, is a valuable tool for informed decision-making. In other words, a statement conveys positivity, negativity, or neutrality, sentiment analysis offers insights into public sentiment regarding a product, individual, event, or other significant topics. This research focuses on the effectiveness of sentiment analysis techniques, using Machine Learning (ML) and Natural Language Processing (NLP) especially pre-trained language models for Persian, in assessing users’ satisfaction with their partner, using data collected from X (formerly Twitter). Our motivation stems from traditional in-person surveys, which periodically analyze societal challenges in Iran. The limitations of these surveys led us to explore Artificial Intelligence (AI) as an alternative solution for addressing contemporary social issues. We collected Persian tweets and utilized data annotation techniques to label them according to our research question, forming the dataset. Our goal also was to provide a benchmark of Persian tweets on this specific topic. To evaluate our dataset, we employed several classification methods to achieve our goal, including classical ML models, Deep Neural Networks, and pre-trained language models for Persian. Following a comprehensive evaluation, our results show that BERTweet-FA (one of the pre-trained language models for Persian) emerged as the best performer among the classifiers for assessing users’ satisfaction. This point indicates the ability of language models to understand conversational Persian text and perform sentiment analysis, even in a low-resource language like Persian.

## introduction assessing people's sentiments, culture, social values, and attitudes is paramount in gaining insights into a society's collective mindset and functioning. human societies are made up of individuals who interact and shape their environment based on shared beliefs and behaviors. therefore, we can comprehend society's challenges, strengths, and weaknesses by investigating these factors. since 2000, traditional assessments have been conducted three times in iran in provincial surveys under the supervision of experts in various fields. one of the critical topics in these surveys is community members' satisfaction in measuring their social and cultural status. the main challenges in this field can be identified by evaluating and understanding people's satisfaction with various factors in this extensive research and surveys. however, this traditional approach to data collection, such as questionnaires, was last conducted in 2015 in iran, where families were interviewed and completed self-report lists to assess their satisfaction. nevertheless, this method has shortcomings in conducting detailed investigations and analyzing surveys responses. one issue is the difficulty attracting many participants due to privacy concerns and disagreement with the research. additionally, social satisfaction is a variable characteristic that changes over time, requiring regular surveys every few years to keep up with changing trends. conducting surveys every few years is costly for the country regarding finances and human resources, and the reporting process is time-consuming, taking several weeks or even months to complete. furthermore, there is a risk of human error in the reports, leading to lower accuracy and higher costs. at the same time, the popularity of social networks has grown with the advancement of internet technology and the widespread use of smartphones. they have become a crucial part of our lives, enabling people to communicate and access information. these platforms also offer a new way of sharing information, exchanging knowledge, and connecting people globally. social networks serve more than just as a tool for users to document their lives and connect with others; they also provide an avenue for expressing personal thoughts and maintaining relationships. x (formerly twitter) is a popular social networking platform with real-time and interactive features. users can express their emotions through texts, emojis, photos, and videos, making it a suitable and essential platform to share happiness and sadness. furthermore, tweets contain short emotional information that holds significant value in shaping public opinion and driving social impact. this feature reflects users' interests and preferences and can significantly influence the spread of online public opinion. therefore, in this research, we decided to analyze social network data to address a crucial societal issue. specifically, we aim to examine the satisfaction levels of individuals within their relationships. with the increasing divorce rates in our society, it is crucial to understand the factors that contribute to satisfaction and dissatisfaction in relationships. hence, we propose a methodology that utilizes artificial intelligence (ai) and machine learning (ml) techniques, especially language models (lms), to minimize the challenges and costs associated with traditional surveys. by analyzing the tweets that users post on social media platforms, we can gather data on a large scale without requiring human resources. this will allow us to design an efficient model, saving time and resources while providing valuable insights into our society's social challenges. however, we acknowledge that this study is based on data collected from the persianspeaking community on x, which may not fully represent the wider population. therefore, this research serves as a preliminary case study, highlighting the potential of social network data to address societal issues, while also acknowledging the limitations of its specific user base. overall, in this paper, we make the following contributions. (1) we provided a new labeled dataset and a benchmark that explores user satisfaction with their partner, specifically targeting persian tweets, to evaluate the performance of different classification models and lms; (2) we designed a new framework to analyze social questions based on social networks using ai that can reduce the drawbacks of traditional surveys; (3) following the results of our classification models, we testified the power of the transformer-based model for the persian language in investigating social problems. ## related work over the past decade, sentiment analysis has emerged as one of the main areas of research in both data mining and natural language processing (nlp). researchers have provided this approach in different applications like analyzing movie reviews @xcite , identifying hateful content on social media @xcite , analyzing mobile reviews in persian @xcite , opinion analysis @xcite and opinion mining @xcite . the mentioned projects are part of the intensive literature in this research area. furthermore, sentiment analysis is a valuable tool for examining and analyzing user characteristics on social networks. in @xcite , researchers conducted a comprehensive analysis of the relationship between users' personalities, including popular users and influencers, using x data. they developed a model to estimate users' personalities based on follower data and used ml algorithms such as support vector machine (svm) @xcite for prediction. their research revealed that emotional stability and extroversion are common traits among all users, while popular users tend to be more imaginative, and influential users are typically more organized. these findings provide valuable insights that were previously difficult to quantify. by predicting user personalities from public data, we can gain important information for various applications. @xcite proposed a social satisfaction prediction model based on research in the field. they used apis to collect micro-blogging data from social networks and conducted surveys to obtain user satisfaction scores. their results showed that regional social satisfaction is linked to local economic indicators. this suggests that the prediction model can accurately identify social satisfaction through social media data. also in @xcite , a novel technique for measuring user satisfaction with a product was introduced using deep neural networks (dnns) like convolutional neural networks (cnns) @xcite instead of traditional ml algorithms. the cnn network achieved a higher accuracy rate than classical algorithms such as svm. after noticing the good performance of neural networks, researchers applied different deep learning techniques and architectures such as word embeddings and long short-term memory (lstm) @xcite , even to develop sentiment analysis systems with higher performance @xcite @xcite . previous research has primarily focused on individual-level opinions, such as those related to films and products. however, there remains a significant gap in the analysis of collective sentiments and opinions on pressing social issues. recognizing this gap and building upon prior research, we propose models to predict users' satisfaction with their partner based on persian tweets. by analyzing social media content related to users' satisfaction with their partner, we hope to better understand the real-life problems families face in our society. furthermore, we believe that these models can serve as a valuable foundation for addressing other cultural and social issues in the future. ## method in this section, we will discuss in detail the approach proposed for assessing satisfaction with their partner, and we will provide more information about our benchmark dataset. our proposed framework is shown in figure 1 , and we will provide additional details in the following sections. ## results to compare classifiers effectively, it is important to maintain a consistent dataset. to achieve this, we randomly selected 82% of the dataset as the training set, while the remaining 18% was assigned to the test set. these two datasets were then saved as separate csv (comma-separated values) files, ensuring that a fixed dataset is used for all classifications. different metrics were used to evaluate the proposed approaches. for the classification evaluation, we utilized accuracy and the macro f1 score (f1-score). ## conclusion the motivation for this study stems from the drawbacks of the traditional surveys conducted in iran every few years. our research aims to enhance traditional survey methods by introducing a new approach to analyzing complex social issues by applying text classification methods and testing the performance of pre-trained language models for persian. in particular, we leveraged ml and nlp techniques to classify the sentiment of tweets from x users regarding their satisfaction with their partner. our data collection took place on the x social network, primarily in persian, given its popularity among persian-speaking individuals. following data preprocessing, we employed human taggers to annotate the tweets according to our research question, forming a labeled dataset as a challenging benchmark for classification models and pretrained lms for persian. as there was no existing foundational model for the subject under investigation, our research explored various classification algorithms, including svm, knn, nb, rf, lr, bilstm, cnn, parsbert, parsbert-bilstm, parsbert-cnn, and bertweet-fa. our comprehensive evaluation shows that bertweet-fa, a pretrained language model for persian, outperformed the other classifiers in accurately classifying sentiment in persian tweets. this result highlights the effectiveness of lms in understanding conversational persian text for sentiment analysis and challenging social problems. in future research, we aim to explore semisupervised learning techniques for data annotation and employ multilingual and large language models (llms) to enhance the dataset and classification models further, respectively. we also plan to investigate data augmentation methods to address the issue of data scarcity and improve the robustness of our models. additionally, we will explore deeper linguistic insights, such as analyzing sentimentbearing idioms and slang unique to persian, to enhance the interpretability and performance of our models in persian nlp.