Biographically Relevant Tweets – a New Dataset, Linguistic Analysis and Classification Experiments (2022)

Abstract: We present a new dataset comprising tweets for the novel task of detecting biographically relevant utterances. Biographically relevant utterances are all those utterances that reveal some persistent and non-trivial information about the author of a tweet, e.g. habits, (dis)likes, family status, physical appearance, employment information, health issues etc. Unlike previous research we do not restrict biographical relevance to a small fixed set of pre-defined relations. Next to classification experiments employing state-of-the-art classifiers to establish strong baselines for future work, we carry out a linguistic analysis that compares the predictiveness of various high-level features. We also show that the task is different from established tasks, such as aspectual classification or sentiment analysis.

## introduction since its beginning, the web has been a valuable data source for natural language processing (nlp) applications. particularly social media contain data complementary to what previous (news) text corpora could offer @xcite , such as information on ordinary people. this work focuses on biographical information on ordinary users of twitter. the task is to identify tweets that contain biographically relevant information on the author of a tweet. we frame this as a binary text classification task: we distinguish between biographically relevant tweets (1)-( 5 ) and biographically irrelevant tweets (6)-( 10 ). biographical information on ordinary users would be valuable for various applications. providers of social-media sites could support their users with effectively building up their social network, for example, by recommending friends based on shared interests @xcite . another obvious application with a high commercial potential is product recommendation @xcite . personalized advertisement using biographical information about a user (e.g. recommending baby care products for parents-to-be) may be much more effective than undirected general advertisement. biographical information may also be used for applications that are questionable from an ethical point of view. for instance, potential employers could do background checks on job seekers. similarly, insurance companies could screen potential insurants before offering them a particular insurance. specific biographic information could well have a negative impact on the customers. for example, a health insurance might decline a customer an offer if they knew about their unhealthy life style. we are fully aware of these ethical issues and we want to make clear that our aim is not to provide basic research for technologies that could be used for such questionable purposes to evolve. on the contrary, we could envisage scenarios that would help users to protect the above misuse of their personal data @xcite . for instance, the functionality to identify biographically relevant information in user-generated content may be incorporated in an application that automatically anonymizes such sensitive text passages @xcite or, at least, warns users prior to posting such content that they are about to disclose sensitive material. the question may arise why we aim for a binary classification task and do not specify the subtype of biographical information. we consider our task just as the first step in the pipeline of processing biographically related material. obviously, processing all microposts of a social-media site manually would be a prohibitive task. a great reduction of work could be achieved by selecting all biographically-relevant material automatically, as proposed in this paper. only a small proportion of tweets within a user timeline from twitter -our estimate based on our data is that this is about 33% -actually contains such information. the aim of this research is two-fold. first, we want to investigate whether biographically relevant utterances can reliably be identified. second, we also want to determine what types of biographically relevant information one may encounter on twitter. our contributions are the following: all data created as part of this research are made available for research purposes upon request. foot_0 2 related work a substantial body of previous research on the extraction of biographical information using nlp focuses on pre-defined relations, such as hasbirthdate(<person>,<@xmath0. in this paper, we depart from this setting. our observation is that in social media there is plenty of biographically relevant information that does not easily fit into this finite set of relations: as a data source for extracting biographical information, wikipedia foot_1 has, so far, been the most frequently used corpus @xcite @xcite . consequently, previous research typically focused on celebrities or people who are notable in some way since it is this group of people who are predominantly represented in wikipedia. by focusing on social media (i.e. twitter), we are able to extract biographical information from ordinary people. this also means that we are able to widen the scope of biographically relevant information. the task addressed in this paper also bears some relation to multi-document (biography) summarization @xcite which is typically framed as a two-step approach: in the first step, biographically relevant utterances (from different documents) are identified while in the second step, a summary of these extracted utterances is generated by eliminating redundancies and creating a plausible order. our task matches the first step although in the context of summarization previous work only dealt with pre-defined relations. with regard to social media, there have been related research efforts in the area of author profiling @xcite @xcite . these works focus on demographic categories, such as predicting gender, age or personality traits. moreover, there has been research on extracting major life events. such events are environmental circumstances that have an identifiable onset and ending and may carry the potential for altering an individual's present state of mental or physical well-being @xcite . examples are getting married or finding a new job @xcite @xcite . as we will provide evidence later, major life events and demographic categories only form a subset of the relations we consider as biographical relevance. in their recent study, @xcite examine what life events are disclosed on social media. the authors establish that a significant number of such events can be found. unlike our work in which we address both data analysis and classification experiments, @xcite present a purely descriptive study. moreover, that work is carried out on data collected from facebook, while our work is carried out on data crawled from twitter. while all life events represent biographically relevant information, the set of biographically relevant utterances considered in this paper also includes relations beyond life events, e.g. likes or habits. in the area of privacy protection @xcite , there has also been research using nlp. @xcite present a descriptive study examining properties of three types of leaks on twitter: divulging vacation plans, tweeting under the influence of alcohol and revealing medical conditions. @xcite present a tool for privacy detection trained on 500 tweets manually labeled as private or not-private using svm. @xcite introduce a system that distinguishes between sensitive vs. non-sensitive information based on recursive neural networks @xcite . information about the training data used are not provided. @xcite establish a set of heuristic rules operating on depen-dency parses for the detection of privacy leaks. the latter are approximated by mentions of locations or emotions. the approach is examined on a random set of 856 facebook statuses. our work differs from these works in that our focus is not just on sensitive data but all different types of biographical information. for example, habits, personality traits or likes and dislikes are biographical information but they need not be categorized as (highly) sensitive data. moreover, our dataset with about 14k tweets that has been annotated manually is considerably larger. it has also been constructed in a more principled and less biased way. for example, @xcite create their dataset by sampling for specific topic keywords while our raw data represent timelines of users ( §3). finally, our classification experiments are carried out using state-of-the-art classifiers which are much more robust than those in previous work (e.g. svm). unfortunately, none of those previous datasets on privacy protection are publicly available, so we could not consider them in our evaluation. ## data we now describe the creation of our novel dataset to detect biographically relevant tweets, henceforth referred to as brt-dataset. we did not sample our dataset using specific keywords since this results in topic biases @xcite . neither did we randomly sample tweets since, as our exploratory experiments revealed, the tweets would have a bias towards topics predominant during collection time, such as covid-19. instead, we extracted timelines of users. using the twitter api foot_2 , we selected users from australia, canada, ireland, new zealand, the uk and the usa who had recently posted a tweet containing the first-person singular pronoun. since we could only annotate a small number of timelines, we tried to come up with a subset of very different users (having different ages, genders, ethnicities, sexual orientations, political beliefs etc.). thus, we hope to appropriately account for the demographic diversity of our society. in order to reduce the manual annotation, we got rid of trivial cases of irrelevant tweets, i.e. tweets that do not contain any mention of the author of the tweet itself. that is, we excluded tweets without a mention of the first-person pronoun. 4 since we did not want to discard cases of implicit protagonists (12) @xcite , which despite the absence of such pronoun refer to the author, we applied some further heuristics (e.g. by including declarative sentences that begin with a verb). we insisted that the biographically relevant information should be unambiguous. for instance, (15) could imply that the author is a regular churchgoer in which case the tweet would be biographically relevant. however, since there are also several other possible explanations (for instance, the author might just be attending a wedding) we did not label such cases as biographically relevant. on a random subset of 300 tweets we measured the agreement between our annotator (a member of the department of one of the co-authors) and one coauthor of this paper. we obtained a substantial interannotator agreement of cohen's κ = 0.73 (landis and @xcite . ## high-level features in this paper, we follow a supervised learning approach. among the different classifiers we consider, we also implement a feature-based classifier using high-level features. these features are also we only identified 4% biographically relevant instances. therefore, we can conclude that our filtering heuristic only misses a negligible proportion of actual biographically relevant tweets. ## linguistic analysis ## classification experiments we now report on our classification experiments. baselines. as baselines, we consider a majorityclass classifier that always predicts biographical irrelevance. furthermore, we consider a sentiment classifier which predicts biographical relevance if a tweet either conveys positive or negative rather than neutral sentiment. (as in §4.2, we obtain sentiment information using tweeteval.) our third baseline is an aspectual classifier which predicts biographical relevance if a tweet conveys habitual aspect. (as in §4.1, we obtain the aspectual classification using sitent.) our fourth baseline is a classifier using meta-information. motivated by our analysis from §5, it predicts biographical relevance if a tweet is neither a retweet, a quote tweet nor represents the answer to a tweet. the more predictive this baseline is, the less important we should consider nlp-based information for this task. supervised classifiers. we used logistic regression as a classifier for our high-level features ( §4) and for bag of words. furthermore, we considered the two transformers roberta @xcite and @xcite . the latter is specifically designed for twitter. since we do not want our classifiers to overfit, we did not tune the hyperparameters on brt but took the settings from nguyen et al. (2020) which are generally considered effective for twitter. 10 experimental set-up. the supervised classifiers were evaluated via 5-fold cross-validation. we created the folds in two different ways: on the one hand, we randomly split the tweets of brt into 5 different folds (randomized folds). these folds may produce training and test splits where tweets of the same timeline occur both in the training and the test set. within a particular timeline, biographically relevant tweets may co-occur with particular topics or constructions. these co-occurrences are idiosyncratic to the user to whom that timeline belongs. however, they may not be representative of the classes to be predicted. supervised classifiers may produce high classification scores by memorizing these user-specific artefacts. as a realistic alternative, we consider a different set-up in which the tweets of a particular timeline are restricted to the same fold (unseen timelines). as a consequence, on this set-up classifiers are not rewarded for learning user-specific artefacts since the tweets in a test set will always originate from timelines not observed in the training data. as far as the transformers are concerned, we further report the average result over 5 different runs (including standard deviation). all other classifiers produce deterministic output. results. the results of our evaluation are displayed in the supervised classifiers outperform the baselines. the most effective classifier is bertweet, i.e. the language model that has been designed for twitter. figure 1 shows the learning curve using our best performing classifier from ## subtypes of biographical relevance we extended our annotated brt-dataset ( §3) by an additional layer. for each biographically relevant tweet, we manually annotated the underlying subtype. thus, we want to give an indication of what type of biographically relevant information can be found on twitter. for the inventory of subtypes, we chose categories that are sufficiently distinctive and frequent on our data. ## error analysis despite the strong performance of our best classifier, i.e. bertweet, we still observed a set of cases that were systematically classified incorrectly. a frequent error source stems from diverse forms of overgeneralization. while according to our annotation guidelines, desires and (dis)likes are considered biographically relevant, trivial desires ( 24 ) and (dis)likes (25) are not. our classifier, however, fails to make this distinction. similarly, while long-term health-conditions are biographically relevant (e.g. asthma, diabetes), common temporary illnesses are not (26). our classifier also fails to make this distinction. moreover, it is not able to appropriately take aspectual information into consideration. episodic needs and desires (27) are thus erroneously considered biographically relevant. finally, an obvious challenge are instances of implicit realizations of biographical relevance where there are no obvious lexical clues that suggest the presence of this category ( 28 )-( 29 ). ## conclusion we presented a new dataset comprising tweets for the novel task of detecting biographically relevant utterances. unlike previous research we do not restrict biographical relevance to a set of pre-defined relations. we showed that state-of-the-art classifiers are able to automatically detect such utterances even on tweets of unseen timelines. our feature analysis using high-level features revealed the relatedness between linguistic features and biographical relevance (e.g. specific supersenses and pos-tags) while there is also a set of different meta features predictive for biographical irrelevance.