Unsupervised Preference-Aware Language Identification (2022)

Abstract: Recognizing the language of ambiguous texts has become a main challenge in language identification (LID). When using multilingual applications, users have their own language preferences, which can be regarded as external knowledge for LID. Nevertheless, current studies do not consider the inter-personal variations due to the lack of user annotated training data. To fill this gap, we introduce preference-aware LID and propose a novel unsupervised learning strategy. Concretely, we construct pseudo training set for each user by extracting training samples from a standard LID corpus according to his/her historical language distribution. Besides, we contribute the first user labeled LID test set called “U-LID”. Experimental results reveal that our model can incarnate user traits and significantly outperforms existing LID systems on handling ambiguous texts. Our code and benchmark have been released.

## introduction language identification (lid) is widely applied in a range of web services where a multitude of languages may be presented, such as translation systems, search engines, and social media @xcite @xcite @xcite . it predicts the natural language that a text is written in, and decides which language-specific model to invoke in downstream natural language processing (nlp) tasks @xcite @xcite . several recent studies have well tackled lid by designing a feature set for a traditional or neural classifier @xcite @xcite . however, these researches merely explore textual information regardless of external knowledge about the user. in a real-world scenario, there exists large amount of ambiguous user inputs, such as texts with falsefriend, code-switching, and misspelling, as shown in when drawing on a multilingual nlp application, every person has his/her own accustomed languages. the historical behavior implicitly mirrors the user language preference and can be exploited for lid. to this end, we propose a task named preference-aware lid, where the historical language distribution of a user is leveraged for the disambiguation of mistakable texts, and guides lid to predict different languages for different users. a major bottleneck for this task lies in the lack of well-labeled training data. in particular, it is unavailable to obtain large amount of ambiguous texts labeled with different languages by different users. to overcome this issue, we propose a novel unsupervised strategy that builds synthetic data for each user via sampling natural training examples according to his/her historical language distribution. we build our model upon transformer @xcite and introduce two kinds of extensions. one is directly revising the predicted probability of lid using the user language preference. in order to maintain the robustness, the other encodes the user traits into inductive bias. our models are trained using a publicly available dataset extracted from wikipedia. towards evaluating the effectiveness, we construct a user-driven lid test set "u-lid". the benchmark consists of 21 languages, each of which contains 500 examples collected from a real-world translation system and labeled by users. extensive analyses demonstrate the superiority and the robustness of our approach on recognizing error-prone cases. ## preliminary problem formulation given an input text x, the vanilla lid model with parameter θ predicts the probability of the language y by p (y|x; θ). as an extension of conventional lid, preferenceaware lid considers the traits of each user, thus facilitating the classifying of ambiguous texts. in this paper, we treat the language preference of user as the external knowledge, which can be implicitly embodied in historical language distribution d (u) of user u. consequently, our task aims to model p y (u) |x, d (u) ; θ , as illustrated in figure 1 . ## conclusion we explore preference-aware lid. major contributions of our work are four-fold: 1) we introduce preference-aware lid task that leverages user language preference to improve lid. we hope our work can attract more attention to explore techniques on this topic; 2) we propose a novel unsupervised strategy to guide model to take user historical language distribution into account; 3) we collect u-lid and make it publicly available, which may contribute to the subsequent researches on lid; and 4) extensive analyses indicate the effectiveness and robustness of our method, verifying that lid can profit from personality information to make the results conform to user intention.