Social media provides an unfiltered stream of user-generated input, leading to creative languageuse and many interesting linguistic phenomena, which were previously not available so abundantly. However, this language is harder to process automatically. One particularly challenging phenomenon is the use of multiple languages within one utterance, also called Code-Switching (CS) . Whereas monolingual social media data already provides many problems for natural language processing, CS adds another challenging dimension. One solution that is commonly used to improve processing of social mediadata is to translate input texts to standard language first. This normalization has shown to improve performance of many natural language processing tasks. In this paper, we focus on normalization in the context of code-switching. We introduce a variety of models to perform normalization on CS data , and analyse the impact of word-level language identification on normalization . We show that the performance of the proposed normalization models is generally high, but language labels are only slightly informative. We also carry out POS taggingas extrinsic evaluation and show that automatic normalization of the input leads to 3.2\% absolute performance increase , whereas gold normalization leads to an increase of 6.8\% .
Lexical normalization, the translation of non-canonical data to standard language, has shown to improve the performance of manynatural language processing tasks on social media. Yet, using multiple languages in one utterance, also called code-switching (CS) , is frequently overlooked by these normalization systems, despite its common use in social media. In this paper, we propose three normalization models specifically designed to handle code-switched data which we evaluate for two language pairs: Indonesian-English (Id-En) and Turkish-German (Tr-De). For the latter, we introduce novel normalization layers and their corresponding language ID and POS tags for the dataset, and evaluate the downstream effect of normalization on POS tagging. Results show that our CS-tailored normalization models outperform Id-En state of the art and Tr-De monolingual models, and lead to 5.4\% relative performance increase for POS tagging as compared to unnormalized input .
R
Social media provides an unfiltered stream of user-generated input, leading to creative languageuse and many interesting linguistic phenomena, which were previously not available so abundantly. However, this language is harder to process automatically. One particularly challenging phenomenon is the use of multiple languages within
Lexical normalization, the translation of non-canonical data to standard language, has shown to improve the performance of manynatural language processing tasks on social media. Yet, using multiple languages in
R
Code-Switching
code-switching
R
. Whereas monolingual social media data already provides many problems for natural language processing, CS adds another challenging dimension. One solution that is commonly used to improve processing of social mediadata is to translate input texts to standard language first. This normalization has shown to improve performance of many natural language processing tasks.
, is frequently overlooked by these normalization systems, despite its common use in social media.
R
focus on normalization in the context of code-switching. We introduce a variety of models to perform normalization on CS data , and analyse the impact of word-level language identification on normalization . We show that the performance of the proposed normalization models is generally high, but language labels are only slightly informative. We also carry out POS taggingas extrinsic evaluation and show that automatic normalization of the input leads to 3.2\% absolute performance increase , whereas gold normalization leads to an increase of 6.8\%
propose three normalization models specifically designed to handle code-switched data which we evaluate for two language pairs: Indonesian-English (Id-En) and Turkish-German (Tr-De). For the latter, we introduce novel normalization layers and their corresponding language ID and POS tags for the dataset, and evaluate the downstream effect of normalization on POS tagging. Results show that our CS-tailored normalization models outperform Id-En state of the art and Tr-De monolingual models, and lead to 5.4\% relative performance increase for POS tagging as compared to unnormalized input