Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparation (2023)

Abstract: Multilingualism is widespread around the world and code-switching (CSW) is a common practice among different language pairs/tuples across locations and regions. However, there is still not much progress in building successful CSW systems, despite the recent advances in Massive Multilingual Language Models (MMLMs). We investigate the reasons behind this setback through a critical study about the existing CSW data sets (68) across language pairs in terms of the collection and preparation (e.g. transcription and annotation) stages. This in-depth analysis reveals that a) most CSW data involves English ignoring other language pairs/tuples b) there are flaws in terms of representativeness in data collection and preparation stages due to ignoring the location based, socio-demographic and register variation in CSW. In addition, lack of clarity on the data selection and filtering stages shadow the representativeness of CSW data sets. We conclude by providing a short check-list to improve the representativeness for forthcoming studies involving CSW data collection and preparation.

## introduction millions of bilingual/multilingual speakers around the world speak more than one language/dialect in their daily lives and/or mix them which (known as code-switching (csw)). some of these languages are also considered as low-resource @xcite . @xcite , there is a wide range of research involving multilingual and csw data across different domains of computational linguistics (e.g., @xcite , @xcite ). furthermore, research in multilingualism and csw has been presented as one of the "next big ideas" at 60th annual meeting of the association for computational linguistics (acl'22). despite these encouraging prospects and availability of mmlms, there is still not much progress in building mixed language systems which can process and produce csw speech and text seamlessly across different language pairs (e.g., spanish-english and hindi-english as exceptions). based on an in-depth analysis of csw data sets (68), we argue that the lack of representative csw data collection and preparation procedures could lead to this drawback. although they claim to be multilingual and capable of handling diverse sets of languages, generative language models (e.g, gpt-3.5 @xcite and bloom @xcite ) perform worse on nlp tasks concerning low-resource languages @xcite . in addition, @xcite show that open-source multilingual language models fail to generate csw for southeast asian language pairs (e.g. english-tamil and english-tagalog). similarly, @xcite reveal the performance gap between small fine-tuned models and llms with zero-shot/few-shot prompting on machine translation, sentiment analysis, and language identification tasks involving texts with csw (spanish-english, malayalam-english, tamil-english, hindi-english, and modern standard arabic-egyptian arabic). performance of mllms on csw data is still much poorer in comparison to their performances on monolingual data @xcite . in addition, multilingual bert @xcite is trained mainly on wikipedia articles, and performs much worse on standard csw benchmarks @xcite than xlm-r @xcite . existing csw evaluation benchmarks may also fail to represent real-life csw accurately. for example, asr models that were trained to perform well on csw speech data tend to perform poorly on monolingual speech data in the same languages and vice-versa @xcite . csw benchmarks for asr typically only contain csw speech but not the monolingual speech which is also part of the real-life communication. in ad-dition, csw evaluation data sets created through social media data (e.g. twitter) 1 may also be limited due to the assumptions made during the data collection and preparation stages. as a result, language models that are overly optimized for some benchmarks and leaderboards have impressive results in terms of system performance, but they are not very useful for the multilingual speakers/users since they do not represent csw as it takes place in real-life communication. although we analyze csw data sets in depth, we do not aim for a literature survey describing all the tasks, experiments and their results about csw across language pairs. considering that labelled data is still necessary for fine-tuning mllms, we only focus on the data collection and preparation stages to assess the issues about representativeness before the modeling stage. this assessment is not only relevant for ethical and scientific purposes but it is also a necessity for product related issues in industrial and/or social good applications which target multilingual speakers/users and their communities. ## defining representativeness for csw data language technologies depend on large data sets of language (i.e., corpora). @xcite defines representativeness in corpora as "the extent to which a sample includes the full range of variability in a population" and it is a core requirement to be able to make generalizations about a language. @xcite define a representative data set as a special subset of an original set which is smaller in size but captures most of the information from the original set. as illustrated by @xcite , csw patterns vary even within the same language pairs depending on various factors (e.g., location, context, socio-demographic factors of speakers/users, historical factors). if this is the case, collecting random csw data sets without taking this variation into account will lead to unrepresentative data sets without external validity (i.e., the collected data will not represent the csw in real-life). as a result, csw systems trained on unrepresentative data sets will fail to meet the needs and preferences of the target multilingual speakers/users. therefore, we posit that researchers should perform quality measures on the representativeness of the datasets 1 https://twitter.com before deploying the csw dataset for training language models. in the subsections below, we expand on the dimensions of variation in relation to the representativeness for multilingual and csw data in terms of data collection and preparation procedures. ## current practices for csw data collection and preparation to have a better understanding about the data collection and preparation procedures, we analyzed csw research published in acl anthology and interspeech between 2008-2023. we mainly focus on the publications describing csw data sets that make use of speaker and user generated data (i.e., speech data, social media posts) and exclude the ones based on written and historical sources (e.g., liu and smith (2020)) and non-user generated content (e.g., movie scripts and information retrieval data as in sequiera et al. (2015), mehnaz et al. (2021), khanuja et al. (2020a), chandu et al. (2019), raghavi et al. (2015), pratapa and choudhury (2017b)). if the same data set (e.g., van der westhuizen and niesler (2018)) was used in other related studies multiple times (e.g., biswas et al. and wilkinson et al. ( 2020 )), we only report the reference associated with the original data set. we are also aware of the artificially created csw data sets which are derived from user-generated data through translation (e.g., @xcite 2017 2019 ). however, it is not always clear how the data is translated into csw in these cases (e.g., @xcite ). additionally, it is possible to generate csw text data synthetically, either by using computational implementations of linguistic theories to generate data from monolingual sentences (e.g., @xcite , @xcite ) or learning patterns from real user-generated csw data (e.g., @xcite ). however, synthetic data provides diminishing returns when used in models that are already trained on real csw data @xcite . in sum, more research is needed to determine what kind of complementary information synthetic csw data provides to linguistic models in comparison to user-generated csw data in real-life communication contexts. therefore, practices around artificial csw data sets are also excluded from our study. ## results in this section, we explain the results of our findings for the data sets we have surveyed based on the quality check and representativeness criteria presented in section (2). ## discussion and conclusion considering data as the backbone of language technologies, our goal was to investigate the reasons behind the lack of progress in csw language technologies through documenting the current data collection and preparation procedures. in line with this goal, we reviewed 52 studies and 68 data csw data sets in speech and social media domains in terms of representativeness in terms of location based variation, coverage of languages, register and socio-demographic variation and filtering practices. our results indicate that current practices around data collection and preparation in csw are far from reporting the rationale behind the choices and procedures systematically. despite the increasing capacities and performances of mmlms, the forgotten lesson is that abundance of csw data which does not represent the variation in real-life multilingual communication is not valuable and it will not serve the needs and preferences of the multilingual users/speakers. to build sustainable and reliable csw systems, there is a need to consider representativeness as the key issue for collecting and preparing csw data for further processing. the need for data statements and/or guidelines in computational linguistics @xcite ; @xcite ) and machine learning @xcite have been voiced earlier due to ethical and bias concerns. although we acknowledge them, there are also issues specific to representativeness in data collection and preparation in multilingual communication. instead of creating a new list of guidelines, we present the reader with a compact check-list to consider when they are collecting and preparing representative csw data sets as follows: although there are many different factors contributing to the success and failure of systems in language technologies, following the above mentioned check-list and our detailed explanations about these items in the previous sections of this paper will improve the representativeness of data collection and preparation stages for the forthcoming csw data sets.