Dialect and Variant Identification as a Multi-Label Classification Task: A Proposal Based on Near-Duplicate Analysis (2023)

Abstract: We argue that dialect identification should be treated as a multi-label classification problem rather than the single-class setting prevalent in existing collections and evaluations. In order to avoid extensive human re-labelling of the data, we propose an analysis of ambiguous near-duplicates in an existing collection covering four variants of French.We show how this analysis helps us provide multiple labels for a significant subset of the original data, therefore enriching the annotation with minimal human intervention. The resulting data can then be used to train dialect identifiers in a multi-label setting. Experimental results show that on the enriched dataset, the multi-label classifier produces similar accuracy to the single-label classifier on test cases that are unambiguous (single label), but it increases the macro-averaged F1-score by 0.225 absolute (71% relative gain) on ambiguous texts with multiple labels. On the original data, gains on the ambiguous test cases are smaller but still considerable (+0.077 absolute, 20% relative gain), and accuracy on non-ambiguous test cases is again similar in this case. This supports our thesis that modelling dialect identification as a multi-label problem potentially has a positive impact.

## introduction in this paper, we argue that dialect foot_0 identification should be treated as a multi-label classification problem unless it can be shown that every text in a given dataset belongs to only one dialect or language variant. this feels like a natural hypothesis, as it seems reasonable that some utterances are equally valid in more than one dialect or variant. however, most datasets for, and evaluations of this task rely on single-label classification, where each utterance is annotated as belonging to a single variant. @xcite previous work shows that manually identifying the language variety of a text is difficult, and that it is actually easier for native speakers to identify texts that are not in their variety (goutte et al., 2016, sec. 4.4). accordingly, proper multi-label manual annotation requires multiple annotators with complementary skills, and therefore massive annotation budget, when run at the usual scale of tens-to-hundreds of thousands of utterances. in this work, we focus instead on analyzing and processing already existing dialect identification data, with minimal annotation need. we argue that automatically assessing differences between two similar texts, as done here, is an easier task. we explore empirically how the data can be enriched with multiple labels, and how switching to the multi-label classification paradigm can potentially improve performance in identifying dialects and variants. we start by analyzing the duplicates and nearduplicates in an existing dataset built for french dialect identification. we search for instances that are identical or highly similar textually, but are annotated with different labels. we find that a considerable number of near-duplicates have different labels, but no obvious differences that could be considered dialectal in nature. we further show that near-duplicate analysis is useful in at least two ways. first, it allows us to inspect and refine a dataset, in a manner similar to measuring data (wang et al., 2022; mitchell et al., 2022, inter alia), by identifying phenomena that might otherwise go unnoticed, e.g. texts that are assigned to different classes but have no actual dialectal differences or spotting artefacts due to the selection of text sources or to the processing pipeline (e.g. boiler plate removal, sentence splitting, etc.). second, by spotting similar texts that have no obvious dialectal differences, it allows us to convert an existing dataset in single-label format into a multi-label dialect classification format. using the results of this analysis, we combine the labels of near-duplicates to create what we argue is a more accurate representation of the data. for further empirical validation of this approach, we use this data to train a multi-label classifier for dialect identification. we compare those results to single-label classification and show that the overall classification performance stays at a similar level, while the performance on the subset of examples that have multiple labels is greatly improved. the experimental code developed in this work is available at https://github.com/gbcolborne/ vardial2023 . ## data for this project, we used the frecdo corpus (gȃman et al., 2022), foot_2 which was used for the cross-domain french dialect identification (fdi) shared task at the vardial 2022 evaluation campaign (aepli et al., 2022). it contains 413,522 short texts belonging to one of four varieties of french from belgium (be), canada (ca), switzerland (ch), and france (fr), cf. second, this dataset features four different dialects of french, which seemed promising in terms of identifying texts that belong to more than one dialect. in particular, the four-variant setting seems more flexible than the situation where only two variants are considered (e.g. portuguese from brazil and portugal), in which case the only multi-label configuration is essentially all labels. third, the authors of this paper are all fluent in (one or more variants of) french and were therefore able to analyze the texts and identify possible dialectal differences between texts or dialectal markers in a given text. it is important to note that this dataset was created using methods that are common for dataset compilation for dialect identification tasks (aside from the cross-domain split). these methods include scraping texts from the internet and assigning them to a language variety based on the top-level domain name of the source. this practice naturally leads to a single-label formulation of the problem, if each unique text is only present in one of the sources. the limitations of this practice was a motivating factor for the dsl-tl (discriminating between similar languages -true labels) shared task at this year's vardial evaluation campaign: the dslcc was compiled under the assumption that each instance's gold label is determined by where the text is retrieved from. while this is a straightforward (and mostly accurate) practical assumption, previous research has shown the limitations of this problem formulation as some texts may present no linguistic marker that allows systems or native speakers to discriminate between two very similar languages or language varieties. @xcite the solution proposed in dsl-tl was therefore to curate a higher-quality, human-annotated subset of an existing collection of dialect identification data, dslcc @xcite , such that some of the resulting examples have multiple labels (zampieri et al., 2023). this is in line with our proposal to reformulate the problem as a multi-label classification task. however, although dsl-tl provides high-quality annotation on a subset of data, we focus on the use of semiautomatic near-duplicate analysis in order to minimize the annotation burden. also, as mentioned earlier, the dataset used in this work contains four different dialects of french, whereas the dsl-tl dataset uses only two dialects for each of three different languages: american and british english, brazilian and european portuguese, and argentinian and peninsular spanish. it is also important to note that deduplication is often applied to datasets for dialect identification, although we have observed duplicates both within and across classes in several such datasets. if deduplication is somewhat common, near-duplicate analysis is not a common step in dataset development as far as we can tell. @xcite we argue that it is a useful tool in the context of dialect identification. it can be carried out efficiently and provides useful additional information. in fact, our analysis shows that many highly similar near-duplicates vary only in minor aspects that have nothing to do with dialectal variation or lexical choice, such as slight changes in punctuation or formatting (for example the choice of double quotes), which are typically missed by standard deduplication pipelines. in the following experiments, we used our own, random split of the texts, because the cross-domain nature of the original split was not relevant for our purposes. we also wanted to eliminate the small amount of leakage of texts between the training, development, and test portions of the original dataset. we therefore created an 85/5/10 split, as this was approximately the size of the partitions in the original dataset, by randomly sampling the train/dev/test from the entire original collection. ## methods in this work, we first identify ambiguous nearduplicates that are present in an existing singlelabel dataset for dialect identification. we perform a light manual inspection (section 3.2), then create an enriched version of the data by combining the labels of near-duplicate texts. finally, we train and evaluate classifiers on the resulting data. ## discussion based on our analysis and experimental results, we argue that the analysis of near-duplicates and particularly ambiguous near-duplicates, should be an integral part of a dataset creation and validation pipeline, and should be described in the documentation for the collection. in the case of french variant identification, this analysis uncovered a number of features and issues with the dataset, such as differing formatting and typological conventions, which evade traditional deduplication, and may cause further problems, such as inconsistent named entity tagging, especially in terms of span. another issue is that the segmentation of the original news stories into text fragments may differ between similar instances. this suggests that we may improve the near-duplicate detection and analysis by integrating sentence splitting into the processing, i.e. further split segments into individual sentences to detect more duplicates or near-duplicates. it is important to remember that we do not believe that the ambiguity of duplicate text pairs and near-duplicates is unique to this dataset. in fact, we have observed similar issues in several datasets used for dialect identification in the past. however, further testing, e.g. on datasets in other languages, may be required to better establish the validity of the proposed approach. although we show that modelling dialect identification as a multi-label problem is useful, the proportion of ambiguous near-duplicates identified by our methods may seem small and therefore of little significance. if another dataset contained more ambiguous near-duplicates, or if a better method of identifying them were to be developed, the utility of this proposal would only be heightened. note that in the dataset developed for the "true labels" shared task at this year's vardial evaluation campaign (zampieri et al., 2023), the number of ambiguous examples was between 12% and 32%, which is much higher than the ∼ 1% proportion we identified in the frecdo dataset using near-duplicate analysis. in the proof of concept presented here, we limited ourselves to semiautomatic methods that exploit a sampling-based re-annotation protocol that is simple and inexpensive. note also that further refinements to this protocol could reduce the number of disagreements between annotators on the sampled cases. ## conclusion this contribution is motivated by the hypothesis that dialect identification is best addressed as a multi-label problem. by analyzing the similarity between instances in a four-class, french-language variant identification collection, we showed that there are a significant number of duplicates or nearduplicates with essentially the same surface representation and content, but differing reference labels. this is likely an artefact of the data acquisition pipeline, which focuses on the source of the data and provides a single label. by leveraging this finding, we were able to re-label some instances with multiple labels, and show that taking those into account by training a multi-label classifier produces a large increase in performance on the instances with multiple labels, while maintaining the performance on instances with a single label. we argue that the analysis of ambiguous nearduplicates should be a standard in dataset creation and validation efforts, hopefully producing data that is labelled in a more informative way than by provenance alone. additional investigations may provide more insight on how to best represent dialect and variant classification. for example, we could encode multiple labels in a single-label model by encoding combinations of dialects as classes. another possibility would be to formulate dialect identification as a word-level sequence tagging problem, identifying parts of a sentence that are dialectal markers, and parts that are not specific. this would likely require much more labelling, modelling and training effort.