Active learning and negative evidence for language identification (2021)

Abstract: Language identification (LID), the task of determining the natural language of a given text, is an essential first step in most NLP pipelines. While generally a solved problem for documents of sufficient length and languages with ample training data, the proliferation of microblogs and other social media has made it increasingly common to encounter use-cases that *don’t* satisfy these conditions. In these situations, the fundamental difficulty is the lack of, and cost of gathering, labeled data: unlike some annotation tasks, no single “expert” can quickly and reliably identify more than a handful of languages. This leads to a natural question: can we gain useful information when annotators are only able to *rule out* languages for a given document, rather than supply a positive label? What are the optimal choices for gathering and representing such *negative evidence* as a model is trained? In this paper, we demonstrate that using negative evidence can improve the performance of a simple neural LID model. This improvement is sensitive to policies of how the evidence is represented in the loss function, and for deciding which annotators to employ given the instance and model state. We consider simple policies and report experimental results that indicate the optimal choices for this task. We conclude with a discussion of future work to determine if and how the results generalize to other classification tasks.

## background language identification (lid) is the task of classifying a document according to the natural language in which it is written @xcite . it is a special case of text classification, where a document is assigned a label l from a finite set of discrete values l. such problems, and lid as a special case, have been widely studied for decades @xcite , with recent state-of-the-art methods focusing on neural architectures over character representations @xcite . most methods share the intuition (verified by many traditional studies) that the signal for lid comes from the character level. this intuition is reinforced by the difficulty that flexible neural architectures have unseating traditional n-gram methods @xcite : frequencies of short character-sequences seem to hold most signal for the task. negative evidence, for a feature or label, is explicit evidence that it is not present in or does not apply to an instance @xcite : in this study, it refers to annotations that say "this document is not language x". a given annotator can only know a handful of languages, so in addition to the positive evidence when presented with one of them, there may be a much higher volume of implicit negative evidence, i.e. the documents whose language they couldn't recognize. among text classification tasks, the lid task is a particularly acute, naturally-occurring example of this imbalance, in contrast to specific phenomena like linguistic structure or small-inventory tasks like named-entity recognition or sentiment, where annotators are expected to have a full grasp of the potential label-space. our use of model estimates to choose annotators has similarities to work on multi-armed contextual bandits @xcite , where the context includes both the new instance and the current model state. similar to bayesian last layer optimization @xcite we focus on the final linear layer in the model, though rather than employing reinforcement learning we directly specify simple policies based on the output distributions. the choice of likeliest annotators is similar to thompson sampling @xcite . figure 1 1 shows the performance of a lid model figure 1 : performance when training only on negative evidence ("this is not language x") and no use of the model for routing instances to annotators or for estimating probabilities (i.e. the random and uniform policies described in section 2). as languages are ruled out, probability-mass is shifted to the set containing the correct label. improvements occur with more labeled instances (x-axis), and with more labels-per-instance. model our model truncates and pads input sentences to 128 characters, and maps each character into a randomly initialized 64-dimension embedding space. the (128x64xbatchsize) tensors increasingly-solid lines: for better readability, we omit this from the legends are fed to a two-dimensional cnn layer with filters of widths @xmath0: 5 followed by relu nonlinearity. the cnn outputs are concatenated, fed to a dense linear layer and softmax to produce distributions over the labels. the choice of characterlevel cnns of the given widths gives the model access to the same statistical information employed by traditional n-gram models. training we simulate l annotators, one per label, each only capable of recognizing their respective label. starting with zero instances and a randomly-initialized model, at each step we are provided with 500 new instances, and for each of them, allowed to query annotators_per_instance of the annotators, ranked according to an annotator policy. each annotator returns positive if the instance is in the language they recognize, negative otherwise. this evidence is represented as a target categorical distribution for the model's loss function, according to a representation policy. the model is then trained via sgd for a maximum of 500 epochs, with learning @xmath1.1, mo-@xmath2.9, maximum @xmath3, minibatch @xmath4, early @xmath5, learning rate reduction of @xmath6.1 and @xmath7, and dropout of 0.5 on the cnn outputs. foot_2 note that we are training to convergence between receiving each new batch of annotated instances. evidence given the annotations received during the training process, we can selectively employ evi-dence: positive and positive+negative are the most important point of comparison, while negative was used in figure 1 to illustrate the potential value of the negative annotations. ## results and discussion figure 4 : comparison of the best approach using positive evidence, and the best approach using posi-tive+negative evidence, both of which use likeliest annotation policy and estimated representation policy. given the same number of annotations-per-instance (line solidity), negative evidence provides significant performance improvements, after a brief initial delay (see figure 6 ). figure 4 shows our primary result: the use of negative evidence, in combination with the estimated representation policy and likeliest annotator policy, produces large improvements over the best baseline approach using positive evidence alone. in particular, when the algorithm is given a small number of annotation opportunities per instance (2, 4, 8), it surpasses the baseline at 20k instances by 7, 15, and 10 points, respectively. positive+negative with 4 annotations per instance exceeds performance of positive with 8 annotations per instance, and with 8 annotations per instance is within 3 points of performance with perfect positive evidence. on the other hand, the positive+negative models show performance delays compared to the corresponding positive models. because the negative instances rely on the quality of the current estimates for constructing target distributions, these are initially quite poor until the model escapes its random initialization. this escape is easier to accomplish with more annotations, which can be seen by comparing where the positive+negative models start outperforming their positive counterparts. figures 6 and 5 compare representation and annotator policies under the simplest configurations. the delayed performance is clearest between the representation policies. the estimated representation policy also provides the most performance improvement, eventually providing benefits for every value of annotators_per_instance. the annotator policies, on the other hand, only provide significant benefits when @xmath8. referring back to figure 4 confirms the combination of both policies outperforms either in isolation. ## conclusions and future work we have demonstrated a general method for exploiting negative evidence using an underlying virtuous cycle, where an improved model leads to better annotator selections and more accurate target distri-butions. these appear to be crucial ingredients for negative evidence to provide benefit: when we remove them, and always select annotators at random and employ cmeds, the positive+negative models provide no advantage over the positive-only models throughout training. it may be that the benefits would only materialize on larger training sets, but since the models already approach the optimum (black line) this is never seen in practice for this lid task. negative evidence can be gathered (and simulated) for any classification task, and it is an open question whether the same approaches will generalize. in particular, we would like to run experiments in speech processing, which has its own language identification problem as well as a number of tasks with even sparser annotation abilities (e.g. speaker identification), and image recognition. the initial performance delays we observed in figures 4 and 6 come from the estimated policy's use of the model before it has sufficiently improved. representation policies that take this into account should be able to shift the early performance curves to the left, similar to work on multi-armed bandits @xcite emphasizing the importance of uncertainty estimates. the annotation policies we considered can only gather annotations for incoming instances: it may be useful to expand this to include all currently-known instances, to allow more flexible shifting of probability mass using the same amount of annotation resources.