Composing Structure-Aware Batches for Pairwise Sentence Classification (2022)

Abstract: Identifying the relation between two sentences requires datasets with pairwise annotations. In many cases, these datasets contain instances that are annotated multiple times as part of different pairs. They constitute a structure that contains additional helpful information about the inter-relatedness of the text instances based on the annotations. This paper investigates how this kind of structural dataset information can be exploited during training. We propose three batch composition strategies to incorporate such information and measure their performance over 14 heterogeneous pairwise sentence classification tasks. Our results show statistically significant improvements (up to 3.9%) - independent of the pre-trained language model - for most tasks compared to baselines that follow a standard training procedure. Further, we see that even this baseline procedure can profit from having such structural information in a low-resource setting.

## introduction datasets that define pairwise relations between sentence-level text instances are widely used in natural language processing (nlp). they describe the relation of sentence pairs with an annotated label. common examples of such pairwise classification tasks are paraphrase identification @xcite , natural language inference @xcite , semantic textual similarity @xcite , or argument convincingness @xcite . in many such datasets (eq. six out of 11 glue tasks), single sentences can occur in multiple pairwise annotations. figure 1 shows such an example where three annotated pairs share a common question. besides the annotations themselves, such figure 1 : example of three pairwise annotations (edges) using four unique questions (nodes), taken from the qqp dataset @xcite . q1 is the common element of all annotations. there are several ways to control the training process considering such external information. contrastive learning @xcite @xcite aims at learning text representations in a self-supervised fashion where similar instances are aligned and dissimilar pairs are separated using an external measure -i.e. semantic similarities. in curriculum learning @xcite the training data order is determined by the estimated difficulty of the instances using a additional heuristic. inspired by such work, we want to examine whether considering the dataset-annotation structures affects the models' performance. but neither we create new pairs nor open a dependency to an external heuristic. more specifically, we present three different strategies to compile training batches that consider that text instances occur in multiple pairwise configurations. this approach is also motivated by recent work @xcite investi-gating the effect of training order and inter-instance correlations on model performance. we evaluate the strategies on 14 heterogeneous tasks from different domains and in two different scenarios to measure the generalizability of our approach. our experimental results show significant performance improvements on a wide range of tasks for both scenarios compared to a standard training setup. our contributions can be summarized as follows: for pairwise text classification tasks to integrate inter-instance relations into the training procedure 2. we show statistically significant performance improvements on a wide range of heterogeneous tasks in our experimental results ## approach by analysing different pairwise annotated datasets, we found that various ones contain single sentences that occur in multiple annotation instances (see degree in ## further analysis based on the previously shown experiments, we further analyse the influence of the graph structure and the stability & computational complexity of our approach. ## related work while not directly comparable, our work is related to (supervised) contrastive learning in natural language processing @xcite . most approaches in this domain @xcite @xcite aim to learn text representations where related samples (positive pairs) are aligned while unrelated samples (negative pairs) are separated. this self-supervised learning uses training objectives like text reconstruction @xcite or using supervision signals @xcite @xcite ) from labelled data like natural language inference (nli) @xcite . in their setup with nli data, @xcite adapt training batches such that entailment relations are treated as positive examples but contradiction relations and all other in-batch instances as negative examples. compared to these approaches, our setup focus on the supervised learning setting for these downstream tasks, rather than learning text representations which can be used latter on for these tasks. in general, our approach adapts the training instance order that a model processes. this idea is also at the core of curriculum learning @xcite where training instances are reordered according to their estimated difficulty. this has been shown to be beneficial for model performance @xcite and faster convergence @xcite . while curriculum learning approaches make use of heuristics to adapt the sample order in one epoch, our approach only relies on the dataset structure to control the composition of training batches. @xcite identified that the order of the training samples is a random factor that influences the non-deterministic learning process of neural networks. further, @xcite found that inter-instance correlations lead to instabilities during training. we acknowledge these effects and investigate if inter-instance relations can be leveraged in the batch composition to improve the task performance for pairwise text classification. ## conclusions we presented three strategies that adapt the composition of batches to encode structural dataset information. we evaluated these batching strategies on 14 heterogeneous tasks from different domains. our results confirm the usefulness of this structural information during model training. edge-i show the best overall results, including different model types (e.g. albert or roberta) and model architectures (bi-or cross-encoder). further, we see its success on tasks with extreme characteristics (high degree) and in situations where annotated data is extremely scarce (25 instances). we interpret our results as a promising step to integrate structural dataset information besides instancelevel annotations. further, we encourage future annotation studies to consciously consider includ-ing pairs that share common text instances for two reasons. first, to exhaust all possibilities later and second, we showed that even baseline approaches can gain from such structures. this work covered a broad set of pairwise classification datasets that provide a structure of annotation pairs that share text instances. we plan to employ our method on datasets that do not meet this requirement by inducing inter-instance relations using similarity metrics for future work.