Combining Shallow and Deep Representations for Text-Pair Classification (2021)

Abstract: Text-pair classification is the task of determining the class relationship between two sentences. It is embedded in several tasks such as paraphrase identification and duplicate question detection. Contemporary methods use fine-tuned transformer encoder semantic representations of the classification token in the text-pair sequence from the transformer’s final layer for class prediction. However, research has shown that earlier parts of the network learn shallow features, such as syntax and structure, which existing methods do not directly exploit. We propose a novel convolution-based decoder for transformer-based architecture that maximizes the use of encoder hidden features for text-pair classification. Our model exploits hidden representations within transformer-based architecture. It outperforms a transformer encoder baseline on average by 50% (relative F1-score) on six datasets from the medical, software engineering, and open-domains. Our work shows that transformer-based models can improve text-pair classification by modifying the fine-tuning step to exploit shallow features while improving model generalization, with only a slight reduction in efficiency.

## introduction text-pair classification determines the class relationship between two sentences; for example, it determines the inference class relationship (entailment, contradiction, or neutral) between a premise and a hypothesis @xcite . such classification requires interpreting the semantic content of sentences to determine their relationships. applications of text-pair classification, which we experiment with here, are natural language inference @xcite , question answering candidate ranking @xcite and duplicate question detection (wang et al., 2019a; 1 code and dataset will be released upon publication. @xcite . for these tasks, we consider open, biomedical, and software engineering domains. contemporary methods in text-pair classification use transformer encoder networks, such as bert @xcite , which are popular in natural language processing @xcite @xcite . for these encoder models, there are several studies improving the model in different aspects. @xcite and @xcite introduce improvements to the pretraining by removing next sentence prediction (nsp) and using a larger batch-size with the lamb optimizer @xcite . scib-ert @xcite pretrains bert over publications from semanticscholar and adjusts the model's vocabulary to be domain-specific. al-bert @xcite increases model depth, adds layer parameter sharing, and includes the sentence coherence over nsp. these studies primarily improve the transformer architecture by scaling up overall model capacity through dataset source @xcite and adjustment of the pretraining task @xcite . they use only the classification token as the primary feature for classification. these improvements also require pretraining the architecture, which requires a high amount of computing resources. increasing model size and complexity is not the only approach to improve transformer-based models. recent research shows that these models capture different levels of information at different layers in the network @xcite . the information encoded in these levels are surfacelevel, structural and syntactic in the lower layers, and semantic in the upper layers @xcite . shallow features are potentially useful because interpreting the semantic content of sentences may be difficult without additional knowledge. for instance, in software engineering question-answering, structural knowledge and program syntax may aid in recognizing jargon in a programming question or the grammar of a program; a program does not work if the syntax is incorrect (e.g., func() vs. func{} only differ by bracketing in the context of the programming language). likewise, in medical queries, the structural and syntactic information helps recognition of medical charts; for instance, blood pressure is written as bp 80/50 and understanding this structure and syntax (numbering order) aids in the semantic interpretation of hypotension. we investigate the value of these structural, syntactic and semantic features in existing pretrained models for text-pair classification tasks in the medical and software engineering fields. our model adjusts the decoder to use the hidden representations of the bert transformer encoder model by combining shallow and deep feature representations of the input sentences. as our method involves adjusting the decoder, there is no requirement to pretrain the network, allowing it to be extended to all transformer-based encoders. our main contributions are summarized as follows: ## related work text-pair classification is a specialization of text classification. early studies on text-pair classification used rule-based inference from a knowledge base of patterns and templates for textual entailment @xcite . this was superseded by supervised probabilistic models such as support vector machine (malakasiotis and androutsopoulos, 2007), naïve bayes, and decision trees @xcite , as well as unsupervised algorithms such as k-nearest neighbours @xcite . since 2014 @xcite , neural network-based techniques dominated the field. they are based on convolutional neural network (cnn)-based encoders @xcite , long short-term memory (lstm) and recurrent neural network (rnn) sentence interaction encoder models @xcite which use shallow reasoning over sentences to capture semantics. these methods are limited by their model's receptive fields, as they cannot model deep semantic contextual knowledge and do not directly capture shallow information @xcite . ## deeply interconnected convolutional transformer network transformer encoders learn different features of a language at different layers @xcite . we explore if combining shallow features alongside deep features would improve the effectiveness of representation learning. we therefore design a new method that exploits all the hidden features within the transformer encoder. we use cnn-based decoders connected to each layer of the transformer encoder as a low-parameter fully-connected network @xcite to combine hidden features in a highly parallelized manner. in doing so, multiple gradient flows @xcite are subsequently introduced, allowing gradient propagation to shallower network layers and aiding the learning of downstream language features in earlier parts of the network. however, the efficacy of multiple gradient flow is determined by convergence, which poses a significant challenge for our method as we use randomly initialized decoders together with a pretrained decoder. these two models expect different input distributions. to tackle this problem, we adopt convolutional components to help in dimensionality reduction and use larger batch-sizes with the lamb optimizer @xcite and one cycle policy @xcite for hyper-convergence. we also include residual connections to ensure a stable gradient throughout the network. we use convolutional components over lstms as the recurrent step is non-parallelizable @xcite and slowdown the parallel computation in the transformer. we adopt two configurations similar to vdcnn and googlenet to test the generality of our method. we demonstrate that a stronger capability of learning is enabled by our method in text-pair classification tasks, especially for domains that require structural and syntactic knowledge such as the medical and software engineering domains. ## datasets we use two datasets from medical and software engineering because these domains may benefit from the structural and syntactic knowledge for downstream tasks. we also use two datasets from the open-domain to test the generality of our method. mediqa the mediqa challenge @xcite was part of the bionlp 2019 shared task. it features three separate tasks: (1) recognizing question entailment (rqe), requiring binary entailment classification between text-pairs for 8,588 medical questions; original question conversion error setting value for 'null converter' -why do i need a converter in jsf? duplicate question selectonemenu with complex objects, is a converter necessary? negative sample conversion error setting value '1' for 'null converter' figure 4: stack overflow dataset examples. stack overflow dataset to test our methods on a substantial, technical dataset, we create one in the software engineering field. our stack overflow duplicate question dataset evaluates the performance of our methods and its ability to generalize on specialized technical domains. to create the dataset, we use okapi bm25 scoring @xcite from elasticsearch with default parameters and word embeddings. specifically, we use question titles, which we expand with word embeddings trained on the stack overflow corpus (efstathiou et al., 2018) for querying. for each word in the query, we found the three most similar words in the embedding space via cosine distance and added this to the original query, q, as expansion terms, e. to promote diversity, we empirically set the weights of the e in to be 1.3 (multiplicative), which is higher than q at 1.0. these expanded queries, (q, e), were used to select candidates with the highest bm25 scores not already marked as a duplicate of q. an example from the dataset is shown in figure 4 . this dataset consists of 1.6 million question pairs, with a balanced label distribution. to our knowledge, this is the first dataset created from stackoverflow with difficult examples for text-pair classification before this work. ## experimental setup we use fastai @xcite , a pytorch-based library. we use a one cycle policy learning rate scheduler over a cycle length of we train our models on a single gpu, v100 tesla 16 gb. training is repeated five times for each configuration to collect reliable statistics for paired t-test significance testing. ## results and discussion effectiveness of the model under different dataset constraints we compare our model in differing constraints. we select the domain type (open or closed), and dataset size as constraints. we first discuss the main results. on the snli dataset, a large open-domain dataset, the te conv model performs the best across all metrics. a similar observation is made for the stack overflow dataset, a large technical domain dataset, where the te conv model performs the best. on both datasets, the te conv model with auxiliary networks also performs better than the baseline. this suggests that the model performs well in datarich environments. this result contrasts with results from the smaller specialized datasets such as the mediqa collection. in these datasets, we see that the te model overfits on all three medical tasks; this is apparent as the training and validation loss is lower on the te model than the convolutional-based models. this means the model failed to generalize as test set performance was low despite performing well on the validation/training sets. this performance may be from vocabulary disparity between the train/test sets and the additional gradients which allowed convolutional models to reach a better optimum through regularization @xcite . on the nli dataset, te conv significantly improves over te and is stronger than te aux , suggesting that usage of all twelve layers of te is useful for inference tasks. by contrast, te aux performed better on rqe and qa as te conv overfit on rqe and performed worse on qa. finally, on the quora dataset, both te conv models performed significantly better (@xmath1.05) with the te aux performing better in recall and f1-score while te conv performed better in terms of accuracy and precision. we note that our models do not match the state-of-the-art performance due to lack of large model ensembling and much lower total parameter count @xcite @xcite as our study is focused on investigating the usefulness of shallow features. however, our framework could potentially be applied to the current state-of-the-art models to improve their performance. to summarize, the te model performs poorly in the low-resource technical setting as reflected by the mediqa dataset results, mediqa has fewer training examples compared to the open-domain which made it challenging to train due to overfitting. however, we found that increased gradients allowed for better generalization. therefore, in this low-resource setting, the convolutional te model seems more suitable. additionally, in more data-rich environments such as the quora, stack overflow and snli datasets, we found the models performed better. however, on the stack overflow dataset, where the model performs better than the baseline, it was not statistically significant due to large variance between runs (σ = 0.02 for f1score). does including additional features from the lower layers of the network help prediction? for each domain, we select a dataset, and we analyze text-pairs to see situations where te conv benefits from additional features over the baseline. from furthermore, the model allows for improved gradient flow in the network, as shown in figure 5 , where the average gradient over three epochs on the snli is depicted. to illustrate each layer, we average all the gradients at that particular level. specifically, the query, value, key attention weights gradients' are all averaged together in each hidden encoder layer. for the te network, there is a diminishing gradient flow, a downwards slope, throughout the network. this slope contrasts with the te conv networks, which show a general increase in gradient flow (positive slope) throughout the network, allowing for learning in shallower layers of the network; which is useful because the shallower features, which are more correlated with the input, are now used for final layer prediction. as the models exhibited similar trends in the embedding layer, we report a singular figure (figure 7 ). a side effect of removing the nsp layer, we observe a large gradient flow in the segment(ation) embeddings, as the model learned sentence segmentation between the two text-pairs. this may explain the capacity of the network to better differentiate between the text-pairs during classification. the magnitude of this segmentation gradient is larger in both te conv models than the te model (figure 5 ), allowing for better modeling of pair semantics. encoder comparisons: efficiency and effectiveness earlier, we hypothesize that using convolutional components should not hinder training speed. from our results indicate that both models are significantly better than the te baseline in most settings. dataset sentence a sentence b gold label baseline prediction stackoverflow why does this settimeout() call work in the console but not as a greasemonkey script? setinterval() and .click() in a greasemonkey script duplicate not duplicate how to resolve error: [@xmath0rootscope/inprog?@xmath2; and how to solve them? not duplicate duplicate mednli in the ed, initial vs were: 8 98 64 131/113 in the ed initial respiratory rate was low entailment not entailment received asa 325mg and nitro 0.4mg x3. the patient has not had any vasodilator drugs. not entailment entailment snli a young child dressed in a scarf, hat, jacket, gloves, pants, and boots, outside playing in the snow. a child plays with a sled in the snow while dressed warmly. ## conclusions we investigate whether using shallow hidden representations-which encode syntactic and structural information in the transformer encoder architecture-aids text-pair classification in the medical, software engineering and open-domains. to exploit these representations, we use deep convolutional neural networks as low-parameter networks to increase gradient propagation to the earlier layers of the network with a minimal decrease in efficiency. we find that including these representations, even as a simple summation over all hidden states, leads to increased system effectiveness. validating if this holds for other variants of transformer encoder architecture is a suitable avenue for future research.