How does Punctuation Affect Neural Models in Natural Language Inference (2020)

Abstract: Natural Language Inference models have reached almost human-level performance but their generalisation capabilities have not been yet fully characterized. In particular, sensitivity to small changes in the data is a current area of investigation. In this paper, we focus on the effect of punctuation on such models. Our findings can be broadly summarized as follows: (1) irrelevant changes in punctuation are correctly ignored by the recent transformer models (BERT) while older RNN-based models were sensitive to them. (2) All models, both transformers and RNN-based models, are incapable of taking into account small relevant changes in the punctuation.

## introduction in recent years models for natural language inference (nli) have reached almost human-level performance. these models frame inference as a classification problem, whose input is a premise/hypothesis pair. it has been noted that small changes in the pair, can flip the prediction @xcite . in this paper, we explore the effect of punctuation 1 in neural models in natural language inference. small changes in a premise/hypothesis pair are of two kinds. first, the change can be of an irrelevant kind. for example, we can expect that removing a sentence-final stop should not change the relationship between a premise and hypothesis sentence. second, a textually small change could flip the relationship between hypothesis and premise. for example, adding a negation word is a small textual change that has a lot of semantic content. but it is not only words that can have a large impact on the meaning of a sentence. commas, 1 the set of punctuation symbols we consider are: '!"#$%&() * +,-./:;<=>?@[]\ˆ'{}| ' for example, may indicate which words belong together and which do not in a list. ideally, an nli model should be insensitive to changes of the first kind, but still, properly recognize changes of the second kind. in this paper, we test both hypotheses for the case of punctuation. namely: this work is part of the larger question concerning the ability of nli models to generalize. there are a number of papers that report several problems of generalizability: @xcite have shown that several nli models break considerably easily when, instead of tested on the original snli @xcite test set, they are tested on a test set which is constructed by taking premises from the training set and creating several hypotheses from them by changing at most one word within the premise. @xcite show that nli models break down when one trains in one dataset, but then test on the test set of a similar dataset (e.g. training on mnli @xcite and testing on snli). @xcite report problems in generalizability when the two pairs are swapped. the idea is that one should expect the same accuracy for contradiction and neutral when the pairs are swapped (neutral remains neutral, and contradiction remains a contradiction foot_0 ), and a lower accuracy for entailment (given that entailment turns neutral when the pairs are swapped). ## datasets and experiments our experiments are performed on the multi-genre natural language inference (mnli) corpus @xcite (and variants thereof, as described below). mnli consists of 433k human-written sentence pairs labeled with entailment, contradiction and neutral. mnli contains sentence pairs from ten distinct genres foot_1 of both written and spoken english. only five genres are included in the training set. the development and test sets have been divided into matched and mismatched, where the former includes only sentences from the same genres as the training data and the latter include sentences from the remaining genres not present in the training data. we consider three variants of mnli: appending a sentence-final stop is in general reasonable, especially for the non-dialogue examples. for the dialogue part of the mnli dataset, this is unnatural as final stops typically are not expressed in dialogue. to convey an idea of the amount of data that our transformation impact, we show the raw and relative count foot_2 of punctuation symbols in ## models the experiments are performed using three models: bilstm the simplest model is a bidirectional lstm that encodes the premise and hypothesis, then applies max pooling. the model then concatenates the premise and hypothesis in the standard fashion @xcite : [p; h; p -h; p * h] where p is the premise representation and h the hypothesis representation. a three-layer perceptron with leaky relu activation between the layers then assigns a class to the example. ## experimental setup for each architecture (bert, hbmp, and bil-stm) we perform experiments by training four models, two trained and validated on the dataset with punctuation and two models trained and validated on the dataset without punctuation. to asses the effect of our data augmentation we test the model on the other dataset, i.e. a model trained and validated without punctuation is tested on the dataset with punctuation. we measure the performance in terms of accuracy. for hbmp and the bilstm models we use the default hyperparameters reported by @xcite with glove @xcite ) word embeddings foot_3 . the bert model is fine-tuned with the default model hyperparameters. we use the adam optimizer with a learning rate of 0.00002 and a batch size of 24. ## conclusions the conclusions of this paper can be summarized as follows: only bert is robust to irrelevant changes in punctuation (h1 is validated for bert). the other models see a significant drop in performance when for any mismatch of the presence of punctuation between training and testing sets. however, the presence or absence of the full stop at the end of a sentence has little effect. this statement rests on the observation that punctuation is generally semantically insignificant in mnli. this fact has not been tested using a model but rather relies on manual inspection of the data. we have evidence that no model is capable of taking into account cases where punctuation is meaningful. at this stage of our research, this evidence does not rely on a large body of data. this result is not surprising because of the above observation (namely, there is not enough meaningful punctuation in the training set). yet, we use pre-trained embeddings (bert) which have been trained on very large dataset, and it could not be ruled out a priori that such embeddings did not contain information related to the meaning of punctuation. as a general remark, it seems to us useful, if not necessary, to extend the present datasets for nli to include examples where punctuation is actually meaningful. in general, this is part of a discussion of extending current datasets to include cases of inference where more fined-grained phenomena are taken into consideration @xcite ; @xcite . this also connects with the generalization capabilities of nli models that were briefly brought up in the introduction. however, the goal should not only be to create many diverse datasets that can get very fine-grained for numerous syntactic phenomena. what we further need are models that will have the ability to generalize well to new data after they have been trained on datasets that represent a much more diverse and rich picture of nli, and are not prone to similar problems as these have been reported in the literature @xcite @xcite .