Mxgra at S em E val-2020 Task 4: Common Sense Making with Next Token Prediction (2020)

Abstract: In this paper, we explore solutions to a common sense making task in which a model must discern which of two sentences is against common sense. We used a pre-trained language model which we used to calculate complexity scores for input to discern which sentence contained an unlikely sequence of tokens. Other approaches we tested were word vector distances, which were used to find semantic outliers within a sentence, and siamese network. By using the pre-trained language model to calculate perplexity scores based on the sequence of tokens in input sentences, we achieved an accuracy of 75 percent.

## introduction much research has been conducted on natural language understanding (nlu) and models based on neural networks have achieved state-of-the-art results in reading comprehension . however, results in tasks regarding common sense making do not rival those of human performance, such as in inference tasks in which state-of-the-art models score upwards of 80 percent @xcite . as the organizers of the semeval 2020 commonsense validation and explanation @xcite ) task highlighted, pinpointing exactly why an utterance is against common sense is not a trivial undertaking. advances in this area could lead to improvements in nlu. we participated only in task a, which was determining which of two sentences was against common sense. for example, given the input "he poured milk on his cereal" and "he poured orange juice on his cereal" we would identify "he poured orange juice on his cereal" as the sentence against common sense. task a was evaluated by accuracy. all data was in english. we used a pre-trained language model to calculate the probability of the next word in a sequence to produce perplexity scores for each of the sentences in the input pairs. a higher perplexity score meant a lower overall probability of the words occurring in this order, thus being more likely against commonsense. we also attempted to solve this task with word vector distances, attempting to find an outlier among tokens in each input sentence, and with a siamese network, which was trained as a feature embedder for valid sentences. ## background the data consisted of pairs of english sentences, one of which made sense and the other did not. the desired output was the index of the sentence against common sense. for example, the model would take the input sentences "he poured orange juice on his cereal." and "he poured milk on his cereal." and output a 0. the official training data from the comev organizers was a collection of 10,000 pairs of sentences. this work is licensed under a creative commons attribution 4.0 international licence. licence details: http:// creativecommons.org/licenses/by/4.0/ . ## system overview we used three distinct and separate approaches to solve this task. our main approach is based on the probability of the next word in the sequence. after analyzing the data, we realized the structure of the input was such that the perplexity score of each sentence would be a strong indicator of its validity. we made use of a pre-trained neural network, the openai gptlm head model @xcite , from the transformers library from huggingface foot_0 to calculate perplexity scores. this model is based on the openai gpt model @xcite , a causal (unidirectional) transformer pre-trained using language modeling on a large corpus with long range dependencies, the toronto book corpus @xcite . the openai gpt head model consists of the basic transformer @xcite , but with a language modeling head on top (linear layer with weights tied to the input embeddings) and is therefore powerful at predicting the next token in a sequence. our classification pipeline was rather small and straight-forward. we loaded the model with pretrained weights and fed one tokenized sentence as input. as output we took the prediction scores of the language modeling head (e.g. the scores for each vocabulary token before a softmax layer that chooses the most probable word). the scores for both sequences of a pair were then compared, the larger one labeled as invalid. another approach was to calculate distances between word vectors of words inside one sentence. the general idea is simple: as language models trained on large corpora are increasingly better at embedding words in semantically meaningful vectors, we tried to use the multidimensional representation of big word embeddings to find semantic outliers in the sentence. if a sentence contained such an outlier, it is marked as the invalid one. only sentences where the keyword (differentiating word between the two sentences) was a subject or object (direct and indirect) were used. the bert base-uncased language model @xcite and the last four of its layers computed the word embeddings. we used the spacy parser @xcite to first create a dependency tree of the sentence, a simple function then extracted the root of the sentence and the desired subjects and objects. next, we computed the cosine distance between the word vectors in the sentence and averaged the score over the number of tokens. the smaller this score, the more likely the sentence to be sensical. born out of the word-vector approach discussed above, our final approach was a siamese network with which our goal was to train it as a feature embedder for valid sentences. from a selection of the provided data we crafted anchor sentences through the bert masked lm model @xcite . the keyword was masked and of the ten most probable predicted words the most viable (same category of part of speech) were used. this was done automatically for all sentences, therefore there is the possibility that some novel sentences are not viable as sensible anchors. a review of all the anchors was not feasible. we implemented a custom variant of the triplet loss based on vector distances, which should push the model to create context embeddings that achieve a higher cosine similarity for the valid sentences to the anchor than the invalid ones. ## experimental setup we limited preprocessing because of the terseness of input data. the shortest input string was of length two. the longest input string was 23 words long. the average length of input was 7.7 words. we decided to alter the data as little as possible in the preprocessing stage because of our next-token-prediction approach. input without stop words, for example, would have resulted in lower perplexity scores from the gpt head model @xcite , as they would be more unlikely to appear in the corpus the model was trained on. additionally, some of the input strings in the training set were against common sense because they were grammatically incorrect, and we opted to preserve the original structure of the input so that these sentences would result in a higher perplexity score from the model. we performed a contraction mapping with the use of the pycontractions package foot_1 to ensure proper tokenization. task a was evaluated on accuracy. ## results the gpt model achieved an accuracy of 75 percent, which places us 25th among 28 teams, who submitted results in the post-evaluation phase. teams placed 15th to 26nd, achieved accuracies between 80 and 90 percent. teams 14th and higher achieved accuracies above 92 percent, the highest being 96.7 percent. ## discussion the main observation we drew from the results is that our model is not able to handle input sentences that are outside of common parlance. our model also had difficulty correctly labeling sentences with spelling or grammatical errors i.e "dog's" instead of "dogs." on the other hand, our model is able to distinguish very well the nonsensical sentence in pairs in which a reasonable noun is exchanged for an outrageous one. for example, "book" was replaced with "tiger" in the sentence "i read a ". we came to the conclusion that, if we were able to repeat this task, we would use a hybrid approach. rather than relying on statistics, however powerful and effective, we would supplement our process with a discriminative function. our model could also benefit from the utilization of a semantic analysis. with the use of, for example, framenet @xcite , a lexical database of annotated english examples of how words are used in actual context from the international computer science institute at uc berkeley, we could establish that, using the above example, a tiger is an animal, and therefore something that cannot fit in with the semantic framework for reading.