Reasoning about implied relationships ( e.g. paraphrastic , common sense , encyclopedic ) between pairs of words is crucial for many cross-sentence inference problems . This paper proposes new methods for learning and using embeddings of word pairs that implicitly represent background knowledge about such relationships . Our pairwise embeddings are computed as a compositional function on word representations , which is learned by maximizing the pointwise mutual information ( PMI ) with the contexts in which the two words cooccur . We add these representations to the cross-sentence attention layer of existing inference models ( e.g. BiDAF for QA , ESIM for NLI ) , instead of extending or replacing existing word embeddings . Experiments show a gain of 2.7 % on the recently released SQuAD 2.0 and 1.3 % on MultiNLI . Our representations also aid in better generalization with gains of around 6 - 7 % on adversarial SQuAD datasets , and 8.8 % on the adversarial entailment test set by Glockner et al . ( 2018 ) . Reasoning about relationships between pairs of words is crucial for cross sentence inference problems such as question answering ( QA ) and natural language inference ( NLI ) . In NLI , for example , given the premise " golf is prohibitively expensive " , inferring that the hypothesis " golf is a cheap pastime " is a contradiction requires one to know that expensive and cheap are antonyms . Recent work ( Glockner et al . , 2018 ) has shown that current models , which rely heavily on unsupervised single-word embeddings , struggle to learn such relationships . In this paper , we show that they can be learned with word pair vectors ( pair2vec 1 ) , which are trained unsupervised , and which significantly improve performance when added to existing cross-sentence attention mechanisms . Unlike single-word representations , which typically model the co-occurrence of a target word x with its context c , our word-pair representations are learned by modeling the three-way cooccurrence between words ( x , y ) and the context c that ties them together , as seen in Table 1 . While similar training signals have been used to learn models for ontology construction ( Hearst , 1992 ; Snow et al . , 2005 ; Turney , 2005 ; Shwartz et al . , 2016 ) and knowledge base completion ( Riedel et al . , 2013 ) , this paper shows , for the first time , that large scale learning of pairwise embeddings can be used to directly improve the performance of neural cross-sentence inference models . More specifically , we train a feedforward network R(x , y ) that learns representations for the individual words x and y , as well as how to compose them into a single vector . Training is done by maximizing a generalized notion of the pointwise mutual information ( PMI ) among x , y , and their context c using a variant of negative sampling ( Mikolov et al . , 2013a ) . Making R(x , y ) a compositional function on individual words alleviates the sparsity that necessarily comes with embedding pairs of words , even at a very large scale . We show that our embeddings can be added to existing cross-sentence inference models , such as BiDAF++ ( Seo et al . , 2017 ; Clark and Gardner , 2018 ) for QA and ESIM ( Chen et al . , 2017 ) for NLI . Instead of changing the word embeddings that are fed into the encoder , we add the pretrained pair representations to higher layers in the network where cross sentence attention mechanisms are used . This allows the model to use the background knowledge that the pair embeddings implicitly encode to reason about the likely relationships between the pairs of words it aligns . Experiments show that simply adding our wordpair embeddings to existing high-performing models , which already use ELMo ( Peters et al . , 2018 ) , results in sizable gains . We show 2.72 F1 points over the BiDAF++ model ( Clark and Gardner , 2018 ) on SQuAD 2.0 ( Rajpurkar et al . , 2018 ) , as well as a 1.3 point gain over ESIM ( Chen et al . , 2017 ) on MultiNLI ( Williams et al . , 2018 ) . Additionally , our approach generalizes well to adversarial examples , with a 6 - 7 % F1 increase on adversarial SQuAD ( Jia and Liang , 2017 ) and a 8.8 % gain on the Glockner et al . ( 2018 ) NLI benchmark . An analysis of pair2vec on word analogies suggests that it complements the information in single-word representations , especially for encyclopedic and lexicographic relations . We presented new methods for training and using word pair embeddings that implicitly represent background knowledge . Our pair embeddings are computed as a compositional function of the individual word representations , which is learned by maximizing a variant of the PMI with the contexts in which the the two words co-occur . Experiments on cross-sentence inference benchmarks demonstrated that adding these representations to existing models results in sizable improvements for both in-domain and adversarial settings . Published concurrently with this paper , BERT ( Devlin et al . , 2018 ) , which uses a masked language model objective , has reported dramatic gains on multiple semantic benchmarks including question-answering , natural language inference , and named entity recognition . Potential avenues for future work include multitasking BERT with pair2vec in order to more directly incorporate reasoning about word pair relations into the BERT objective . Bishan Yang and Tom Mitchell . 2017 . Leveraging knowledge bases in LSTMs for improving machine reading . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( Volume 1 : Long Papers ) , pages 1436 - 1446 . Association for Computational Linguistics .

Challenge: Existing methods for word pair relationship reasoning suffer from learning such relationships because they heavily rely on unsupervised single-word embeddings.
Approach: They propose a compositional function on word representations which learns maximizing the pointwise mutual information with contexts of two words to compute pairwise embeddings.
Outcome: The proposed representations coupled with neural cross-sentence inference models achieve gains on SQuAD 2.0 and MultiNLI and generalize better on adversarial datasets.