Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning (2020)

Abstract: We propose to solve the natural language inference problem without any supervision from the inference labels via task-agnostic multimodal pretraining. Although recent studies of multimodal self-supervised learning also represent the linguistic and visual context, their encoders for different modalities are coupled. Thus they cannot incorporate visual information when encoding plain text alone. In this paper, we propose Multimodal Aligned Contrastive Decoupled learning (MACD) network. MACD forces the decoupled text encoder to represent the visual information via contrastive learning. Therefore, it embeds visual knowledge even for plain text inference. We conducted comprehensive experiments over plain text inference datasets (i.e. SNLI and STS-B). The unsupervised MACD even outperforms the fully-supervised BiLSTM and BiLSTM+ELMO on STS-B.

## introduction humans are not supervised by the natural language inference (nli). supervision is necessary for applications in human-defined domains. for example, humans need the supervision of what is a noun before they do pos tagging, or what is a tiger in wordnet before they classify an image of tiger in imagenet. however, for nli, people are able to entail that a a man plays a piano contradicts b a man plays the clarinet for his family without any supervision from the nli labels. in this paper, we define such inference as a more general process of establishing associations and inferences between texts, rather than strictly classifying whether two sentences entail or contradict each other. inspired by this, we raise the core problem in this paper: given a pair of natural language sentences, can machines entail their relationship without any supervision from inference labels? in his highly acclaimed paper, neuroscientist moshe bar claims that "predictions rely on the existing scripts in memory, which are the result of real as well as of previously imagined experiences" @xcite . the exemplar theory argues that humans use similarity to recognize different objects and make decisions @xcite . analogy helps humans understand a novel object by linking it to a similar representation existing in memory @xcite . such linking is facilitated by the object itself and its context @xcite . context information has been widely applied in selfsupervision learning (ssl) @xcite @xcite . adapting context to nli is even more straightforward. a simple idea of constant conjunction is that a causes b if they are constantly conjoined. although constant conjunction contradicts "correlation is not causation", modern neuroscience has confirmed that humans use it for reasoning in their mental world @xcite . for example, they found an increase in synaptic efficacy arises from a presynaptic cell's repeated and persistent stimulation of a postsynaptic cell in hebbian theory @xcite . as to the natural language, the object and its context can be naturally used to determine the inference. for example, a contradicts b because they cannot happen simultaneously in the same context. the context representation learned by ssl (e.g. bert @xcite ) has already achieved big success in nlp. from the perspective of context, these models @xcite learn the sentence level contextual information (i.e. by next sentence prediction task) and the word level contextual information (i.e. by masked language model task). besides linguistic contexts, humans also link other modalities (e.g. visions, voices) to novel inputs @xcite . even if the goal is to reason about plain texts, other modalities still help (although they are not provided as inputs) @xcite . for example, if only textual information is used, it is difficult to entail the contradiction between a and b . we need the commonsense that a man only has two arms, which cannot play the piano and clarinet simultaneously. this commonsense is hard to obtain from the text. however, if we link the sentences to their visual scenes, the contradiction is much clearer because the two scenes cannot happen in the same visual context. we think it is necessary to incorporate other modalities for the unsupervised natural language inference. the idea of adapting multimodal in ssl is not new. according to @xcite , we briefly divide previous multimodal ssl approaches into two categories based on their encoder infrastructures. as shown in fig. 1a , the first category uses one joint encoder to represent the multimodal inputs @xcite @xcite @xcite . obviously, if the downstream task is only for plain text, we cannot extract the representation of text separately from the joint encoder. so the first category is infeasible for the natural language inference. the second category @xcite @xcite first encodes the text and the image separately by two encoders. then it represents the multimodal information via a joint encoder over the lower layer encoders. this is shown in fig. 1b . although the textual representation can be extracted from the text encoder in the lower layer, such representation does not go through the joint learning module and contains little visual knowledge. in summary, the encoders in previous multimodal ssl approaches are coupled. if only textual inputs are given, they cannot effectively incorporate visual knowledge in their representations. thus their help for entailing the contradiction between a and b is limited. in order to benefit from multimodal data in plain text inference, we propose the multimodal aligned contrastive decoupled learning (macd) network. this is shown in fig. 1c . its text encoder is decoupled, which only takes the plain text as inputs. thus it can be directly adapted to downstream nli tasks. besides, we use multimodal contrastive loss between the text encoder and the image encoder, thereby forcing the text representation to align with the corresponding image. therefore even if the text encoder in macd only takes the plain text as input, text encoder text image encoder image joint encoder ssl loss (b) multimodal ssl with two single-modal encoders and one joint encoder. text encoder text image joint encoder for image multimodal nce loss downstream nli tasks (c) our proposed multimodal aligned contrastive decoupled network. when adapting to downstream nli tasks, we directly leverage the representation by the text encoder through the red lines, which only requires text as input. it still represents visual knowledge. in the downstream plain text inference tasks, without taking images as input, the text encoder of macd still implicitly incorporating the visual knowledge learned by the multimodal contrastive loss. note that we do not need a decoupled image encoder in the ssl. so the image encoder in fig. 1c in macd takes texts as inputs to provides a more precise image encoder. we will elaborate this in section 2.1. ## problem formulation we outline the general decoupled ssl process of macd in section 2.1, and the downstream unsupervised nli task in section 2.2. ## methods this section elaborates our major methodology. in section 3.1, we show how we maximize the crossmodal mutual information (mi) for the decoupled representation learning. in section 3.2, we show how we incorporate the mutual information (mi) of local structures. we elaborate the encoders in section 3.3. in section 3.4, in order to solve the catastrophic forgetting problem, we use lifelong learning regularization to anchor the text.