Semantic-aware Contrastive Learning for More Accurate Semantic Parsing (2022)

Abstract: Since the meaning representations are detailed and accurate annotations which express fine-grained sequence-level semtantics, it is usually hard to train discriminative semantic parsers via Maximum Likelihood Estimation (MLE) in an autoregressive fashion. In this paper, we propose a semantic-aware contrastive learning algorithm, which can learn to distinguish fine-grained meaning representations and take the overall sequence-level semantic into consideration. Specifically, a multi-level online sampling algorithm is proposed to sample confusing and diverse instances. Three semantic-aware similarity functions are designed to accurately measure the distance between meaning representations as a whole. And a ranked contrastive loss is proposed to pull the representations of the semantic-identical instances together and push negative instances away. Experiments on two standard datasets show that our approach achieves significant improvements over MLE baselines and gets state-of-the-art performances by simply applying semantic-aware contrastive learning on a vanilla Seq2Seq model.

## introduction semantic parsing aims to translate natural language utterances into formal meaning representations(mrs), which has attracted much attention for many years @xcite @xcite . recent studies mostly treat semantic parsing as a neural sequence to sequence translation task via encoder-decoder frameworks @xcite @xcite @xcite . ## base seq2seq parser this paper uses the classical seq2seq semantic parser as our base model due to its simplicity and effectiveness @xcite . encoder. given a sentence @xmath0, w 2 , ..., w n , a bidirectional lstm @xcite or bert @xcite can be used to map words into h @xmath1, h 2 , ..., h n . attention-based decoder. given the sentence representation, the tokens of the logical forms are generated sequentially. specifically, the decoder is first initialized with the hidden states of the encoder. then at each step t, let φ(y t-1 ) be the vector of the previous predicted token, the current hidden state s t is obtained from φ(y t-1 ) and s t-1 . we calculate the attentioned source context representations for the current step t: the seq2seq model is trained by maximizing the likelihoods of the tokens in an autoregressive fashion: ## semantic-aware contrastive learning for semantic parsing in this section, we describe how to address the sequence-level semtantics and fine-granularity challenges via semantic-aware constrastive learning. the contrastive learning aims to disperse apart semantic-distinct instances and pull closer semantic-identical instances on vector representation space. specifically, to learn to differentiate contrast similarity: 0 > 1 > 2 1 rank  2 rank  0 rank  the labeled mr mr with right exe results other mrs _ _ 0 rank  utterances labeled the mr 1 rank  utterances with right answer 1 rank  paraphrases 2 rank  other utterances similarity: > > push pull utterances mrs , , ' . . . . . . , ' { , ' | = 0} { , ' | = 1} { , ' | = 2} multi-level parition ', . . . . . . ', { ', | = 0} { ', | = 1} { ', | = 2} multi-level parition (, ) similarity (, ) similarity mr sampling '~ ∪ () utterance sampling '~ ∪ () figure 2: the architecture of our method, in which (1) we first collect samples via online sampling where p arse(x) are the parses of the current model and aug(x) are the paraphrased augmentations; (2) then we divide samples into multi-levels, where @xmath2, @xmath3, and @xmath4; (3) finally we use ranked contrastive loss for optimization based on the similarities between the multi-level samples. in following we describe them in detail. ## conclusions this paper proposes semantic-aware contrastive learning -an effective contrastive learning framework for semantic parsing, which takes the sequence-level semantics and the fine-granularity into consideration. specifically, we propose a multi-level online sampling algorithm for accurately collecting confusing and diverse samples, and design three semantic-aware similarity functions to measure the similarity between utterances and mrs. we also propose ranked contrastive loss to optimze the representations from the multi-level samples. experimental results show that our approach can achieve state-of-theart performance in several domains and datasets.