Enhancing Unsupervised Semantic Parsing with Distributed Contextual Representations (2023)

Abstract: We extend a non-parametric Bayesian model of (Titov and Klementiev, 2011) to deal with homonymy and polysemy by leveraging distributed contextual word and phrase representations pre-trained on a large collection of unlabelled texts. Then, unsupervised semantic parsing is performed by decomposing sentences into fragments, clustering the fragments to abstract away syntactic variations of the same meaning, and predicting predicate-argument relations between the fragments. To better model the statistical dependencies between predicates and their arguments, we further conduct a hierarchical Pitman-Yor process. An improved Metropolis-Hastings merge-split sampler is proposed to speed up the mixing and convergence of Markov chains by leveraging pre-trained distributed representations. The experimental results show that the models achieve better accuracy on both question-answering and relation extraction tasks.

## introduction the goal of semantic parsing is to map natural language input into a formal meaning representation (mr), which is one of the long-standing challenges in natural language understanding. unlike shallow semantic analysis tasks such as relation extraction and semantic role labeling, the output of semantic parsing is complete and unambiguous to the point where it is machine interpretable or even can be executed by a computer program in order to enable various tasks including question answering, reading comprehension, parsing utterances in conversational agents, and translating natural language to database queries @xcite . early semantic parsing systems were built using hand-crafted rules @xcite @xcite . after the seminal work of @xcite , much attention has been given to statistical approaches that can learn models on a corpus of pairs of sentences and their desired outputs @xcite @xcite . both rule-based and statistical approaches require a large amount of labor-intensive annotation. many methods have been proposed to reduce the number of annotated examples including active learning @xcite , weak supervision @xcite , using auxiliary information @xcite , supervision from conversations @xcite , and learning from user feedback @xcite . however, writing hand-crafted rules or creating training datasets by manual annotation is still a formidable task so they are hard to scale and only work well in certain domains. over the last decade, there has been a rise in endto-end trainable neural network-based approaches using encoder-decoder frameworks for semantic parsing @xcite @xcite . arguably, the biggest disadvantage of these approaches is their "black box" nature-it is hard to know how or why a neural network comes up with a certain output. it is still unclear whether the machine truly "understands" natural language or just uses some tricks and shortcuts to fulfill the tasks @xcite . even though neural network-based approaches greatly reduce the burden of defining lexicons, templates and manually selected features, it is hard for them to model meaning and composition at varying levels of granularity by disentangling higher-and lowerlevel semantic information and capturing meaning from low-level to high-level via compositionality. unsupervised approaches are more widely applicable than supervised ones because they do not require humans to manually annotate training data. the work of @xcite is the first attempt to learn a semantic parser in an unsupervised way. they use markov logic networks @xcite to model the joint probability of dependency trees and their la-figure 1 : two example sentences with different syntactic structures but sharing the same meaning representation of border(usa,canada). these syntactic structures can be better represented by their contextual embeddings. inspired by @xcite , @xcite considered the goal of semantic parsing is to decompose the dependency tree of a sentence into fragments, assign each fragment to a semantically equivalent cluster, and predict predicate-argument relations between the fragments. they use hierarchical pitman-yor processes to model dependencies between the meaning representations of predicates and those of their arguments. however, their approach fails to model polysemy while many words in languages are polysemous, carrying multiple related and distinct meanings. as the examples shown in we extend the work of @xcite in the following five aspects: (i) the features derived from the contextual word and phrase representations pre-trained on large-scale unlabelled texts are integrated into a non-parametric bayesian model for unsupervised semantic parsing; (ii) capturing phenomena of homonymy and polysemy by leveraging introduced distributed representations that cannot be modeled before; (iii) phrase-level representations or embeddings are used to better determine whether adjacent words should be composed into a fragment as the smallest semantic unit; (iv) the similarity scores estimated by distributed contextual representations are taken into account in selecting which two semantic classes could be merged into one with priority, which greatly speeds up the mixing and convergence of markov chains; (v) unlike the situation where only discrete features are used, language semantics can be modeled in a more compact feature space of distributed representations to alleviate the problem of data sparsity. with the above improvements, the enhanced models achieved better performance on both question-answering and relation extraction tasks. the source code of our model can be downloaded from https://github.com/ narcissuslzx/usp . ## method similar to @xcite , we consider the problem of semantic parsing as a process that seeks to split the words of a sentence into fragments, assign each fragment to a cluster consisting of semantically equivalent expressions, and identify predicateargument relations between the fragments, given the dependency parse tree of the sentence. as two example sentences shown in figure 1 , we should compose three adjacent words of "the united states" to a fragment and assign it to the semantic class "usa". the fragments of "shares a border with" and "is adjacent to" also need to be grouped into a semantic class "border". therefore, a major challenge in semantic parsing is syntactic variations of the same meaning, which abound in natural languages. by our definition of unsupervised semantic parsing (usp) problem, two main matters need to be addressed. one is to determine whether neighboring words should be composed into fragments, and the other is to cluster fragments into groups based on their similarity in semantic meaning. we demonstrate that pre-trained contextual word and phrase embeddings are quite useful to better revolve those two matters. ## related work as one of the major challenges in natural language processing, many methods have been proposed for semantic parsing, which generally can be divided into three categories: rule-based @xcite @xcite @xcite @xcite , and neural network-based approaches @xcite @xcite . existing approaches differ in the form of meaning representations and the amount of annotation required. in the following, we mainly review prior work on unsupervised statistical methods by which manually labeled training examples are no longer required to build parsing models and refer to two recent surveys @xcite for the other methods. @xcite proposed the first unsupervised approach to semantic parsing which defines a probabilistic model over the dependency tree and semantic parse using markov logic. their model recursively clusters and composes the fragments of dependency trees using a hard em-style procedure. since they use non-local features and operate over partitions, exact inference is infeasible. they thus resort to a greedy algorithm to find the maximum-a-posteriori parse by searching over partitions. although it is a powerful model, it is too computationally expensive to run on large corpora. besides, the methodology of markov logic networks (innately undirected models) might not be suitable for modeling the semantic structure of a sentence derived from its directed parse tree. @xcite introduced an unsupervised learning algorithm for semantic parsing, which takes a self-training method driven by confidence estimation. the algorithm iteratively identifies high-confidence self-labeled examples with several simple scoring models and uses the identified samples to re-train the model. to compensate for the absence of direct supervision, @xcite proposed a grounded-learning approach to leverage database schema for indirect supervision. @xcite showed that converting a knowledge graph (kg) to its description in natural language (i.e., text generation) and mapping a text back to the kg (i.e, semantic parsing) can be done jointly in an unsupervised manner. @xcite first used an unsupervised paraphrase model to convert natural language utterances into their canonical utterances that were automatically generated by grammar rules and associated with the logic forms, and then trained a semantic parser on a collection of pairs of natural language utterances and the corresponding logic forms in a supervised way. those approaches are different from our bayesian model as they rely on either pseudo examples generally without human annotation or external resources such as database schemata or knowledge graphs. ## conclusion we improved the unsupervised learning algorithm proposed by @xcite for semantic parsing based on a non-parametric bayesian model. pre-trained contextual word and phrase embeddings were introduced to capture the linguistic phenomena of homonymy and polysemy. those embeddings and the similarity scores derived from them are also used to determine whether adjacent words can be composed and which semantic classes should be merged during the sequential importance sampling, which can greatly improve computational efficiency. we demonstrate empirically that the semantic parser learned by our approach achieved better performance over the baselines on both question-answering and relation extraction tasks, and show that contextual distributed representations play a vital role in capturing the polysemous variants of words and phrases.