Parsing Natural Language into Propositional and First-Order Logic with Dual Reinforcement Learning (2022)

Abstract: Semantic parsing converts natural language utterances into structured logical expressions. We consider two such formal representations: Propositional Logic (PL) and First-order Logic (FOL). The paucity of labeled data is a major challenge in this field. In previous works, dual reinforcement learning has been proposed as an approach to reduce dependence on labeled data. However, this method has the following limitations: 1) The reward needs to be set manually and is not applicable to all kinds of logical expressions. 2) The training process easily collapses when models are trained with only the reward from dual reinforcement learning. In this paper, we propose a scoring model to automatically learn a model-based reward, and an effective training strategy based on curriculum learning is further proposed to stabilize the training process. In addition to the technical contribution, a Chinese-PL/FOL dataset is constructed to compensate for the paucity of labeled data in this field. Experimental results show that the proposed method outperforms competitors on several datasets. Furthermore, by introducing PL/FOL generated by our model, the performance of existing Natural Language Inference (NLI) models is further enhanced.

## introduction semantic parsing is the task of mapping natural language utterances into logical expressions. as two major logical forms of text representation, propositional logic (pl) and first-order logic (fol) play an increasingly important role in a wide range of downstream tasks including inductive logic programming (ilp) @xcite , question answering (qa) @xcite and interpretable reinforcement learning @xcite of discovering and representing knowledge in an explicit symbolic structure that can be understood and examined by human @xcite . it is worth noting that all of these tasks have a prerequisite, i.e., parsing natural language utterances into pl/fol. the results of parsing directly affect the performance of downstream tasks. thus, it is crucial to have a strong semantic parser for pl/fol. some solutions have been proposed to parse natural language into pl/fol. one of the most typical methods is to model the parsing task as a sequenceto-sequence (seq2seq) generation problem, including using a character-level recurrent neural network @xcite and introducing a variable alignment mechanism @xcite . however, these approaches have the following issues. first, a large amount of labeled data is required for these approaches to achieve good results, which inevitably suffers from the paucity of labeled data in this field. one of the solutions is to generate labeled data by templates @xcite , but it leads to a lack of diversity in the data which makes the model prone to overfitting the training data. second, previous works @xcite only consider unidirectional generation (from utterance to pl/fol), while intuitively bidirectional generation can further enhance the performance of the models. regarding these issues, we propose an effective framework for parsing natural language into pl/fol named dual-(m)t5, and introduce unlabeled data to alleviate the impact of insufficient labeled data. inspired by @xcite ; @xcite , we model the learning of logical expressions and natural language generation as dual tasks. both of the tasks are jointly trained via reinforcement learning (rl) since the training process is non-differentiable. however, we encounter the following challenges when applying dual reinforcement learning to pl/fol: 1) the validity reward in dual reinforcement learning is rule-based @xcite , which results in a lot of manual attempts to get an effective one, and needs to be redesigned when the type of logical expressions changes (e.g. from lambda-calculus to fol) since it is customized for a specific type of logical expression. 2) the effectiveness of the validity reward needs to be improved because it only considers a lexical-level matching between utterances and logical expressions, and cannot achieve deep semantic matching. 3) an effective training strategy needs to be explored since the training process easily collapses when the model is trained with only the rewards from dual reinforcement learning. to address the first two issues mentioned above, we propose a scoring model to automatically learn a model-based validity reward that is applicable to various types of logical expressions. the scoring model aims to evaluate whether the semantics of utterances and logical expressions match. for issue 3, we propose an effective training strategy to stabilize the training process. specifically, curriculum learning @xcite is employed to initialize the parameters of the model, aiming to get the model in a good initial state, and unlabeled data is introduced to prevent models from crashes due to insufficient labeled data. experimental results on different datasets show that our framework effectively parses natural language into pl/fol and consistently improves performance compared to competitors. to further demonstrate the value of pl/fol, we take natural language inference (nli) as a downstream task, and the performance of existing nli models is further enhanced by introducing pl/fol generated by our model. our contributions are three-folds: ## overview in this section, we formalize the problem and outline our framework. ## methodology in this section, we first present the details of dual reinforcement learning in § 3.1. the scoring model is introduced in § 3.2, and the training strategy is provided in § 3.3. ## dataset collection to compensate for the paucity of labeled data in this field and verify the effectiveness of our proposed framework, we construct a dataset containing natural language and pl/fol pairs. in the previous work @xcite , the authors define templates first and then obtain samples by filling slots. however, the resulting dataset is limited by the lack of diversity of templates. crowdsourcing is another option but is not applicable for this task, since professional knowledge about pl/fol is required. therefore, we use expert annotation to ensure the quality and diversity of the dataset. the annotation team consists of 8 chinese graduate students who are familiar with pl/fol. if the annotators are required to construct data without any reference, this will introduce inevitable troubles and labeling errors, since pl/fol is not intuitive to humans. aiming to reduce nontrivial human labor and ensure the quality of the dataset, the data collection process consists of the following steps: we first obtain pl/fol exercise sets and exam papers that require students to convert natural language into pl/fol from baidu wenku foot_1 . then, the annotators are asked to organize these exercises in a uniform format. each sample consists of three parts: natural language sentence s, symbolic definition d, and logical expression e (see in in this way, we obtain a total of 2,727 samples consisting of 1,263 pl and 1,464 fol with the corresponding utterances and symbolic definitions. to establish human performance and conduct consistency assessments, we ask an additional 3 undergraduate and 2 graduate students who have acquired basic knowledge of pl/fol to provide logical expressions given natural language sentences and symbolic definitions from the entire test set. the detailed statistics of the dataset are shown in ## experiments in the experimental section, we investigate the following research questions: 1) how is the overall performance of dual-(m)t5 in comparison to competitors? 2) how does dual-(m)t5 perform on other types of logical expressions? 3) what is the optimal ratio of unlabeled data to labeled data? 4) are all the components in dual-(m)t5 necessary? 5) which samples are not yet well processed by the model? 6) can the logical expressions generated by dual-(m)t5 help downstream tasks? 5.1 setup datasets. 1) english-fol @xcite is generated by pre-defined templates and contains natural language utterances paired with fol. we follow the training/validation/test splits as @xcite . 2) chinese-pl/fol the details of our dataset have been introduced in § 4. since symbol definition only appears in this dataset, we concatenate it with the original input, i.e., natural language sentence in the prime module and logical expression in the dual module. due to the small amount of training data, pl and fol are trained together. 3) atis @xcite consists of queries about flight information and logical expressions in lambda-calculus syntax. for fairness in model comparison, we keep the same preprocessing settings as @xcite . baselines. to verify the effectiveness of our approach on english-fol and chinese-pl/fol, we reproduce several strong baselines as there are not many existing works on these two datasets. for atis, we directly compare our method with stateof-the-art works. refer to appendix.c for details. ## conclusion in this paper, we introduce dual-(m)t5, an effective dual reinforcement learning framework for parsing natural language into pl/fol. a novel reward mechanism is proposed to avoid manually defining the validity reward in rl. an effective training strategy is further proposed to stabilize the training process. experimental results show that the proposed method outperforms competitors on several datasets. by introducing logical expressions, we further enhance the existing nli model. in addition to the technical contribution, a new dataset called chinese-pl/fol is constructed to aid further research in this field. a algorithm algorithm 1 training scoring model input: supervised dataset @xmath0, e⟩}; number of nucleus sampling k; fine-tuned nl2le model output: scoring model 1: p ← {}, n ← {} 2: for all ⟨s, e⟩ ∈ l do 3: p ← p ∪ {⟨s, e⟩} 4: given s, fine-tuned nl2le model generates k logical expressions {e i } via nucleus sampling 5: for all e @xmath1: if e i ̸ = e then 7: n ← n ∪ {⟨s, e i ⟩} 8: end if 9: end for 10: end for 11: repeat 12: update scoring model w.r.t. eq.(10) 13: until scoring model converges