Semantic Parsing for E nglish as a Second Language (2020)

Abstract: This paper is concerned with semantic parsing for English as a second language (ESL). Motivated by the theoretical emphasis on the learning challenges that occur at the syntax-semantics interface during second language acquisition, we formulate the task based on the divergence between literal and intended meanings. We combine the complementary strengths of English Resource Grammar, a linguistically-precise hand-crafted deep grammar, and TLE, an existing manually annotated ESL UD-TreeBank with a novel reranking model. Experiments demonstrate that in comparison to human annotations, our method can obtain a very promising SemBanking quality. By means of the newly created corpus, we evaluate state-of-the-art semantic parsing as well as grammatical error correction models. The evaluation profiles the performance of neural NLP techniques for handling ESL data and suggests some research directions.

## introduction there are more people around the world learning english as a second language (esl) than there are native speakers of english with this gap continually and steadily expanding @xcite . accordingly, an extremely large volume of nonnative english texts are generated every day. we need an automatic machinery to annotate such large-scale atypical data with in-depth linguistic analysis. high-performance automatic annotation of learner texts, from an engineering point of view, enables it possible to derive high-quality information by structuring the specific type of data, and from a scientific point of view, facilitates quantitative studies for second language acquisition (sla), which is complementary to hands-on experiences in interpreting interlanguage phenom-ena @xcite . this direction has been recently explored by the nlp community @xcite @xcite . different from standard english, esl may preserve many features of learners' first languages foot_0 . the difference between learner texts and benchmark training data, e.g. penn treebank (ptb; @xcite , is more related to linguistic competence, rather than performance @xcite . this makes processing esl different from almost all the existing discussions on domain adaptation in nlp. despite the ubiquity and importance of interlanguages at both the scientific and engineering levels, it is only partially understood how nlp models perform on them. in this paper, we present, to the best of our knowledge, the first study on semantic parsing for english as a second language. motivated by the interface hypothesis @xcite in sla, we emphasize on the divergence between literal and intended meanings. to obtain reliable semantic analyses in order to represent the two types of meanings, we propose to combine english resource grammar @xcite , which is a wide-coverage, linguistically-precise, hand-crafted grammar and tle, which is a manually annotated syntactic treebank for esl in the universal dependency (ud; @xcite framework. in particular, we introduce a reranking model which utilizes the partial constraints provided by gold syntactic annotations to disambiguate among the grammar-licensed candidate analyses. experiments on deepbank @xcite demonstrates the effectiveness of our proposed model. by means of the newly created corpus, we study semantic parsing for esl, taking elementary de-pendency structure (eds; @xcite as the target representation. we probe the semantic parsing of multiple state-of-the-art neural parsers for literal meaning and intended meaning, and investigate how grammatical error correction (gec) can contribute to the parsing. in addition, we give a detailed analysis of the effect from grammatical errors. results reveal three facts: 1) semantic parsing is sensitive to non-canonical expressions, and the distribution as well as types of grammatical errors have an effect on parsing performance; 2) factorization-based parser is the most effective and robust parser to process learner english; and 3) automatic gec has a positive, but limited influence on the parsing of intended meaning. ## related work early work regarding the collection of learner corpora mainly concentrates on tagging alleged errors @xcite . the past decade has seen a tendency to directly annotate the linguistic properties in learner sentences @xcite @xcite . the lack of precisely annotated data has limited the systematic analysis of interlanguages. there are several attempts to set up annotation schemes for different linguistic layers of learner languages, such as pos tags and syntactic information @xcite @xcite @xcite . but it is challenging to elucidate the exact definition of "syntax" for learner languages. @xcite defines multiple layers (morphological dependencies, distributional dependencies, and subcategorization) based on different evidence to capture non-canonical properties. similarly, motivated by the interface hypothesis @xcite , we employ a principled method to create parallel semantic representations for learner english by discriminating between the literal and intended meanings. with regard to the semantic analysis for learner languages, @xcite takes the first step in this direction. based on a parallel semantic role labeling (srl) corpus, they prove the importance of syntactic information to srl for learner chinese. in this paper, we provide a much deeper semantic analysis for learner english. ## literal versus intended meaning there is a classic distinction between two aspects of meaning: the literal meaning (conventional meaning or sentence meaning) versus the intended meaning (speaker meaning or interpretation). the former puts an emphasis on the linguistic code features appearing in the sentence, while the latter is derived from the author's intention. when we consider an interlanguage, the divergence between literal and intended meanings is much larger due to various cross-lingual influences. it is reasonable to consider both aspects to develop a principled method to process outputs from l2 learners. contemporary research on sla has extensively argued and empirically supported the claim that linguistic properties pertaining to the interface between syntax and other linguistic modules are vulnerable in l2 and integrating linguistic phenomena relevant to such interfaces imposes much difficulty to l2 learners @xcite . according to this view, the interaction or mapping between syntactic and semantic representations is less likely to be acquired completely than structures within one single module, either syntactic or semantic. with respect to outputs of l2 learners, mismatches between syntactic structures and intended meanings are frequently observable. figure 1 presents an example from the tle corpus. although discussion is misused, the whole fragment is grammatical and thus interpretable according to syntactic analysis. however, the literal meaning along with a sound syntactic analysis is far from the intended meaning that a native speaker can infer from intra-and inter-sentence contexts. it is quite obvious that discussion should be regarded as a verb coordinating with give. nodes represent concepts, while edges represent semantic dependencies. following morphosyntax, "discussion" acts as the conjunct of the previous noun "topic". however, according to discourse, it should be juxtaposed with the verb "give" because these are two successive actions. ## two state-of-the-art parsers existing work in data-driven semantic graph parsing can be roughly divided into four types, namely composition-, factorization-, transitionand translation-based ones @xcite . according to experimental results obtained on benchmark datasets with various target structures including abstract meaning representation(amr; @xcite , elementary dependency structures (eds; @xcite , semantic dependency parsing (sdp) as well as universal conceptual cognitive annotatio (ucca; @xcite , the compositionand factorization-based approaches are the leading approaches obtained by now @xcite . in this paper, we use these two kinds of parsers (compositionand factorization-based parsers) described in @xcite as state-of-the-art representatives. following the principle of compositionality, a semantic graph can be viewed as the result of a derivation process, in which a set of lexical and syntactico-semantic rules are iteratively applied and evaluated. the core engine of the composition-based parser is a graph rewriting system that explicitly explores the syntacticosemantic recursive derivations that are governed by a synchronous hyperedge replacement grammar (shrg; @xcite . the parser constructs dmrs graphs by explicitly modeling such derivations. it utilizes a constituent parser to build a syntactic derivation, and then selects semantic hrg rules associated to syntactic cfg rules to generate a graph. when multiple rules are applicable for a single phrase, a neural network is used to rank them. we use the parser in @xcite based on both the lexicalized grammar and the constructional grammar (refer to @xcite for the distinction). henceforth, they are called lexicalized and constructional compositionbased parsers respectively. figure 3 shows an example of the shrg-based syntactico-semantic derivation from the constructional composition-based parser. the derivation can be viewed as a syntactic tree enriched with semantic interpretation rules that are defined by an hrg. each phrase in the syntactic tree is assigned with a sub-graph of the final semantic structure. moreover, some particular nodes in a sub-graph are marked as communication channels to other meaning parts in the same sentence. in hrg, these nodes are summarized as a hyperedge. two subgraphs are glued according to a construction rule following the graph substitution principle of hrg. the factorization-based parser explicitly models the target semantic structures by defining a score function that is able to evaluate the goodness of any candidate graph. it needs to know how to find the highest-scoring graph from a large set of each phrase in the syntactic tree ("and then" and "discussion about it") is assigned with a sub-graph of the final semantic structure, as illustrated in the boxes. some particular nodes (filled nodes) in a sub-graph are marked as communication channels to other meaning parts. according to the construction rule (shown in double-framed box), we glue the two sub-parts via the filled nodes, forming a larger graph with the syntactic label "np". more details are illustrated in @xcite possible candidates. the parser works with a twostage pipeline structure, for concept identification and relation detection, as illustrated in figure 4 . in the first phase, sequence labeling models are used to predict nodes, and in the second phase, we utilize the dependency model introduced by @xcite to link nodes. the two models in both stages use a multi-layer bilstm to encode tokens. in the first stage, another softmax layer is utilized to predict concept-related labels, while in the second stage, the dependency model is utilized to calculate a score for selecting token pairs. ## parsing to intended meanings previous evaluation indicates the difficulty to adopt a standard semantic parsing model to handle competence errors. motivated by this fact, we are concerned with whether it is feasible to automatically normalize the texts first. specifically, our strategy is correcting the grammatical errors contained in the input sentences, and then parsing the revised texts into semantic struc-tures with standard models. the first step can resort to grammatical error correction (gec), the task of correcting different kinds of errors in text. it has attracted a lot of attention and considerable effort has been made to promote the performance on specific benchmark data. we utilize two off-the-shelf gec models. one is a multilayer convolutional encoder-decoder neural network proposed in @xcite . we choose the basic model introduced in the paper. the other model copies the unchanged words from the source sentence to the target sentence using a pretrained copy-augmented architecture with a denoising auto-encoder @xcite . it achieves the state-of-the-art performance without extra pseudo data. performances of the two gec models on conll-2014 test set are shown in we train the factorization-based model on deepbank and examine the performance on l2 and l1 sentences as well as the revised sentences by two gec models. the produced graphs are compared with i-silver which represents the intended meaning. we notice that during the computation of smatch, some disagreements of nodes result from the discrepancy of morphological variation or different collocations between the input and the standard sentence. hence the node score may be underestimated. therefore, we relax the standards of matching nodes. we establish a paraphrase table based on the statistical machine translation between a parallel learner corpus foot_2 . as long as the labels of two aligned nodes have the same stem or they form a paraphrase pair in our table, then the two nodes can be considered "matching". we call the new evaluation metric as "noderelaxed smatch". ## conclusion and future work in this paper, we formulate the esl semantic parsing task based on the divergence on literal and intended meanings. we establish parallel meaning representations by combining the complementary strengths of knowledge-intensive erg-licensed analysis and dependency tree annotations through a new reranking model. for literal meaning, we probe the semantic parsing of multiple state-ofthe-art neural parsers and give detailed analysis of effects from grammatical errors. for intended meaning, we investigate how grammatical errors affect the understanding of sentences as well as how grammatical error correction (gec) can contribute to the parsing. results reveal three facts: 1) semantic parsing is sensitive to non-canonical expressions, and the parsing performance varies with regard to the distribution as well as types of grammatical errors; 2) factorization-based parser is the most promising parser to process learner english; and 3) gec has a positive, but limited influence on the parsing of intended meaning. this paper shows a pilot study on the semantic parsing for learner language. future research may involve tailoring existing parsers to learner data, combining literal and intended meanings in a unified framework, evaluating gec models in terms of speakers' intention and parsing for other languages.