Building Analyses from Syntactic Inference in Local Languages: An HPSG Grammar Inference System (2022)

Abstract: We present a grammar inference system that leverages linguistic knowledge recorded in the form of annotations in interlinear glossed text (IGT) and in a meta-grammar engineering system (the LinGO Grammar Matrix customization system) to automatically produce machine-readable HPSG grammars. Building on prior work to handle the inference of lexical classes, stems, affixes and position classes, and preliminary work on inferring case systems and word order, we introduce an integrated grammar inference system that covers a wide range of fundamental linguistic phenomena. System development was guided by 27 geneologically and geographically diverse languages, and we test the system’s cross-linguistic generalizability on an additional 5 held-out languages, using datasets provided by field linguists. Our system out-performs three baseline systems in increasing coverage while limiting ambiguity and producing richer semantic representations, while also producing richer representations than previous work in grammar inference.

## introduction machine-readable grammars for human languages that are grounded in theoretical syntactic formalisms can be useful tools in the context of endangered language documentation and revitalization. first, they support treebanking @xcite , which in turn supports data exploration @xcite ; and second, they facilitate the development of tools such as grammar checkers @xcite and automated tutors @xcite . in spite of these advantages, the use of such grammars is hindered by the time-consuming process of developing them together with the need of a specific skillset required for grammar engineering, which is distinct from the skills involved in documentation itself. we are therefore motivated to investigate whether we can create machinereadable grammars automatically. foot_0 endangered languages represent scenarios where the type of resources required for typical natural language processing techniques are scarce to non-existent. furthermore, the output we are targeting goes well beyond simple labels or even structured representations, but rather must be a coherent and well-formed formal object -a grammar. fortunately, we have two rich sources of linguistic knowledge from which to work: the first is corpora of interlinear glossed text (igt), annotated by field linguists during the process of documentation and analysis. due to the efforts of field linguists and archivists, a number of archives (many of which we list in appendix a) make igt data publicly available. an example from chintang [iso 639-3: ctn] is shown in (1). such annotations are linguistically rich, showing what grammatical information is marked morphologically and providing further information implicitly via a translation into a language of broader communication (in all examples we work with, this language of broader communication is english). using the methodology of annotation projection, as applied to igt @xcite , we can leverage parsers available for the translation language and project structural information such as part-of-speech (pos) tags and syntactic dependencies onto words in the target language. leveraging these sources, the question we investigate here is whether and how we can create machinereadable hpsg grammars for typologically diverse local 3 and/or endangered languages on the basis of corpora of igt and the grammar matrix. in particular, we build on the open-source code base provided by the ag-gregation project @xcite to produce the following contributions: (1) we integrate all existing inference modules into a single system to which (2) we add modules for additional grammatical phenomena and (3) where previous end-to-end testing treated only a single language, we use 27 diverse languages in development, doing end-to-end system testing on 9 of the 27, and then evaluate on 5 additional held-out languages not considered during system development. we begin by situating our work on grammar inference against the broader background of automatic grammar generation in section 2 and then provide background on the aggregation project in section 3. section 4 describes our methodology for grammar inference, including lexical, morphological and syntactic aspects of an inferred grammar. in section 5, we describe the languages we used in system development and how we use the delph-in suite of so ware tools to evaluate the grammars we create by parsing and treebanking held-out data from each language. we use that same methodology for held-out languages to evaluate the generalizability of the system, finding that though the coverage of the grammars is still limited, the proposed methodology generally produces higher quality grammars than three baseline approaches. the languages we test on and the results of this evaluation are presented in sections 6 and 7. finally, section 8 provides error analysis and discussion. we conclude in section 9 with discussion of applications of grammars produced in this fashion. ## automatic grammar generation interest in creating machine-readable grammars is likely as old as the field of computational linguistics itself, with published work in grammar engineeringthe process of creating machine-readable grammars by hand -going back at least as far as @xcite and continuing into the present day. our work in grammar inference builds on grammar engineering work (in the form of the grammar matrix; @xcite @xcite , but also fits into a tradition of work on automatic grammar generation, which is the development of systems that automatically create grammars on the basis of data. within automatic grammar generation, we distinguish four broad categories of approaches, differentiated by the types of inputs they take: grammar induction from stringsautomatic grammar generation based on text alone ( §2.1); grammar extraction -automatic grammar generation based on treebanks ( §2.2); grammar induction from meaning representations -automatic grammar generation based on strings paired with some form of semantic representation ( §2.3); and grammar inferenceautomatic grammar generation based on text annotated with partial grammatical information but not full parse trees or logical forms ( §2.4). just as these four approaches to grammar generation differ in their input, they also differ in the types of grammars they can produce. grammar induction, if working from strings alone, will produce noisy representations that align only partially with structures created by linguists. grammar extraction will produce grammars that provide the same kind of representations as given in the source treebank and similarly, grammar induction based on strings paired with semantic representations will produce grammars that can output those semantic representations. in each of these cases, the generated grammar will also typically include a parse selection model, based on observed pa erns in the corpus. grammar inference systems, by contrast, draw on both partial annotation in their input data and some external source of grammatical knowledge. for this reason, the inferred grammars can generate richer representations than those found in the input. ## the aggregation project the aggregation project @xcite @xcite @xcite , describes its primary goal as providing the benefits of implemented, formal grammars to documentary linguists, without their having to invest time in develop-ing those grammars by hand. such grammars are useful for testing linguistic hypotheses against data @xcite @xcite @xcite as well as building treebanks which are useful for discovering examples of phenomena in a language @xcite @xcite . the task of developing a grammar by hand is very time consuming and not likely to be taken up by field linguists already busy with the work of language documentation and description. however, the detailed analysis involved in annotating igt data (another time consuming task that documentary linguists are doing anyway) provides a very rich starting point for producing these grammars automatically. therefore, an end-to-end pipeline that begins with an igt corpus and results in a machine-readable grammar has the potential to serve the language documentation community without requiring additional work on their end, either in the form of data curation or grammar engineering. 5 the aggregation project has produced many key components towards this goal, as well as a rudimentary end-to-end pipeline (tested on chintang in @xcite . in this work, we build on those components to create a more robust and full-featured pipeline. in this section, we present the overall aggregation pipeline as it is developed in our work, with reference to previous work. in (2; repeated from 1) we present an example of interlinear glossed text (igt) from the chintang language research project (clrp; @xcite . based on the information encoded in this igt and others in the corpus, our goal is a grammar that parses this sentence to produce an hpsg syntactic representation, like the one in figure 2 , and an mrs semantic representation, as in figure 3 . the enriched corpus provides four key components that are necessary for grammar inference: morpheme segmentation, glossing, pos tags and syntactic dependencies, which can be seen in the final box in figure 4 . the morpheme segmentation and glossing are provided by the linguist in the source igt and are necessary to extract a lexicon, infer the morphotactic system and associate morpho-syntactic and morpho-semantic information with the corresponding morphemes. pos tags are o en provided in the source igt, but if they are not, they can be acquired from intent. intent creates alignments between the english translation and the sentence by leveraging the one-to-one alignment between words of the sentence and words in the gloss line and noisy alignment between the gloss words (frequently english lemmas) and the english translation line. it then parses the english sentence and projects the pos and syntactic dependency tags from the english parse onto the aligned words in the source language. while this approach only provides an approximation, as pos and dependencies do not necessarily map across languages, it serves as a useful starting point for inference. finally, the projected dependencies allow us to discriminate between arguments, modifiers and conjuncts and to identify different types of constituents in the sentence in order to infer syntactic properties. our grammar inference system uses these four components to produce a grammar specification file. as an example of our target output, figure 5 illustrates some of the values we infer that are relevant to sentential negation in chintang. chintang expresses sentential negation with a verbal suffix -n1ŋ. we indicate that negation is expressed with a single morpheme by se ing the negation exponence (neg-exp) to 1 in the grammar specification. in the morphology section of the grammar specification, we define one or more lexical rules for a morpheme with orthography n1ŋ and morpho-semantic feature negation: plus. this grammar specification can be input to the grammar matrix customization system @xcite , which uses stored syntactic analyses to produce customized grammars for languages based on the specification. the customized grammar generated by the grammar matrix for this specification will contain the appropriate lexical rule(s) to model negation @xcite , which are illustrated in figure 6 . figure 6: the relevant lexical rule for negation in the chintang grammar, produced from the specification in figure 5 the lexical rule in figure 6 licenses the topmost v node in figure 2 and introduces the neg predication in figure 3 . this rule is expressed in the delph-in joint reference formalism (called tdl; @xcite , which can be used to implement hpsg-style typed feature structures. a grammar encoded in this way can be loaded into delph-in processing tools like the lkb @xcite and @xcite for parsing and [incr tsdb()] @xcite and fftb @xcite for treebanking. previous work in the aggregation project has produced grammar specifications that contain a lexicon of nouns and verbs, morphological rules and descriptions of the language's word order and case system as well as case frames for individual words. the lexicon and morphotactic rules are inferred using mom @xcite , which we describe in sections 4.2 and 4.3. these rules abstract away from morphophonology, so the inferred grammars are tested by parsing the morpheme-segmented line of the igt. inference algorithms for basic word order and case system were developed by @xcite and this inference together with lexical inference was used to generate grammars by @xcite and @xcite . in this work, we present basil, an inference system that extends the number of phenomena that can be inferred by building on the existing morphotactic and syntactic inference systems. this system, also described in howell 2020, infers additional lexical items including determiners, case-marking adpositions, coordinators and auxiliaries as well as properties including argument optionality, sentential negation and coordination. we also integrate syntactic and morphological inference to handle person, number and gender information on nouns, agreement between verbs and their arguments, and tense, aspect and mood contributed morphologically or by auxiliaries. finally, whereas previous work has either evaluated the correctness of the grammar specifications on a variety of languages @xcite or grammar performance on a single language @xcite , we evaluate our system on grammar performance using 14 genealogically and geographically diverse languages. ## methodology: inferring grammar specifications this section focuses on our approach to inferring the grammar specifications illustrated in the previous section. we take as our starting point the system of zamaraeva et al. (2019a) which integrates the morphological inference module (called mom; @xcite @xcite and a module for inference of a few syntactic properties @xcite . to this integrated system we add extended inference for morphologically marked syntactic and semantic features, additional lexical classes and further syntactic properties to create basil, building analyses from syntactic inference in local languages. basil takes an enriched (using intent; georgi, 2016) corpus of the xigt @xcite data type as input and produces a grammar specification file which can be input into the grammar matrix to generate a custom grammar for the language. this grammar specification ( §4.1), o en referred to as a 'choices file' in the grammar matrix literature, contains specifications for a lexicon ( §4.2), a collection of morphological rules ( §4.3), definitions of syntactico-semantic features ( §4.4) and definitions of syntactic properties ( §4.5) for the language at hand. during development, we used a set of 9 core languages to design and tune basil's algorithms and consulted an additional 18 languages that were illustrative of particular phenomena we wished to test (see §5.1). in this section, we describe each of basil's inference modules, including the typological range covered, what specifications the grammar matrix customization system requires, and how we infer appropriate specifications for a language based on igt. foot_2 ## development languages we developed the inference algorithms described in section 4 using a data-driven approach in which we consulted the typological literature for each phenomenon and actively tested each algorithm on a diverse set of languages throughout implementation. in this section, we describe the languages and datasets we used during development ( §5.1), phenomena that appear in our datasets, both targeted by basil and otherwise ( §5.2) and basil's performance on the development datasets ( §5.3). ## evaluation methodology in section 5, we present results for our development languages, where system development benefited from close error analysis. we use the same methodology to evaluate the system on held-out data from held-out languages. as above, we use the full end-to-end pipeline described in section 3, with 10-fold cross-validation, and report the same five metrics from section 5.3: lexical coverage, parse coverage, correct predicateargument structure, correct predicate-argument structure and semantic features, and ambiguity. in this section, we describe how we measured these ( §6.1), and present our baseline system ( §6.2) and test languages ( §6.3). the following sections ( § §7-8) present our results and error analysis on the held-out languages. ## results using the methodology in section 6, we performed tenfold cross-validation on the evaluation languages for the basil inference system and the three baselines described in section 6.2. 24 we show lexical coverage in for each language, we treebanked n folds such that the number of parsed sentences in n folds is greater than 100. the results for lexical coverage, parse coverage and ambiguity are averages across ten folds, while the results for coverage with correct predicate-argument structure and coverage with correct predicate-argument structure and features are averages across n folds where n is given in there is a great deal of variation in how well any of the systems did at inferring grammars that can parse held-out sentences for each language, as illustrated by the graph in figure 14 . coverage for arapaho was very low, at roughly 3% lexical coverage for each system and similar parse coverage for basil and to understand the impact of syntactic inference on automatic grammar generation, we compare basil with three baselines that use the same morphotactic and lexical inference system as basil, but must specify the syntactic portions of the grammar specification through some other means. the broad-cov system uses the specifications that are expected to parse the most sentences, whether correctly or incorrectly. typ uses the typologically most common specification and rand uses a random choice (for details, see §6.2). each of these baselines uses a random choice for at least one specification, where no clear determination could be made for broad coverage or typological frequency, so ten-fold cross validation (given that a new random choice is made when specifying the grammar for each fold) is important to reduce the effect of chance on the overall performance of each baseline. because the same morphotactic and lexical inference system was used for the baselines as for basil, the lexical coverage across systems is roughly compa- for other languages, it is slightly higher because basil strategically excludes ditransitive and clausal complement-taking verbs (which it would not handle correctly) from the lexicon. 26 additional variation in the lexical coverage across systems can be a ributed to variations in the morphological graph: it is different for each baseline, because it is sensitive to verb valence assignments and these are done at random in each run for the typ and rand baselines. a larger and more meaningful difference between the systems is seen in parse coverage. here, the typ and rand baselines have much lower coverage than basil and broad-cov. while the typ baseline has a be er chance of using the correct value for each individual specification, it will not necessarily be correct for enough phenomena to produce a grammar that can parse simple sentences: for example, even if the order of verbs with respect to subjects and objects is correct, sentences with determiners won't parse if the determiner-noun order is incorrect. by design, the broad-cov system has the highest parse coverage, often outperforming basil; however, without syntactic in-ference this coverage could be spurious, so we must consider correct coverage (described in §6.1). again, the typ and rand baselines under-perform the other systems, as there is a relatively low chance that their specifications will correctly model any given language. in terms of correct predicate-argument structure, basil outperforms broad-cov for south efate and wakhi, while broad-cov does be er for arapaho and titan. they tie on hixkaryana. as broad-cov is designed to maximize coverage, it specifies asyndetic coordination for each language, enabling it to parse sentences for languages where basil failed to infer this strategy. for correct predicate-argument structure and semantic features, basil outperforms all baselines, as they cannot posit semantic features. only in rare cases did broadcov have the 'correct features', because the semantic representation shouldn't include any features at all. so far, we have shown that basil and broad-cov out-perform the other two baselines in parse coverage and correct predicate-argument structure, while basil out-performs all of the baselines in correct predicateargument structure and semantic features, as illustrated in figure 14 . the last thing to consider is how much ambiguity each of the grammars contain. typ and rand produced grammars with very li le ambiguity. these grammars only parsed simple sentences, so low ambiguity is not surprising. broad-cov was designed to maximize coverage, but this comes at the cost of increased ambiguity. for example, positing free word order for each language will ensure that all word orders will parse, but will also allow parses where the wrong constituents are identified as subjects and objects. as a result, the broad-cov baseline has significantly higher ambiguity than basil for all languages but south efate. while the results show a great deal of variation across the test languages, basil and broad-cov outperform the typ and rand baselines for most metrics. basil and broad-cov perform fairly comparably for a number of the metrics, but basil excels in two areas. first, basil generally has fewer parses per test item than broadcov, suggesting that there is less spurious ambiguity in the inferred grammars than in that baseline. while typ and rand have even lower ambiguity scores, they also have such low coverage that this is not an advantage. second, the semantic representations produced by basil are more correct in that they contain semantic features, resulting in higher scores for the correct predicate-argument structure and features metric.