Unified Syntactic Annotation of E nglish in the CGEL Framework (2023)

Abstract: We investigate whether the Cambridge Grammar of the English Language (2002) and its extensive descriptions work well as a corpus annotation scheme. We develop annotation guidelines and in the process outline some interesting linguistic uncertainties that we had to resolve. To test the applicability of CGEL to real-world corpora, we conduct an interannotator study on sentences from the English Web Treebank, showing that consistent annotation of even complex syntactic phenomena like gapping using the CGEL formalism is feasible. Why introduce yet another formalism for English syntax? We argue that CGEL is attractive due to its exhaustive analysis of English syntactic phenomena, its labeling of both constituents and functions, and its accessibility. We look towards expanding CGELBank and augmenting it with automatic conversions from existing treebanks in the future.

## introduction ask a linguist about a detail of english grammar, and chances are they will reach for the cambridge grammar of the english language (cgel; @xcite . the product of the labors of two editors and 13 other chapter authors over more than a decade, cgel is the most recent comprehensive reference grammar of english, describing nearly every syntactic facet of present-day standard english in its 1700+ pages @xcite ). as but one example, a section foot_0 is devoted to the form and function of sentences like the first sentence of this paragraph, where the part before and is grammatically imperative but interpreted as a condition, and the part after and is interpreted as a consequence. cgel is a gold mine for such idiosyncrasies that a sharp-eyed english student (or linguist, or treebanker) might want to look up, in bottom-up fashion. it is also a systematic top-down survey of the building blocks of the language-in this respect, aided by a lucid companion textbook (sieg2; @xcite . in a review for computational linguistics, @xcite argued that cgel is a descriptive reference that echos precise formal thinking about grammatical structures; and as such, it holds considerable relevance for computational linguistics, supplementing formal grammars and treebanks like the venerable penn treebank (ptb; @xcite . brew exhorts: "it should become a routine part of the training of future grammar writers and treebank annotators that they absorb as much as is feasible of this grammar". it has certainly had an impact, for example, on the universal dependencies project (ud; @xcite @xcite , whose annotation guidelines cite cgel many times in discussing particular phenomena foot_1 (though the ud trees themselves, for reasons of lexicalism and panlingualism, diverge significantly from the representations given in cgel). we ask: what would it take to develop an annotation scheme based on cgel? if cgel's attention to terminological precision and rigor is as strong as brew suggests, it should not be nearly as difficult as mounting an effort of a completely new annotation framework. most substantive questions of grammatical analysis should be addressed by cgel, leaving only minor points to flesh out for treebanking. on the other hand, because cgel was not designed for annotation, and therefore not tested systematically on corpora, perhaps it has substantial holes, regularly missing constructions that occur in real data. to answer this question, we bootstrap an annotation manual and small set of sentences based on the descriptions from cgel. we examine what blanks in the cgel specifications need to be filled in to realize full-sentence trees in our data-both qualitatively (through working sentence-by-sentence) and quantitatively (by conducting an interannotator agreement study). what practical benefits hinge on the answer to this question? we are cognizant that substantial english treebanks already exist-constituency treebanks following the penn treebank standard, dependency treebanks, and others ( §2). thus, we do not anticipate a significant amount of from-scratch annotation in the cgel framework. yet we see practical benefits of the cgel style of description, perhaps induced automatically as a new "view" of gold ptb trees. first, exhaustiveness: cgel trees systematize both constituent categories and functions in a unified framework, whereas mainstream approaches for english prioritize either constituent structure (like ptb) or dependency structure (like ud). and second, accessibility: the trees would be consistent with human-readable descriptions and linguistic argumentation in the cgel and sieg2 texts, allowing users of a treebank (or parser) to look up the constructions in question. 3 through developing guidelines and annotating data, we find that cgel offers a powerful foundation for treebanking, though there are points where further specification is needed. our small but growing treebank-which we call cgelbank-and accompanying code for validation and measuring interannotator agreement are available at https: //github.com/nert-nlp/cgel/ . we also publish our annotation manual, which stands at about 75 pages (mostly of example trees): @xcite . ## related work even considering just english, there have been many formalisms deployed for syntactic annotation. a sample is given in a corpus that likewise integrates constituent categories and functions in a single tree is the tiger treebank for german @xcite . ## the cgel framework an example parse in the cgel framework appears in figure 1 . 4 its building blocks are constituents, each of which receives a category indicating the type of unit it is and a function (notated with a colon) indicating its grammatical relation within the higher constituent. the constituent structure is a hierarchical bracketing of the sentence, which is projective with respect to the order of words in the sentence. terminals consist of lexical items (omitting punctuation) as well as gaps used to handle constructions with noncanonical word order. categories. cgel posits a distributionallydefined set of lexical categories, on which basis we developed a part-of-speech tagset with 11 tags: n (noun), n pro (pronoun), v (verb), v aux (auxiliary verb), p (preposition), d (determinative), foot_4 adj (adjective), adv (adverb), sdr (subordinator), coordinator, and int (interjection). (see framework representative citations constituency ptb penn treebank (marcus et al., 1994; bies et al., 2012; pradhan et al., 2013) tag tree-adjoining grammar (chen and vijay-shanker, 2000) mg minimalist grammars (torr, 2018) rrg role and reference grammar (bladier et al., 2018) dependency sd stanford dependencies (de marneffe et al., 2006) ud universal dependencies (nivre et al., 2016) sud surface universal dependencies (gerdes et al., 2018) fgd functional generative description @xcite constraint-based lfg lexical-functional grammar @xcite ) hpsg head-driven phrase structure grammar @xcite @xcite categorial ccg combinatory categorial grammar @xcite a determinative phrase (dp), not to be confused with the notion of a determiner phrase in generative grammar @xcite . gaps. cgel employs gap constituents and coindexation, as in figure 1 , to handle unbounded dependency constructions (udcs) and other constructions that deviate from the canonical declarative order, showing where there is a clear structural gap. nevertheless "the account is quite informal" (r. huddleston, personal communication). to make it more formal, we have restricted the use of gaps to udcs, subject-auxiliary inversion (sai), and pre-and post-posing of complements. we also use gaps for adjunct fronting when it triggers sai (e.g., only once had i -seen it -). subject-dependent inversion (sdi) is a double-gapped construction with a subject gap and a complement gap in the vp (e.g., here -is -jim). all subject-relatives have a gap, as do delayed right constituent coordination and end-attachment coordination. coindexation is used with and only with a gap. every gap must be coindexed with exactly one overt constituent (and possibly other gaps). there are no gaps for ellipsis. fusion. certain constructions are analyzed with fusion of functions, in which a constituent participates in two different higher constituents @xcite . this is shown in figure 1 for the np which, short for something like "which items": the dp is taken to fulfill both the determiner function in the np and the head in its nominal. other constructions where cgel employs fusion of functions include compound determinatives (e.g. someone), other determinatives or adjectives as np heads (the rich, the tallest, those three foot_5 ), and fused (a.k.a. free or head- ## towards cgelbank despite its detail and richness, in 1700+ pages, cgel includes just 40 trees, and on some points is inexplicit. annotating naturally occurring sentences ( §5) brought many of these ambiguities to the fore. here we identify questions we faced and the decisions we made. ## annotation process what began as a pet project to make cgel-style trees for interesting sentences found in the wild eventually became a corpus-building effort, with two linguists interested in the cgel framework (the first and third authors) serving as annotators. to date, this has resulted in over 200 trees of naturally occurring sentences-some handpicked, others sampled at random from a corpus. statistics for cgelbank are presented in tables 2 and 3 . cgelbank trees were drawn from multiple sources, and were annotated in four phases. 1. twitter: exploratory annotation of real-world sentences taken from twitter by annotator 2 resulted in this set of 65 trees. at this point, there were no agreed-upon guidelines for cgel annotation and the project was largely informal. 2. ewt: a set of 100 sentences sampled from the english web treebank @xcite was annotated by annotator 2 and simultaneously the guidelines were composed in discussion with annotator 1. to maintain consistency with the guidelines, both the ewt and twitter treebanks were validated and iteratively corrected. 3. ewt-trial, twitter-trial: once the guidelines and validation script were mostly complete, and as the browser-based annotation workflow was under development, the two annotators used it to make 37 more trees (27 from additional ewt sentences, 10 from twitter and other sources). these trees were singly annotated and validated but not adjudicated. 4. iaa and pilot: for an interannotator study, both annotators independently annotated and then adjudicated a pilot set of 5 trees and then a larger set of 50 trees. these were also drawn from ewt. figure 4 : illustration of the .cgel data format for the clause from figure 1 . note that the bracketed notation forms a proper tree: the reentrancy of the fused determiner-head is automatically added post hoc. the verb lemma is included as it differs from its inflected form. features on nodes are extensible: for example, cgelbank uses :p for punctuation, :note to offer commentary on a construction (with cgel page references), and :correct to indicate corrections to typos. finergrained morphosyntactic information (inflectional features, clause types, etc.) may be added in the future. the initial 165 twitter and ewt sentences were annotated in l a t e x using the forest package and converted into the .cgel format using an ad hoc python script. later annotation was done with a customized version foot_9 of active dop, a browserbased graphical treebanking tool @xcite . active dop incorporates disco-dop @xcite , an active learning parser, which considerably sped up annotation. we trained disco-dop on the 202 trees created prior to the start of the iaa pilot. as input to the active dop tool, ewt sentences were preannotated with pos tags and gaps heuristically derived from gold ud and ptb trees; the tagging was then manually edited in a text editor. 10 for the 50 iaa sentences, after trees were exported to the .cgel format, adjudication was performed cooperatively between the two annotators using a text editor with a file comparison mode. each split is stored in a separate file in the .cgel data format illustrated in figure 4 . this combines the sentence metadata style from the conll-u format foot_11 with trees in a bracketed format adapted from penman notation @xcite . cgel-bank includes a python api for working with this format, including a script to export it to l a t e x for visualization (with any reentrancies due to fusion of functions). during the initial phases of development, it became clear that certain structural properties (like the number of nom layers in a complex np) were sources of annotator inconsistency. we therefore developed a validator, a script to check structural properties for obvious errors (e.g., misspelled labels; phrases with no head) as well as less obvious errors (a category occurring in an unusual position in the tree; an unnecessary level of nesting of a phrase; a modifier forming a ternary-branching structure; invalid coindexation of a gap; improper structure of coordinates in coordination). some of the validation rules are conservative and need to be broadened as new data is encountered; but flagging them is an opportunity for the annotator to check for an error or inconsistency. in our experience, the rules (implemented in 500 lines of python) often find small problems that might otherwise have gone undetected. we quantify the impact of the validator in the next section. ## interannotator study to test the consistency of our cgel annotation guidelines, we conducted an interannotator study. as a pilot, five sentences sampled from the english web treebank @xcite were annotated independently by the first and third authors. after adjudicating annotation disagreements and adapt-ing to the annotation tool, we sampled 50 new sentences from ewt for the interannotator study. the annotators independently annotated the new set and then jointly adjudicated disagreements. ## conclusion using the analysis developed in cgel @xcite , we introduced a new expressive and linguistically-informed syntactic formalism to corpus annotation of english, which unifies constituent and dependency information in an accessible format. creating annotation guidelines confirmed that cgel was a strong foundation for syntactic analysis, but also revealed some minor points of underspecification for which new policies were necessary. using our guidelines, we have created trees from naturally occurring sentences in multiple genres, and we conducted an interannotator study. we find high annotator agreement overall and even on the complex phenomenon of gapping. overall, we are confident that the formalism of cgel is suitable for consistent annotation of real-world text. in the future, we intend to take advantage of existing resources in other frameworks to obtain cgel-style trees and parsers on a larger scale and in a wider range of genres. kaizhong zhang and dennis shasha. 1989. simple fast algorithms for the editing distance between trees and related problems. siam journal on computing, 18(6):1245-1262.