Adding linguistic information to parsed corpora (2019)

Abstract: No matter how comprehensively corpus builders design their annotation schemes, users frequently find that information is missing that they need for their research. In this methodological paper I describe and illustrate five methods of adding linguistic information to corpora that have been morphosyntactically annotated (=parsed) in the style of Penn treebanks. Some of these methods involve manual operations; some are executed by CorpusSearch functions; some require a combination of manual and automated procedures. Which method is used depends almost entirely on the type of information to be added and the goals of the user. Of course, the main goal, regardless of method, is to record within the corpus additional information that can be used for analysis and also retained through further searches and data processing.

## method 3: embedding information in coding strings coding strings are strings of characters, each character representing a linguistic or extralinguistic variable, which are inserted as nodes in the tokens of a corpus file. method 3, the construction of coding strings, is the traditional and perhaps most widely used method of adding information to corpus data. coding strings had their origin in quantitative sociolinguistic research and were used decades before the creation of parsed corpora. the coding function of cs is used to construct coding strings based on the morphosyntactic annotation and the lexical content of the token; once created, coding strings may be manually extended to encode information that is not represented in the corpus. since coding strings are part of the token structure, they may be searched and manipulated by cs. coding strings may also be used as input to software for statistical analysis, like r; this is perhaps their most important function. case study 3: taylor and pintzuk 2015 (t&p 2015) examine the position of objects in old english and look at the effect of verb order and the length and information structure of the object to support their conclusion that there are two sources for post-verbal objects in old english, object postposition and base-generation. as shown in (4) below, objects can appear both before and after the verb cluster in clauses with non-finite main verbs before finite auxiliaries (o v aux and v aux o), and before and after non-finite verbs in clauses with auxiliary -main verb order (aux o v and aux v o). in these examples, finite auxiliaries are underlined, non-finite main verbs are italicised, and objects are in bold face. if all post-verbal objects were derived by postposition in both v aux and aux v clauses, i.e. if structure (5)e didn't exist, we would expect the factors influencing post-verbal position to be the same in both clause types. to test this null hypothesis, t&p 2015 looked at the influence of weight (as measured by the length of the object in words) and informational status (given vs. new) on the position of objects in auxv and vaux clauses. this was a four-step process. as a first step, cs was used to code each token for three factors: the order of finite auxiliary and non-finite main verb (auxv vs. vaux); the position of the object with respect to the non-finite main verb (ov vs. vo); the length of the object in words (1 . . . 11). the coding query file is given below in (6); @xcite an example of a token coded for the first three factors is given in ( 7 ). the second step was to manually code the informational status of the object (given vs. new). examples of tokens coded for all four factors are given in ( 8) through (11 ); (8) is the token in (7) with informational status added as the fourth factor. (8) /~* +t+at we god don sceolon that we good do should (coaelive,+als_[auguries]:245.3644) *~/ (0 (1 ip-sub-spe (2 coding vaux:ov:1:new) (4 np-nom (5 pro^n we)) (7 np-acc (8 n^a god)) (10 vb don) (12 mdpi sceolon)) (14 id coaelive,+als_[auguries]:245.3644)) (9) /~* gif +du +t+as ecan lifes ges+al+te habban wylt if you the eternal life's happiness have will (coaelive,+als_[alban]:65.4038) *~/ (0 (1 ip-sub-spe (2 coding vaux:ov:4:new) (4 np-nom (5 pro^n +du)) (7 np (8 np-gen (9 d^g +t+as) (11 adj^g ecan) (13 n^g lifes)) (15 n ges+al+te)) (17 hv habban) (19 mdpi wylt)) (21 id coaelive,+als_[alban]:65.4038)) (10) /~* +t+at se l+ace sceolde asceotan +t+at geswell that the leech should lance the tumour (coaelive,+als_[+athelthryth]:61.4177) *~/ (0 (1 ip-sub (2 coding auxv:vo:2:given) (4 np-nom (5 d^n se) (7 n^n l+ace)) (9 mdd sceolde) (11 vb asceotan) (13 np-acc (14 d^a +t+at) (16 n^a geswell))) (18 id coaelive,+als_[+athelthryth]:61.4177)) (11) /~* for +tan +te he sylf sceal sw+arran witu +trowian because he himself shall heavier torments suffer (coaelive,+als_[vincent]:89.7849) *~/ (0 (1 ip-sub-spe (2 coding auxv:ov:2:new) (4 np-nom (5 pro^n he) (7 adjp-nom (8 adj^n sylf))) (10 mdpi sceal) (12 np-acc (13 adjr^a sw+arran) (15 n^a witu)) (17 vb +trowian)) (19 id coaelive,+als_[vincent]:89.7849)) the third step was to use the print_only function of cs to create an output file containing only the coding strings of the data file. the file is shown in (12) below. cs separates the factors by ':', and the user must manually insert a header naming the factors for input to statistical processing, the last step. the results for this study are shown in and new objects are much more likely to postpose than light objects and given objects. since auxvo clauses are derived by two different processes, postposition and base-generation, the effects of weight and informational status are weakened; this is why the effect of weight is weaker in auxv clauses and the effect of informational status is reduced to non-significance. method 4, the modification of corpus annotation, may be done manually, but it is much more efficient (and safer) to use the corpus-revision tool of cs. this tool enables the addition, deletion, and modification of annotation in the corpus, including not only node labels but also structure. any search that can be made using cs can act as the basis for corpus revision; the output of corpus revision is a new version of the corpus -i.e., the original version of the corpus is not deleted, in case of catastrophic errors. corpus revision can be used to build an annotated corpus starting from a straight text corpus with only part-of-speech. i frequently find it useful to mark particular structures so that they are easy to identify, and also to evade some cs restrictions, as will be seen in case study 4 below. case study 4: haeberli and pintzuk 2017 (h&p) look at verb placement in 'true v2' contexts in old english. h&p analyse in detail one particular clause type: clauses with an initial gif/þa/þonne 'if/when/ when' subordinate clause, followed by a resumptive adverb (e.g. þa/ þonne 'then') and the rest of the main clause; an example is given in (13). note that initial þa/þonne in old english main clauses is considered a 'true v2' context: 97.4% (6546/6719) of these clauses exhibit strict v2 order, with the verb in second position followed by the subject. in order to simplify the searches for and coding of these clauses, i wanted to flag the relevant ip-mats and cp-advs by modifying the label. in addition, i wanted to 'remove' @xcite the subtrees of all subordinate clauses other than the ip-sub dominated by the relevant cp-adv. three steps were necessary, as shown below; a red font is used for highlighting. step 1: flag the relevant ip-mat, cp-adv, and ip-sub using the query file below. the ip-mat and cp-adv are flagged by appending '-z' to the label; the ip-sub is flagged by prepending 'x-' to the label. notice that the first token (coeust . . . 26) contains an ip-sub that is not dominated by the clause-initial cp-adv; the second token (cobede . . . 3530) contains an ip-mat that does not dominate a cp-adv as the first constituent; and the third token (cocanedgx . . . 82) contains both a cp-adv that is not the first constituent of the ip-mat and an ip-sub that is not dominated by the relevant cp-adv. these are all nodes that are irrelevant to the investigation. ## conclusions in this article i have described and illustrated five methods for adding information to corpora that have been parsed in the penn treebank style. these methods may involve manual operations, or they may be executed by cs functions, or they may require a combination of manual and automated procedures. some of the methods overlap: for example, method 2 (inserting code nodes) is functionally equivalent to method 3 (embedding information in coding strings). which method is used depends almost entirely on the type of information to be added and the goals of the user. within a unix platform, batch files are run by typing 'source [filename]'. blank lines and lines that start with the hash character (#) are ignored; in this way comments can be added to describe the procedures and searches. in the file below, the executable lines are shown in red. (ii.2) a. generic interpretation (code: <nptype:bsg-@xmath0. . . ' (cmhali,147.282) b. generic interpretation (code: <nptype:an-@xmath1,292.c2.187) (ii.3) ambiguous between existential and generic (code: <nptype:an-@xmath2. . . ' (cmctmeli,224.c1.277) (ii.4) existential specific nominal (code: <nptype:an-exs-@xmath3, mighty and rich, begat upon his wife, who was called prudence, a daughter who was called sophie.' (cmctmeli,217.c1b.5)