To Boldly Query What No One Has Annotated Before? T he Frontiers of Corpus Querying (2020)

Abstract: Corpus query systems exist to address the multifarious information needs of any person interested in the content of annotated corpora. In this role they play an important part in making those resources usable for a wider audience. Over the past decades, several such query systems and languages have emerged, varying greatly in their expressiveness and technical details. This paper offers a broad overview of the history of corpora and corpus query tools. It focusses strongly on the query side and hints at exciting directions for future development.

## introduction annotated corpora have always been the backbone for many fields in nlp and other disciplines related to linguistics. whether serving as an invaluable source of empirical evidence for foundational research or doubling as gold-standard training input for fueling the furnaces of our machine learning factories, their importance cannot be overemphasized. but especially for the empirically motivated user base, corpora are only ever as good as the means available to explore them. and the primary means of exploring linguistically annotated corpora have always been (dedicated) corpus query tools and corpus query languages in their manifold shapes. in this paper we intend to give a thorough chronology of the major interplay between corpus progression and query tool evolution, with a strong focus on the latter. we start with an overview on relevant aspects of corpora and how they changed over the past ~30 years in section 2. section 3 elaborates on the observable phases in query tool development. in section 4 we discuss alternative corpus query approaches based on general purpose data(base) management solutions and provide pointers to related work in section 5. section 6 summarizes some of our observations and with section 7 we finally hint at our vision for future directions in corpus query system development. ## once upon a corpus -trends in corpus evolution though corpus linguistics dates back further, major online catalogs such as those from ldc foot_0 and elra foot_1 list corpora starting from the early 1990s. in the following decades corpus trends have varied along several dimensions, both technical and content-related. this section discusses such features and gives examples for their evolution. since this overview is an introduction to digital corpus query systems, we mainly focus on written and annotated corpora. with a focus on written corpora, character encoding is a decisive factor when estimating the publication date. starting from plain ascii (everts, 2000 3 , @xcite and language/script specific encodings, such as iso/iec 8859 @xcite , nowadays many corpora come with a (mostly) language independent utf-8 encoding @xcite ; @xcite and compare @xcite with @xcite ), which is also able to capture symbols relevant for transcription and annotation. similar to character encoding, the preferences regarding the representation format for corpus content changed over time. many corpora established in the 1990s come in an sgml format @xcite @xcite . in the next decade, xml-based corpora followed @xcite and compare @xcite and @xcite ) and since corpora were also made accessible over the web, relational database management systems (rdbmss) became a valuable backend for corpus storage @xcite . today we face a multitude of formats ranging from sophisticated and specialized xml encodings to simple tabular formats and often a corpus comes with more than one representation @xcite . especially since the first conll shared tasks foot_3 , their tabular format to encode sequence-based annotations and relations has been majorly developed @xcite . regarding included languages, multilingual and (partly) parallel corpora appear early @xcite @xcite , however, there was a rise of parallel corpora in the first decade of the current century. prominent examples are europarl @xcite , the cesta evaluation package @xcite and the prague czech-english dependency treebank 1.0 @xcite . on the other hand, with the rise of web corpora, language detection became more important to only crawl (or keep) web data for a specific language. corpus size is a less discriminative factor than one might think, since many early corpora came as collections of sub-corpora. @xcite already contains 90 million words and ldc's gigaword initiative started in 2003 @xcite , while many small corpora for specific topics or containing manual annotations are constantly being created. nevertheless, with recent web corpora, e.g. encow16 foot_4 and iweb foot_5 , several billion tokens pose new challenges for the design of both storage and search facilities. while for spoken corpora domain selection is often tailored to the research question at hand (cf. talkbank @xcite ), for written corpora (and especially annotated ones) there is a bias towards news and official documents, which was superseded by multi-domain web corpora starting in the late 2000s (e.g. the wacky initiative @xcite and cow) and, in the follow up, the increasing number of corpora of computermediated communication and social media foot_6 . like with the language setting, for web-corpora the challenge is no longer to include more languages or domains, but to identify and/or restrict them to a sensible subset. collections of historical language data have also been available for some time, e.g. the corpus of middle english prose and verse foot_7 and with the rise of the digital humanities many further corpora are created and/or enhanced with linguistic annotations, such as the drama corpora project foot_8 , where some corpora have been enhanced with lemma information. most corpora come with annotations, the earlier ones mainly with flat and word-based annotations, mostly including part-of-speech, such as the eci-elsnet italian & german tagged sub-corpus foot_9 . regarding the structural aspect, stand-off syntactic annotations became more feasible with emerging treebanks, while over time the focus changed from phrase-based @xcite to dependency tree structures @xcite . the current decade has also seen an increase in the richness of annotation layers of morphological, syntactical and semantical description, including highly concurrent annotations belonging to the same description layer, e.g. @xcite or @xcite . ## a brief history of querying we observed three major phases or generations in the history of corpus query systems, which are roughly aligned to the last three decades. the following is meant as a comprehensive but not exhaustive chronology of corpus query systems and approaches. space does not permit we provide indepth descriptions for every system mentioned but instead refer to section 5 for pointers to existing work that discusses and compares certain (families of) query systems in detail. ## technological alternatives many of the systems we presented in section 3 use various forms of database technology as their storage or evaluation backend. typically every such database or information management system already ships with its dedicated query language, such as sql for rdbmss, sparql for the rdf format, xpath and xquery for xml documents, cypher for neo4j and other graph-based databases or apache lucene with its own query dialect for accessing the text database. 38 the high level of abstraction it implements and the verbosity required to express simple queries combined with json syntax results in limited human readability. ## related work a lot of work has been invested already into laying the theoretical foundations for various aspects of and approaches to corpus querying, as well as into evaluating and comparing existing query systems. we distinguish between three types of contributions, namely (i) requirement analyses, (ii) evaluations of individual query languages or approaches and (iii) actual performance comparisons between multiple systems (feature-based or benchmarks). several contributions listing requirements for corpus query systems have been previously mentioned in section 4. in addition, @xcite provides a list of required language features for querying pdt and @xcite do so for treebanks in general, specifically related to navigation, closures over relations and going "beyond ordered trees" in order to query more complex structures. this list of functional requirements is later extended on in @xcite with features such as temporal organization and non-navigational requirements. while not exclusive to corpus query systems, technical aspects related to feasibility (e.g. scalability or computational complexity) or longterm maintainability (e.g. interoperability and extensibility) are also frequently emphasized by @xcite , @xcite and others. besides the usability-focused scholarly position of @xcite around aspects of answer time, maintenance cost and the management of search results, we previously discussed additional non-technical requirements related to the general readability or postprocessing capabilities of a query language and its learnability in @xcite , the latter being a crucial factor for achieving wide-spread use in humanistic fields. formal evaluations of query languages are somewhat rare, e.g. @xcite for lpath and lpath + , @xcite for monasearch or in part @xcite for fsq. instead the vast majority of evaluations use example queries of varying complexity to compare different query languages or systems. notable early work on query complexity was done by @xcite , comparing several query languages foot_20 based on a set of linguistic information needs of increasing complexity. the example queries they provide have proven to be a good baseline for comparing the capabilities of query languages and subsequently found their way into many later tool evaluations, such as @xcite for emdros or in @xcite when highlighting features of particular query languages. yet another evaluation approach was used by @xcite when they applied the classes defined in cqlf-1 as evaluation criteria in the comparison of cosmas ii, poliqarp and aql. @xcite provides a very thorough reflection and categorization of the various families of corpus query languages: text corpus, treebank, path-based 46 and logic-based. a point he makes that resonates well with other surveys is the importance of striking the right balance between usability and technical aspects in any practical situation. in some cases actual performance benchmarks have been published, such as testing emdros with different rdbms backends @xcite , comparisons between tigersearch and emdros in @xcite , monasearch and tigersearch in @xcite and @xcite benchmarking sets against icarus. however, due to the rapid change in technologies and the architectural differences between query systems, it tends to be very difficult to provide accurate and meaningful performance comparisons and readers are advised to carefully examine whether the reported use cases are applicable to their own. ## key observations & shortcomings in this section we intend to condense some of our observations after analyzing a large number of query systems. we focus on the following two aspects suitable for pointing out challenges (stemming from past shortcomings) and motivating directions in development of future corpus query systems, protocols or architectures. ## the final frontier -an outlook with several dozens of systems contributing their individual variations, the pool of available corpus query tools and languages has become quite large. navigating this ocean in order to find the right tool for the job and then learn to use it can already be as much effort as manually investigating the data at hand. fortunately the cqlf standardization initiative aims at providing developers with the means of locating their tools on a map of query features, so that prospective users may find them without an odyssey. while this effort is still in an early stage, we are looking forward to having catalogs available in the not too distant future, allowing us to browse for query languages based on our individual information needs. however, many questions regarding the future of corpus querying still remain, two of which we consider of particular importance and will discuss in the following sections.