ET : A Workstation for Querying, Editing and Evaluating Annotated Corpora (2021)

Abstract: In this paper we explore the functionalities of ET, a suite designed to support linguistic research and natural language processing tasks using corpora annotated in the CoNLL-U format. These goals are achieved by two integrated environments – Interrogatório, an environment for querying and editing annotated corpora, and Julgamento, an environment for assessing their quality. ET is open-source, built on different Python Web technologies and has Web demonstrations available on-line. ET has been intensively used in our research group for over two years, being the chosen framework for several linguistic and NLP-related studies conducted by its researchers.

## introduction annotated corpora are the basis for several natural language processing (nlp) tasks, serving as material from which machine learning systems learn how to perform linguistic analysis and as evaluating material against which system analyses can be confronted. however, manipulating annotated corpora is a costly activity for humans alone. in this context, we present et: a workstation for querying, editing and evaluating annotated corpora 12 . the underlying idea is to facilitate linguistic research using annotated corpora in the conll-u format foot_2 . et is composed of two integrated webbrowser-based interfaces that are easily manipulated by non-developers at the same time that it provides tools to explore annotated texts driven by simple-to-build yet linguistically complex queries. the system main language is portuguese but comes with english translation for the majority of its modules, it was built using different python web technologies and has web demonstrations available online. one can use the available demonstration pages for working with small corpora or install a local copy 4 . et has been intensively used in our research group for over two years, being the chosen framework for several linguistic and nlp-related studies conducted by its researchers. in section 2, we discuss other tools that inspired et and how different it is from them. in section 3, we explore the interface for querying and editing annotated corpora, and in section 4 we demonstrate how to assess their quality using the workstation. finally, in section 5, we outline some future perspectives for the tool. ## related tools et is a workstation that focuses on building quality corpora for natural language processing, but not from scratch. thus, it should not be confused with tools aimed at corpus annotation from raw pieces of text, such as arborator @xcite , conllueditor @xcite and ud annotatrix @xcite , nor with tools aimed at corpus linguistics studies, such as antconc @xcite and cqpweb @xcite . from the point of view of text analysis tools, although et query environment was largely inspired by ac/dc @xcite , from linguateca @xcite -one of the most important repositories for nlp in portuguese -, ac/dc is based on an early version of cwb @xcite , a robust and widely used processor capable of quickly and reliably processing corpora of millions of tokens, a task which et does not propose to do with such high quality. the query-ing module from et allows for querying syntactic dependencies in the universal dependencies format, sorting the results based on their annotation distribution, such as part-of-speech, morphological features and dependency labels distribution, and affords the addition of filters in order to further specify the query. in addition to the search tools, et comprises both a manual and a rule-based treebank editing system that was inspired by ac/dc's corte e costura @xcite , where the user codes linguistic rules that will search and correct annotation mistakes. from the point of view of corpora annotation, tools like arborator will suit better as they allow the editing of trees with features such as graphical editing and user management for collaborative annotation, which facilitate the process for annotators and project coordinators. what is different about et, in turn, is the integration of the querying environment with evaluation methods to assess corpora that were previously annotated by humans or nlp systems. with the application of linguistic rules and the verification of inconsistent patterns, the tool makes it possible to linguistically guide the work of reviewing annotated corpora for nlp. ## querying and editing annotated corpora interrogatório (portuguese word for "interrogatory") is the name of the first of the two environments that compose et. its purpose is to make it easy for anyone to query and revise annotated corpora. the system was built using python cgi technology in the back-end and jquery in the front-end. a server and a client machine are needed, but the same machine can work as both server and client, although installing and running a server-side will require intermediate technology skills. installation steps and requirements are available in the workstation github page. ## evaluating annotated corpora julgamento (portuguese word for "judgment") is the name of the second of two environments that compose et. its purpose is to evaluate the quality of annotated corpora by different methods that search for inconsistencies in the annotation. currently, julgamento provides three methods that help to search for annotation inconsistencies -n-grams, linguistic rules and contrastive analysis -presented below. the system was built using python flask framework technology in the back-end and jquery in the front-end. its installation process and architecture are the same as interrogatório: a server and a client machine are needed, one for setting the system up and the other for the user to browse through the interfaces. installation steps and requirements are available in the workstation github page. managing corpora from the top menu in any screen one can have access to the "manage corpora" hub in julgamento. interrogatório and julgamento are integrated foot_5 , which means that whatever corpora are uploaded to one environment will also be accessible through the other, since the corpus file is the same. some of julgamento's evaluation methods are based on the idea of contrastive analysis, which will require the user to upload a corpus with two different annotations (e.g. annotations provided by two different systems, two different human annotators or a gold-standard and a system counterpart), basing the evaluation on the confrontation of both. it is in this page that the user is able to upload the main corpus and its alternative annotation, that is, two different conll-u files with the same sentences but different annotation. finding inconsistent n-grams this is a method for detecting inconsistencies in the annotation of a corpus uploaded to julgamento. it is largely based on de marneffe et al. ( 2017 ) method, although some important changes were applied and are still under test. the general idea is that hardly two dependency pairs with the same context and same lemmas will have different dependency relations, as in figure 6 , in which "antónio" and "oliveira" are a dependency pair (antónio being the head) but have a different dependency relation in each sentence (nmod and flat:name), indicating an inconsistency that needs fixing. the method will display all the n-grams in the corpus that, although similar, have different relations, leaving it to the user to judge whether they are wrongly annotated and giving them the ability to correct their annotation. checking for validation errors two methods for checking validation errors in a corpus are available in julgamento. one is the official universal dependencies project script for validating a new corpus 7 , in which several rules will be applied to a conll-u file to ensure that the format is correctly encoded and that basic points from ud annotation guidelines have not been skipped while annotating that corpus. another script was built by our team and focuses on portuguese grammar rules that, when skipped, provide evidence of incorrect annotation or inconsistency. the rules were built using the same syntax from interrogating a corpus in interrogatório (as discussed earlier) and can be edited 8 to conform to any project annotation guidelines. comparing corpora other way of judging a corpus quality is by comparing two different annotations of the same sentences. these two annotations can be provided by two different systems, by 7 the script is named "validate.py", which can be called from the workstation interface. the returned sentences can be edited from inside it, as well. the source code is available at: https://github.com/ universaldependencies/tools . accessed on 5 jan. 2021. 8 rules can be edited from the file "validar_ud.txt". the "accuracy" for each part-of-speech tag and dependency relation are described through tables, as in figure 7 . clicking on any dependency head attachment percentage will lead to a page where the user will find sentences in which a token is attached to different heads when comparing both versions of the corpus. in this paper we presented et, a workstation for querying, editing and evaluating annotated corpora. its aims are to facilitate linguistic research and evaluate annotated corpora in the conll-u format, reuniting functionalities that are not new ideas along with innovative ways of characterizing and judging annotated corpora. both interrogatório and julgamento, integrated parts of the workstation, are available on-line in the project github page for download and usage, as well as a live demonstration which does not need previous installation. although et is not to be confused with corpus analysis tools and corpora annotation tools alone, the workstation can benefit from features of both kinds of tools. in the future, thus, it is possible to expand its functionalities so it will work as both kinds of tools as well, increasing the range of tools available to the user without leaving the workstation.