Antarlekhaka: A Comprehensive Tool for Multi-task Natural Language Annotation (2023)

Abstract: One of the primary obstacles in the advancement of Natural Language Processing (NLP) technologies for low-resource languages is the lack of annotated datasets for training and testing machine learning models. In this paper, we present Antarlekhaka , a tool for manual annotation of a comprehensive set of tasks relevant to NLP. The tool is Unicode-compatible, language-agnostic, Web-deployable and supports distributed annotation by multiple simultaneous annotators. The system sports user-friendly interfaces for 8 categories of annotation tasks. These, in turn, enable the annotation of a considerably larger set of NLP tasks. The task categories include two linguistic tasks not handled by any other tool, namely, sentence boundary detection and deciding canonical word order, which are important tasks for text that is in the form of poetry. We propose the idea of sequential annotation based on small text units, where an annotator performs several tasks related to a single text unit before proceeding to the next unit. The research applications of the proposed mode of multi-task annotation are also discussed. Antarlekhaka outperforms other annotation tools in objective evaluation. It has been also used for two real-life annotation tasks on two different languages, namely, Sanskrit and Bengali. The tool is available at https://github.com/Antarlekhaka/code

## introduction and motivation manual annotation plays an important role in natural language processing (nlp). it is particularly important in the context of low-resource languages for the creation of datasets. there are a number of syntactic and semantic tasks in nlp which can utilize annotation by domain experts. lemmatization, morphological analysis, parts-of-speech tagging, named entity recognition, dependency parsing, constituency parsing, co-reference resolution, sentiment detection, discourse analysis and so on are some examples of such common nlp tasks. there's a need of considering historical context and respecting the perspectives of indigenous language speaking communities when conducing nlp research involving these languages @xcite . sanskrit, a classical language, has a large amount of text available digitally; however, it still suffers from poor performance in standard nlp tasks. hence, manual annotation of text in sanskrit is of prime necessity. further, most of the classical sanskrit literature is in poetry form following mostly free word order @xcite , without any punctuation marks. therefore, certain specialized tasks, such as sentence boundary detection and canonical word ordering, are needed for sanskrit text processing consider an example foot_0 from a sanskrit text, valmiki ramayana, @xcite shown in fig. 1 . the sentence boundaries are denoted using square brackets @xcite ). the half-verse boundaries are marked by single forward slashes (/) and the verse boundaries by two forward slashes (//). it can be observed that the sentence boundaries do not coincide with the verse boundaries. in particular, there may be multiple sentences present in a single verse, or a sentence may extend across multiple verses. further, the right side of the arrow shows that the canonical word order is different from the order in which words appear in the original text. for such languages that either do not use punctuations or use them in a limited amount, sentence boundary detection is an important task. additionally, in languages with relatively free word order, decision of a canonical word order is also relevant. these two tasks also play a vital role when dealing with the corpora in the form of poetry, making them potentially relevant for all languages. in this paper, we describe antarlekhaka foot_2 , a tool for distributed annotation that provides userfriendly interfaces to facilitate the annotation process of various common nlp tasks in a straightforward and efficient way. we propose a sequential annotation model, where an annotator carries out multiple annotation tasks relevant for a small text unit, such as a verse, before proceeding to the next. the tool has full unicode support and is designed to be language-agnostic, meaning it can be used with corpora from any language, making it highly versatile. the tool sports eight taskspecific user-friendly annotation interfaces corresponding to eight general categories of nlp tasks: sentence boundary detection, canonical word ordering, free-form text annotation of tokens, token classification, token graph construction, token connection, sentence classification and sentence graph construction. the goal of the tool is to streamline the annotation process, making it easier and more efficient for annotators to complete multiple nlp tasks on the same corpus. annotators can participate in the annotation without any programming knowledge. additionally, the tool's easy setup and intuitive administrator interface make it accessible to administrators with minimal technical expertise. ## background an annotation tool is crucial for the successful completion of any annotation task, and its success relies heavily on its usability for the annotators. apart from this, the tool should be easily installable and should support web deployment for distributed annotation, allowing multiple annotators to work on the same task from different locations. the administration interface of the tool should also be intuitive and should provide convenient access to common administrative tasks such as corpus upload, ontology creation, and user access management. additionally, often there is a need for multiple annotation tasks to be completed on the same corpus. a well-designed annotation tool should encompass these features to ensure a smooth, efficient, and accurate annotation process. numerous text annotation tools are available that target specific annotation tasks, such as webanno @xcite , inception @xcite , gate teamware @xcite , flat (van gompel, 2014), brat @xcite , doccano @xcite and more. however, each of these tools falls short in fulfilling all the requirements of an ideal annotation tool. for instance, webanno is rich in features but becomes complex to use and experiences performance issues as the number of lines displayed on the screen increases. both webanno and brat lack full support for firefox @xcite , an issue that was rectified in inception. gate teamware suffers from shortcomings such as inadequate support for relation and co-reference annotation @xcite , installation issues @xcite and complexity for average users @xcite . flat uses a non-standard folia xml data format and the system is not intuitive @xcite . brat has not been actively foot_3 developed and exhibits issues such as slowness, limited scope for configuration and limitations regarding file formats @xcite . the tool doccano, although simple to set up and intuitive, only supports labeling tasks. sangrahaka @xcite , while being easy to set up and use, focuses only on the annotation towards creation of knowledge graphs and lacks support towards general-purpose nlp annotation tasks. use spans for marking most annotations, which a user by selecting and dragging mouse cursor over the corpus text. this method, while versatile, has a trade-off that the annotation process is slower and more tedious. importantly, none of these tools address crucial tasks like canonical word ordering. ## architecture antarlekhaka is a language-agnostic, multi-task, distributed annotation tool presented as a webdeployable software. the tool makes use of several technologies, including python 3.8 , flask 2.2.2 , and sqlite 3.38.3 for the backend, and html5, javascript, and bootstrap 4.6 for the frontend. flask web framework powers the backend of antarlekhaka providing a robust and scalable in-5 https://universaldependencies.org/format.html frastructure. a web framework is responsible for a range of backend tasks, including routing, templating, managing user sessions, connecting to databases and others. the recommended way to run the tool in a production environment is using a web server gateway interface (wsgi) http server, such as gunicorn , which can operate behind a reverse proxy server such as nginx or apache http server . however, any wsgi server, including the built-in server of flask, can be utilized to run the application. sqlite is used as the database management system to store and manage the data and metadata associated with the annotation tasks. an object relational mapper (orm) sqlalchemy foot_4 is used to interact with the relational database. this allows the user to choose any supported dialect of traditional sql, such as sqlite, mysql , postgresql , oracle , ms-sql , firebird , sybase and others foot_5 . the frontend of the tool, built using html5, javascript, and bootstrap, provides user-friendly interfaces for annotators and administrators. the tool provides a feature-rich administrative interface to manage user access, corpus, tasks and ontology. the tool also includes eight types of intuitive annotation interfaces, explained in detail in sec. 3.3. the tool simplifies setup with a single configuration file that controls various customizable aspects. overall, by combining the state-of-the-art technologies, antarlekhaka offers a powerful and flexible solution for large-scale annotation projects. ## evaluation the tool is being used for annotation of a large corpus in sanskrit, namely, valmiki ramayana. the details of this project are described in sec. 4.3. additionally, the tool is also being used for the annotation of plain text corpus in bengali language. following @xcite , we have evaluated our tool using a two-fold evaluation method of subjective and objective evaluation. we have not used the time taken for annotation as an evaluation metric since annotators often spend more time processing the text to identify the relevant information than physically annotating. hence, it may not be a reliable measure. ## potential for nlp research a natural language annotation tool simplifies the process of creating datasets for machine learning models, which is useful for nlp tasks such as lemmatization, ner, pos tagging, co-reference resolution, text classification, sentence classification, and relation extraction. annotator-friendly and intuitive interfaces can simplify the otherwise tedious process of manual annotation process to a great extent. the effectiveness of higher-level tasks such as question-answering, grammatical error correction and machine translation often relies on the success of several low-level tasks, which can be handled by a multiple task annotation tool. the tool's ability to handle large amounts of data and multiple users simultaneously can contribute to faster completion of these tasks. ## conclusions we have developed a web-based multi-task annotation tool called antarlekhaka for sequential annotation of various nlp tasks. it is available at https://github.com/antarlekhaka/code . the tool is language-agnostic and has full unicode support. the tool sports eight categories of annotation tasks and an annotator-friendly interface for each category. multiple annotation tasks from each category are supported. the tool enables creation of datasets for computational linguistics tasks without expecting any programming knowledge from the annotators or administrators. it is actively being used for a large-scale annotation project, involving a large sanskrit corpus and a significant number of annotators, as well as another annotation task in bengali language. antarlekhaka has a potential to propel research opportunities in nlp by simplifying the conduction of large-scale annotation projects.