An Open-source Web-based Application for Development of Resources and Technologies in Underresourced Languages (2023)

Abstract: The paper discusses the Linguistic Field Data Management and Analysis System (LiFE), a new open-source, web-based software that systematises storage, management, annotation, analysis and sharing of linguistic data gathered from the field as well as that crawled from various sources on the web such as YouTube, Twitter, Facebook, Instagram, Blog, Newspaper, Wikipedia, etc. The app supports two broad workflows - (a) the field linguists’ workflow in which data is collected directly from the speakers in the field and analysed further to produce grammatical descriptions, lexicons, educational materials and possibly language technologies; (b) the computational linguists’ workflow in which data collected from the web using automated crawlers or digitised using manual or semi-automatic means, annotated for various tasks and then used for developing different kinds of language technologies. In addition to supporting these workflows, the app provides some additional features as well - (a) it allows multiple users to collaboratively work on the same project via its granular access control and sharing option; (b) it allows the data to be exported to various formats including CSV, TSV, JSON, XLSX, , PDF, Textgrid, RDF (different serialisation formats) etc as appropriate; (c) it allows data import from various formats viz. LIFT XML, XLSX, JSON, CSV, TSV, Textgrid, etc; (d) it allows users to start working in the app at any stage of their work by giving the option to either create a new project from scratch or derive a new project from an existing project in the app.The app is currently available for use and testing on our server (http://life.unreal-tece.co.in/) and its source code has been released under AGPL license on our GitHub repository (https://github.com/unrealtecellp/life). It is licensed under separate, specific conditions for commercial usage.

## introduction field linguists constantly need tools for collecting, storing, annotating, analysing, sharing and managing linguistic data. as field linguists gather a lot of data for lots of languages, including comparatively under-represented, lesser-known, minoritised, and endangered languages around the globe, this data has to be properly preserved, quickly and accurately analysed and processed and made available to the larger community for social good. on the other hand, there are very few publicly available and accessible datasets for developing language tools and technology for a vast array of languages around the world including most of those which have been worked upon by field linguists -the field linguists', if made available in a structured format, could help in alleviating this situation to a certain extent. to address this dual issue of accelerating the process of collecting and processing datasets of underresourced languages (possibly with assistance from available state-of-the-art language technologies) and developing language technologies for these under-resourced languages, a unified platform with an easily available/accessible and handy interface targeted for linguists is required. our application, linguistic field data management and analysis system "life" aims to offer a practical intervention in the field. the app creates a structured framework for the management, analysis and sharing of primary linguistic field data. it also provides interfaces for producing the derivatives of this data such as digital and print lexicons, sketch grammars, and basic language processing tools like part-ofspeech taggers and morphological analyzers and generators, automatic speech recognition systems, machine translation and others. the software focuses on automating the different tasks as much as possible through a handy, intuitive interface for carrying out all the tasks. for instance, by giving initial input, the system gradually trains automatic techniques for inter-linear glossing of the dataset and subsequent production of sketch grammar as well as nlp tools for the language. the app integrates popular machine learning model hubs such as huggingface hub foot_0 and universal language contribution api (ulca) foot_1 as well as other popular models for individual tasks to provide automation for various tasks. this has enabled us to, for example, give access to the most recent unsupervised and transfer learning-based asr models based on transformers (such as wav2vec 2.0 @xcite , wav2vec-u @xcite and whisper @xcite ). it has given field linguists direct access to the most recent state-of-the-art models available for language processing tasks. on the other hand, the app has also provided access to a no-code environment for training or fine-tuning models for new languages and new tasks using some of the most popular libraries such as huggingface transformers and scikit-learn -this environment provides simple point-and-click options to train baseline models that could be quickly integrated into the field linguists' workflow. these two together have helped us provide an automated pipeline for transcription, inter-linear glossing and free translation of the dataset collected from the field. ## related work the growth of field linguistics and nlp has mostly taken place independently of one another. thus the tools for speech and multimodal data collection, management and analysis used by linguists and the tools used for data collection and annotation by nlp practitioners are not developed to be interoperable and are generally used exclusively by the two communities. for the storage, management and gathering of multimodal data as well as the creation of a lexicon, there are certain programmes and technologies designed specifically for field linguists (or community members engaged in fieldwork for their own language). one of the earliest pieces of software created by sil (the summer institute of linguistics), toolbox @xcite , formerly known as shoebox foot_2 , served as a forerunner to flex and was primarily designed for use by anthropologists and field linguists to input their text data and create dictionaries. fieldworks language explorer (flex) 6 @xcite and @xcite , which is used for the gathering, management, analysis, and sharing of linguistic and cultural data, is one of the most well-liked tools in the field. a software called lexiquepro foot_4 @xcite makes it simple to create and format lexicon databases for sharing with others. wesay foot_5 was developed by sil to assist native speakers and non-linguists in creating dictionaries for their own languages @xcite & @xcite . there have been various attempts to create tools that are primarily used for data collection. @xcite talks about the creation of an app called woefzela foot_6 . it is a smartphone-based (android operating system) data-gathering tool that @xcite . it supports several sessions for data collection and can operate without an internet connection. for the purpose of data collection quality control, it works well. in the south african data collection experiment, where nearly 800 hours of voice data were gathered from remote and rural locations, this technique is displayed. similarly, saymore foot_7 is designed to collect data for building dictionaries. while these different tools provide adequate support for different tasks, there are some serious limitations -• all of these are standalone desktop/mobile applications and most of these tools are not compatible with linux systems, thereby, forcing users to use them on windows /mac. on the other side of the spectrum, nlp practitioners, make use of a different set of tools and applications for data management and annotation. for example, label studio foot_11 is a specific opensource, web-based application for data labelling. with a clear and simple user interface, it enables users to label a variety of data kinds, including speech, text, image, video, and time series, and export to several model formats. to create more accurate machine learning models, it can be used to prepare raw data or enhance current training data @xcite . similarly, an open-source platform named shoonya foot_12 is being created with the goal of enhancing the digital presence of india's underrepresented languages. it allows users to annotate and classify data at scale. this is a crucial necessity to produce larger datasets for neural machine translation training on a wide range of indian languages. some other open-source tools for token, span and document-level text annotation include brat rapid annotation tool (brat) 16 @xcite , doccano foot_14 is an open-source text annotation tool. for text categorization, sequence labelling, and sequence-to-sequence applications, it offers annotation features. users can produce labelled data for sentiment analysis, named entity recognition, text summarising, and other purposes @xcite , inception foot_15 @xcite , among several others. as is evident, none of these tools even attempt to support data collection from the field and its management and analysis. given this, life attempts to provide the following -• an integrated interface that caters to the needs of both the field linguists and nlp practitioners for their data management and processing workflows. we discuss the architecture and features of the app in more detail in the following sections. ## life interface life has a full-fledged interlinked pipeline of four customisable modules for questionnaire creation, multimodal data labelling (speech, text, image etc) and validation, production of output based on data annotation and analysis (viz lexicon generation) and model training (figure 1 ). all these four modules of life are discussed in detail in the following subsections and their interrelationship is demonstrated in figure 2 . ## supported workflows in life life does not enforce any specific workflows or pipelines. the app is currently being used by over 200 users and at least three organisations for various purposes and they all have different workflows. the app allows users to start working at any point in their data collection and analysis project. while users could define their own workflow and use different modules accordingly, the has been consciously designed to support two broad kinds of workflow - these two are discussed in the following subsections. ## summary in this paper, we have presented an open-source app, life for linguistic data management, analysis and sharing. the app intends to accelerate the development of language technologies for extremely underresourced languages by providing a link between field linguists and computational linguists. the app allows field linguists to use state-of-theart nlp models for aiding and accelerating their work and also training baseline models for new languages and tasks in a no-code environment. at the same, it stores the data collected in the field in a structured format that could be used by computational linguists for their research. the app is currently being actively developed and is also used by multiple teams for their research.