Demo of the Linguistic Field Data Management and Analysis System - L i FE (2021)

Abstract: In the proposed demo, we will present a new software - Linguistic Field Data Management and Analysis System - LiFE - an open-source, web-based linguistic data management and analysis application that allows for systematic storage, management, sharing and usage of linguistic data collected from the field. The application allows users to store lexical items, sentences, paragraphs, audio-visual content including photographs, video clips, speech recordings, etc, along with rich glossing / annotation; generate interactive and print dictionaries; and also train and use natural language processing tools and models for various purposes using this data. Since its a web-based application, it also allows for seamless collaboration among multiple persons and sharing the data, models, etc with each other. The system uses the Python-based Flask framework and MongoDB (as database) in the backend and HTML, CSS and Javascript at the frontend. The interface allows creation of multiple projects that could be shared with the other users. At the backend, the application stores the data in RDF format so as to allow its release as Linked Data over the web using semantic web technologies - as of now it makes use of the OntoLex-Lemon for storing the lexical data and Ligt for storing the interlinear glossed text and then internally linking it to the other linked lexicons and databases such as DBpedia and WordNet. Furthermore it provides support for training the NLP systems using scikit-learn and HuggingFace Transformers libraries as well as make use of any model trained using these libraries - while the user interface itself provides limited options for tuning the system, an externally-trained model could be easily incorporated within the application; similarly the dataset itself could be easily exported into a standard machine-readable format like JSON or CSV that could be consumed by other programs and pipelines. The system is built as an online platform; however since we are making the source code available, it could be installed by users on their internal / personal servers as well.

## introduction linguistic data management and analysis tools have always been a requirement of field linguists. a huge amount of data is collected and analysed by field linguists for a large number of languages including relatively lesser-known, minoritised and endangered languages of the world and these need to be properly stored, analysed and made accessible to the larger community. on the other hand, there are a huge number of languages across the globe (including the kinds mentioned above), whose data is not available for building any kind of language technology tools and applications. in order to tackle this multi-faceted problem of storing, processing, retrieving and analysing the primary linguistic data, an integrated system with an easily-accessible and user-friendly interface aimed at linguists needs to be made available. "life" is developed with the intent of providing a practical intervention in the field by making available an organised framework for management, analysis, sharing (as linked data) and processing of primary linguistic field data including development of digital and print lexicons, sketch grammars and fundamental language processing tools such as part-of-speech tagger and morphological analysers. the software provides an easy-to-use, intuitive interface for performing all the tasks and there is an emphasis on automating the tasks as far as possible. for example, given some initial input, the system incrementally trains automated methods for inter-linear glossing of the dataset (which improves as more data is stored in the system) and subsequent generation of sketch grammar as well as nlp tools for the language. similarly, the system automatically in-fers and links the entries in the lexicon and interlinear glossed data using lemon (more specifically ontolex-lemon) @xcite and ligt @xcite . ## motivation and features linguistic field data storage, management, sharing and linked data generation has largely developed independent of each other. as such while there are quite a few tools and applications aimed at field linguists (or community members interested in fieldwork for their own language) for collection and management of data as well as generating lexicon, such as fieldworks language explorer (flex) foot_0 @xcite ; toolbox foot_1 @xcite ; lexique-pro foot_2 @xcite ; wesay foot_3 (perlin, 2012) @xcite and a few other platforms for archiving and providing access to the data, the prominent ones being endangered languages archive (elar) foot_4 @xcite ; the language archive (tla) foot_5 @xcite ; sil language and culture archive foot_6 , etc. the open language archives community (olac) foot_7 , which is a consortium of over 60 participating linguistic archives of various kinds (including the ones mentioned above and others for storage and access of linguistic data, especially of endangered languages) has also recently joined the linguistic linked data open cloud which paves the way for providing a large amount of such data as linked data @xcite . however none of the tools and platforms directly provide an interface for storing or (largely) automatically generating the primary linguistic data as linked data or provide a seamless two-way between the nlp tools and libraries and linguistic data management softwares. on the other hand, the linked data community has developed tools for supporting generation of linked data, especially linked data lexicons. one of the best-known tools for this is vocbench (vb), which is a fully-fledged open-source web-based thesaurus management platform with the feature of collaborative development of multilingual datasets compatible with semantic web standards. it provides the facilities of generating lexicons, thesauri, and linked data ontologies to the large organisations, companies, and user communities @xcite . however tools like these focus on generating linked data which is generally not very user-friendly for field linguists nor do they provide options for automating the tasks or linking to the nlp ecosystem. the primary motivation for building this platform is to provide a tool that acts as a bridge between field linguists (who are primarily engaged in data collection from low-resource and endangered languages, building lexicons, writing grammatical descriptions and also producing educational and other kinds of materials for the communities that they work with), linked data community (who are primarily engaged in meaningfully connecting data from different languages and resources using the semantic web techniques) and the nlp community (who primarily makes use of the linguistic data from multiple languages; could potentially provide support in automating the tasks carried out by field linguists; and also provide tools and technologies for the marginalised and under-privileged linguistic communities). as such in its current state the app provides the following functionalities - during the demo we will present these features and the interface of the tool in detail and also briefly train the participants in using it.