Closing the NLP Gap: Documentary Linguistics and NLP Need a Shared Software Infrastructure (2022)

Abstract: For decades, researchers in natural language processing and computational linguistics have been developing models and algorithms that aim to serve the needs of language documentation projects. However, these models have seen little use in language documentation despite their great potential for making documentary linguistic artefacts better and easier to produce. In this work, we argue that a major reason for this NLP gap is the lack of a strong foundation of application software which can on the one hand serve the complex needs of language documentation and on the other hand provide effortless integration with NLP models. We further present and describe a work-in-progress system we have developed to serve this need, Glam.

## introduction the labor that is required for documenting a language is complicated, repetitive, and timeconsuming. as many have pointed out, methods from nlp and computational linguistics have great potential for expediting the documentary process @xcite @xcite , and researchers in nlp/cl foot_0 have made great progress in advancing the ability of their models and algorithms to serve the needs of documentary linguistics. at the same time, interest in "low-resource languages" has surged in the past few years in the nlp research community, and there ought to be no better time than now for documentary projects to benefit from the contributions of researchers in nlp. but somehow, most documentary linguistic work done even today in 2022 proceeds without any of the assistance that methods in nlp could be providing. this has been the state of affairs for quite some time-the introductory paragraph of the preface for the proceedings of the first computel conference @xcite explains (emphasis ours): contemporary efforts to document the world's endangered languages [...] are dependent on the widespread availability of [...] software to annotate [documentary data]. however, despite well over a decade of dedicated funding efforts aimed at the documentation of endangered languages, the technological landscape that supports the work of those involved in this research remains fragmented, and the promises of new technology remain largely unfulfilled. moreover, the efforts of computer scientists, on the whole, are mostly disconnected from the day-to-day work of documentary linguists, making it difficult for the knowledge of each group to inform the other. on the one hand, this deprives documentary linguists of tools making use of the latest research results to speed up the time-consuming task of describing an underdocumented language. on the other hand, it severely limits the ability of computational linguists to test their methods on the full range of world's linguistic diversity. eight years later, at computel-5, these words for the most part read as though they could have been written today. why is it that these "promises of new technology" remained unfulfilled for documentary linguists 2 ? we argue here that the fundamental issue preventing vigorous exchange between documentary linguistics and nlp is a lack of application software which can adequately serve both communities: while it is true that apps exist and are commonly used in documentary linguistics, they are ill-suited for integration with nlp models. we therefore claim that documentary linguistics will not benefit from advances in nlp until significant investments are made in developing application software which can compete with existing apps in functionality and provide first-class support for nlp model integration. in this work, we present and discuss this thesis, outlining ideals for what application software ought to accomplish for the documentary linguistics community. in addition, we present a work-in-progress system we have developed which attempts to implement these ideals as practical, usable application software aimed at catalyzing research relationships between documentary linguists and computationalists by taking the needs of both seriously and in equal proportion. ## the nlp gap why does the nlp gap exist? that is, why is it that language documentation is still being carried out without the help of nlp models despite their great potential to help? we argue here that the single most important reason why the nlp gap exists is a rather simple one: there is not a foundational infrastructure of application software that can serve both nlp researchers and documentary linguists. 9 7 to our knowledge, montage was never implemented, and nothing has been published on it since 2005, though some of its conceptual threads have been continued in the aggre-gation project ( http://depts.washington.edu/ uwcl/aggregation/). if models are good enough to deliver value, and documentary linguists want to reap the benefits of nlp and know where in their workflows they'd like models to assist, and computationalists are often available to assist in getting their models to process documentary data, then what else remains? the only possible answer seems to be that it is the lack of support in language documentation apps that is to blame. as noted in §2, documentary linguists cite difficulty in using models, as to the extent that they are available at all, they are usable only in awkward ways which grate against their workflows. nlp models, if they are to be unobtrusive, must have deep integration with documentary workflows, and since these workflows occur in software, nlp models must be deeply integrated into documentary software, the only substrate in which vigorous exchange between these two communities may occur. this is not a small challenge, as this software, if it were to succeed in its goal of catalyzing cooperation between computationalists and documentary linguists, would need to serve well the needs of both parties. from the perspective of the documentary linguists, the whole point of using an nlp model is that it ought to reduce their labor, and as we have seen, existing ways of using nlp models with apps like flex and elan are unergonomic to the point of often being more work than the alternative. from the perspective of nlp researchers, we must make it easy for them to do something more than make their model publicly available, which is a necessary but unfortunately insufficient step in making them usable by all but the most technically experienced and motivated documentary linguists. beyond these design challenges, there is also the challenge of how to find the labor necessary to develop this software, which has been noted as a severe issue @xcite . despite the fact that a path forward for excellent research and positive outcomes for language communities requires significant investment in application software infrastructure, the cultural currents within both linguistics and nlp, for better or worse, dictate that software engineering (which also happens to be incredibly time-intensive) does not constitute research activity. the obvious outcome is that no researcher in either discipline would be well advised to make this kind of work more than a side-interest in their research interests, and it is telling that the two most popular apps, flex and elan, were developed by software engineering staff at language-related organizations rather than academics themselves. that is a bleak outlook-is a shift in how our fields reward software development too much to hope for? it is worth digressing for a moment here to note that academic communities do have the power to change how the field views and rewards software artefacts as contributions, if they choose to prioritize bringing about such a cultural shift. for example, in the field of astronomy, academics have been publishing software packages providing implementations of commonly needed statistical and simulation algorithms for decades, though traditionally, such packages were only viewed as "contributions" worthy of the attention of, say, a hiring or tenure committee if there was an associated publication in a journal @xcite . securing such a publication could be difficult if a package was very specialized or small, and as the need for new packages has risen sharply, the field of astronomy has responded by lowering the requirements for a "software publication" (see kelley 2021 for an example). in the future, the field may be moving towards treating a package in itself as a "publication" (in the academic sense, i.e. something that can appear on a c.v. or be indexed by google scholar). in sum, the field was able to recognize that its traditional assessment and treatment of certain research activity was no longer appropriate, and needed to be changed so that activity that used to be thought of as marginal would be recognized and rewarded as a first-class scholarly activity. despite these challenges, we believe it is possible and vitally important for researchers in lan-guage documentation and nlp to try to find ways of building the backbone of application software which is needed for interchange between the two fields to progress, which as we hope is clear by now is crucially necessary for achieving widespread use of nlp models in language documentation. in the short term, we hope that individuals will be able to overcome career risks that come with working on something that is not "research" by cooperating with others, thereby amortizing the loss of time spent on more traditional research topics. in the long term, we challenge senior academics, and especially senior academics in nlp who have presented their models and algorithms as beneficial for language documentation, to consider whether it is not time to reassess whether the software work we have described is deserving of more recognition and support, and if it is, how the community's cultural values and institutions could be changed to reward such work. we close our discussion of the nlp gap on this note. in the remainder of this work, we turn to describe what we believe would be key goals for an app aimed at closing the nlp gap, and further describe a prototype-grade system we have constructed which aims to achieve these goals. ## system description glam is an alpha-quality system we have developed which aims to serve the needs we have described. while for the rest of this section we speak mostly of design instead of the state of the implementation, we take a moment to note its progress. in its present state, glam is capable of surfacelevel interlinear annotation of texts, and there is work underway to add support for lexical inventories (as in flex). this is the bare minimum necessary to conduct a small-scale language documentation project, such as for a semester-long field methods course that might be offered at a university. support for nlp models has not yet been implemented, which may seem strange. the reason is that, as we have noted, it is important for this app to fully satisfy the needs of both documentary linguists and nlp researchers, and we have viewed the former as the much harder problem and prioritized solving it first. we have however naturally been considering the problem of nlp integration from the very initial stages of design, and have made implementation decisions with care in order to facilitate its eventual implementation. the latest state of the project can be tracked by visiting the repository. foot_3 ## conclusion we have discussed the problem of why nlp models have not seen more use in documentary linguistics, and concluded that the single most important barrier to adoption of nlp models is the lack of a substrate of application software that can serve the needs of both documentary linguists and nlp models well. we have moreover presented design goals and implementations of a system which we think shows potential to meet this need. regardless of the ultimate fortunes of our system, glam, we reprise our invitation to readers to consider whether our assessment of the nlp gap is correct (i.e., that it cannot be closed without serious investment in application software, which in turn might require a cultural shift in some academic communities), and if it is, what there is to be done about it. nlp researchers have gained much from endangered languages, not least by sourcing unique data from them for publications-if they are in dire need of assistance that the nlp community is singularly able to provide, and which is not forthcoming from any other community or organization in the world, should the nlp community not act? moreover, beyond this matter of deserts, there is also the exciting prospect of opportunity for new methods and models that could come from a deeper relationship between these two fields, mediated by a substrate of application software. for junior researchers without a faculty position or tenure, a helpful action might be to find collaborators to work on this software problem with. for researchers in nlp working on low-resource nlp models aimed at application in documentation of endangered languages, it might be right to consider whether they ought to have more involvement in making this application actual instead of potential. for senior researchers with tenure, who wield the most influence, it may be appropriate to reexamine the reasons why the current norms around what constitutes "research activity" are what they are, and whether it might be right to reform them given the unmet needs of endangered languages.