Machine-in-the-Loop with Documentary and Descriptive Linguists (2024)

Abstract: This paper describes a curriculum for teaching linguists how to apply machine-in-the-loop (MitL) approach to documentary and descriptive tasks. It also shares observations about the learning participants, who are primarily non-computational linguists, and how they interact with the MitL approach. We found that they prefer cleaning over increasing the training data and then proceed to reanalyze their analytical decisions, before finally undertaking small actions that emphasize analytical strategies. Overall, participants display an understanding of the curriculum which covers fundamental concepts of machine learning and statistical modeling.

## introduction this paper outlines the curriculum for "machinein-the-loop for language documentation" workshops and shares from the experience we gained while teaching the material in different settings. 1 the workshop material gives instruction in foundational machine learning concepts, natural language processing (nlp) techniques, and statistical modeling. the full version of the material is designed to create an opportunity where documentary and descriptive linguists learn the concepts and apply nlp models to assist them in their documentary and descriptive tasks. in hopes of increasing the effectiveness of instructors who teach nlp to linguists or who wish to improve inter-disciplinary collaboration and communication, this paper describes patterns that we observed while teaching this material in different settings. we aim the discussion towards instructors who come from a computational angle and will be teaching participants whose degrees or training is primarily in linguistics or related social sciences (we use the term "linguist" as a shorthand for any of these learners.) after teaching the material in whole or part, we noticed similar patterns of interaction with the nlp models emerged. these patterns make sense in light of typical linguistic training that promotes analytical skills and in-depth investigation of language data. ## motivation and background unfortunately, current documentary and descriptive methods cannot scale up to match the pace of the language endangerment or provide automated methods to assist documentation @xcite because annotating large corpora of potential training data is still done mostly by hand @xcite . this situation can be addressed by providing non-computational linguists with a basic understanding of statistical nlp so they are better equipped to integrate machine learning assistance into their work. the potential for nlp to increase and improve documentary tasks has been clearly demonstrated @xcite @xcite . however, nlp research with underdocumented languages often does not consider realistic factors sufficiently. for example, research tends to take a linear approach, where initial annotation of training data is the only input from human experts that the nlp system receives. in contrast, the workshop promotes human-in-the-loop approaches in realistic settings. the goal of humanin-the-loop techniques is to make optimal use and also reduce expensive human annotation while improving model performance @xcite . we use the term "machine-in-the-loop (mitl)" to emphasize our conviction that technology's role is to assist humans @xcite , not vice versa. although @xcite indicate that "reactive" learning that uses simple uncertainty sampling for denoising a corpus is not ideal, the workshop code implements this basic method, we choose this simple method because, unfortunately, a short pedagogical event is not ideal for active learning experimentation or complex strategies. it may be worth noting that @xcite assume that the examples sampled by the algorithm as most "uncertain" are the only examples that will be relabeled, whereas we allow the linguists to choose what examples they will work with. humans can analyze why a data point might have been selected as most uncertain by the algorithm and whether they agree that relabeling it for the next training cycle is likely to be impactful. also, humans can generalize from the algorithm's simple calculations and then find and re-label multiple examples that they feel are similar in nature. these abilities may counteract some of the noted drawbacks of simple uncertainty sampling. ## overview of curriculum and workshops the goal of the curriculum is to help linguists better understand concepts that will, hopefully, make them comfortable integrating nlp systems in their work. the material is aimed at linguists or others engaged in language-related work who do not have a background in advanced mathematics or computer science. the curriculum progresses through the learning objective listed below. the material uses lectures and activities to convey concepts and guide participants through a mitl approach to language documentation and description task. it includes python code that we used to preprocess data and train the nlp models. other instructors may find the code helpful but will probably wish to adjust it to suit their context. the activities are of two types. the first are short activities usually in the form of games or worksheets that are intended to break up lectures and reinforce concepts. these can be done in groups or individually. the final activity is essentially the same as learning objective 8. it assumes participants can be grouped into teams of 2-6 members. for the final activity, the one-day version of the curriculum uses the same data for all teams. details about this data is provided in the released material. the three-day version assumes teams will be formed before the workshop begins and each team will work on their own field data during the final activity. with this in mind, team leaders are encouraged to invite members who are knowledgeable about the language and the task. in a workshop setting with more than two or three teams, we found it necessary to recruit "tech coaches" for each team. these are participants who have skills to download and run the workshop code. tech coaches do not need to run the code for preprocessing data if the instructor can do this more complicated task beforehand. the tech coaches do not necessarily need to be familiar with the language. the material was developed from stand-alone lectures and assignments in university classrooms which were gradually combined and redesigned for non-credit workshop settings aimed at faculty, graduate students, or others working with endangered languages. the material released with this paper are designed for all-day workshops either one or three days in length. the three-day version of the workshop aims to coach participants to apply nlp systems to their own data. below we described how we covered the material in a university classroom in some detail because we did release curriculum for this setting and then outline more briefly how a one-day and three-day workshop can be conducted. ## the machine-in-the-loop approach the workshop curriculum centers around a machine-in-the-loop (mitl) approach comprising active learning cycles of training and annotation. the goal is to guide humans to new annotation that will have most impact towards gradually improving nlp performance so that the improved model offers useful assistance to the humans. we implemented re-active labeling @xcite using least confidence uncertainty sampling @xcite . other sampling strategies can be used by instructors, but uncertainty sampling is relatively easy to explain. as implemented for the workshop, least confidence uncertainty sampling ranks the model's predicted annotations by the differences between the model's confidences for each predicted sequence. when using conditional random fields (crf), the model's confidence c is defined as its calculated probability p that unit x should be annotated with a particular class ŷ. confidence for a pre-dicted sequence is calculated by normalizing each unit probability by the number of units in the sequence. for pos tagging, units are words and sequences are sentences; for morphological parsing, units are morphemes and/or morpheme glosses and sequences are characters in a word. the same calculation of confidence can be used with other models, such as the transformer, but probability can be substituted for the absolute value of the models' negative log likelihood for a unit. the sequences and the predicted annotations are ranked from lowest to highest confidence scores and written to a plain text file without the scores. since participants naturally begin at the top of a file, this ranking nudges them to start correcting the new annotations with examples that the model found most confusing. the goal of the mitl activity in the three-day workshops is to provide ai assistance for basic language documentation tasks, specifically annotation (interlinerization), and thereby increase the amount and the quality annotated data in endangered languages. where possible, the activity should be prepared before the workshop begins. teams can choose their task, and if ethically sound, share their field data with the instructor two to six weeks before the workshop begins. the instructor can run the preprocessing code to separate the annotated and unannotated units (units causing problem for the code are removed to the unannotated corpus) and divide the annotated corpus into a 9/1 training/test split. an initial model can then be trained, tested, and then used to annotate unlabeled data. ideally, participants bringing field data should first create gold standard labels by correcting the portion of the data withheld to serve as the test set. if this is all done before the workshop begins, participants can start the mitl activity (objective 8) on the first day by examining the initial model's classification report, confusion matrix, and predicted annotations. then they can attempt to clean manually labeled training data, correct the model's annotations, or customize the model architecture of feature selection. at the end of each day or whenever teams feel ready, new models can be trained on the version of the training data. ## observations we taught the material in all settings described in section 3, primarily to participants with higher education degrees primarily in linguistics or related social science fields. in all three settings, similar patterns of interaction with the mitl approach emerged. we feel that a description of these patterns may assist instructors who come from a computational background. we have no data to confirm these patterns quantitatively, but they match our broader experience as computational and quantitative linguists teaching in linguistic departments. 2 we observed that linguists' interaction with the mitl approach tend to reflect their analytical training rather than statistical or engineering solutions. given free choice of the sub-steps under objective 8, the linguists followed three stages. first, they remove noise in the manually annotated data by correcting mistakes and inconsistencies. second, they revise their previous analytical decisions and change manual annotations accordingly. third, they take strategic actions aimed to improve the representation of the training data. the first two stages held true across all settings, but the third one was primarily observed in the in our one threeday workshop. we include it because, based on our broader teaching experience, it seems likely to hold true generally for linguists newly introduced to nlp and mitl concepts, and so may be helpful for other instructors to look for. first, clean data. field data is inherently noisy, either due to the dynamic nature of linguistic analysis, or as a by-product of manual work. given a choice between the tasks under objective 8, linguists showed a preference to clean training data. this held true even though the instructors emphasized the impact of data size on statistical models. interestingly, research @xcite suggests, given limited time, cleaning rather than adding more data is a wise choice. cleaning meant correcting typos and making glosses consistent (e.g. '1.sg' and 'i' → 1.sg). we noted that teams tended not to clean all data at once, but asked to retrain the model 2-5 times and leaned on the model's output to find mistakes for the next round of corrections. it seems they found that although language data can be cleaned without nlp assistance, the mitl approach helped them more quickly identify issues. second, reanalyze. as obvious mistakes in the training data were corrected, we observed the linguists started to lean more on the mitl approach, but not in the way we expected. they looked more closely at the model's predictions for the unlabeled data. noting that some of the model's "mistakes" were due to isomorphism or other ambiguity, they asked questions about outliers and weighting in statistical modeling. we assumed they would focus on adding new annotated sequences to the training data. instead, they shifted their focus to the analytical decisions which had governed their original annotation. several mentioned how the model highlighted a tension between lumping versus splitting choices they had felt while analyzing and annotating the original data. the linguists began reanalyzing their previous annotation choices. they used the highest listed sequences (ones with lowest confidence scores) in the computer-annotated file to guide their reanalysis. this indicates that they had grasped the concept of least confidence sampling. it seems they hypothesized that if they changed their entire annotation schema, either by adding new classes or lumping others and then bulk-edited the training data, this would solve with the model's "confusions". after one or two rounds, reanalysis resulted improvement and sometimes decreased performance. this led to conversations about how an mitl approach encourages iterative work just solving the next lowhanging fruit may be more effective than linear work that provides a comprehensive analysis. finally, strategize additional annotation. once the original training data had been cleaned and the teams finished their reanalysis we observed the linguists really began to integrate the principles of statistical modeling that were covered during the lectures. their questions and discussions turned to strategies for increasing annotated training data as much as possible. a notable pattern at this stage was they either did not fully grasp the impact of data size or were daunted by the task of providing enough new annotated data. instead of bulk-editing, they tended to correct the model's annotations in small strategic efforts that seemed guided as much by their knowledge of the language as by the uncertainty sampling strategy. then they would request for the model to be retrained so they could see the effect. for example, they might correct the machine annotations only on the first six (least confident) sequences or they might search for sentences with a rare part of speech. sometimes the latter approach improved the model's performance on one class. one group who had a member with python programming skills and worked with a crf for postagging decided to experiment with the model architecture which was optimized for english. for example, because their language is more morphologically complex than english, they programmed the model to use the first and last six letters of a word as features, instead of the first and last four letters. this was the only example of participants attempting to customize the model, and they also seemed to prefer leveraging strategic knowledge of the language, rather than a statistical strategy (i.e. lots of annotation). what's next? participants progressed through these three stages independently. in general, they demonstrated understanding how noise and ambiguity present significant issues for statistical models with limited data. in the closing discussions, a repeated theme was the amount of annotation needed in order to make a significant impact would exceed their time limitations. this presented a chance to introduce "engineering" solutions not covered in the workshop materials, such as data hallucination and synthetic augmentation or crosslingual transfer learning. ## conclusion while teaching this material we observed that documentary and descriptive linguists bring their analytical strengths to the mitl approach. the material does not include formal assessments but we consider it successful because participants who brought their own data left with a better quality documentary corpus. a recent grant submitted by a workshop attendee to fund this workshop in another location indicates that the participants also assessed the material positively. we conclude with specific recommenations. first, despite many successful remote collaborations in the post-pandemic age, we found that inperson events, removed from regular work, promote focused work and encourages social interaction that counteracts the intense schedule. second, we recommend adding one day to the longer workshop schedule just to deal with anticipated and unanticipated issues (remote server not set up in time, flight delays, code bugs, forming teams, etc.).