Ye Olde F rench: Effect of Old and M iddle F rench on SIGMORPHON - U ni M orph Shared Task Data (2024)

Abstract: We offer one explanation for the historically low performance of French in the SIGMORPHON-UniMorph shared tasks. We conducted experiments replicating the 2023 task on French with the non-neural and neural baselines, first using the original task splits, and then using splits that excluded Old and Middle French lemmas. We applied a taxonomy on our errors using a framework based on Kyle Gorman’s “Weird Inflects but OK” 2019 annotation scheme, finding that a high portion of the French errors produced with the original splits were due to the inclusion of Old French forms, which was resolved with cleaned data.

## introduction the annual sigmorphon-unimorph shared task on morphological (re)-inflection has been a locus for developing language resources, algorithms, and tasks in the computational morphology community since @xcite . while the details of the task settings have varied over the years, the basic task has been: given training data consisting of triples ⟨lemma, inflection features, inflected form⟩, (e.g., ⟨désarmer, 2.sg.subj, désarmerais⟩ for french 'disarm',) train a model to infer inflected forms given pairs ⟨lemma, inflection features⟩. already in 2016, @xcite noted the remarkable average accuracy achieved in the basic task (95.56% averaged across languages in the top system) and the "surprising" huge performance gap between neural and non-neural approaches (e.g., with the best performing neural approach exceeding the best non-neural one by as much as 60% in accuracy within a language). seven years and three versions of unimorph later, the most recent sigmorphon-unimorph shared task, @xcite similarly remarks that performance over individual languages was "quite impressive" and that "all neural systems outperformed the non-neural systems" on average across languages. however, @xcite also notes the mysteriously poor performance of systems on french in two senses. first, no system achieved higher than 77.7% accuracy on french. the other languages for which models' accuracies peaked at 80% were navajo, ancient greek, sanskrit, belarusian, and sami. @xcite implicitly note the oddness of french (a suffixing, fusional, high-resource language) in this group: "while there is no one characteristic shared between all of these languages, it is worth noting that this list includes the only two extinct languages tested in this task, and the only mostly prefixing language. perhaps further development of tailored models could help fill this gap." second, neural systems did not outperform nonneural systems on french-while the non-neural baseline achieved 77.7% accuracy, the best performing neural system achieved 74.7% accuracy. in fact, the non-neural baseline was the best performing system in english, danish, and french. @xcite again point out the oddness of french in this group: "partial explanation may be the small size of the inflection tables in danish and english that necessitated inclusion of many lemmas in the training set and may facilitated better generalization ability of the non-neural baseline. admittedly, this explanation is not valid for french, foot_0 but this language was proven difficult in previous shared tasks @xcite and in other works @xcite ." why has french been a particularly challenging language for inflection tasks since it was first added to unimorph in 2017? in this paper, we show that old and middle french lemmas/forms have been erroneously included in french unimorph data in all sigmorphon-unimorph shared tasks involving french, as well as @xcite ; @xcite . we also provide evidence that including these old and middle french forms has caused anomalously poor performance via three replication experiments of the 2023 shared task for french-two excluding old and middle french lemmas-and an error analysis of the results. ## background to contextualize our claim that old and middle french forms have resulted in poor performance, we will first provide background information on old and middle french and explain its presence in wiktionary, as well as a brief history of poor performance on french in past inflection tasks. hereafter "old french" will be used as shorthand to encompass both old french and middle french. ## materials and methods all source data, scripts for processing data, output files, and the error analysis spreadsheet can be found at https://github.com/prophecyo ak/tigre-2023sigmorphon , which includes a readme explaining how to run the scripts. the script reproduce.sh can be run to repeat the steps used to produce the output data from the sigmorphon-unimorph 2023 shared task replication experiments used for error analysis. ## conclusion when shared tasks include dozens and dozens of languages, it is hard to interpret results when each individual language could be affected by data issues like those we have discussed in this paper. such problems underscore the need for shared tasks to include qualitative, language-by-language analysis of data and results in addition to reporting accuracy. it is admittedly a tall order to do analyses like the one in this paper for each of the over two dozen languages from the sigmorphon-unimorph 2023 shared task, but perhaps shared tasks could explicitly focus on probing and improving data quality and otherwise emphasize languageby-language error analysis as an essential step of analyzing results. this kind of work would naturally encourage collaboration between language experts/linguists and modelers, as suggested in malouf et al. (2020)'s statement of best practices for computational modeling of cross-linguistic morphology. by closely examining the distribution of errors produced, future projects can concentrate on eliminating prevalent error categories that have previously hindered model performance, enabling focused improvements in shared tasks.