Findings from the B ambara - F rench Machine Translation Competition ( BFMT 2023) (2023)

Abstract: Orange Silicon Valley hosted a low-resource machine translation (MT) competition with monetary prizes. The goals of the competition were to raise awareness of the challenges in the low-resource MT domain, improve MT algorithms and data strategies, and support MT expertise development in the regions where people speak Bambara and other low-resource languages. The participants built Bambara to French and French to Bambara machine translation systems using data provided by the organizers and additional data resources shared amongst the competitors. This paper details each team’s different approaches and motivation for ongoing work in Bambara and the broader low-resource machine translation domain.

## bfmt 2023 -competition introduction orange silicon valley, hosted the "bambara-french machine translation competition 2023" (bfmt 2023) a low-resource machine translation (mt) competition that ended on february 15, 2023. the competition was launched on december 15, 2022. participants had access to a github repository with a training dataset of parallel french-bambara aligned sentences foot_0 . the participants were also invited into a slack community to share their approaches and data. an additional development dataset was provided to the teams and fewer than 48 hours before the submission deadline, a test dataset was released for generating text output to be sent to the competition organizers to evaluate translation performance using bleu scores @xcite . the goals of the competition were to improve not only french to bambara and bambara to french automated translation systems, but also support a transparent and collaborative community to work on these and other language pairs, especially those (low-resource) languages spoken by west africans. 50 people joined the online community and fourteen people competed in 6 teams. the teams contained participants from mali, senegal, namibia, nigeria, ireland, germany, russia, spain, france, the us, and the uk. many of the participants speak or have working knowledge of a "low-resource language" or a language that does not have the digital resources that support highly accurate natural language processing tool development. bambara is a tonal language with a rich morphology spoken by five million people as a first language and approximately 15 million people as a second language. approximately 30-40 million people speak a language in the mande language family, to which bambara belongs @xcite . a predominately oral language, several competing writing systems have developed. a majority of bambara speakers have not been taught to read or write in a standard format. bambara's standardization is evolving and this poses challenges to automated text processing such as machine translation @xcite . additional contest information may be found in both french and english on the orange silicon valley website foot_1 . ## background current state-of-the-art low-resource mt is surveyed in @xcite . google translate has integrated more low-resource languages into their language library sharing innovations as detailed in blog posts @xcite . mt for the bambara -french language pair has been explored in recent years in @xcite ; @xcite ; @xcite . this work is in part motivated by an increased financial and cultural focus on bringing machine learning to the sahel region (diarra and leventhal, 2020). ## team-by-team machine translation findings from bfmt 2023 six teams submitted system output that could be evaluated using sacrebleu. team peter-sokhar (section 3.7) built an mt system but did not submit an output for scoring. nonetheless, their findings from training and error analysis are included in this paper. in the following sections, each team first describes their methodology, then they describe their error analysis. see ## bfmt 2023 results and discussion nonetheless, there were key themes to the submissions. all of the teams used the same core datasets, with two teams bootstrapping alternatives as shown in subsequent insights were that the winning team used a backtranslation approach, cyclic backtranslation, and another successful team used a beam search optimization. also, we learned that smaller distilled models could beat larger models with limited amounts of data (i.e., fine-tuning distilled models yields more accurate results). only one team had members that spoke bambara but many participants are speakers of other low-resource languages and hope to extend their experience with mt system development to languages that their families and friends speak. ## conclusion and future work because of bfmt 2023, researchers have successfully implemented innovative low-resource machine translation systems. these implementations are extensible to other language pairs, which is helpful since low-resource languages continue to face numerous challenges in terms of research focus and funding. we believe bfmt 2023 has not only supported increased visibility of the bambara language, but it has also showcased the talent that is working on using creative techniques to address these technical challenges globally. the bfmt 2023 competition community would like to extend this work by holding other competitions. ideally, the next competition will utilize automatic speech recognition data. including spoken data in mt might circumvent a challenge in low-resource language, where only a few online datasets support predominately oral language text processing. the output of bfmt 2023 is a viable baseline for french -bambara and bambara -french machine translation. in addition, the competition dataset is now available to researchers seeking to exceed this baseline or evaluate their translation systems. similar to the practice in some kaggle competitions, we can also provide a baseline model in the next competition iteration that is based on the top scoring competition submission 6 . finally, we would like to provide greater financial support to the participating teams by sponsoring equal and standard access to computational resources. this could better illuminate which machine learning models are the highest performers.