P icard understanding Darmok: A Dataset and Model for Metaphor-Rich Translation in a Constructed Language (2022)

Abstract: Tamarian, a fictional language introduced in the Star Trek episode Darmok, communicates meaning through utterances of metaphorical references, such as “Darmok and Jalad at Tanagra” instead of “We should work together.” This work assembles a Tamarian-English dictionary of utterances from the original episode and several follow-on novels, and uses this to construct a parallel corpus of 456 English-Tamarian utterances. A machine translation system based on a large language model (T5) is trained using this parallel corpus, and is shown to produce an accuracy of 76% when translating from English to Tamarian on known utterances.

## introduction science fiction and fantasy literature has long created constructed languages for their characters, from elvish in lord of the rings and klingon in star trek to heptapod in arrival @xcite . these languages often have many of the same syntactic or semantic features as human languages, and some (such as klingon) have been developed to a level where full dictionaries @xcite and online translators are available. 2 an unconventional language was proposed in an episode of star trek: the next generation called "darmok", where a race of aliens called the tamarians speak a language that is communicated exclusively through metaphors. instead of direct reference (e.g. "i want to give this to you"), tamarians speak in metaphorical references grounded in stories (e.g. "temba, his arms wide") that (like symbols) have learned associations with their true meaning meaning. in the darmok story, the unusual nature of the language poses a challenge for both the automated translation systems and the 1 data and code available at: https://github.com/ cognitiveailab/darmok 2 https://www.translate.com/klingon-english t5 "translate-tamarian: they put aside their differences and worked towards a common goal." tamarian utterance inferred meaning english example 1 darmok and jalad at tanagra working together knowing they would both be needed, they went together. 2 temba, his arms wide. giving the child offered his toy to his friend. 3 kira at bashi. story-telling they described what had happened to those who listened. 4 chenza at court, the court of silence. incontestability the results were beyond reproach. 5 zima at anzo, zima and bakor. persistence they continued their task, undeterred from past failures. 6 fendit, refusing the flame. refusing help she preferred to work alone, without assistance. 7 chatha and teribium, the fire warm. hospitality their household was offered for rest and comfort. 8 jeral, her arms weary. being tired she was spent at the end of the day. 9 pirakee, with clouds parted. visibility she turned on a flashlight, making it easier to see. 10 hammat dancing. liking something it filled them with delight. tamarian-to-english dictionary: to create a parallel english-tamarian corpus, first a tamarianto-english dictionary that captures the inferred meaning of each tamarian utterance was required. the meanings of the twenty broadcast utterances was ascertained from a reddit thread with extensive discussion of the topic. 3 the meanings of the remaining forty-eight utterances was inferred as best as possible from the surrounding context of where they appeared in their respective novels. tamarian-english parallel corpus: training a machine translation system requires a parallel corpus, where utterances of one language are paired with utterances of a second language, where the utterances in both languages have the same meaning. tamarian utterances abstractly refer to specific types of situations that could be applicable to many circumstances. thus, for each tamarian utterance a set of k english examples were manually authored, with ten examples authored for thirty-nine utterances, and five examples authored for eleven utterances. eighteen tamarian utterances were not included in the parallel corpus as they have relatively narrow meanings, and generating a large number of parallel examples for them in english proved challenging. the final parallel corpus contains fifty tamarian utterances, paired with 456 parallel english utterances ( ## translation model approach: here, english-to-tamarian is modeled as a sequence-to-sequence (seq2seq) learning task, using english utterances as the source sentence, and a single tamarian translation of that english utterance as the target sentence. models: modeling used t5 @xcite , a large pre-trained multi-task language model. t5 includes pre-training for a variety of tasks, including question answering, summarization, and translation. several model sizes were explored, including t5-small (66m parameters), t5-base (220m parameters) and t5-large (220m parameters). the model prompt took the form of: translate english to tamarian: {src} where {src} is the english source sentence to translate (e.g. "she offered it to them"). the model then generated a corresponding target sequence corresponding to the tamarian translation of the source sentence (e.g. "temba. his arms wide."). the model was implemented using the huggingface transformers library @xcite . dataset splits: due to small dataset, we use 5fold crossvalidation: with 60% of data used for training, 20% for development, and 20% for test. for utterances with ten examples, this corresponds to six train, two development, and two test samples per run, while for utterances with five examples, this corresponds to three train, one development, and one test sample per run. evaluation metrics: translation performance was evaluated using sacrebleu @xcite , a metric that measures translation performance using n-grams, while taking partial matches into account. ## results models were trained until performance (bleu) asymptoted on the development set, at thirty epochs. the best performing model achieves a translation accuracy of 76% on the unseen test set, which corresponds to translating approximately three out of four english utterances from the corpus correctly into tamarian ( ## discussion as a constructed language for a fictional universe, tamarian is a low resource language with fewer than one hundred known utterances. what might it take to grow tamarian (or a metaphoricallygrounded tamarian-like language) into a more complete artificial language similar to klingon? this section attempts to address the challenges of scaling beyond this work in the context of two central difficulties: growing the parallel corpus of metaphors, and challenges associated with the semantics of translating complex ideas in tamarian. ## related work: (computational) linguistics for constructed languages the elephant in the room is whether it is worthwhile to study constructed languages at all. this section seeks to answer that question with a resounding yes by discussing the other insights that have come from scholarly investigations of constructed languages. tamarian is from the star trek universe, so it is instructive to spend a little time first with the oldest star trek language, klingon. klingon is often used in nlp education because it has features that are rare in natural languages but it is incredibly regular: a morphological analyzer can get 100% accuracy but still have fascinating properties like affixes for honorifics, completion, and tense @xcite . likewise, because klingon is by construction meant to feel literally alien, its ovs structure can also upend students' part of speech tagging expectations @xcite . but klingon is not just a fun exercise for programmers and linguists; the creation of parallel data (as discussed above for tamarian) also explores the interplay between culture and translation. for the translation of hamlet into klingon, cultural adaptation @xcite is also needed: for example, fortinbras becomes "the most insuborinate head of the house of duras" @xcite . the art of translation often relies on metaphor @xcite and cultural knowledge @xcite , and just as exploring klingon can reveal limitations of our understanding of affix morphology and ovs word order, tamarian can help illuminate the limitations of metaphor in communication. all extant constructed languages are low resource languages, which typically pose challenges for machine translation @xcite . like how klingon can emphasize particular aspects of a language (word order, morphology), tamarian helps focus attention on the role of mythology, inter-personal relationships, and multiword expressions for translation. ## conclusion this paper is an initial english-tamarian translation model. this task is difficult because it not only maps words to words but also maps metaphor to typical translation phrases. while tamarian is a constructed language, it shows large language models' ability and limitations for metaphor.