How can machine translation help generate A rab melodic improvisation? (2023)

Abstract: This article presents a system to generate Arab music improvisation using machine translation (MT). To reach this goal, we developed a MT model to translate a vocal improvisation into an automatic instrumental oud (Arab lute) response. Given the melodic and non-metric musical form, it was necessary to develop efficient textual representations in order for classical MT models to be as successful as in common NLP applications. We experimented with Statistical and Neural MT to train our parallel corpus (Vocal → Instrument) of 6991 sentences. The best model was then used to generate improvisation by iteratively translating the translations of the most common patterns of each maqam (n-grams), producing elaborated variations conditioned to listener feedback. We constructed a dataset of 717 instrumental improvisations to extract their n-grams. Objective evaluation of MT was conducted at two levels: a sentence-level evaluation using the BLEU metric, and a higher level evaluation using musically informed metrics. Objective measures were consistent with one another. Subjective evaluations by experts from the maqam music tradition were promising, and a useful reference for understanding objective results.

## introduction the purpose of this paper is to present a method for using machine translation (mt) to generate automatic instrumental improvisation in arab maqām music, particularly in two contexts: responsive accompaniment to vocal improvisation (mawwāl), and free instrumental improvisation (taqāsīm). this method could then be adapted to other melodic musical traditions. we situate this project within the efforts to maintain, preserve, and develop these musical forms in arab music using (mt) paradigms. we construct our own corpora and tools for their collection and processing, explore neural and statistical methods, and test them using the bilingual evaluation understudy (bleu) measure, which is a common mt metric @xcite . the study presents the results of using a bleu score along with musically informed metrics @xcite and subjective evaluations. more broadly, we view this musicbased project as a mt challenge within the broader context of under-resourced languages @xcite . in arab music, mawwāl is a non-metric vocal improvisation and is often applied to narrative poetry. upon the completion of each vocal sentence, the instrumentalist performs a recapitulation, or a translation of that sentence @xcite . in other words, the duo (singer, instrumentalist) play a musical conversation. at time t i , the singer produces an improvisatory phrase p(t i ), then the instrumentalist produces a musical answer corresponding to p(t i ), which we will call a(t i+1 ). when the instrumentalist finishes, the singer responds with a new improvisatory sentence p(t i+2 ), and this process is repeated until the end of the improvisation. in this paper, we first review our approach for using mt to propose an instrumental responsive accompaniment to mawwāl. we then explain how to use the same model to generate a full instrumental improvisation in the maqām context using iterative translation. we particularily aim to answer the following questions: ## machine translation of mawwal data structuring and representation are key in this application in order for statistical and neural mt models to be as satisfying as in nlp applications. we therefore begin this section by describing these details before discussing the details and evaluation of the mt models. ## conclusion we proposed a mt system for automatic instrumental improvisation in maqām music. by reducing the dimensions of the textual representation of musical sentences to only scale degree and quantized duration, it was possible to train smt and nmt models using a parallel dataset (vocal and instrumental) that is both relatively small (6991 sentences) and diverse (8 different maqamat). the superior mt model was then used as a basis to generate real-time instrumental improvisation conditioned to listener feedback. to this end, we constructed a fully instrumental dataset of 717 improvisations from which we extracted frequent repre-sentative patterns (n-grams) for each maqam. mt was then applied, iteratively, starting first with the n-grams and then conditioned to listener feedback. results were found promising based on subjective evaluations by experts from the maqām music tradition, as well as objective evaluation applied at two levels: the sentence level using the bleu measure, and a higher level using statistical, musically informed metrics. the two objective measures were found consistent with each other. future work will include investigating the influence of the following factors on the performance of music mt models: musical quality, size, and average sentence length of the (sub-)dataset.