Hierarchical Multi-task learning framework for Isometric-Speech Language Translation (2022)

Abstract: This paper presents our submission for the shared task on isometric neural machine translation in International Conference on Spoken Language Translation (IWSLT). There are numerous state-of-art models for translation problems. However, these models lack any length constraint to produce short or long outputs from the source text. In this paper, we propose a hierarchical approach to generate isometric translation on MUST-C dataset, we achieve a BERTscore of 0.85, a length ratio of 1.087, a BLEU score of 42.3, and a length range of 51.03%. On the blind dataset provided by the task organizers, we obtain a BERTscore of 0.80, a length ratio of 1.10 and a length range of 47.5%. We have made our code public here https://github.com/aakash0017/Machine-Translation-ISWLT

## introduction reaching a worldwide audience is a critical aspect of audio-visual content localization. this automation necessitates source language speech translation and seamless integration of target language speech with the original visual information. the uniqueness of this task is to generate length-controlled outputs. a significant application of isometric translation is in automatic dubbing, where the most crucial part is to sync the length of translated subtitles with the audio of the source language. these types of translations give a holistic experience to the user while reading the translated sentences. this paper will explain our hierarchical architecture for generating such isometric outputs. initially, we experimented with a verbositycontrolled multi-task model. we used two prompt types: (i) task prompt and (ii) length prompt. the task prompt decides what task the model should perform. for example, an empty prompt means that the model will receive english inputs and generate translated french outputs, whereas "para" prompt means that the model will receive french input and generate paraphrased french sentences. para prompt always accompanies a length prompt that ensures that the paraphrased output is of the desired length. to illustrate, if the initial translated output of the model falls short of the source text, we will append the prompt: "para long." this prompt will help the model paraphrase this generated output to an optimal length. we experimented with various combinations of this translate-paraphrasing approach. finally, our best architectures consist of three separately trained models for translation and paraphrasing. we use helsinki opus-mt and google's mt5 for machine translation & paraphrasing, respectively, while google translation api for short-length sentences. we use must-c v1.2 fr and paws-x en-fr datasets to train these models. ## shared task overview this task entails creating translations that are similar in length to the source. the shared task's outcome can help with the following issues: auto standardized dubbing to achieve coupling between the source and target speech, improved subtitling to fit the translated content into a specified video frame, layout constrained translation to control the generated text to fit in the document tables or database fields, and more general simultaneous speech translation for ease of reading or listening. participants in the shared task can create text-to-text mt systems for languages such as german (de), french (fr), and spanish (es) using either the must-c or 3 background our approach towards controlling the output length of translated sequences is based on the recent advancement in the transformer architecture @xcite towards multi-task training. ## system overview in this section, we will explain our architecture in detail. as mentioned in the above sections, we implement a hierarchical architecture consisting of 3 separate models. our model is a complex fusion of two distinct functionalities, resulting in a differentiated pipeline that adds to improved performance for text generation tasks. the entirety of the model is fragmented into neural machine translation and a text paraphrasing system. while the former converts text from the source (en) to target (fr) language, the latter, which is trained independently of the nmt model, assists in deforming the generated text into a more useful form specific to the task. additionally, we are also using google's translation api for short-length sentences. ## 256 1.13274 not isometric the climate getting worse means that many years, their crops won't grow: there will be too much rain, not enough rain; things will change in ways their fragile environment simply can't support. translation, we use the previously mentioned verbosity control prompt engineering method. the ## experimental setup during the experimentation, we used three datasets: 1) wmt, 2) must-c 3) paws-x. we also train a separate system constituting google's mt5 pre-trained model for text paraphrasing. for this we're using an ada-factor optimizer (13), with a cross-entropy loss as objective. also, we use a beam size of 5 and select the top 3 hypotheses accordingly. the model is initialized with pre-trained weights from the transformers library. we use the base version with a total of 580m parameters. we use a batch size of 32 and epochs equal to 1. each model is trained on a cluster of 4 tesla v100-pcie gpu with a memory size of 32510mib each. ## result and analysis as shown in secondly, the length range matrix represents the percentage of total translated sentences falling under the ideal length ratios range. two of our suggested models are close to 50%, suggesting that almost half of the predictions are isometric with high bleuscore and bertscore. the reason of decrease in the bertscore of system 3 is that the model loses essential information while predicting the output. our analysis shows that verbosity control can sometimes lead to abrupt shortening of results, where the model skips words after a specific limit. along with length compliance(lc) metrix, outputs are evaluated for their adequacy and quality of translation. this task emphasizes more towards bertscore rather than bleuscore. when the length of source and target varies, bleuscore does not adapt well; however, bertscore can evaluate based on semantics. the challenge is to translate the source text to the target language with ideal length compliance while also maintaining the semantic meaning of the output. while our suggested models are also performing equally well on the blind dataset provided by the organizer, however, a significant dip can be seen with the length ratio & bert score for the predicted outputs. the reason being is that the blind data covers a versatile range of source input with a word count ranging from 1 to 44. the paws-x dataset has an average length of 10-15 words and cannot provide a variety of training examples with a much lower token count. thus, while predicting, the model performs rather poorly for short-length examples. to solve this we have employed google translate api. however, for some instances within the 5-8 word count, the model can still not convert the input sequence to its target language ("french") counterpart. our experiments with the google mt5 model, which is fine-tuned for machine translation and text paraphrasing, have shown considerable promise. however, it still needs rigorous experimentation and hyper-parameter tuning. in addition to quantitative, we vouch for qualitative analysis of our results in ## conclusion & future work in this work, we propose a hierarchical mt approach, using prompt engineering to attribute the opus-mt and mt5 paraphrasing model. we evaluate the proposed approach in the isometric machine translation case, where translated text is expected to match the source length to synchronize the source and target text. our finding shows that though the model has been trained precisely for generating constrained output, however, a lot of improvements can be employed to produce more optimal results. firstly, the paraphrasing model could not generalize for short sentences (i.e., @xmath0.95). secondly, the must-c dataset has an unequal distribution of instances for all three categories of length ranges, which imposes an uncertain suspicion over the model predictions. moreover, our finding shows that the proposed approach can perform better than lakew et al. @xcite , length aware positional encoding based nmt approach.