Enhancing Neural Machine Translation with Semantic Units (2023)

Abstract: Conventional neural machine translation (NMT) models typically use subwords and words as the basic units for model input and comprehension. However, complete words and phrases composed of several tokens are often the fundamental units for expressing semantics, referred to as semantic units. To address this issue, we propose a method Semantic Units for Machine Translation (SU4MT) which models the integral meanings of semantic units within a sentence, and then leverages them to provide a new perspective for understanding the sentence. Specifically, we first propose Word Pair Encoding (WPE), a phrase extraction method to help identify the boundaries of semantic units. Next, we design an Attentive Semantic Fusion (ASF) layer to integrate the semantics of multiple subwords into a single vector: the semantic unit representation. Lastly, the semantic-unit-level sentence representation is concatenated to the token-level one, and they are combined as the input of encoder. Experimental results demonstrate that our method effectively models and leverages semantic-unit-level information and outperforms the strong baselines.

## introduction neural machine translation (nmt) @xcite @xcite @xcite @xcite has achieved significant success and has been continuously attracting significant attention. currently, the mainstream language models use tokens as meaningful basic units, which are typically words or subwords, and in a few cases, characters @xcite . however, these tokens are not necessarily semantic units in natural language. for words composed of multiple subwords, their complete meaning is expressed when they are combined. for some phrases, their overall meanings differ from those of any individual constituent word. consequently, other tokens attend to each part of a semantic unit rather than its integral representation, which hinders the models from effectively understanding the whole sentence. recent studies have been exploring effective ways to leverage phrase information. some works focus on finding phrase alignments between source and target sentences @xcite . others focus on utilizing the source sentence phrase representations @xcite @xcite . however, these approaches often rely on time-consuming parsing tools to extract phrases. for the learning of phrase representations, averaging token representations is commonly used @xcite . while this method is simple, it fails to effectively capture the overall semantics of the phrases, thereby impacting the model performance. some researchers proposed to effectively learn phrase representations @xcite using bert @xcite , but this introduces a large number of additional parameters. therefore, we propose a method for extracting semantic units, learning the representations of them, and incorporating semantic-level representations into neural machine translation to enhance translation quality. in this paper, a semantic unit means a contiguous sequence of tokens that collectively form a unified meaning. in other words, a semantic unit can be a combination of subwords, words, or both of them. for the extraction of semantic units, it is relatively easy to identify semantic units composed of subwords, which simplifies the problem to extracting phrases that represent unified meanings within sentences. generally, words that frequently cooccurent in a corpus are more likely to be a phrase. therefore, we propose word pair encoding (wpe), a method to extract phrases based on the relative co-occurrence frequencies of words. for learning semantic unit representations, we propose an attentive semantic fusion (asf) layer, exploiting the property of attention mechanism @xcite that the length of the output vector is identical with the length of the query vector. therefore, it is easy to obtain a fixedlength semantic unit representation by controlling the query vector. to achieve this, pooling operations are performed on the input vectors and the result is used as the query vector. specifically, a combination of max-pooling, min-pooling, and average pooling is employed to preserve the original semantics as much as possible. after obtaining the semantic unit representation, we propose a parameter-efficient method to utilize it. the token-level sentence representation and semantic-unit-level sentence representation are concatenated together as the input to the encoder layers, with separate positional encoding applied. this allows the model to fully exploit both levels of semantic information. experimental results demonstrate this approach is simple but effective. ## method in this section, we present our methods in detail. as depicted in figure 2 , the proposed method introduces an additional attentive semantic fusion (asf) module between the encoder layers and the token embedding layer of the conventional transformer model, and only a small number of extra parameters are added. the asf takes the representations of several tokens that form a semantic unit as input, and it integrates the information from each token to output a single vector representing the united semantic representation. the sentence representations of both token-level and semanticunit-level are then concatenated as the input of the first one of the encoder layers. note that the token-level representation is provided to the encoder layers to supplement detailed information. we also propose word pair encoding (wpe), an offline method to effectively extract phrase spans that indicate the boundaries of semantic units. ## experiments in this section, we conduct experiments on three datasets of different scales to validate the effectiveness and generality of the su4mt approach. ## analysis 5.1 ablation study to further understand the impact of providing different forms of source sentence representations to the encoder layer on model performance, we conduct three sets of ablation experiments based on the default settings of asf, as shown in as demonstrated by the experimental results, using only one perspective of sentence representation leads to a decrease in translation quality, indicating that diverse sentence representations complement each other and provide more comprehensive semantic information. furthermore, we remove the duplicated portion to investigate the impact of adding redundant information on translation. surprisingly, when doubling the token-level sentence representation, sacrebleu increases by 0.6, but when doubling the semantic-unit-level sentence representation, sacrebleu even exhibits a slight decline. we speculate that repeating the information-sparse tokenlevel sentence representation allows the model to learn detailed semantic information and to some extent learn overall semantic representation separately from the two identical representations. however, semantic-unit-level sentence representation is semantically dense. it's already easy for the model to understand the sentence and duplicating it may introduce some noises. in conclusion, using both perspectives of semantic representation together as input maximizes the provision of clear, comprehensive semantic and detailed linguistic features. ## conclusions in this work, we introduce the concept of semantic units and effectively utilize them to enhance neural machine translation. firstly, we propose word pair encoding (wpe), a phrase extraction method based on the relative co-occurrence of words, to efficiently extract phrase-level semantic units. next, we propose attentive semantic fusion (asf), a phrase information fusion method based on attention mechanism. finally, we employ a simple but effective approach to leverage both token-level and semantic-unit-level sentence representations. experimental results demonstrate our method significantly outperforms similar methods.