NECAT-CLWE: A S IMPLE B UT E FFICIENT P ARALLEL D ATA G ENERATION A PPROACH FOR L OW R ESOURCE N EURAL M ACHINE T RANSLATION
A BSTRACT
Many languages lack sufficient data to train qualitative translation systems, particularly those based on the cutting-edge neural machine translation architectures.
Recently, it has been demonstrated that using an exact copy of the monolingual target data as the source data improves the quality of translation systems, allowing them to benefit from proper nouns and such similar words that do not require translation.
However, using an exact copy of the target data contaminates the source data with terms in the target language that needs translation.
As a result, we describe in this paper a similar but more effective parallel data generation approach for improving low resource neural machine translation using named entity copy ing and approximate translations using cross lingual word embedding (NECATCLWE
The work will be evaluated on the low resource English Hausa neural machine translation.
I NTRODUCTION
Since the dawn of time, language and communication have been important to human interactions.
As a result, linguistic translation has played a crucial role in sociological and cultural achievements.
To simplify and enable communication between different languages, an means is required.
Because these means can be expensive or difficult to obtain, researchers have explored creating translations automatically.
Machine Translation (MT) was one of the first applications that was designed to be achieved using a computer. Warren Weaver’s ”translation memorandum” coined this idea, and the IBM created a word-for-word translation system in 1954 (Hutchins, 1997).
Since then, various strategies were created to handle the challenge of MT, the most notable of which is Statistical
Machine Translation (SMT) (Koehn, 2009). Neural Machine Translation (NMT) (Bahdanau et al., 2015; Vaswani et al., 2017), a deep learningbased approach to machine translation is currently being considered as a superior alternative to the prior state-of-the-art machine translation paradigm, the phrase-based SMT (PBSMT) (Koehn et al., 2003), due to a variety of factors, including its end-to-end learning flexibility and capacity to create superior translation (Toral & S´anchez-Cartagena, 2017). Because the quantity of parallel sentence pairs available for training has a direct impact on the performance of the NMT system and many languages lack sufficient amounts of this data, many research works have explored using other available resources, e.g., through Language Modelling (Lample et al., 2018) or Transfer Learning (Zoph et al., 2016).
A strong emphasis has also been placed on producing parallel datasets (also known as bitext), in addition to research on novel MT approaches. Recently, the back-translations (Sennrich et al., 2016a) and forward translations (Zhang & Zong, 2016) of the monolingual target and source data respectively are used to generate additional data.
While the approaches have been shown to be successful at improving translation accuracy, they are very expensive to implement and the quality of the additional data is also not assured.
A simple and cost-effective approach for utilizing the monolingual target data has been copying the target data into the source language (Currey et al., 2017; Burlot & Yvon, 2018).
This has been shown to benefit the translation model especially where the words are proper nouns or words that do not require translation.
This approach, though, will litter the source language with words that are not in its vocabulary and require to be translated.
This research work, therefore, aims to improve upon the baseline work of Currey et al.
While the named-entities will be shared between the source and target sentences, the unknown words that require translation will be replaced with the nearest words in the vocabulary of the source language using a pre-trained cross-lingual word embedding (JP et al., 2021).
Cross-lingual word embedding have been shown to generate a similar vector representation for similar words across two or more languages (Zhang et al., 2019).  P ROBLEM S TATEMENT , R ESEARCH O BJECTIVES AND S IGNIFICANCE This section presents the motivation for conducting this research and the associated objectives or contributions.
2. P ROBLEM S TATEMENT ( MOTIVATION )
Because of the diversity of languages and the never-ending need for inter-lingual communication among people, a means for automatic translation has never been so important. Neural Machine Translation (NMT) has shown a great potential for training excellent systems that can automatically translate between languages when the available manually-translated data between these languages are in abundance (Edunov et al., 2018).
However, this method suffers from the scarcity of parallel data between the majority of languages, the Low Resource Languages (LRL). Copying the target data into the source language has been shown as a straightforward and costeffective method for utilizing the monolingual target data to improve low resource machine translation (Currey et al., 2017). The approach has been demonstrated to help the translation model, particularly with words that are proper nouns or words that do not need to be translated. However, using an exact copy of the target data will clog up the source data with words that are not part of its lexicon and, therefore, require translation.
2. R ESEARCH O BJECTIVES (C ONTRIBUTIONS )
The aim of this study is to create an efficient parallel data generation technique based on named entity copying and approximate translation using cross-lingual word embedding.
The following objectives are presented to achieve the stated aim.
To conduct an extensive review of current parallel data creation algorithms.
To develop a parallel data generation algorithm using named entity copying and approximate word translation with cross-lingual word embedding. 3.
To evaluate the proposed approaches against current data generation algorithms on EnglishHausa low resource NMT.
Due to the importance of having qualitative translation systems especially in languages that are categorized as low-resourced, the successful conduct of this research work will provide an efficient method for improving the performance of machine translation systems in low and even high resource languages.
The work, being evaluated on English-Hausa NMT, will also help further develop research in the local Hausa language.  R ELATED L ITERATURE This section reviews literature that are related to the proposed approach: additional parallel data generation algorithm for low resourced languages. 3. L OW R ESOURCE L ANGUAGES
A low-resource problem in Natural Language Processing (NLP) can develop primarily because the investigated languages or domains are under-resourced (Hedderich et al., 2021).
Researchers have attempted to define low-resourced languages (LRLs) by investigating several factors such as the number of mother-tongue speakers and the amount of unlabeled, labeled, or supplementary datasets available, as well as the availability of other NLP resources and tools (Hedderich et al., 2021). Over the years, there have been many initiatives to categorize languages according to the aforementioned different criteria. Machine translation (MT) occurs between two languages. This means that in MT, the resourcefulness of a language pair is decided by the number of parallel corpora accessible between the target languages.
When referring to the parallel corpora at hand, the terms extremely low-resource (or zero-resource), low-resource (LR) or high-resource (HR) have been used regularly (Hedderich et al., 2021).
However, there is no minimum size requirement for parallel corpora to classify a language pair as high, low, or extremely low-resource.
Some early research considered 1 million parallel sentences and below as LR (Zoph et al., 2016).
Recent research appears to regard a language pair as
LR or severely LR if the available parallel corpora for the studied pair for NMT trials is less than 0.5 million and 0.1 million respectively (Lakew et al., 2020; Liu et al., 2020; Platanios et al., 2018).
However, these do not represent absolute numbers for the size of dataset.
3. A UGMENTING L OW R ESOURCE D ATASET
Translation into or out of under-resourced languages has always been viewed as a difficult endeavor (Durrani & Koehn, 2014; Haque et al., 2012; Nießen & Ney, 2004; Resnik & Smith, 2003; Utiyama & Isahara, 2007).
Despite the ability of NMT to produce state-of-the-art outcomes in a wide variety of translation tasks, it continues to offer numerous challenges for language pairings with scarce resources (Cheng et al., 2019; Sennrich & Zhang, 2019).
Researchers in MT have tried a variety of strategies to solve this challenge, such as using monolingual data from the source and/or target side for training (Cheng et al., 2019; Sennrich et al., 2016a), augmenting bilingual training data (Fadaee & Monz, 2018), and exploiting training data in other language pairs (Zoph et al., 2016).
As proposed in Currey et al.
(2017), a novel method for incorporating target-side monolingual data entails copying a monolingual corpus and converting it into a bitext with identical source and target sides.
This copied bitext is then combined with the parallel and/or back-translated data to train an NMT system with no distinction made between the copied, back-translated, and parallel data during training.
They experimented a mixing ratio of 1:2:2 between the parallel, copied, and backtranslated data and reused the same monolingual data for generating both the copied and backtranslated corpora.
Burlot & Yvon (2018) undertook a comprehensive investigation of back-translation (BT), evaluating other uses of monolingual data and multiple data production processes.
They also investigated several approaches to integrating monolingual data in an NMT framework, with an emphasis on the influence on quality and domain adaptability.
While verifying the usefulness of BT, the researchers also recommended substantially less expensive alternatives to increase baseline performance, such as employing a slightly modified copy of the target rather than its complete BT.
When no high-quality
BT is available, they suggested doing domain adaptation by employing Generative Adversarial Networks (GANs) to get the pseudo-source sentences closer to natural source sentences.
As stated by Wu et al.
(2019), back-translation has been widely employed in prior techniques to unsupervised neural machine translation, in which synthetic sentence pairings are constructed to train the models with a reconstruction loss.
However, because translation errors accrue throughout training, these synthetic sentences are frequently of poor quality.
To avoid this fundamental issue, they suggested a different but more effective approach, extract-edit, in which genuine phrases are extracted and subsequently edited from the target monolingual corpora.
They also proposed a comparative translation loss for evaluating translated target sentences and thereby training unsupervised translation systems.
Experiments show that their proposed approaches consistently outperforms previous state-of-the-art unsupervised machine translation systems.
Recently, Sennrich & Zhang (2019) adapted an NMT system to low-resource environments using a variety of strategies, including modifying the model’s major hyper-parameters (learning rate, model depth, label smoothing rate), and lowering byte-pair encoding (BPE) (Sennrich et al., 2016b)
Increasing the size of the vocabulary and decreasing the size of the batch.
Their findings show that, contrary to earlier research, NMT can outperform PBSMT in low-data scenarios.
In the works of Abdulmumin et al.
(2021a;b), the authors presented novel approaches that combine self-learning and back-translation to allow both backward and forward models to benefit from monolingual target data.
Experimental results showed that the proposed strategy outperformed the classic back-translation method on English-German low resource neural machine translation.
They also proposed an iterative self-learning strategy that outperforms iterative back-translation while requiring less model training and only monolingual target data.
Wu et al.
(2021) discovered that the two methods can complement each other in the sense that the former can exploit context information via language-independent features but sees no task-specific information in the target language, whereas the latter generates pseudo target-language training data via translation but its exploitation of context information is hampered by inaccurate translations.
Furthermore, previous studies have infrequently used unlabeled data in the target language, which is easily obtained and may provide useful information for improved findings.
To address both issues, they offer Unifying Transfer, a novel approach that unifies both model and data transfer for cross-lingual named entity recognition, as well as better knowledge distillation to use the available information from unlabeled target-language data.
T HE P ROPOSED NECAT-CLWE P ARALLEL D ATA G ENERATION A PPROACH
We illustrate the proposed Named-Entity Copying and Approximate Translation using CrossLingual Words Embedding approach in Figure 1 and Algorithm 1. Algorithm 1 Named-Entity Copying and Approximate Translation using Cross-Lingual Words Embedding Input : X = { ( x ( u ) ) } Uu
=
& Y
= {
( y (
v Vv = exhausted 10: end procedure [Equation]
[Equation]
[Equation] Firstly, we get the monolingual data in each of the target and source languages, mix the two data and train a cross-lingual words embedding. Secondly, we copy all the named entity words into the generated source sentences at the same positions as in the target sentences. Then, for each word in the target sentences that require translation, we select and insert into the source sentences the nearest word in the source vocabulary using the pretrained cross-lingual words embedding.
These steps are iterated until all the target sentences are paired with their pseudo-parallel, approximate translations.
[Figure]
We also experiment replacing the words that require translation with the special unknown token: ‘ < unk > ’ – see Algorithm 2.
Although the special token is another form of noising, it is still not a word that will confuse the decoder into thinking all the words in the target sentences do not require translation.
Also, we hypothesize that the translation model will learn to ignore the special token.
Algorithm 2 Named-Entity Copying with Unknown Word Replacement
C ONCLUSION AND F UTURE W ORK
Due to the absence of sufficient parallel datasets for training machine translation models, several research works are proposing efficient approaches for utilizing other resources to improve the accuracy of these models.
In this work, we presented a conceptual framework, called NECAT-CLWE— Named-Entity Copying and Approximate Translation using Cross-Lingual Words Embedding, a work-in-progress Masters Dissertation study, that aims to utilize the abundant monolingual target data to improve low resource neural machine translation using named-entity copying and word substitution with cross-lingual words embedding.
We aim to implement the proposed approach and evaluate it against current data generation algorithms.