Creating the best development corpus for Statistical Machine Translation systems (2018)

Abstract: We propose and study three different novel approaches for tackling the problem of development set selection in Statistical Machine Translation. We focus on a scenario where a machine translation system is leveraged for translating a specific test set, without further data from the domain at hand. Such test set stems from a real application of machine translation, where the texts of a specific e-commerce were to be translated. For developing our development-set selection techniques, we first conducted experiments in a controlled scenario, where labelled data from different domains was available, and evaluated the techniques both with classification and translation quality metrics. Then, the bestperforming techniques were evaluated on the e-commerce data at hand, yielding consistent improvements across two language directions.

## introduction tuning is a critical step in every system that presents a weighted combination of features. by adjusting the weights so that they best fit the target distribution, this process typically yields important improvements on the performance of the system developed. however, selecting an appropriate development set is key for this process to reach its goal. in statistical machine translation (smt), the tuning step implies optimizing the log-linear c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. with x and y being the source and target sentences. such optimization has become de-facto standard in smt, thanks to the wide-spread adoption of tuning algorithms such as minimum error rate training (mert) @xcite or the margin infused relaxed algorithm (mira) @xcite . the purpose of these algorithms is to adjust the log-linear weights such that the model distribution best fits the target distribution, or the target metric by which the system is evaluated. given that the amount of weights λ m is typically around 10 or 20, the size of the development corpus required for tuning is typically in the range of hundreds or a few thousands of sentences. however, such corpus is typically required to be disjoint from the training corpus, used to estimate the features h m , and its selection is critical, having an important impact on the system's performance if the development set of choice is too different from the test set at hand @xcite . the data selection (ds) task is stated as the problem of selecting the best sub-corpus of sentences from an available pool of sentences, with which to train a machine learning system. this paper deals with ds, but here the aim is to select, out of an available pool of sentences, the best development corpus for a given test set, for the purpose of log-linear weight optimization in smt. we study our development ds techniques in two different tasks. in the first case, the purpose is to analyse the behaviour of our techniques in a controlled scenario where the data is labelled according to domain. the goal is to study our methods' capacity of correctly predicting the domain labels, in addition to the translation quality achieved. in the second scenario, we evaluate the techniques presented in a real task, where a specific test set belonging to the texts of a real e-commerce site is provided, without domain labels. the main contributions of this paper involve the necessary steps required to assess our novel development set selection techniques: this paper is structured as follows. sections 3 and 4 present our different dds methods. section 5 presents the experiments: in section 5.2, we present the analysis derived from the controlled experiment; section 5.3 presents the results achieved with the real e-commerce task. section 2 summarises related work. conclusions and future work are discussed in section 6. ## related works the work presented here is close in concept to the domain adaptation scenario. domain adaptation in smt systems received considerable attention from the research community. different domain adaptation techniques, including data selection, mixture models, etc., have been developed for different scenarios. a wide variety of data selection methods have been used over the years, where the main principle is to measure the similarity of sentences from the out-of-domain corpus to some in-domain corpus, either the development or the (source side of the) test set. such similarity is often based on information theory metrics, like perplexity or cross entropy. in the last years, perplexitybased, or cross-entropy based, methods have become more common @xcite @xcite @xcite . cross-entropy difference is a typical and well-established ranking function. techniques based on information retrieval have also been widely used for data selection @xcite . furthermore, @xcite leveraged neural language models to perform ds, reporting substantial gains over conventional n-gram language model-based ds. finally, many researchers have used convolutional neural networks (cnn) in the domain adaptation field @xcite @xcite . all the above ds approaches assume that the selection corpus is used to train or combine the smt models. however, previous research on selecting the appropriate development corpus also exists. such research can be split into two categories: in the first category, a development set is chosen, from among several "closed" development sets, based on the test set at hand (transductive learning) @xcite @xcite . the second category deals with the problem without knowing the test set beforehand, but knowing the domain of the test set (inductive learning). previous work on development data selection for unknown test sets includes @xcite . note that the work presented here has an important difference with both transductive and inductive learning: even though it is closer to the transductive learning setting, all these works are based on selecting the most adequate development corpus from a collection of "closed" development corpora, with the purpose of choosing the one that belongs to the test set domain. in our case, we want to construct a specific development corpus for a given test corpus, without knowing the domain of the test set. ## levenshtein distance dds the first dds technique proposed involves computing the edit distance (levenshtein distance) between a candidate sentence and the closest sentence in the test set. here, the intuition is to consider that a given sentence to be included in the development set d is a good candidate if it is not too far away from the sentences in the test set t , as measured by the levenshtein distance. we will refer to this technique as ld-dds. the levenshtein distance (ld) @xcite is a string metric for measuring the difference between two sequences (words or sentences). the ld between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to make them match. algorithm 1 shows the procedure. here, p is the pool of sentences available, [x p , y p ] is an outof-domain sentence pair ([x p , y p ] ∈ p ), and |p | is the number of sentences in p . then, our objective is to select data from p such that it is the most suitable for translating data belonging to the test corpus t (composed only of source sentences). algorithm 1: pseudo-code for ld-dds. algorithm 1 introduces the ld(•, •) function, which computes the ld between two given sentences. note that threshold τ establishes the size of the development corpus, and will need to be fixed empirically (section 5.2). ## dds with vector-space representations here, we present two other dds selection techniques, where the common point is that they both leverage a continuous vector-space representation of the sentences involved. first, we will describe our technique in abstract terms, and then we will present two different candidates for obtaining a continuous vector-space representation f (x) (or f x for short) of a given sentence x. here, the intuition is to select as candidate sentences those whose vector-space representation is similar to those in the test set, assuming that similar sentences will have similar vectors. the advantage of having a continuous vectorspace representation of the test sentences is that a centroid can be computed, which can be assumed to be a sort of prototype of the sentences present in the test set. note it was not possible to compute such centroid in the case of ld-dds (section 3). perhaps the best way to explaining this intuition is graphically, as shown in figure1. this figure is a graphical example of the idea that we follow in this section, where sentences are represented in a two-dimensional vector-space. here, blue points are the representation of the test sentences and red points represent the vectors of the sentences of the available pool of sentences, from which the development set is to be selected. assuming that similar sentences will have a similar vector-space representation, the vectors of the test corpus will be very closer to each other, but the vectors for the general pool of sentences will be more disperse. the idea in our method is to draw a circle boundary, containing all test-sentences within it, and (hopefully) only a few of the sentences in the candidate pool. the radius of this circumference (or hyper-sphere in a multi-dimensional vector-space) is established as the distance between the centroid of the test set, and the furthest of the test sentences. algorithm 2 shows the procedure. here, p is the pool of candidate sentences, [x p , y p ] is a candidate sentence pair, with [x p , y p ] ∈ p , f x is the vectorspace representation of x, and |p | is the number of sentences in p . then, our objective is to select data from p such that it is the most suitable for translating data belonging to the source test data t . for this purpose, we define f t as the vectorspace representation of a sentence @xmath0. algorithm 2 introduces several functions: data: pool p ; test data t result: development corpus d f @xmath1; ρ = inf forall t in t do if cos(f t , f t ) ≤ ρ then ρ = cos(f t , f t ) end end forall [x p , y p ] in p do if cos(f xp , f t ) ≥ ρ then add [x p , y p ] to d end end algorithm 2: pseudo-code for dds leveraging vector-space representations of sentences. in addition, ρ represents the radius of the circumference, which is computed in lines 2 to 6 (the first forall loop) in algorithm 2. once the selection algorithm has been established, we need to define how to represent sentences in a z-dimensional space. using vectorspace representation for textual data (word, sentence or document) is not a new idea and has been widely employed in a variety of nlp applications. these representations have recently demonstrated promising results across a variety of tasks. in this paper, we used two different approaches for representing sentences in a continuous vectorspace: the popular term frequency -inverse document frequency (tf-idf), and sentence embeddings @xcite . the basic idea is to represent a sentence x with a real-valued vector of some fixed dimension z, i.e., f (x) ∈ r z that is able to capture similarity (lexical, semantic or syntactic) between a given pair of sentences. ## experiments in this section, we describe the experimental framework employed to assess the performance of the dds methods described in sections 3 and 4. for this purpose, we studied their behaviour in two separate tasks: a controlled scenario with labelled data, and a real e-commerce translation task. we will first detail the experimental setup employed, which is common to both tasks, and then we will report on each one of the tasks and their results. ## conclusions in this paper, we have presented different techniques for building a test-specific development corpus, leveraged for optimizing the log-linear weights of the smt system. we proposed three new development data selection methods: ld-dds, tf-dds, and cvr-dds. we analysed the performance of these methods in a controlled scenario, where domain labels are available, and evaluated the methods in a real translation task where e-commerce data was to be translated, without a development set being readily available. the empirical results show that cvr-dds, which leverages a continuous vector-space representation of the sentences, is able to improve over baseline translation quality, and provide a development set that leads to similar translation quality as than the one obtained whenever an in-domain development set is readily available. in addition, the results obtained with cvr-dds consistently and significantly improve over those obtained with a random sampling baseline, across different languages. in the future, we will further investigate the selection of development corpus, since there is more room for improvements, as reported by the oracle setting. we also intend to test our methods on other domains and test data so as to establish their robustness. finally, we are providing the ecommerce corpus cachitos de plata, used as test data, free for research purposes.