Cross-Lingual Document Retrieval with Smooth Learning (2020)

Abstract: Cross-lingual document search is an information retrieval task in which the queries’ language and the documents’ language are different. In this paper, we study the instability of neural document search models and propose a novel end-to-end robust framework that achieves improved performance in cross-lingual search with different documents’ languages. This framework includes a novel measure of the relevance, smooth cosine similarity, between queries and documents, and a novel loss function, Smooth Ordinal Search Loss, as the objective function. We further provide theoretical guarantee on the generalization error bound for the proposed framework. We conduct experiments to compare our approach with other document search models, and observe significant gains under commonly used ranking metrics on the cross-lingual document retrieval task in a variety of languages.

## introduction in modern search engines, cross-lingual information retrieval tasks are becoming prevalent and important. for example, when searching for products on a shopping website in english, immigrants tend to use their native languages to form the queries and would like to see the most desired products which are in english. another example is in international trading, investors might use english to describe their product to search the sentiment in other languages on online forums from different countries, in order to understand the customers' attitude towards it. despite that these tasks can be naturally formulated as information retrieval (ir) tasks and resolved by mono-lingual methods, the rising need of cross-lingual ir techniques also requires robust models to deal with queries and documents from different languages. early studies on documents retrieval mostly rely on lexical matching, which is error-prone in crosslingual tasks as vocabularies and language styles usually change across different languages, as well the contextual information is largely lost. with the recent surge of deep neural networks (dnns), researchers are able to go beyond lexical matching by building neural architectures to represent textual information of query and documents by vector representations via non-linear transformations, which have shown great successes in many applications @xcite @xcite . despite the success of those aforementioned advances, several existing difficulties have not been well addressed, e.g. exploding gradients and convergence guarantee. the most widely used optimization method for training dnns is stochastic gradient descent, which updates model parameters by taking gradients of the weights. meanwhile, cosine similarity, a commonly used measure of relevance between query and documents, is not stable as its gradient may go to infinity when the euclidean norm of the representation vector is close to zero, leading to exploding gradients and resulting in irrational training. figure 1 demonstrates the gradients of cosine similarity and our proposed smooth cosine similarity. in addition, in most of the previous works, the loss functions are either heuristically designed or margin-based @xcite , resolving to account for the particular property of the model but lacking interpretability and convergence guarantee. these new scenarios pose a new challenge towards the model: the robustness of the cross-lingual information process needs to be well addressed. to tackle these issues, we introduce an end-to-end robust framework that achieves high accuracy in the cross-lingual information retrieval task with different document languages. particularly, for each query q from language a, we are given a set of documents @xmath0, d 2 , ..., d m ) from a different language b and the degrees of relevance @xmath1, where the entries (y 1 , y 2 , ..., y m ) in y typically belong to ordered class {1, ..., k}. the goal of learning-to-rank is to return an ordered or unordered, depending on the evaluation metrics, subset of the documents d that are more relevant to the query. most common evaluation metrics, such as the normalized discounted cumulative gain (ndcg) and precision, however, are discontinuous and thus cannot be directly optimized. as a result, researchers usually assume an unknown continuous relevance score r, where a higher value r means a larger y and higher relevance between the query and the document. to optimize the model, typically a surrogate loss function is used, which is easier to optimize using the relevance score r. subsequently a subset of the documents are being chosen by ordering r from high to low, to serve the ranking purpose. our contributions in this paper can be summarized as follows: 1. first, we propose a novel measure of relevance between queries and documents, smooth cosine similarity (scs), whose gradient is bounded such that exploding gradients can be avoided, stabilizing the model training. 2. second, we propose a smooth loss function, smooth ordinal search loss (sosl), and provide theoretical guarantees on the generalization error bound for this proposed framework. 3. third, we empirically show significant gains with our approaches over other document search models under commonly used ranking metrics on the cross-lingual document retrieval task, by conducting experiments in a variety of languages. ## smooth neural document retrieval in this section, we propose a novel smooth cross-lingual document retrieval framework. this framework consists of three parts. first, we use neural models to encode queries and documents from different languages, and represent them by low-dimensional vectors. second, we propose a smooth cosine similarity to indicate the relevance score r, which avoids gradient explosion and therefore stabilizes the training process. finally, we introduce smooth ordinal search loss for optimizing r. ## theoretical analysis generalization error measures the difference between training error and testing error. typically, if the generalization error of a model is bounded and converges to zero, then minimizing empirical loss on training set implies that the expected loss on unseen testing set is also minimized. although previous works gave generalization bounds for surrogate losses in learning-to-rank model @xcite @xcite , to the best of our knowledge, no theoretical result has been derived in search models with neural architecture. here, we prove a generalization error bound for the commonly used sgd procedure. this error bound suggests that the generalization gap at any training step t converges to zero when the number of training pairs of query and document goes to infinity. we show the detailed proof in the appendix. the following proposition 1 and lemma 2 show that sosl is both smooth and lipschitz continuous with respect to not only relevance score r but also embedding parameters. proposition 1. sosl is smooth and lipschitz continuous with respect to r. lemma 2. let l(r, y) be a smooth and lipschitz continuous loss function with respect to r, then in our search ranking model, l(r (q, d), y) = l(f q , f d , y) is also smooth and lipschitz continuous with respect to model parameters, i.e., embeddings. from lemma 2, we can assume l(f q , f d , y) to be α-lipschitz and γ-smooth. next, suppose @xmath2, d 1 , y 1 ), ..., (q n , d n , y n )) is a training set sampled from the data distribution with sample size n, and assume unseen testing data is from distribution e. let @xmath3, f d ) be the neural models for queries and documents. by defining l l s (f ) as the mean training error and l l e (f ) the expected error on testing set, the following theorem establishes the upper bound for generalization error l l e (f ) -l l s (f ). theorem 3. (generalization error bound) let l be a smooth loss function bounded by m . suppose we run sgd for t steps (with step size α @xmath4, @xmath5, 2, . . . , t }), with probability at least 1 -δ over the draw of s, note that in theorem 3, the bound does not depend on thresholds θ. we ignore the dependency on and α as both are constants. theorem 3 suggests that for any t , the generalization error after t steps for sgd converges as the sample size of s increases. for some proper step size, we can allow t to increase with n. in particular, if γc < 2, then t may increase at a rate of n while the generalization error still converges. evaluation metrics we apply the commonly used ranking metrics in our experiments for evaluation, including precision, ndcg, map and mrr. for each query, the corresponding documents are sorted by their relevance score, and the metrics are averaged over all queries. p mr @1 represents the precision for mr document at top 1 position. p r @5 represents the precision for mr and sr documents combined at top 5 positions. ndcg@5 is normalized discounted cumulative gain for top 5 documents. map is calculated as the mean of the average precision scores for each query. m rr mr is the mean reciprocal rank of mr document. ## conclusion in this study, we propose a smooth learning framework for cross-lingual information retrieval task. we first suggest a novel measure of relevance between queries and documents, namely smooth cosine similarity (scs), whose gradient is bounded thus able to avoid exploding gradients, enforcing the model to be trained in a more stable way. additionally, we propose a smooth loss function: smooth ordinal search loss (sosl), and provide theoretical guarantees on the generalization error bound for the whole proposed framework. further, we conduct intensive experiments to compare our approach with existing document search models, and show significant improvements with commonly used ranking metrics on the cross-lingual document retrieval task in several languages. both the theoretical and the empirical results imply the potentially wide application of this smooth learning framework. proof of lemma 2 to have a high-level view of the proof, we decompose the loss function into three functions, and will show that each function is lipschitz continuous and smooth with respect to its domain. proof. let r(f ) = f t f d ( f + )( f d + ) be a function of @xmath6. for a @xmath7, the gradient of ∇r(f ) exists, where to show r(f ) is also smooth, we calculate the hessian matrix of r(f ). let @xmath8, similarly, this is also true with respect to f d . deep neural networks normally do not enjoy lipschitz continuity or smoothness due to their expressive power, which make it difficult to analyze its theoretical performance. we now show that our particular models are lipschitz continuous and smooth for both query and document with respect to embedding parameters. for a text query q with token length t, we represent it by @xmath9, ..., q t ) ∈ r nq×t where q @xmath10. the output of query encoding networks f q is then note that hyperbolic tangent tanh(x) = e x -e -x e x +e -x is smooth, and t @xmath11. we can claim that f q (q) is smooth and lipschitz continuous with respect to w q . using lemma a.1, lemma a.2 and proposition 1, we can prove lemma 2.