Interactive Refinement of Cross-Lingual Word Embeddings (2020)

Abstract: Cross-lingual word embeddings transfer knowledge between languages: models trained on high-resource languages can predict in low-resource languages. We introduce CLIME, an interactive system to quickly refine cross-lingual word embeddings for a given classification problem. First, CLIME ranks words by their salience to the downstream task. Then, users mark similarity between keywords and their nearest neighbors in the embedding space. Finally, CLIME updates the embeddings using the annotations. We evaluate CLIME on identifying health-related text in four low-resource languages: Ilocano, Sinhalese, Tigrinya, and Uyghur. Embeddings refined by CLIME capture more nuanced word semantics and have higher test accuracy than the original embeddings. CLIME often improves accuracy faster than an active learning baseline and can be easily combined with active learning to improve results.

## introduction modern text classification requires large labeled datasets and pre-trained word embeddings @xcite @xcite . however, scarcity of both labeled and unlabeled data holds back applications in low-resource languages. cross-lingual word embeddings @xcite can bridge the gap by mapping words from different languages to a shared vector space. using clwe features, models trained in a resource-rich language (e.g., english) can predict labels for other languages. the success of clwe relies on the domain and quality of training data @xcite . while these methods have impressive word translation accuracy, they are not tailored for downstream tasks such as text classification (glavas * indicates equal contribution mediocre lackluster disappointing uninspiring figure 1 : a hypothetical topographic map of an english-french embedding space tailored for sentiment analysis. dots are english words, and squares are french words. positive sentiment words are grouped in a clime (red), while negative sentiment words are grouped in another clime (blue). these climes help sentiment analysis. @xcite . we develop classifying interactively with multilingual embeddings (clime), that efficiently specializes clwe with human interaction. foot_0 given a pre-trained clwe, a bilingual speaker in the loop reviews the nearest-neighbor words. clime capitalizes on the intuition that neighboring words in an ideal embedding space should have similar semantic attributes. in an analogy to geographic climes-zones with distinctive meteorological features-we call areas in the embedding space where words share similar semantic features climes. our goal is to convert neighborhoods in the embedding space into classification climes with words that induce similar labels for a given classification task. for example, in the embedding for english-french sentiment analysis, positive sentiment words such as "excellent", "exceptional", and their french translations are together, while "disappointing", "lackluster", and their translations cluster together elsewhere (fig-ure 1 ). curating words in the embedding space and refining climes should help downstream classifiers. first, clime uses loss gradients in downstream tasks to find keywords with high salience (section 2.1). focusing on these keywords allows the user to most efficiently refine clwe by marking their similarity or dissimilarity (section 2.2). after collecting annotations, clime pulls similar words closer and pushes dissimilar words apart (section 3), establishing desired climes (figure 1 ). quickly deploying cross-lingual nlp systems is particularly important in global public health emergencies, so we evaluate clime on a crosslingual document classification task for four lowresource languages: ilocano, sinhalese, tigrinya, and uyghur (section 4). clime is effective in this low-resource setting because a bilingual speaker can significantly increase test accuracy on identifying health-related documents in less than an hour. clime is related to active learning @xcite , which also improves a classifier through user interaction. therefore, we compare clime with an active learning baseline that asks a user to label target language documents. under the same annotation time constraint, clime often has higher accuracy. furthermore, the two methods are complementary. combining active learning with clime increases accuracy even more, and the user-adapted model is competitive with a large, resource-hungry multilingual transformer @xcite . ## interactive neighborhood reshaping this section introduces the interface designed to solicit human feedback on neighborhoods of clwe and our keyword selection criterion. suppose that we have two languages with vocabulary v 1 and v 2 . let e be a pre-trained clwe matrix, where e w is the vector representation of word type w in the joint vocabulary @xmath0. our goal is to help a bilingual novice (i.e., not a machine learning expert) improve the clwe e for a downstream task through inspection of neighboring words. ## fitting word embeddings to feedback after receiving user annotations, clime updates the embeddings to reflect their feedback. the algorithm reshapes the neighborhood so that words near a keyword share similar semantic attributes. together, these embeddings form desired taskspecific connections between words across languages. our update equations are inspired by attract-repel @xcite , which fine-tunes word embeddings with synonym and antonym constraints. the objective in attract-repel pulls synonyms closer to and pushes antonyms further away from their nearest neighbors. this objective is useful for large lexical resources like babelnet @xcite with hundreds of thousands linguistic constraints, but our pilot experiment suggests that the method is not suitable for smaller constraint sets. since clime is designed for low-resource languages, we optimize an objective that reshapes the neighborhood more drastically than attract-repel. ## cross-lingual classification experiments we evaluate clime on cross-lingual documentclassification @xcite , where we build a text classifier for a low-resource target ilocano ... nagtalinaed dagiti pito a balod ti bureau of jail management and penology (bjmp) ditoy ciudad ti laoag iti isolation room gapo iti tuko ... english ... seven inmates from the bureau of jail management and penology (bjmp), laoag city, have been transferred to the isolation room due to chicken pox ... our experiments confirm that a bilingual user can quickly improve the test accuracy of crosslingual models through clime. alternatively, we can ask an annotator to improve the model by labeling more training documents in the target language. therefore, we compare clime to an active learning baseline that queries the user for document labels; clime often improves accuracy faster. then, we combine clime and active learning to show an even faster improvement of test accuracy. comparing active learning to clime may seem unfair at first glance. in theory, document labeling only requires target language knowledge, while clime learns from a bilingual user. in practice, researchers who speak a high-resource language provide instructions to the annotator and answer their questions, so bilingual knowledge is usually required in document labeling for low-resource languages. moreover, clime is complementary to active learning, as combining them gives the highest accuracy across languages. we also experiment with refining the same set of keywords with multiple rounds of user interaction. the repeated sessions slightly improve test accuracy on average. finally, we compare with xlm-r @xcite , a state-of-the-art multilingual transformer. despite using fewer resources, clime has competitive results. 0 25 50 75 accuracy 54.3% 56.3% 62.6% 73.2% ilocano, rcsls 10 users, 40 min. (avg) method base active clime a+c 55.2% 60.2% 62.0% 63.4% sinhalese, cca 25 users, 45 min. (avg) 53.6% 60.9% 56.0% 63.7% sinhalese, rcsls 25 users, 40 min. (avg) 0 25 50 75 55.9% 57.5% 73.0% 77.1% tigrinya, cca 1 user, 34 min. (avg) 53.0% 69.7% 69.9% 73.2% uyghur, cca 1 user, 27 min. (avg) we select and label fifty target language documents by uncertainty sampling and combine them with the source language training set. clime uses the clwe refined by clime and the original training set. a+c uses the clwe refined by clime and a training set augmented by active learning. we control the number of user interactions so that active, clime, and a+c require the similar interaction time (section 4.2). the sinhalese and ilocano results are averaged over multiple users, while we only have one user for other languages. each subcaption indicates the target language, embedding alignment, number of users, and average time per user. clime has higher accuracy than active on four of the five embeddings, and the combined a+c model has the highest. ## conclusion and future work clime is an interactive system that enhances clwe for a task by asking a bilingual speaker for word-level similarity annotations. we test clime on cross-lingual information triage in international health emergencies for four low-resource languages. bilingual users can quickly improve a model with the help of clime at a faster rate than an active learning baseline. combining active learning with clime further improves the system. clime has a modular design with three components: keyword ranking, user interface, and embedding refinement. the keyword ranking and the embedding refinement modules build upon existing methods for interpreting neural networks @xcite and fine-tuning word embeddings @xcite . therefore, future advances in these areas may also improve clime. another line of future work is to investigate alternative user interfaces. for example, we could ask bilingual users to rank nearest neighbors @xcite or provide scalar grades @xcite instead of accepting/rejecting individual neighbors. we also explore a simple combination of active learning and clime. simultaneously applying both methods is better than using either alone. in the future, we plan to train a policy that dynamically combines the two interactions with reinforcement learning @xcite .