How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation (2020)

Abstract: Sentence encoders map sentences to real valued vectors for use in downstream applications. To peek into these representations—e.g., to increase interpretability of their results—probing tasks have been designed which query them for linguistic knowledge. However, designing probing tasks for lesser-resourced languages is tricky, because these often lack largescale annotated data or (high-quality) dependency parsers as a prerequisite of probing task design in English. To investigate how to probe sentence embeddings in such cases, we investigate sensitivity of probing task results to structural design choices, conducting the first such large scale study. We show that design choices like size of the annotated probing dataset and type of classifier used for evaluation do (sometimes substantially) influence probing outcomes. We then probe embeddings in a multilingual setup with design choices that lie in a ‘stable region’, as we identify for English, and find that results on English do not transfer to other languages. Fairer and more comprehensive sentence-level probing evaluation should thus be carried out on multiple languages in the future.

## introduction sentence embeddings (a.k.a. sentence encoders) have become ubiquitous in nlp @xcite , extending the concept of word embeddings to the sentence level. in the context of recent efforts to open the black box of deep learning models and representations @xcite , it has also become fashionable to probe sentence embeddings for the linguistic information signals they contain @xcite , as this may not be clear from their performances in downstream tasks. such probes are linguistic micro tasks-like detecting the length of a sentence or its dependency tree depth-that have to be solved by the majority of approaches for probing sentence embeddings target english, but recently some works have also addressed other languages such as polish, russian, or spanish in a multiand cross-lingual setup @xcite . motivations for considering a multi-lingual analysis include knowing whether findings from english transfer to other languages and determining a universal set of probing tasks that suits multiple languages, e.g., with richer morphology and freer word order. our work is also inspired by probing sentence encoders in multiple (particularly low-resource) languages. we are especially interested in the formal structure of probing task design in this context. namely, when designing probing tasks for lowresource languages, some questions arise naturally that are less critical in english. one of them is the size of training data for probing tasks, as this training data typically needs to be (automatically or manually) annotated, an inherent obstacle in low-resource settings. 1 thus, at first, we ask for the training data size required for obtaining reliable probing task results. this question is also relevant for english: on the one hand, @xcite claim that training data for a probing task should be plentiful, as otherwise (highly parametrized) classifiers on top of representations may be unable to extract the relevant information signals; on the other hand, hewitt and liang (2019) note that a sufficiently powerful classifier with enough training data can in principle learn any task, without this necessarily allowing to conclude that the representations adequately store the linguistic signal under scrutiny. second, we ask how stable probing task results are across different classifiers (e.g., mlp vs. naive bayes). this question is closely related to the question about size, since different classifiers have different sensitivities to data size; especially deep models are claimed to require more training data. we evaluate the sensitivity of probing task results to the two outlined parameters-which are mere machine learning design choices that do not affect the linguistic content stored in the sentence representations under scrutiny-and then determine a 'region of stability' for english (en), where outcomes are predicted to be similar for the majority of parameter choices. overall, our research questions are: these questions are important because they indicate whether or not probing tasks (and their relation to downstream tasks) have to be re-evaluated in languages other than en. foot_2 2 related work our goal is to probe for sentence-level linguistic knowledge encoded in sentence embeddings @xcite in a multilingual setup which marginalizes out the effects of probing task design choices when comparing sentence representations. sentence embeddings have become central for representing texts beyond the word level, e.g., in small data scenarios, where it is difficult to induce good higher-level text representations from word embeddings @xcite or for clustering or text retrieval applications @xcite . to standardize the comparison of sentence embeddings, @xcite proposed the senteval framework for evaluating the quality of sentence embeddings on a range of downstream and 10 probing tasks. probing tasks are used to introspect embeddings for linguistic knowledge, by taking "probes" as dedicated syntactic or semantic micro tasks @xcite . as opposed to an evaluation in downstream applications or benchmarks like glue @xcite , probing tasks target very specific linguistic knowledge which may otherwise be confounded in downstream applications. since they are artificial tasks, they can also be better controlled for to avoid dataset biases and artifacts. probing is typically either executed on type/token (word) @xcite or sentence level @xcite . for sentence level evaluation, senteval thus far only includes en data. each probing task in senteval is balanced and has 100k train, 10k dev, and 10k test instances. the effects of these design choices are unclear, which is why our work addresses their influence systematically. in the multilingual setting, sahin et al. (2019) propose 15 token and type level probing tasks. their probing task data is sourced from unimorph 2.0 @xcite , universal dependency treebanks @xcite and wikipedia word frequency lists. to deal with lower-resourced languages, they only use 10k samples per probing task/language pair (7k/2k/1k for train/dev/test) and exclude task/language pairs for which this amount cannot be generated. their final experi-ments are carried out on five languages (finnish, german, spanish, ru, tr), for which enough training data is available. they find that for morphologically rich (agglutinative) languages, several probing tasks positively correlate with downstream applications. our work also investigates correlation between probing and downstream performance, but we do so on sentence level. on sentence level, @xcite train an infersent-like encoder @xcite on en and map this encoder to four languages (ru, french, german, spanish) using parallel data. subsequently, they probe the encoders on the probing tasks proposed by @xcite using wikipedia data for each language, with the same size of probing task data as in senteval, i.e., 100k/10k/10k for train/dev/test. their interest is in whether probing tasks results are higher/lower compared to en scores. they find particularly the ru probing scores to be low, which they speculate to be an artifact of cross-lingual word embedding induction and the language distance of ru to en. in contrast to us, their focus is particularly on the effect of transferring sentence representations from en to other languages. the problem of such an analysis is that results may be affected by the nature of the cross-lingual mapping techniques. krasnowska-kieraś and wróblewska (2019) probe sentence encoders in en and polish (pl). they use tasks defined in @xcite but slightly modify them (e.g., replacing dependency with constituency trees), reject some tasks (bigram-shift, as word order may play a minor role in pl), and add two new tasks (voice and sentence type). since pl data is less abundant, they shrink the size of the pl datasets to 75k/7.5k/7.5k for train/dev/test and, for consistency, do the same for en. they extract probing datasets from an enpl parallel corpus using combo for dependency parsing @xcite . they find that en and pl probing results mostly agree, i.e., encoders store the same linguistic information across the two languages. ## approach in the absence of ground truth, our main interest is in a 'stable' structural setup for probing task design-with the end goal of applying this design to multilingual probing analyses (keeping their restrictions, e.g., small data sizes, in mind). to this end, we consider a 2d space x comprising probing data size and classifier choice for probing tasks. 3 for a selected set of points p 0 , p 1 , . . . in x , we evaluate all our encoders on p i , and determine the 'outcomes' o i (e.g., ranking) of the encoders at p i . we consider a setup p i as stable if outcome o i is shared by a majority of other settings p j . this can be considered a region of agreement, similarly to inter-annotator agreement @xcite . in other words, we identify 'ideal' test conditions by minimizing the influence of parameters p i on the outcome o i . below, we will approximate these intuitions using correlation. ## experiments experimental setup to the senteval toolkit @xcite , which addresses both probing and downstream tasks and offers logistic regression (lr) and mlp classifiers on top of representations, we added implementations of random forest (rf) and naive bayes (nb) from scikit-learn as other popular but 'simple' classifiers. senteval defines specific model validation techniques for each task. following sen-teval, we tune the size of the hidden layer in {50, 100, 200}, dropout in {0.0, 0.1, 0.2} and l 2 regularization in {10 -5 , 10 -1 } when training an mlp. for rf, we tune the maximum tree depth in {10, 50, 100, ∞}. for lr, we tune the l 2 regularization in {10 -5 , 10 -1 }. we do not tune any hyperparameters for nb. for all probing tasks and trec, we use predefined splits. for am and sentiment analysis, we use 10-fold inner cross validation.