On the Effectiveness of Dataset Embeddings in Mono-lingual,Multi-lingual and Zero-shot Conditions (2021)

Abstract: Recent complementary strands of research have shown that leveraging information on the data source through encoding their properties into embeddings can lead to performance increase when training a single model on heterogeneous data sources. However, it remains unclear in which situations these dataset embeddings are most effective, because they are used in a large variety of settings, languages and tasks. Furthermore, it is usually assumed that gold information on the data source is available, and that the test data is from a distribution seen during training. In this work, we compare the effect of dataset embeddings in mono-lingual settings, multi-lingual settings, and with predicted data source label in a zero-shot setting. We evaluate on three morphosyntactic tasks: morphological tagging, lemmatization, and dependency parsing, and use 104 datasets, 66 languages, and two different dataset grouping strategies. Performance increases are highest when the datasets are of the same language, and we know from which distribution the test-instance is drawn. In contrast, for setups where the data is from an unseen distribution, performance increase vanishes.

## introduction the performance of natural language processing systems is dependent on the amount of training data, which is often scarce. to complement existing training data, supplementary data sources can be used. especially data annotated for the same task from other sources can be beneficial to exploit. however, because of heterogeneity in language or domain this might lead to sub-optimal performance. in early work on combining training sources, data was selected at training time @xcite for a given test set. a more nuanced way to exploit heterogeneous data is to encode properties of the language as features @xcite . recently, @xcite showed that encoding the language of an instance as an embedding in a neural model is beneficial for multi-lingual learning. 2 follow-up work found that multiple datasets within the same language can also be combined by encoding their origin @xcite , thereby implicitly learning useful commonalities, while still encoding datasetspecific knowledge. these dataset embeddings are employed in groups of datasets which usually range in size from 2 to 10 datasets. however, it remains unclear in which situations these dataset embeddings thrive best. furthermore, two often overseen issues with dataset embeddings are that they are commonly learned from the gold data-source labels attached to each training and test instance and it is assumed that the test data is from a distribution which is seen during training. in many real world situations these assumptions are clearly violated. a common strategy when the test data is drawn from a different distribution as the training datasets (zero-shot), is to use a manually assigned proxy treebank @xcite @xcite . recent work showed that for unseen datasets in mono-lingual setups @xcite , interpolated dataset embeddings can be used to improve performance for zero-shot settings. we use automatically predicted proxy data sources instead, and focus on mono-linugal as well as cross-lingual setups. in this paper, we provide an extensive evaluation of the usefulness of dataset embeddings in existing setups and beyond. more concretely, we ask: 1) what are good indicators to predict the usefulness of dataset embeddings? 2) can we effectively use dataset embeddings in the absence of gold datasource information? ## results we report results for all tasks in two main settings: in-dataset, for setups where we assume that input data is from a distribution present during training (3.1); and zero-shot, (3.2), a setup where this is not the case. for all reported experiments, we use labeled attachment score (las) for parsing @xcite , f1 score for morphological tagging, and accuracy for lemmatization. we do not perform any tuning, and thus only report results on development data (if no dev-split is available we use test). as a control, we compare dataset embeddings to a simple concat, training on concatenation of all the data sources from a dataset group without dataset embeddings. reported results are average over 3 runs for the uuparser, for the multi-team tagger we did only a single run because of the computational costs (see appendix for more details). ## conclusion we provide an extensive evaluation of dataset embeddings in two large-scale settings where they were used successfully @xcite . in setups where in-distribution training data is available, we found dataset embeddings more useful in monolingual dataset groups, compared to cross-lingual ones. in general, performance gains were the largest for 1) datasets for which another same-language datasets was available during training 2) small datasets 3) datasets which were part of a large dataset group. however, with predicted id's, their benefit is limited, contrary to gold information. when moving to zero-shot setups, the performance increases become negligible (except for some particular datasets). in particular, without in-source training data, dataset embeddings work in some cases when another treebank for the language exists; but this gain is not consistent and often small. overall, we find dataset embeddings fail to be a viable adaptation method when no insource data is available. hence, in many realistic out-of-distribution setups, their benefit vanishes. irshad bhat, riyaz a. bhat, manish shrivastava, and dipti sharma. 2017. joining hands: exploiting monolingual treebanks for parsing of code-mixing data. in proceedings of the 15th conference of the european chapter of the association for computational linguistics: volume 2, short papers, pages 324-330, valencia, spain. association for computational linguistics. c ¸agrı c ¸öltekin and taraka rama. 2018. tübingen-oslo at semeval-2018 task 2: svms perform better than rnns in emoji prediction. in proceedings of the 12th international workshop on semantic evaluation, pages 34-38, new orleans, louisiana. association for computational linguistics. alexis conneau and guillaume lample. 2019. crosslingual language model pretraining. in advances in neural information processing systems, pages 7059-7069. yoav goldberg and joakim nivre. 2013. training deterministic parsers with non-deterministic oracles. transactions of the association for computational linguistics, 1:403-414. alex graves and jürgen schmidhuber. 2005. framewise phoneme classification with bidirectional lstm and other neural network architectures. neural networks, 18(5-6):602-610. mohammad khan, markus dickinson, and sandra kübler. 2013. towards domain adaptation for parsing web data. in proceedings of the international conference recent advances in natural language processing ranlp 2013, pages 357-364, hissar, bulgaria. incoma ltd. shoumen, bulgaria. christo kirov, ryan cotterell, john sylak-glassman, géraldine walther, ekaterina vylomova, patrick xia, manaal faruqui, sabrina j. mielke, arya mc-carthy, sandra kübler, david yarowsky, jason eisner, and mans hulden. 2018. unimorph 2.0: universal morphology. in proceedings of the eleventh international conference on language resources and evaluation (lrec 2018), miyazaki, japan. european language resources association (elra). marco kuhlmann, carlos gómez-rodríguez, and giorgio satta. 2011. dynamic programming algorithms for transition-based dependency parsers. in proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 673-682, portland, oregon, usa. association for computational linguistics. miryam de lhoneux, sara stymne, and joakim nivre. 2017. arc-hybrid non-projective dependency parsing with a static-dynamic oracle. in proceedings of the 15th international conference on parsing technologies, pages 99-104, pisa, italy. association for computational linguistics. yu-hsiang lin, chian-yu chen, jean lee, zirui li, yuyan zhang, mengzhou xia, shruti rijhwani, junxian he, zhisong zhang, xuezhe ma, antonios anastasopoulos, patrick littell, and graham neubig. 2019. choosing transfer languages for cross-lingual learning. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 3125-3135, florence, italy. association for computational linguistics. arya d. mccarthy, ekaterina vylomova, shijie wu, chaitanya malaviya, lawrence wolf-sonkin, garrett nicolai, christo kirov, miikka silfverberg, sebastian j. mielke, jeffrey heinz, ryan cotterell, and mans hulden. 2019. the sigmorphon 2019 shared task: morphological analysis in context and cross-lingual transfer for inflection. in proceedings of the 16th workshop on computational research in phonetics, phonology, and morphology, pages 229-244, florence, italy. association for computational linguistics. maria medvedeva, martin kroon, and barbara plank. 2017. when sparse traditional models outperform dense neural networks: the curious case of discriminating between similar languages. in proceedings of the fourth workshop on nlp for similar languages, varieties and dialects (vardial), pages 156-163, valencia, spain. association for computational linguistics. ailsa meechan-maddon and joakim nivre. 2019. how to parse low-resource languages: cross-lingual parsing, target language annotation, or both? in proceedings of the fifth international conference on dependency linguistics (depling, syntaxfest 2019), pages 112-120, paris, france. association for computational linguistics. tahira naseem, regina barzilay, and amir globerson. 2012. selective sharing for multilingual dependency parsing. in proceedings of the 50th annual meeting of the association for computational linguistics (volume 1: long papers), pages 629-637, jeju island, korea. association for computational linguistics. joakim nivre, marie-catherine de marneffe, filip ginter, jan hajic, christopher d. manning, sampo pyysalo, sebastian schuster, francis tyers, and daniel zeman. 2020. universal dependencies v2: an evergrowing multilingual treebank collection. in proceedings of the 12th language resources and evaluation conference, pages 4034-4043, marseille, france. european language resources association. karl pearson. 1901. liii. on lines and planes of closest fit to systems of points in space. the london, edinburgh, and dublin philosophical magazine and journal of science, 2(11):559-572. f. pedregosa, g. varoquaux, a. gramfort, v. michel, b. thirion, o. grisel, m. blondel, p. prettenhofer, r. weiss, v. dubourg, j. vanderplas, a. passos, d. cournapeau, m. brucher, m. perrot, and e. duchesnay. 2011. scikit-learn: machine learning in python. journal of machine learning research, 12:2825-2830. barbara plank and gertjan van noord. 2011. effective measures of domain similarity for parsing. in proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 1566-1576, portland, oregon, usa. association for computational linguistics. vinit ravishankar. 2018. parsing of texts with codeswitching. master's thesis, institute of formal and applied linguisti, prague. aaron smith, bernd bohnet, miryam de lhoneux, joakim nivre, yan shao, and sara stymne. 2018. 82 treebanks, 34 models: universal dependency parsing with multi-treebank models. in proceedings of the conll 2018 shared task: multilingual parsing from raw text to universal dependencies, pages 113-123, brussels, belgium. association for computational linguistics. sara stymne, miryam de lhoneux, aaron smith, and joakim nivre. 2018. parser training with heterogeneous treebanks. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), pages 619-625, melbourne, australia. association for computational linguistics. ahmet üstün, rob van der goot, gosse bouma, and gertjan van noord. 2019. multi-team: a multiattention, multi-decoder approach to morphological analysis. in proceedings of the 16th workshop on computational research in phonetics, phonology, and morphology, pages 35-49, florence, italy. association for computational linguistics. joachim wagner, james barry, and jennifer foster. 2020. treebank embedding vectors for out-ofdomain dependency parsing. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 8812-8818, online. association for computational linguistics. marcos zampieri, shervin malmasi, nikola ljubešić, preslav nakov, ahmed ali, jörg tiedemann, yves scherrer, and noëmi aepli. 2017. findings of the vardial evaluation campaign 2017. in proceedings of the fourth workshop on nlp for similar languages, varieties and dialects (vardial), pages 1-15, valencia, spain. association for computational linguistics. daniel zeman, jan hajič, martin popel, martin potthast, milan straka, filip ginter, joakim nivre, and slav petrov. 2018. conll 2018 shared task: multilingual parsing from raw text to universal dependencies. in proceedings of the conll 2018 shared task: multilingual parsing from raw text to universal dependencies, pages 1-21, brussels, belgium. association for computational linguistics.