P uzz L ing M achines: A C hallenge on L earning F rom S mall D ata (2020)

Abstract: Deep neural models have repeatedly proved excellent at memorizing surface patterns from large datasets for various ML and NLP benchmarks. They struggle to achieve human-like thinking, however, because they lack the skill of iterative reasoning upon knowledge. To expose this problem in a new light, we introduce a challenge on learning from small data, PuzzLing Machines, which consists of Rosetta Stone puzzles from Linguistic Olympiads for high school students. These puzzles are carefully designed to contain only the minimal amount of parallel text necessary to deduce the form of unseen expressions. Solving them does not require external information (e.g., knowledge bases, visual signals) or linguistic expertise, but meta-linguistic awareness and deductive skills. Our challenge contains around 100 puzzles covering a wide range of linguistic phenomena from 81 languages. We show that both simple statistical algorithms and state-of-the-art deep neural models perform inadequately on this challenge, as expected. We hope that this benchmark, available at https://ukplab.github.io/PuzzLing-Machines/ , inspires further efforts towards a new paradigm in NLP—one that is grounded in human-like reasoning and understanding.

## meta-linguistics meta-linguistics is defined by @xcite as "the knowledge of the characteristics and structures of language" as realised on the level of phonology, morphology, syntax and semantics. any english speaker would likely have the linguistic capacity to produce the word undo when asked "what is the opposite of do?" only a speaker with some level of meta-linguistic awareness, however, would further be able to reflect on the structure of the word they have produced: to identify unas a unit that serves to negate words, to spot its similarity in function to other units like disand de-. he/she would also be aware that unis not interchangeable with disand de-, since it attaches to the front of verbs and adjectives but not to nouns. meta-linguistic awareness is especially useful (and often improved) in the process of learning a new language, as it allows the learner to compare and contrast the structure and characteristics of the new language to those that he/she is already familiar with. it is desirable that systems for natural language processing possess meta-linguistic awareness, too, as that could hugely improve their crosslingual generalizability, a problem that remains open after being approached from various engineering perspectives, often with little recourse to linguistics. however, measuring the meta-linguistic awareness of a system is not trivial. existing probing techniques are mostly designed to measure how well neural models capture specific linguistic phenomena, e.g., whether a specific layer of an english language model can capture that undo is negative, instead of testing for meta-linguistic awareness. our challenge takes a step further and tests whether the model can apply the underlying morphological processes, e.g. of verbal negation through prefixing. in addition, our challenge spans a wide-range of language families and covers a variety of linguistic phenomena (see §3.1), that qualifies it as a favorable testbed for measuring meta-linguistic awareness. let us demonstrate how meta-linguistic reasoning skills are used to solve the "chickasaw puzzle" given in ## the dataset the puzzles from linguistic olympiads cover many aspects of language such as phonetics, morphology, syntax and semantics. they are carefully designed by experts according to several key criteria: (1) the puzzles should be self-contained and unambiguous, meaning that no prior knowledge in the foreign language is requires, just the command of one's own native language and some level of meta-linguistic awareness and that a solution is guaranteed; (2) they should require no specialized external knowledge or formal linguistic knowledge, i.e. linguistic terms are either excluded from the instructions that accompany a puzzle or they are explicitly defined; (3) the foreign language used in a puzzle should be from a truly lesser known language family (e.g. chickasaw, lakhota, khmer, ngoni), such that there is no unfair advantage to participants whose native language is related. we based our data collection efforts on a rich and publicly available database of language puzzles maintained by the organizers of naclo. 2 this resource contains puzzles from iol and a wide range of local competitions foot_2 . we only included puzzles written in english (or translated to english) to ensure a quality transcription and to enable error analysis. expert solutions are available for most puzzles; we excluded the rest. in addition to the translation puzzle type shown in the final dataset contains 96 unique puzzles from 81 languages that span 39 different language families from all over the world, as well as two creoles and two artificial languages (see appendix a.6 for the full list). some of the large language families have multiple representatives, e.g. there are 13 indo-european languages, seven austronesian and six from the niger-congo family. but the majority of languages are single representatives of their respective family. this genealogical diversity leads to a great diversity in the linguistic phenomena attested in the data. some puzzles are designed to explore a specific aspect of the unknown language in isolation, e.g. case markers on demonstrative pronouns in hungarian @xcite . in general, however, the correct solution of a puzzle involves processing on the level of syntax, morphology, phonology, and semantics all at once. ## baselines we attemp to solve these puzzles with models of varying complexity, i.e. from random guessing to state-of-the-art neural machine translation systems. random words (rw): since the vocabularies of source and target languages are quite small, we test what random word picking can accomplish. we simply tokenize the training sentence pairs and then randomly choose a word from the target language's vocabulary for each token in the source sentence. 5 fastalign (fa): we use the translation alignment tool fastalign @xcite , to test whether the puzzles can be solved by early lexical translation models @xcite . since fa produces alignments for each training pair, we postprocess the output to create a translation dictionary separately for each direction. we then randomly choose from the translation entries for each token in source test sentence. 6 phrase based statistical machine translation (pbsmt) since koehn and knowles (2017) report that pbsmt models outperform vanilla nmt models in case of small parallel training data, we use pbsmt as one of the baselines. for the @xmath0, we implement two models-one using no external mono-lingual english data and one otherwise. ## results and analysis we report the results for the best models in fig. 2 . the hyperparameter configuration and the development set results are given in appendix a.4. the maximum exact match score among all results is only 3.4%; and the highest scores are consistently achieved by pbsmt models on both directions and dataset splits. the overall results for @xmath1. this may be due to (a) having longer sentences for english; (b) the scores (except from em) being more suitable for english (even the character-based ones) or (c) the more challenging nature of translation into foreign languages, which needs another dedicated study. @xmath2: initializing the nmt encoder with roberta has severely worsened the results, compared to standard transformer model. we believe the main reason is the imbalance between encoder (huge encoder) and the decoder (tiny decoder), that makes training very challenging. the gap between the simplest baselines (rw, @xmath3: we observe that the gap between the simple and more sophisticated baselines are higher in this direction by means of all measures, as we would expect. using roberta features in the decoder does not hurt the performance while providing a small increase in em score compared to standard transformers. it should be noted that the decoder is still tiny and lm features are only incorporated via a separate linear layer at a very late stage, which prevents the imbalance problem we saw in @xmath4. we see similar results for the validation data with the exception that transformer-based models achieve either higher or the same em scores than pbsmt while surpassing pbsmt's bleu-2 scores in @xmath5. it supports the findings of @xcite , drawing attention to the importance of hyper-parameter tuning for low-resource nmt models. ## related work recently, reasoning tasks and datasets that require natural language processing have been introduced, such as common-sense reasoning in the form of pronoun resolution e.g., wsc @xcite , multiple-choice question answering e.g., swag @xcite and arc @xcite ; inference tasks in the form of binary or multi-label classification problems e.g., the glue benchmarks @xcite ; and visual reasoning in the form of question answering @xcite 2019) e.g., gqa @xcite . in these tasks, the required level of semantics is mostly limited to single sentences rather than a collection; almost all tasks target english; data is derived from running text and is mostly close-domain. in addition, some require external knowledge bases or high-level knowledge on physical models or experiments as in arc classified by @xcite , which leaves room for accumulating errors from external parts and complicates the analysis of individual parts like reasoning. another body of early work on symbolic ai provides a different set of tools to model reasoning such as rule-engines, rule-induction algorithms, logic programs and case-based reasoning models @xcite . however, it is not trivial to represent and model our task in these frameworks, since they mostly require defining primitives, expressions, discrete features and cases. furthermore, the strength of statistical/neural models has been repeatedly shown to surpass rule-based models. our goal is to encourage researchers to incorporate reasoning into statistical models, rather than replacing them with symbolic models. ## conclusion and future work the field of nlp has developed deep neural models that can exploit large amounts of data to achieve high scores on downstream tasks. still, the field lacks models that can perform human-like reasoning and generalization. to mitigate this gap, we draw inspiration from the linguistic olympiads that challenge the meta-linguistic and reasoning abilities of high-school students. we create a new benchmark dataset from available linguistic puzzles that spans over 81 languages from 39 language families, which is released at https:// ukplab.github.io/puzzling-machines/ . we implement and evaluate simple baselines such as alignment, and state-of-the-art machine translation models with integrated a pretrained english language model. we show that none of the models can perform well on the puzzles, suggesting that we are still far from having systems with meta-linguistic awareness and reasoning capabilities.