S yntax G ym: An Online Platform for Targeted Evaluation of Language Models (2020)

Abstract: Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models. However, this line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technical proficiency needed to train and deploy large-scale language models. We present SyntaxGym, an online platform designed to make targeted evaluations accessible to both experts in NLP and linguistics, reproducible across computing environments, and standardized following the norms of psycholinguistic experimental design. This paper releases two tools of independent value for the computational linguistics community: 1. A website, syntaxgym.org, which centralizes the process of targeted syntactic evaluation and provides easy tools for analysis and visualization; 2. Two command-line tools, ‘syntaxgym‘ and ‘lm-zoo‘, which allow any user to reproduce targeted syntactic evaluations and general language model inference on their own machine.

## introduction recent work in evaluating neural network language models focuses on investigating models' fine-grained prediction behavior on carefully designed examples. unlike broad-coverage language modeling metrics such as perplexity, these evaluations are targeted to reveal whether models have learned specific knowledge about the syntactic structure of language (see e.g. @xcite @xcite . research in this line of work requires an uncommon intersection of skills: a) the engineering strength of nlp researchers necessary to train and ## background before presenting the syntaxgym framework, we briefly introduce the targeted syntactic evaluation paradigm as a way to assess the quality of neural language models. ## syntaxgym the targeted syntactic evaluation paradigm allows us to focus on highly specific measures of language modeling performance, which more directly distinguish models with human-like representations of syntactic structure. syntaxgym was designed to serve as a central repository for these evaluations, and to make the evaluations reproducible and accessible for users without the necessary technical skills or computational resources. section 3.1 first describes the standards we designed for specifying and executing these targeted syntactic evaluations. section 3.2 then offers a tour of the syntaxgym site, which is built around these standards. ## command-line tools while the syntaxgym website allows for easy centralization of test suites and public access to results, all of its underlying infrastructure is also available independently for researchers to use. we release two command-line tools, available to any user with python and docker installed. ## conclusion this paper presented syntaxgym, an online platform and open-source framework for targeted syntactic evaluation of neural network language models. syntaxgym promises to advance the progress of language model evaluation by uniting the theoretical expertise of linguists with the technical skills of nlp researchers. the site is fully functional at syntaxgym.org, and the entire framework is available as open-source code. syntaxgym is continually evolving: we plan to add new features to the site, and to develop further in response to user feedback. in particular, we plan to incorporate human performance as a reference metric, integrating psycholinguistic experimental results and supporting easy experimental design starting from the test suite format. we also plan to further incorporate language models into the lm-zoo tool, allowing broader access to state-of-the-art language models in general. we welcome open-source contributions to the website and to the general framework, and especially encourage the nlp community to contribute their models to the lm-zoo repository.