ELQA : A Corpus of Metalinguistic Questions and Answers about E nglish (2023)

Abstract: We present ELQA, a corpus of questions and answers in and about the English language. Collected from two online forums, the >70k questions (from English learners and others) cover wide-ranging topics including grammar, meaning, fluency, and etymology. The answers include descriptions of general properties of English vocabulary and grammar as well as explanations about specific (correct and incorrect) usage examples. Unlike most NLP datasets, this corpus is metalinguistic—it consists of language about language. As such, it can facilitate investigations of the metalinguistic capabilities of NLU models, as well as educational applications in the language learning domain. To study this, we define a free-form question answering task on our dataset and conduct evaluations on multiple LLMs (Large Language Models) to analyze their capacity to generate metalinguistic answers.

## introduction language is so powerful that it can be reflected back on itself. statements like "in informal usage, a steep learning curve means something that is difficult (and takes much effort) to learn" or "in some cases, an adjective has both -ic and -ical forms, with no difference in meaning" expressly concern linguistic inventories, structures, and behaviors. in other words, they are metalinguistic-they use language to discuss language (cf. @xcite . they may concern a particular instance of language use, or properties of a language or speaker in general; either way, they are metalinguistic in making linguistic phenomena the subject matter of a linguistic utterance. for the rest of this paper, the term metalanguage is used for natural language text in which natural language is also the subject matter. while nlp models have become powerful at predicting text in many settings, it remains to be seen whether such capability extends to metalanguagewhere linguistic strings are not being deployed to contribute to the discourse with their normal denotations, but rather, are treated as entities with linguistic properties (e.g., grammar, meaning). one way this can be explored is in a question answering framework, which requires suitable datasets, ideally based on questions that are realistic and paired with high-quality answers. in this paper, we present a corpus of metalinguistic questions and answers about english. the corpus is collected and carefully processed from two stack exchange forum sites: english language & usage (eng) and english language learners (ell). it covers more than 70k questions on numerous topics about english such as grammar, meaning, fluency, and etymology along with answers. our corpus, elqa (english language questions and answers), can serve as a tool to facilitate metalinguistic studies. moreover, since questions in elqa cover a variety of topics in english, it can be used in the educational and english language learning domains. as the first case study of elqa, we investigate the performance of current state-of-the-art nlp technology on free-form question answering in the english language domain. additionally, we explore the possibility of building nlp models that can directly answer questions from language learners. we process a subset of elqa and make it appropriate for this task. then, we report on the results of both automatic and human evaluations using different experimental settings of t5 foot_0 and gpt-3 foot_1 models. although most of these models achieve high ratings for well-formedness, the validity of their answers is significantly lower than that of human-authored answers, indicating that this type of metalinguistic qa task is challenging even for large language models. our main contributions are: 1) we release the first publicly available metalinguistic qa dataset, 3 focused on the english language; 2) we present a taxonomy of questions in the corpus along with analysis; and 3) we investigate to what extent llms are able to articulate appropriate generalizations about language in response to these questions. ## related work stack exchange is a network of numerous cqa sites (originally and most famously, stack over-3 https://github.com/shabnam-b/elqa flow) built on a common platform. stack exchange forums have been featured in a number of previous datasets @xcite @xcite @xcite @xcite , including the english site (our eng) along with others such as ask ubuntu, android, gaming and wordpress (dos @xcite . we focus on eng and ell as they concern the english language itself; we show that these datasets cover a wide range of metalinguistic questions. our use of these forums contrasts with previous work on metalanguage in corpora, which annotated and quantified mentions @xcite @xcite @xcite , but did not consider entire questions and answers about language. taylor (2015) studied metalanguage in online forums, but with a focus on the usage of metalinguistic expressions of mock politeness. more recently, @xcite published the first corpus of contemporary slovene, croatian and serbian media metalanguage texts. so far, metalanguage has not been a focus in the qa domain-ours is the first publicly available english metalinguistic qa dataset. most qa tasks are set up to have a question and a reference document, where the objective is to find the answer based on the document @xcite . in this paper, we explored a type of "closed-book" question answering task @xcite . to the best of our knowledge, this task has not been explored to date within the realm of english language questions that require significant generalization and adaptation rather than looking up facts. ## constructing the dataset we collect our data from two sites on stack exchange: english language & usage (eng) 4 and english language learners (ell). 5 sample screenshots of the site are shown in figure 1 . the stack exchange data is publicly released under the cc-by-sa 3.0 license. we preprocessed the data until 2021-12-06 collected from the internet archive 6 to be suitable for nlp studies and release it as elqa. additionally, some cleanup (e.g., removing posts marked as "spam" or "offensive") was done. fields for each entry (question) include the title, body, user bio (if available), score (which is calculated based on up-votes and down-votes by other users), tags (user-assigned, related to the area/topic of the question), favorite count, and a list of answers. textual content (body and user bio) is provided in two formats: html and plain text without html tags. we release two versions of elqa based on different preprocessing steps. in elqa-large, we keep questions as long as they don't include any images (<@xmath0. for elqa-small, we applied further filtering to ensure that the data has the least amount of noise: a) questions should have a score of at least 4 https://english.stackexchange.com/ question has an answer with a score higher than 3 and c) there are no hyperlinks in at least one of the high-rated answers. the last step reduces noise and facilitates a fair comparison for the closed-book question-answering task ( §4) with model-generated answers, as models cannot be expected to have access to the web to suggest valid urls compared to humans who would search the web for appropriate resources to include in their answers. for quality assurance, we also did a human annotation on elqa-small. two of the authors annotated 250 question and answer pairs for the following: 1) is the question answerable? and 2) does the answer fully address the question? we found 99.2% of the questions answerable and 91.8% of the answers acceptable. as can be seen from the examples in ## english language question answering large language models can produce output that is fluent and (at times) informationally adequate when presented with factual questions about entities in the world @xcite . but how do such models perform when asked questions about the language itself? in this section, we investigate the free-form english language question answering task. this task has the potential to benefit educational applications for language learners. research on nlp for educational purposes has investigated tasks such as automated grammatical error correction @xcite @xcite , question and quiz generation for language learning @xcite @xcite , and automated essay scoring @xcite . nevertheless, an application that has not been taken up by the educational nlp community is free-form question answering about language. second language learners possess a degree of metalinguistic awareness about the language they are learning, and often turn to teachers or more advanced speakers with explicit questions about vocabulary, grammar, and usage. community question answering (cqa) websites such as stack exchange have sites for language learners' questions and answers. these sites require consid ## conclusion we presented elqa, a dataset containing metalinguistic questions and answers about the english language. we provided analysis and a taxonomy of the data, along with experiments on free-form answer generation and investigated the extent to which language models can articulate their generalizations about language. since many of the questions in elqa were asked by language learners, it forms a potentially useful and so far untapped resource for educational nlp purposes and metalinguistic question answering. we release the dataset to enable further studies of this task.