To Ask LLM s about E nglish Grammaticality, Prompt Them in a Different Language (2024)

Abstract: In addition to asking questions about facts in the world, some internet users—in particular, second language learners—ask questions about language itself. Depending on their proficiency level and audience, they may pose these questions in an L1 (first language) or an L2 (second language). We investigate how multilingual LLMs perform at crosslingual metalinguistic question answering. Focusing on binary questions about sentence grammaticality constructed from error-annotated learner corpora, we prompt three LLMs (Aya, Llama, and GPT) in multiple languages, including English, German, Korean, Russian, and Ukrainian. Our study reveals that the language of the prompt can significantly affect model performance, and despite English being the dominant training language for all three models, prompting in a different language with questions about English often yields better results.

## introduction mastering a second language is a skill acquired over time. writing or speaking with proper grammar in a second language can be challenging, primarily due to the complexities of form, meaning, and the relationship between the two @xcite . learners often become confused when constructing sentences and seek help from teachers, peers, or online resources for clarity. many of these queries relate to grammar; in fact, the grammar tag is the most popular one on online question-andanswer forums like english language learners stackexchange. 1 behzad et al. (2023) collected elqa, a dataset of english questions and answers from such forums and their study showed that around 38% of the questions are fluency questions. a common format within the fluency category involves com-1 https://ell.stackexchange.com/tags paring multiple sentences to determine which one is correct: example 1 which one is grammatically correct? he is believed to be awarded the prize at the age of 17 when he was in london. or he is believed to have been awarded the prize at the age of 17 when he was in london. 2 behzad et al. (2023) investigate the capabilities of large language models (llms) in responding to learner questions including fluency and grammar questions and demonstrate that, although the performance has not yet reached human levels, the answers are accurate in many instances. this study was limited to asking questions in english and did not take into account that some learners may prefer to interact with llms in their native language, especially when they are at a novice level of proficiency. moreover, @xcite study only included questions about the english language. this motivated us to study the capabilities of llms in a multilingual setting as a first step towards building better interactive multilingual nlp technologies that can help learners practice a new language and answer their questions. since learner question answering datasets similar to elqa are not available in other languages, and since we also wanted to ensure a controlled multilingual comparison, we synthetically created them by filling a template with naturally occurring sentences written by l2 learners of different languages. these sentences are selected and processed from grammatical error correction (gec) datasets, ensuring they follow the same error patterns as those made by learners in the real world. using this template format, we experiment with multiple llms, comparing their performance across different languages from several perspec-tives. in summary, we answer the following research questions in this paper: ## related work as large language models (llms) impact communities worldwide, researchers are increasingly studying their behavior across different languages and tasks @xcite @xcite @xcite . ahuja et al. (2023) introduced mega for benchmarking llms, covering 16 nlp datasets across 70 typologically diverse languages. tasks include classification, question answering, sequence labeling, generation (summarization), and responsible ai. the authors show that there is a consistent performance gap between high-resource, latin script, and low-resourced languages. lai et al. (2023) studied chatgpt's behavior in a multilingual setting for several tasks (pos tagging, named entity recognition, relation extraction, natural language inference, question answering, common sense reasoning, and summarization) and they argue that english task descriptions for non-english inputs can be better processed by chatgpt, although other studies argue otherwise @xcite @xcite . our focus in this paper is grammatical question answering for l2 language learners. a similar task to the task studied in this paper is the grammatical error correction (gec) task in which the goal is to automatically identify and correct errors in a given text. the field has made remarkable progress over the past decade, largely inspired by multiple shared tasks introducing new benchmarks and agreedupon standards for evaluating gec systems @xcite @xcite @xcite . so far, core approaches to gec include rulebased, classification and classic machine learning approaches, statistical and neural machine translation, and edit-based approaches @xcite . more recently, the growing prominence of llms has led researchers to increasingly use them as zero-shot or few-shot generators for the error correction task in english @xcite or other languages such as chinese @xcite , @xcite and korean @xcite . most of these studies focus on gpt family models and many have come to the conclusion that the results are not yet satisfactory because even though output sentences are fluent and grammatical, llms tend to over-correct sentences as they go beyond minimal edits @xcite @xcite . @xcite reveals that for multiple languages, including czech, german, russian, spanish, and ukrainian, gpt makes significant alterations to the source sentences, and sometimes even changes their semantics which creates major challenges for evaluation with reference-based metrics. automatic evaluation in text generation has been a challenge for some time @xcite , and gec is no exception @xcite @xcite . ## proposed study to evaluate llms' multilingual capabilities in l2 learner question answering, we create a binary question answering template that is easily adaptable to different languages. the template is similar to patterns observed by @xcite and shown in example 1. the body of the question in english is which sentence is more grammatical and native-like? this can be translated to any other language, such as german: welcher satz ist grammatikalischer und muttersprachlicher? therefore, we can extend the template to encompass other languages. the question body comprises two sentences: one grammatical and one containing an error. the order is randomized to establish 50% as the accuracy expected due to chance. this template format also allows us to avoid the challenges associated with reference-based text generation evaluation. while simple, this step is crucial for developing question-answering systems for learners. before generating any type of explanation for the learner, we must first confirm the model's capability to differentiate between erroneous and grammatical sentences. to ensure experimentation with learner-like data, we incorporate sentence pairs from gec benchmarks into our template. this setup maintains highquality data, and simplifies the process of switching between different languages, both for the question itself and for the example sentences. examples of our benchmark instances are available in ## evaluation and results in this section, we report and discuss our findings. examining the results from experimenting with all combinations of languages, we observe the highest accuracy for grammar questions about english is 80% using llama; for questions about german it is 84% using gpt; korean, 69.6% using llama; russian, 78.4% using gpt; and ukrainian, 62.4% using gpt (results for all different combinations are available in appendix a, next, we answer the following research questions based on our empirical results: are llms consistent in performance when the same question is asked in different languages? no. to evaluate this, we keep english sentence pairs constant, but translate the first part of the prompt ("which sentence is more grammatical and nativelike?") into different languages. since the knowledge required to answer the question remains the same (english grammar) we expect language models to perform consistently among different languages. however, results in first, we note that aya's performance is below random chance (all test benchmarks are balanced) in all languages except for english, where it is roughly at chance. llama performs fairly well in all languages, getting the highest accuracy (80%) when the question is asked in german, which is 8 percentage points higher than when it is asked in english. gpt also performs better when the question is asked in a different language. there is a 12% increase in accuracy when question is asked in ukrainian instead of english. this is particularly interesting since, for both models, english is the dominant language in the training data. we hypothesize that since the model is heavily fine-tuned for a variety of english tasks that are not metalinguistic, the english prompts are less likely to be interpreted metalinguistically. so when the model encounters different languages, it anticipates metalinguistic usage and also focuses more on sentence structure. another possible explanation could be that these models undergo extensive instruction fine-tuning specifically for english tasks. this intensive fine-tuning may potentially impair some of the model's broader language modeling capabilities, particularly within english contexts. in contrast, the performance observed in other languages may reflect the advantage of less instruction fine-tuning, especially for less common tasks. do llms perform better when questions are asked in specific languages? yes. in except for questions about russian grammar, gpt seems to perform better when the question is asked in ukrainian rather than in the same language as the sentence pair. we hypothesize that the reason in deu about rus gives better performance than in ukr about rus is because ukrainian and russian both use the cyrillic script, whereas a different script such as latin may help the model's attention. it is interesting to note that not all different scripts work similarly; for instance, gpt achieves 70% accuracy in the in kor about rus setup. we setup llama aya gpt in ukr about ukr 56.0 46.0 60.0 in eng about ukr 55.6 48.4 60.0 in deu about ukr 58.4 50.8 54.8 in rus about ukr 60.0 49.2 62.4 in kor about ukr 57.2 45.6 49.2 with llama, it appears that code-switching generally improves performance, particularly with ukrainian grammar, which is considered a midresource language. there is an 11.6% performance boost when questions are asked in german rather than in ukrainian. does prompting in a high-resource language aid grammatical error detection in a lower-resource language? yes (for ukrainian). in gpt achieves its highest performance when the question is asked in russian. similarly, llama also performs best when the question is asked in russian. these findings suggest that in studies involving grammar knowledge in a mid-resource language and when models are less finetuned on a language, prompting in a high-resource language with the same script can enhance performance. we focused solely on ukranian in this study and, therefore, do not intend to make broad claims at this stage. we believe this hypothesis should be validated across different languages before drawing any major conclusions. are the same sentence pairs equally challenging for different models? for some datasets, yes. for this analysis, we compare the predictions of llama and gpt and compute agreement percentage. results can be found in figure 1 . first looking at kor-learner, we observe that, regardless of the language of the prompt, models exhibit the most disagreements on korean grammar, highlighting distinct strengths and weaknesses between llama and gpt in this language. next, we observe that the two models have the least number of disagreements on ukrainian grammar. looking back at chain-of-thought prompting @xcite ) has improved performance on tasks like math problems and common sense reasoning. prior work shows that debating or providing feedback to llms can enhance their performance @xcite @xcite . here, we investigate with a small-scale experiment how models explain an incorrect prediction, and whether they would change their answer when asked for an explanation. we experiment with both llama and gpt, focusing on the in eng about eng setup due to budgetary constraints. if the model selects the incorrect sentence, we follow up with the question: can you explain why? one of the authors manually reviewed 50 examples from each model. both models corrected their responses in more than 60% of cases. for the remaining cases, two main issues were ob-served: 1) both sentences were correct but conveyed slightly different meanings (30% of cases. example: first row in ## analysis and discussion in this section, we take a look at cases where llms struggle. mainly, we study this from two perspectives: i) the proficiency level of the learner who wrote the sentence and ii) the type of error that exists in the sentence. ## conclusion in this paper, we study the multilingual capabilities of several language models in a learner metalinguistic question answering task. our results show that i) there is still room for improvement in this task across different languages including english; ii) models are too sensitive to the prompt and the language of the prompt; and iii) asking questions in languages other than english can help with performance, even for questions about english grammar. a particularly worrying result, which may be counter-intuitive to downstream users of llms, is the substantial degree of variance in how models respond to the same sentence pairs based purely on the language in which the question is posed. here we see a clear trend favoring asking in a different language, with questions in a different script showing particularly notable gains, possibly because they sensitize the model to a meta-linguistic environment (i.e. asking questions about a sentence, rather than asking the model to respond to the contents of the sentence itself). we hope the findings of this research may shed light on the challenges of multilingual, metalinguistic q&a and enable future work on this task. that results vary widely across models-without a clear correlation with parameter count, language, or error type-suggests that more attention needs to be paid to metalinguistic q&a in order to obtain satisfactory and consistent performance.