Using C - LARA to evaluate GPT -4’s multilingual processing (2023)

Abstract: We present a cross-linguistic study in which the open source C-LARA platform was used to evaluate GPT-4’s ability to perform several key tasks relevant to Computer Assisted Language Learning. For each of the languages English, Farsi, Faroese, Mandarin and Russian, we instructed GPT-4, through C-LARA, to write six different texts, using prompts chosen to obtain texts of widely differing character. We then further instructed GPT-4 to annotate each text with segmentation markup, glosses and lemma/part-of-speech information; native speakers hand-corrected the texts and annotations to obtain error rates on the different component tasks. The C-LARA platform makes it easy to combine the results into a single multimodal document, further facilitating checking of their correctness. GPT-4’s performance varied widely across languages and processing tasks, but performance on different text genres was roughly comparable. In some cases, most notably glossing of English text, we found that GPT-4 was consistently able to revise its annotations to improve them.

## introduction and motivation as soon as chatgpt became available in november 2022, it was obvious that there were huge im- * * authors in alphabetical order. given the wide variability in chatgpt-4's performance, we were curious to obtain a more nuanced understanding of the issues involved. in this paper, we use the open source c-lara platform @xcite to carry out an initial cross-linguistic study. c-lara, a reimplementation of the earlier lara @xcite , uses the underlying gpt-4 model to create multimodal texts designed to support learner readers, performing all the key operations: it writes the l2 text, segments it into lexical units, glosses it in the designated l1, and adds lemma and part-of-speech tags. support is provided so that the user can easily edit the output and compare different versions. it is thus straightforward to get an initial estimate of chatgpt's ability to perform several key callrelated tasks, in the context of building potentially useful learning resources. the rest of the paper is organised as follows. section 2 briefly describes c-lara. section 3 presents the experiments and results, and section 4 discusses their significance. the final section concludes and suggests further directions. ## c-lara c-lara ("chatgpt-based learning and reading assistant"; @xcite ) is an international open source project initiated in march 2023 and currently involving partners in australia, china, iceland, iran, ireland, israel and the netherlands. the goal was to perform a complete reimplementation of the earlier lara project @xcite , keeping the same basic functionality of providing a flexible online tool for creating multimodal texts, but adding chatgpt-4 as the central component. chatgpt-4 is used in two separate and complementary ways. in the form of gpt-4, it appears as a software component, giving the user the option of letting it perform the central language processing operations; it also appears as a software engineer, working together with human collaborators to build the platform itself. as described in the initial c-lara report @xcite , the software engineering aspect has proven very successful, with chatgpt not only writing about 90% of the code, but greatly improving it compared to the earlier lara codebase. in the present paper, however, our concern will be exclusively with chatgpt's performance as a language processing component. c-lara is a web app implemented in python/django. 1 an initial deployment for testing and development purposes is currently hosted on the heroku cloud platform, foot_1 and was used to perform the experiments described here. the functionality which will primarily concern us is that used in the sequence of operations which create and annotate a new piece of multimedia content. as outlined in appendix a of @xcite , the user starts by opening a new project. they then move to a screen where they provide a prompt instructing chatgpt-4 to produce the plain text. the following screens are used to add annotations to the plain text, in the sequence segmentation, followed by glossing and lemma/part-of-speech tagging. we describe each of these operations. in the segmentation phase, c-lara passes the plain text to gpt-4, together with instructions requesting it to be divided into sentence-like segments, with words further divided when appropriate into smaller units. the prompt used to make this request is created from a template, which is instantiated with both the text to be segmented and a list of few-shot examples primarily illustrating how words are to be split up. the templates and sets of examples can be made language-specific. for example, in swedish they show how compound nouns should be split into smaller components, and in french they show how clitics should be split off verbs. for mandarin, where text is normally written without interword spaces, segmentation is an important and well-studied problem @xcite @xcite , and c-lara also includes an integration of the popular jieba chinese segmentation package. 3 in the glossing phase, c-lara passes the segmented text to gpt-4, formatting it as a jsonencoded list and requesting a response in the form of a list of ⟨word, gloss⟩ pairs. the request is again created from a template instantiated with the list to be processed and a few-shot set of examples. the lemma-tagging phase is similar, with a json-formatted list passed to the ai and a list of ⟨word, lemma, pos-tag⟩ triples returned, where the pos-tag is taken from the universal dependencies v2 tagset @xcite . post-editing plain text: they lived with their mother in a sand-bank. ## experiments and results using the c-lara infrastructure outlined in the previous section, we created six short annotated texts in each of the languages english, faroese, farsi, mandarin and russian. in all languages, the texts were generated by the prompts shown in in some cases, we also experimented with using the "improvement" operation. due to limited time (hand-correcting the texts is quite laborious), we concentrated on three operations where "improvement" appeared to be having a positive effect, or the original error rate was high: english glossing, faroese segmentation, and farsi writing. all experiments were carried out in august and early september 2023, using the versions of gpt-4 current at the time. in all the experiments, a native speaker of the text the most contentious phase in this respect was glossing; it is sometimes impossible to say either that a gloss is categorically correct or that it is categorically incorrect. two important borderline cases are multi-words and grammatical constraints, where we made choices in opposite directions. we marked glosses as incorrect when they did not respect intuitive classification of words as components of multi-word expressions. thus for example in the en/fr glossing a#un# classic#classique# fairy#conte de fées# tale#histoire# we considered the gloss histoire added to tale as wrong and corrected it to conte de fées; this is a french phrase that means "fairy tale", and thus needs to be attached to both fairy and tale. in contrast, since glossing is not translation, we considered that we did not need to require glosses to respect all potentially applicable grammatical constraints, as long as they conveyed meaning correctly. so in the example a#un# cozy#confortable# little#petite# house#maison# we accepted the gloss un on a, even though un is the masculine form, and in a translation would be required to agree with feminine petite and maison. of course, it is clearly preferable here to gloss a with the feminine form une. we return to these issues in sections 4.4 and 4.5. ## discussion we divide up the discussion under a number of headings: variation across languages, variation across genre, variation across processing phase, types of problems, the "improvement" operation, random variability, and languagespecific/qualitative aspects. ## conclusions and further directions in general, c-lara seems to be a good environment for investigating aspects of gpt-4's linguistic performance more complex than simply writing text. a publicly available version of the platform, hosted at the university of south australia, will be released before the date of the conference. the material presented in this paper should only be considered a preliminary study: obviously, one would ideally use more than five languages and multiple annotators. but given the rapid evolution of chatgpt, it seemed more important to prioritise speed, and quickly gain some insight into the largescale patterns. we summarise what we consider the main results. the study examined the four tasks of writing, segmenting, glossing and lemma-tagging, all of which are key to a wide variety of text-based call systems. there is a great deal of variation across languages, and a great deal of random variation in general. however, for languages given a high enough priority by openai, gpt-4 can write engaging, fluent text with an error rate of well under 1%, and perform the glossing and lemma-tagging tasks with average error rates in the mid single digits. english is not the only language in the high-priority group: mandarin appears to be another. it is important to note that there are no generally available packages that can perform these tasks well, since they do not take proper account of multi-words, of key importance in call applications. we generated texts in six widely different domains, with roughly equal results cross-domain. this suggests that gpt-4's abilities are quite wide-ranging. for some tasks, including the common and important one of glossing english, it is possible to improve performance substantially by instructing gpt-4 to revise its output.