Large Language Models in Linguistic Research: the Pilot and the Copilot (2024)

Abstract: In this paper, we present two experiments focussing on linguistic classification and annotation of examples, using zero-shot prompting. The aim is to show how large language models can confirm or reject the linguistic judgements of experts in order to increase the productivity of their work. In the first experiment, new lexical units evoking a particular FrameNet semantic frame are selected simultaneously with the annotation of examples with the core frame elements. The second experiment attempts to categorise verbs into the aspectual classes, assuming that only certain combinations of verbs belonging to different aspectual classes evoke a semantic frame. The linguistic theories underlying the two experiments, the development of the prompts and the results of the experiments are presented.

## introduction until recently, natural language processing (nlp) relied on specialised language resources such as monolingual, bilingual and multilingual corpora as well as lexical and conceptual resources to develop functional applications. however, breakthroughs in artificial intelligence and the emergence of large language models (llm) have changed the field, enabling the successful completion of a variety of nlp tasks in a completely different way. widely studied applications of llms include document intelligence tasks, such as sentiment analysis @xcite ), text classification @xcite , risk prediction @xcite , information extraction @xcite , and many others. in addition, llms are used for machine translation @xcite , content creation tasks such as creative writing, automatic sentence completion, paraphrasing, personalised decision making and code generation. llms play a crucial role in virtual assistants that facilitate various applications such as language understanding, speech generation and speech recognition @xcite . the increasing use of large language models can be expected not only for traditional nlp tasks, but also to tackle typical linguistic challenges. the recent application of llms in linguistic research can be outlined in several directions. most attempts aim at using llms for linguistic annotation to facilitate corpus-based linguistic studies by automatically annotating texts with targeted linguistic information @xcite , among others, but there are also attempts to use llms for theoretical investigations within specific linguistic frameworks @xcite . in this paper we offer two experiments focusing on the linguistic classification and annotation of examples. the aim is to show how some newly released llms can confirm or reject the linguistic hypotheses of experts in order to increase the productivity of their work. in the first experiment, new proposals for lexical units evoking a particular framenet semantic frame are classified simultaneously with the annotation of examples with the core frame elements. in the second experiment, verb lexical units are categorised into different semantic classes, whereby it is assumed that a certain semantic frame can only be evoked by verbs that belong to certain combinations of semantic classes. the linguistic theories on which the two experiments are based, the development of the prompts and the results of the experiments are briefly presented. the paper deals with the following topics: section 2 gives an overview of the use of large language models in linguistic research. section 3 briefly introduces the large language models used in the experiments. section 4 describes experiments with two types of prompts: a) for the si-multaneous augmentation of framenet semantic frames with new lexical units and annotated examples and b) for the classification of lexical units into semantic classes. the presentation of both experiments includes a description of the linguistic theories, the prompt structures and an overview of the results. section 5 contains a conclusion on the benefits and challenges of using llms in linguistic research based on the experiments conducted. ## llms and linguistic research prompt engineering has already been used to solve linguistically relevant tasks. in few-shot learning, only a few examples are given to the model during inference and task definition @xcite : 5-7). one-shot learning and zero-shot learning are similar to few-shot learning @xcite , but as the names suggest, only one or zero demonstrations are allowed to formulate a task description in natural language. recently, the ability of llms to recognise and classify specific language constructions, to analyse data within a theoretical framework, to annotate texts with relevant linguistic information to support corpus-based linguistic studies, etc. has been investigated. the following is a brief overview of the applications of llms in the field of linguistics, formally divided into: recognition of language constructions, analysis of data within a theoretical framework and linguistic annotation, as some of the studies combine more than one approach. ## large language models used in the experiments there are already several surveys on large language models describing the history of pre-training and proceedings of clib 2024 breakthroughs @xcite , the scaling and impact of pre-trained models @xcite , the prompting methods in natural language processing @xcite , the multimodal pre-trained models @xcite , the recent advances in llms with introduction to the background, key results and mainstream techniques @xcite , the evaluation methods for llms @xcite , the comparison between some of the most popular llms, including the three families: gpt, llama and palm, and the discussion of their features, contributions and limitations @xcite , among others. here we will briefly introduce the llms that were used in the experiments to illustrate the capabilities of llms to assist the creation of large semantic language resources: (in alphabetical order) claude 3.5 sonnet, gemini 1.5 pro, gpt-4o, gpt-4o mini. the models were last accessed on 21 and 22 july 2024. it is not our aim to compare these llms for several reasons: a) models are evolving very fast and new models and updates of existing models are constantly appearing; b) some of the models have been improved in a clearly defined direction; c) for some of the selected llms, detailed information about architecture, training data, model size, etc. is not available. therefore, we do not focus on comparing models, but on exploring a possible way to work with llms on a specific linguistic task. the claude 3 is a family of multimodal models released by anthropic 1 : claude 3 opus, claude 3 sonnet and claude 3 haiku. claude 3 has been trained with a mixture of public and proprietary data and is subject to rigorous cleaning and filtering methods. all models have image processing and show good performance in logical reasoning, maths and coding @xcite . claude 3, especially the opus model, reportedly outperforms other state-of-the-art models in various evaluation benchmarks such as the google-proof question-answering benchmark (gpqa), measuring massive multitask language understanding (mmlu) and others. sonnet has been updated to claude 3.5, the latest version of the claude llms 2 . in anthropic's tests, the claude 3.5 sonnet outperforms some of the latest llms and other claude models. the claude 3.5 sonnet has powered the claude chatbot. the gemini 1.5 family of google deepmind aims to retrieve and analyse millions of context tokens, including multiple long documents, video and audio materials (gemini team, google, 2024). to achieve this, the models are trained on multiple 4096-chip pods of tpuv4 accelerators, using a wide range of multimodal and multilingual data that includes image, audio and video content. gemini 1.5 pro foot_0 is a sparse mixture-of-expert (moe) transformer-based model that builds on the advances of gemini 1.0 (gemini team, google, 2023) in a variety of multimodal tasks such as visual understanding, classification, summarisation and content creation from image, audio and video. the extensive evaluations with diagnostic multimodal long-context benchmarks show that gemini 1.5 pro is able to retrieve and understand large amounts of data. the model is available via the gemini chatbot and the gemini api. gpt-4 is a large multimodal model that accepts both image and text input and generates text output @xcite . it is reported that gpt-4 understands and generates text in more languages compared to its predecessor and outperforms gpt-3.5 and other large language models in a number of traditional nlp benchmarks and the mmlu benchmark. the next generation, gpt-4o (omni) foot_1 , has been integrated into the text, vision and audio modalities through techniques such as filtering training data and refining model behaviour through posttraining. according to openai, gpt-4o matches the performance of gpt-4 turbo in the areas of text, reasoning and coding intelligence evaluated with traditional benchmarks, and offers multilingual, audio and vision processing. gpt-4o is available via the chatgpt plus and the openai api. the gpt-4o mini foot_2 is a smaller version of the gpt-4o model. gpt-4o mini has reportedly outperformed gpt-3.5 turbo in several llm benchmarks and is trained with an instruction hierarchy method @xcite ) that improves the model's resistance to jailbreaks and system prompt extraction. in chatgpt, gpt-3.5 was replaced by gpt-4o mini. it is also available via the openai api. ## prompting-based experiments to facilitate linguistic research we present two experiments with zero-shot prompting aimed at linguistic classification and annotation of examples. the aim is to demonstrate the effectiveness of llms in making linguistic decisions, thus increasing the expert's confidence in creating semantic resources with reliable information without the need for a second or third expert to perform the same activities. the first experiment (augmenting framenet semantic frames with lexical units and annotations) simultaneously aims to select new lexical units that evoke a particular framenet semantic frame, suggest relevant examples and annotate them with the core elements of the frame. the second experiment (classification of framenet lexical units into semantic classes) aims to categorise verbs that evoke a particular framenet semantic frame into relevant semantic classes. the general framework can be summarised as follows: a) formulation of a linguistic task; b) selection of suitable llms (mainly according to two criteria: novelty and accessibility); c) formulation of a prompt template aimed at fulfilling the linguistic task; d) execution of the experiment and collection of data from the selected llms; e) human evaluation of the results obtained. ## conclusions in general, llms cope quite successfully with the linguistic annotation of words or phrases expressing a given framenet core frame element, so it can be expected that the automatic annotation of the (core) frame elements for previously selected examples can be successfully performed by llms. furthermore, the use of llms as a bank of examples illustrating different linguistic phenomena can be perfected with more specific instructions in the prompts. some difficulties in using llms for linguistic classification into aspectual classes may arise from the nature of linguistic tests that only work for a particular linguistic context, while there may even be a shift between aspectual classes within a verb meaning for different constructions and contexts. this means not only that linguistic tests need to be further elaborated in order to work with llms, but also that a theoretical justification needs to be provided for the contextual conditions under which aspectual class shifts occur, as has already been done for a number of cases in english. the use of more than one llm with the same prompts allows the expert to confirm or reject an initial hypothesis, i.e. a second and third annotator may be superfluous. on the other hand, carefully analysing the errors of the llms can also help the expert to make one or the other decision. to summarise, it can be said that the use and importance of llms in linguistic work, as in many other areas, will increase. in any case, llms are useful to linguists as interlocutors who can surprise them with unexpected linguistic usages.