How well C hat GPT understand M alaysian E nglish? An Evaluation on Named Entity Recognition and Relation Extraction (2023)

Abstract: Recently, ChatGPT has attracted a lot of interest from both researchers and the general public. While the performance of ChatGPT in Named Entity Recognition and Relation Extraction from Standard English texts is satisfactory, it remains to be seen if it can perform similarly for Malaysian English. Malaysian English is unique as it exhibits morphosyntactic and semantical adaptation from local contexts. In this study, we assess ChatGPT’s capability in extracting entities and relations from the Malaysian English News (MEN) dataset. We propose a three-step methodology referred to as educate-predict-evaluate . The performance of ChatGPT is assessed using F1-Score across 18 unique prompt settings, which were carefully engineered for a comprehensive review. From our evaluation, we found that ChatGPT does not perform well in extracting entities from Malaysian English news articles, with the highest F1-Score of 0.497. Further analysis shows that the morphosyntactic adaptation in Malaysian English caused the limitation. However, interestingly, this morphosyntactic adaptation does not impact the performance of ChatGPT for relation extraction.

## introduction with the recent emergence of large language models (llm), we have observed a paradigm shift in natural language processing (nlp). these llm include palm @xcite , chat-gpt @xcite , @xcite , and llama 2 @xcite . chatgpt, inarguably the most popular llm currently, is developed by openai and has demonstrated remarkable ability in language understanding and generating coherent responses. the use of chatgpt has been observed in various nlp tasks, including sentiment analysis @xcite , topic classification (reiss, 2023; gilardi * corresponding author. @xcite , and information extraction @xcite @xcite . there have been several research works conducted to evaluate the capabilities of chatgpt for ner and re @xcite @xcite . while most of the evaluation outcomes focused on standard english, it raises a question: is chatgpt capable of extracting entities and relations from malaysian english news? originating from standard english, malaysian english (me) has evolved into a unique form of english incorporating local words from languages like bahasa malaysia, chinese and tamil @xcite . malaysian english exhibits usage of loan words, compound blends and derived words @xcite . some example sentences with the usage of loan words, compound blends and derived words are provided, such as: 1. "... billion of jobs in the next five to seven years, as well as bukit bintang city centre with rm600 million jobs awarded so far". from this sentence, bukit bintang city centre is a compound blend where "bukit bintang" refers to the name of location in bahasa malaysia, and this entity refers to a shopping mall (location). the existence of loan words, compound blends, and derived words in the usage of entity mentions has motivated us to assess the performance of chat-gpt in malaysian english, specifically for named entity recognition (ner) and relation extraction (re). prompting techniques like zero shot, few shot, and chain of thought (cot) have been proven to improve the performance of chatgpt in various nlp tasks @xcite @xcite . in-context learning helps chatgpt to understand more about the task in hand and define the scope on the task to be completed. it has been proven effective for domain-specific tasks, such as legal reasoning @xcite . keeping these in mind, we propose a novel three-step method to extract the entities and relations from malaysian english news articles, called "educate-predict-validate". section 3 discusses these three steps in details. chatgpt's ability to extract entities and relations is measured based on its agreement with humanannotated labels using the f1-score. our evaluation aims to establish a benchmark for chatgpt's performance in malaysian english texts. the code for this experiment is available at github foot_0 for reproducibility. the contributions of this research can be summarised as follows: ## educate-predict-evaluate chatgpt is one of the widely used large language models. it can be easily interacted through the pro-vided web interface, by asking questions and make conversation with the model. providing additional context helps chatgpt to learn and better understand the tasks in hand. in this paper, we propose a systematic methodology called educate-predictevaluate, which aims to carry out a comprehensive evaluation on chatgpt capability in ner and re within malaysian english context. figure 1 shows detailed view of proposed approach. ## result and analysis in this section, we present the outcome of the experiment that we conducted. in section 5.1, we discuss how chatgpt performs ner and re on men-dataset, together with the observed limitations. ## conclusion in this paper, we comprehensively evaluated and analyzed chatgpt's ability to extract entities and classify relations from malaysian english news articles. our extensive experiment was conducted with 18 different prompting approaches. the experimental results prove that morphosyntactic adaptation impacted the performance of chatgpt in extracting entities from malaysian english news articles. we discussed our findings from the experiments, including an analysis of the limitations of chatgpt. chatgpt could not achieve satisfying performance when extracting entities from malaysian english news articles. apart from the limitation in understanding the context of inputs, there are a few factors that influenced the performance of chatgpt. these include the dataset's characteristics, additional contexts like guidelines and explanations, and several few-shot examples. the morphosyntactic adaptation exhibited by malaysian english influenced the performance of chatgpt for ner. given the annotation of our men-dataset, we could only assess the performance of chatgpt in ner and re. for future work, we plan to expand our evaluations by incorporating a broader range of nlp downstream tasks. furthermore, we will extend our assessment to include other language models, such as gpt-4 (openai, 2023) and llama 2 @xcite , for ner and re tasks, specifically in the context of malaysian english. finally as a future work, we will also expand the coverage of our experiment with different prompting techniques to ensure our evaluation is statistically significant.