Characterised LLM s Affect its Evaluation of Summary and Translation (2023)

Abstract: In today’s widespread use of Large Language Models (LLMs), there have been significant achievements in various text domains such as generating summaries and translations. However, there is still room for development and improvement in evaluating the outputs of LLMs. In this paper, we propose an innovative scoring system that assesses the quality of summaries and translations using multiple metrics, we also enhance LLM’s performance in scoring tasks by assigning it different roles, effectively making it act as an expert. We test four roles in the study: a teacher, a proofreader, a travel writer, and an internet troll, comparing the advantages and disadvantages of each role in the scoring task. Our research results demonstrate that emphasizing LLM’s multilingual capabilities and strict standards as its identity can effectively boost its performance. Additionally, imbuing LLM with a more critical thinking ability enhances its performance in translation tasks compared to a milder LLM identity. In summary, we show that assigning different identities to LLM can influence its performance in scoring tasks. We believe that this research will contribute to the use of LLMs for scoring purposes.

## introduction since ghatgpt's emergence, large language models (llm) have been flourishing in the natural language processing (nlp) field. thanks to the growth of llms, tasks such as automatic summaries and translations are becoming more commonly generated by llms. however, we realized that most existing evaluation methods for llms output lack thorough explanation, making the research process in this domain considerably challenging. we believe that by inventing a metric for evaluating summarization and translation, research on article generation would be much more practical. inspired by previous work on using llms to generate scores and evaluate text @xcite @xcite , as well as research exploring having llms play the role of experts @xcite , we present an evaluation system employing multiple metrics (jinlan @xcite by carefully designed prompt @xcite and make llm act as an expert. for generating scores, we employed the model openorca-platypus2-13b @xcite to generate scores, which is a merge of platypus2-13b @xcite and openorcaxopenchat-preview2-13b @xcite . we selected this model because of its strong performers on the leaderboard and its small size for local inference. in order to bolster the large language model's (llm) evaluation capabilities, we implemented a strategy where the llm simulates an expert. this study is also a system description for the eval4nlp 2023 shared task @xcite which in the small track. ## method to use the large language model to better evaluate summarization and translation, we divided the task into a few parts. first, we separated an evaluation task into several metrics. then we made llm role different characters such as a proofreader, writer, or internet troll. llm would evaluate summation and translation in the expert role. in the end, we added scores from different metrics by xgboost and post-processing. ## conclusion we investigate the performance of llms with different character in generating scores to evaluate translation and summary tasks by incorporating characterised-prompts into the prompts. we find that emphasizing multilingual capabilities and stringent criteria in the llm's identity can effectively improve the llm's performance. by endowing the llm with stronger critical thinking compared to a more benign llm, we improve its performance on translation tasks. in summary, we demonstrate that assigning different identities to llms influences their performance on scoring tasks.