System Report for CCL 23-Eval Task 9: HUST 1037 Explore Proper Prompt Strategy for LLM in MRC Task (2023)

Abstract: “Our research paper delves into the Adversarial Robustness Evaluation for Chinese Gaokao Read-ing Comprehension (GCRC advRobust). While Chinese reading comprehension tasks havegained significant attention in recent years, previous methods have not proven effective for thischallenging dataset. We focus on exploring how prompt engineering can impact a model’s read-ing comprehension ability. Through our experiments using ChatGLM, GPT3.5, and GPT4, wediscovered a correlation between prompt and LLM reading comprehension ability, and found thatprompt engineering improves the performance of each model. Our team submitted the results ofour system evaluation, which ranked first in three indexes and total scores. Keywords— LLM, Prompt, Chinese Reading Comprehension”

## introduction machine reading comprehension (mrc), involves machines reading and comprehending human natural language text. based on this, the machines are expected to answer questions related to the information in the text. this task is often used to evaluate the machine's ability to comprehend natural language, which can help humans quickly identify relevant information from a large amount of text. additionally, it can help reduce the cost of manual information acquisition. mrc has strong application value in the fields of text q&a, information extraction, and dialogue systems. in recent years, machine reading comprehension has gained significant attention from both industry and academia and has become one of the research hotspots in the field of natural language processing. the competition, entitled gcrc advrobust: adversarial robustness evaluation for chinese gaokao reading comprehension, adds inference logic perturbation strategies to the regular reading comprehension task to improve the robustness of the machine reading comprehension model. neural mrc has shown superiority over traditional rule-based and machine-learning-based mrc, and has gradually become the mainstream in the research community. however, compared with the large model, this method has obvious shortcomings. for example, bert @xcite cannot handle too long input, and its input is only 512 tokens, which cannot solve the task at all. taking into account the setting of the competition questions and the shortcomings of the existing solutions, we chose a suitable large language model that works efficiently with chinese and adjusted the prompting strategy continuously to enable the model to reason based on the original text. considering the length of the paragraphs in the reading comprehension task, we first want to find sentences that are highly relevant to the options and questions from the paragraphs and then analyze them, but the effect is not very good. so we tried to feed the entire paragraph into the model, and then analyzed the test results to improve the prompt continuously. our final approach involved combining the model output with the original text to obtain the correct answer. our team achieved significant improvements in all indicators, with a 39.2% increase in score, 44.1% in acc0, 46.87% in acc1, and 32.4% in acc2 compared to the official benchmark model. ## related work assessing the scalability of machine reading comprehension models relies heavily on their robustness in practical applications @xcite . although current models have made significant strides in performing well on closed test datasets, their robustness in open, dynamic, real-world environments for reasoning and decision-making remains inadequate @xcite . to evaluate model robustness, previous studies have introduced text noise @xcite or rephrased the problem @xcite . however, these methods have limitations in measuring model performance due to their narrow focus on a single attack and relatively low topic difficulty. thanks to the fast evolution of big language models and their vast amount of semantic information, along with their impressive reasoning abilities, adopting a prompt modification paradigm can outperform the original models in certain nlp tasks. for instance, in reading comprehension tasks, the model can read the original text and select the correct answer from the options. this task is especially fitting as it involves a lengthy original text, and the options have overlapping content. in our research, we examined the common methods that are currently used for large language models, which inspired our approach. we use p θ to denote an llm with parameters θ, and lowercase letters to represent an input language sequence, e.g. zero-shot large language models like gpt-3 are capable of performing certain tasks without any prior training due to their ability to follow instructions and being trained on vast amounts of data. recent studies have shown that instruction tuning can enhance zero-shot learning @xcite . few-shot although large-language models have impressive zero-shot abilities, they struggle with more complex tasks when relying solely on this approach. to address this, few-shot prompting can be used to facilitate in-context learning. by providing demonstrations within the prompt, we can guide the model towards better performance. these demonstrations act as conditioning for subsequent examples where we want the model to generate a response. in a study by @xcite , the importance of these demonstrations for the success of in-context learning was explored. cot introduced in @xcite , chain-of-thought (cot) prompting enables complex reasoning capabilities through intermediate reasoning steps. by using cot with few-shot prompting, one can achieve even better results on challenging tasks that require reasoning before responding. in our research, considering that the original paragraph is long, we first construct the prompt based on zero-shot, then add examples using the few-shot prompt model to the extent allowed by the token, and finally construct cot prompt samples based on different question types. ## model selection the open track does not limit the models that can be used. considering that the task needs to process hundreds of data, we selected the llm that can be called in the form of api, and introduced some models below. ## conclusion our team used a large language model based on gpt4 for the adversarial robustness evaluation in the chinese gaokao reading comprehension task. we modified and tested various prompting strategies to enable the model to make logical inferences from the original text. this method utilizes the semantic information and reasoning ability of the large model more effectively compared to the original approach to solving reading comprehension tasks. however, there are still some limitations to the current system. we were unable to try more hinting strategies due to the display limitation of the gpt4 model api. additionally, the original text in this task was lengthy, and the model input length was restricted, leading to a shorter scalable content. we aim to compress the original text information to enable us to try out more hinting strategies in the future.