YNU - HPCC at S em E val-2024 Task 1: Self-Instruction Learning with Black-box Optimization for Semantic Textual Relatedness (2024)

Abstract: This paper introduces a system designed for SemEval-2024 Task 1 that focuses on assessing Semantic Textual Relatedness (STR) between sentence pairs, including its multilingual version. STR, which evaluates the coherence of sentences, is distinct from Semantic Textual Similarity (STS). However, Large Language Models (LLMs) such as ERNIE-Bot-turbo, typically trained on STS data, often struggle to differentiate between the two concepts. To address this, we developed a self-instruction method that enhances their performance distinguishing STR, particularly in cases with high STS but low STR. Beginning with a task description, the system generates new task instructions refined through human feedback. It then iteratively enhances these instructions by comparing them to the original and evaluating the differences. Utilizing the Large Language Models’ (LLMs) natural language comprehension abilities, the system aims to produce progressively optimized instructions based on the resulting scores. Through our optimized instructions, ERNIE-Bot-turbo exceeds the performance of conventional models,achieving a score enhancement of 4 to 7% on multilingual development datasets.

## introduction semeval-2024 task 1 @xcite addresses the challenge of semantic textual relatedness (str), which goes beyond paraphrasing and entailment of semantic textual similarity (sts) @xcite @xcite by considering topics and logical connections between sentence pairs. this task is particularly complex due to the nuanced context required for str, a feature not fully captured by existing models trained predominantly on sts data. this gap can lead to black-box large language models (llms) misinterpretations like ernie-bot-turbo foot_0 . our study introduces a self-instruction method to enhance the distinction between sts and str in llms @xcite @xcite . in our approach, back translation @xcite converts low-resource language sentence pairs into english as inputs for llms. with a task description as the starting point, the black-box llms generate a new task instruction, which will be refined based on human feedback. the system iteratively refines the enhanced instruction by assessing it against the original and using the resulting score to produce increasingly optimized instructions. our method improves how llms deal with tricky cases of similar but unrelated texts. using our optimized instructions, ernie-bot-turbo outperforms standard models and boosts scores by 4 to 7% on multilingual development datasets in track a. the ranking of each tack a's test dataset is as follows: english (36), amharic (11), algerian arabic (24), telugu (24), spanish (24), moroccan arabic (24), marathi (25), kinyarwanda (20), and hausa (20). the remainder of this paper is organized as follows. section 2 describes the model and method used in our system, section 3 discusses the results of the experiments, and finally, conclusions are drawn in section 4. ## methodology figure 1 illustrates the overall framework of our self-instruction method. we employ back translation for datasets encompassing multiple languages to render sentence pairs into english as the input for llms. with a task description as the starting point, the black-box llms generates a new task instruction which will be refined based on human feedback. the enhanced instruction is subsequently assessed against the original, generating a score that informs the optimization cycle. the system progressively refines the instructions in response to this score, resulting in progressively more optimized directives. the ensuing section will delve into a detailed analysis of this iterative optimization process. ## conclusion in this paper, we developed a self-instruction method that enhances llms' ability to distinguish between semantic textual similarity (sts) and semantic textual relatedness (str), particularly in hard samples(high sts but low str).through this method, ernie-bot-turbo (score llm) not only surpasses the performance of conventional models, achieving a score enhancement of 4 to 7 % on multilingual development datasets, but also effectively reduces confusion between sts and str in large language models (llms). additionally, it achieved a commendable ranking in the final test evaluation. our work demonstrates that optimized instructions, chain of thought, and easily confused examples enable llms to mitigate errors even in few-shot samples. future research will aim to refine llms' capacity to grasp the overall semantic meaning of sentences further.