Mashee at S em E val-2024 Task 8: The Impact of Samples Quality on the Performance of In-Context Learning for Machine Text Classification (2024)

Abstract: Within few-shot learning, in-context learning(ICL) has become a potential method for lever-aging contextual information to improve modelperformance on small amounts of data or inresource-constrained environments where train-ing models on large datasets is prohibitive.However, the quality of the selected samplein a few shots severely limits the usefulnessof ICL. The primary goal of this paper is toenhance the performance of evaluation metricsfor in-context learning by selecting high-qualitysamples in few-shot learning scenarios. We em-ploy the chi-square test to identify high-qualitysamples and compare the results with those ob-tained using low-quality samples. Our findingsdemonstrate that utilizing high-quality samplesleads to improved performance with respect toall evaluated metrics.

## introduction the advent of large language models (llms) like gpt-3.5 has brought about transformative capabilities, seamlessly handling tasks like question answering, essay writing, and problem-solving @xcite @xcite . however, this technological advancement necessitates careful consideration of its associated challenges. concerns regarding the potential impact on creativity and ethical implications, particularly concerning the generation of deepfakes @xcite , warrant careful attention @xcite . additionally, the limitations of llms, including the possibility of producing erroneous information, require rigorous evaluation and verification. the substantial energy consumption required for training llms on massive datasets raises environmental concerns, contributing to their carbon footprint. moreover, plagiarism issues emerge as users may misuse the generated content, either inadvertently or intentionally @xcite . various models have been introduced in recent years designed to distinguish text generated by humans from that created by machines @xcite . examples include gptzero(gpt), ai content detector(cop), and ai content detector by writer (wri) among others. some of these models are trained on specific datasets, while others are commercially available. designing and implementing llms for classification tasks requires substantial resources and computational power, which are often only accessible to institutions and governments. therefore, various optimization models, such as lora @xcite , distillation @xcite , quantization @xcite , and in-context learning @xcite , have been developed to reduce the resource requirements for llm implementation. this paper focuses on in context learning (icl) @xcite , which utilizes the capabilities of other models to enhance their ability to classify ai-generated text. in context learning (icl) is a natural language processing (nlp) technique utilized to enable large language models (llms) to learn new tasks based on minimal examples. this technique proves powerful in scenarios where training models on extensive datasets is impractical or when there are constraints on dataset availability for a specific task. icl operates on the premise that humans can often acquire new tasks through analogy or by observing a few examples of task performance. it can be employed without any examples and is referred to as zero-shot learning. alternatively, if the input includes one example, it is termed one-shot learning, and if it contains more than one, it is known as few-shot learning. this paper focuses on the application of few-shot learning within the context of icl @xcite . in this study, our focus lies exclusively on fewshot learning. we present a methodology that leverages the chi-square statistic @xcite to select samples for few-shot learning and evaluate its impact on the performance of a machine-generated text classification model. we work on task a english language only @xcite . ## dataset the dataset employed for task a comprises two main components. the first part, derived from human writing, was collected from diverse sources including wikibidia, wikihow, reddit, arxiv, and peerread. the second part consists of a machine-generated text produced by chatgpt, cohere, dolly-v2, and bloomz @xcite . for further details, please refer to the associated paper @xcite . ## chi-square chi-square is a statistical test used to assess the independence of two categorical variables. it calculates the difference between observed and expected frequencies of outcomes, and a larger chisquare value indicates a stronger rejection of independence. in text analysis, chi-square can be used to identify keywords that are more likely to occur in one category than another, making it useful for feature selection and text classification. we computed the chi-square values for each training sample and recorded the sample index with the highest and lowest chi-square values for both human-generated and machine-generated samples. table i displays the index and corresponding chi-square values for each of these instances. we will use x 2 to refer to chi-square @xcite . 4 system overview the system architecture is illustrated in figure 1 . the process starts with feeding the entire training dataset to a chi-square computation, where the chisquare value for each sample is calculated. subsequently, the indices of the samples with the highest and lowest chi-square values are selected for both human-generated and machine-generated datasets using information from table i . next, context learning is prepared. initially, multiple templates were tested, and the one presented in figure 1 yielded the best results. this template is then fed with two samples: the first being the machine-generated sample with the highest chi-square value, and the second being the human-generated sample with the highest chi-square value. due to context window size limitations, only the first 5000 characters of each sample are incorporated. this is applied to training samples exceeding 5000 characters to ensure the context learning size is not exceeded. finally, the test sample is fed into the context-learning process. the flan-t5 model large version is used. the results are then recorded and evaluated. the dev/test sample size was truncated to 3000. we also evaluated the system using samples with the lowest chi-square values and doing the same process. ## findings and analysis we employed the flan-t5 large model for both the development and testing datasets. we selected samples from both human-generated and machinegenerated sources, with each sample limited to 5000 characters to avoid exceeding the token size limit. a total of four experiments were conducted. the first experiment utilized samples with high chi-square values from the development set. the second experiment focused on samples with the smallest chi-square values from the development set. the third experiment involved samples with high chi-square values from the test set. finally, the fourth experiment utilized samples with low chisquare values from the test set. table ii presents all achieved results. based on the results presented in table ii , we can discuss several key points. ## conclusion this work presents a system for classifying humangenerated and machine-generated text. the system leverages the combined strengths of in-context learning and chi-square analysis. chi-square is employed to select high-quality samples from the trainin dataset for few-shot learning in the incontext learning. we implement flan-t5 model large version for in-context learning. evaluation using accuracy, recall, precision, and f1-score demonstrates that selecting high-quality samples improves system performance for both dev and test. furthermore, the results indicate that relying solely on in-context learning for new tasks like machine-generated text detection yields relatively low performance.