JUAGE at S em E val-2023 Task 10: Parameter Efficient Classification (2023)

Abstract: Using pre-trained language models to implement classifiers from small to modest amounts of training data is an area of active research. The ability of large language models to generalize from few-shot examples and to produce strong classifiers is extended using the engineering approach of parameter-efficient tuning. Using the Explainable Detection of Online Sexism (EDOS) training data and a small number of trainable weights to create a tuned prompt vector, a competitive model for this task was built, which was top-ranked in Subtask B.

## introduction misogyny is a growing problem online: inflicting harm on the women who are targeted, making online spaces inaccessible and unwelcoming, and perpetuating social asymmetries and injustices. automated tools are now widely deployed to find, and assess sexist content at scale but most only give classifications for generic, high-level categories, with no further explanation. the explainable detection of online sexism (edos) semeval task 10 was introduced to address this problem @xcite , challenging english-language systems to flag what is more generally sexist content and also explaining why it is sexist. the juage team participated by building one ensemble with bert models for the most general subtask (a) and for the more detailed subtasks with taxonomies for sexism (b and c) submissions were made by tuning a palm derived model @xcite , a 62-billion parameter large pre-trained language models (plm). the palm model variant was instruction-tuned @xcite and optimized for inference using quantization @xcite . while it is possible to fine-tune these large language models directly on user tasks, parameterefficient tuning techniques keep the large language model weights fixed and enable sharing of resources across tasks and massively reduce the number of parameters to be updated and stored. the smaller number of parameters that are updated also enable the rest of the model to act as a regularizer, allowing the model to learn from much smaller datasets @xcite . prompt engineering @xcite ) is another popular way to use plms. however, given the constraints of the model input buffer, only a few training examples may be used (i.e., few-shot prompts for in-context learning). a recently proposed parameter-efficient alternative is to prepend pseudo-tokens to the input and then adjust the embedding weights of these tokens ("soft prompt learning"). this combines the resource sharing of a large, frozen model with a procedure for optimizing weights over a training set. this technique has come to be known as prompt-tuning @xcite . we opted for prompt-tuning in our submissions, except for subtask a, where we opted for fine-tuning bert. follow-up experiments showed, however, that prompt tuning would also have been better for this subtask. ## background hateful language, such as hate speech and misogyny, is very present nowadays also due to its propagation through social media. its subjective nature makes hateful language detection a multi-faceted task. therefore, most existing datasets are taskspecific while the lack of common benchmarks leads the researchers to compile their own datasets @xcite . detection models, however, trained on these datasets, do not generalize well @xcite , and unintended biases can impair the models' performance @xcite @xcite . in this paper, we discuss these aspects with regard to misogyny. ## system overview our method is primarily based on agile classification via prompt tuning on large plms. we did not use any misogyny-specific resources beyond those provided by the task organizers. ## experimental setup 4.1 subtask a we fine-tuned bert for 7 epochs, using patience of 5, and a max-length of 50 tokens. we trained 3 different models by changing the random state value during the train, validation, and test split, and built an ensemble by majority voting. ## conclusions the juage team participated in the three subtasks of semeval-2023 task 10 regarding explainable detection of online sexism. our submission was ranked first on the second subtask and 7th on the third. our analysis shows that prompt-tuning would have won subtask a also had it been submitted. we left for future work the investigations on the interpretability of the learned soft prompts. for each of their learned pseudo-token's embeddings, @xcite retrieved the nearest neighbor frozen vocabulary token's embdeddings. they suggest that prompt-tuning may learn natural language context specific to a task. we plan to use the same method to interpret our learned prompts as well as compare the learned prompts for each task.