Fralak at S em E val-2024 Task 4: combining RNN -generated hierarchy paths with simple neural nets for hierarchical multilabel text classification in a multilingual zero-shot setting (2024)

Abstract: This paper describes the submission of team fralak for subtask 1 of task 4 of the Semeval-2024 shared task: ‘Multilingual detection of persuasion techniques in memes’. The first subtask included only the textual content of the memes. We restructured the labels into strings that showed the full path through the hierarchy. The system includes an RNN module that is trained to generate these strings. This module was then incorporated in an ensemble model with 2 more models consisting of basic fully connected networks. Although our model did not perform particularly well on the English only setting, we found that it generalized better to other languages in a zero-shot context than most other models. Some additional experiments were performed to explain this. Findings suggest that the RNN generating the restructured labels generalized well across languages, but preprocessing did not seem to play a role. We conclude by giving suggestions for future improvements of our core idea.

## introduction task 4 of the semeval 2024 workshop deals with the identification of persuasion techniques in meme data @xcite . subtask 1 regarded only the textual content of the memes. training, validation and development data is only available in english, but the test phase includes data in three more languages (bulgarian, north macedonian and arabic) for multilingual zero-shot classification. the 20 persuasion techniques are organized in an hierarchical directed acyclic graph (available on the task website). each meme can have zero, one or multiple persuasion techniques associated with it, making this a hierarchical multilabel classification problem. assigning a parent node of the target label results in partial points. our system (team fralak) implements an ensemble model including a seq2seq module, using some innovations to avoid common pitfalls and exploit the hierarchy information. our approach transforms the problem by restructuring the labels into strings in a way that captures all possible paths through the hierarchy and uses these as target sequences to train a rnn that learns the relationships between the labels on different levels. it combines the power of a rnn with a simple fully connected architecture. these non-sequential modules are also expected to mitigate the error propagation effect (also called exposure bias), where a wrongly predicted label in the beginning of the generated sequence results in more errors down the line @xcite . rnns for multilabel classification also depend on the ordering of the labels, even though the class labels are essentially an unordered set @xcite . we address this by sorting the labels by frequency. the textual content of the memes is represented using multilingual sentence embeddings. although we only participated in subtask 1, our architecture can easily be expanded to also take into account the visual content of the meme (subtask 2a+b) foot_0 . our system performed below average on the english-only test set, but generalized better than most other systems. the goal of this paper is to explain our methodology and system architecture (sections 2 and 3) and explore why the system performs relatively well at the zero-shot task (section 5). ## methodology we aimed to implement rather simple nn modules in order to explore their usefulness for a complicated task like this. the main idea behind our system is to transform the labels into strings that reflect the hierarchical acyclic graph containing the different persuasion techniques. these are be used to train an rnn that is supposed to learn the relationships between both the labels and the different levels of the hierarchy. we expect that the relations between labels are a feature that generalizes especially well across languages, making our approach especially adapt for multilingual zero-shot learning for this specific task. ## experimental setup the training+validation data consisted of 7,500 memes. after restructuring the labels this gave us 21,968 training instances. including the development data (1,000 memes) resulted in 24,664 instances spread over 8,500 memes. all of these instances were in english. as typical for multilabel settings, the class labels are extremely unbalanced: the most common label, smears, occurs 1,990 times in the training data, whereas the least common label, intentional vagueness, occurred only 21 times. our teams original test submission was only trained on the train and validation data, as we used the development data to validate our approach, but we added the development data in subsequent experiments as we theorized that having more training data would give more robust results; as we did not do additional finetuning for the posthoc experiments, no development set was used (see section 5). due to the way the semeval challenge was set up, the validation set was a dataset that was available from the beginning, whereas the gold labels for the development set only became available a couple of weeks before the test submission closed; we only used the validation set for some preliminary testing and setup, after which it was joined with the test set. all results are reported on the test data. the modules were trained separately, but on the same data. we conducted some preliminary experiments training module 1 and 2 on 75% of the data and module 2 on the remaining 25% (random split) but this led to a drop in performance. the optimal amount of epochs for each module was decided based on plots of the average loss per epoch. each separate module took less than 30 minutes to train on an apple m3 8-core cpu. we used an adam optimizer with a learning rate of 1e-3. modules 1 and 3 were trained with a crossentropyloss; module 2 with a smoothl1loss. the task evaluation metric was the hierarchical-f1, calculated using hierarchical precision and recall @xcite . this measures gives partial points for assigning a label higher up in the hierarchy, and full points for assigning the specific technique. ## results ## discussion we hypothesize that two mechanism that might have contributed to the system's generalization capacity. first, the preprocessing (particularly the ner step) might have made the model more generalizable. second, the seq2seq rnn module to learn the labels might have been particularly good at capturing the relationships between the labels in order to investigate these hypotheses, we ran additional experiments in which we left out parts of the system to investigate what happened to the performance. the results are summarized in our first hypothesis was that the preprocessing, especially the ner, helped the system generalize better to unseen languages. however, this seems not to be the case. taking out only the ner module let to a slight drop in performance in english, but a better performance in the other languages. a possible explanation is the non-ubiquity of the name mark: replacing people with 'mark' might not actually be helpful if 'mark' is not adapted to the specific language. skipping all preprocessing steps (other than adding the embeddings) actually improved performance for english (even though taking out only the ner led to a drop, suggesting this might actually have been a very helpful step for the english data), but made hardly any difference for the other languages when compared to the setting without ner. our second hypothesis was that the rnn module was especially helpful for generalization. either module alone performed worse than the three modules combined for all languages (apart from the first module, that reached the same f1 for arabic), so the influence of the label-generating rnn should not be overestimated. on the other hand, when comparing the performance of module 1 with the performance of module 2, we see that the difference is the same for english and bulgarian, and bigger for both macedonian and arabic. this might mean towards the second module actually being a bit more important in the zero-shot setting, but more research is required. our full model had remarkably high recall, but low precision. looking at the performance of modules 1 and 2 separately suggests that this is mainly due to module 1 (and, possibly, module 3, that is very similar to module 1 in architecture). this pattern is the same across languages and modifications. this is not very surprising; erroneously generating the eos token once makes the module stop predicting labels, and given that every training instance has an eos token, it is very common and the chance of it being produced erroneously is relatively high. moreover, the rnn stops generating strings when the maximum string length of 50 is reached. we thus re-ran the base model (including development model) with a maximum string length of 100 for the rnn ( adding the development data (i.e. training the model on more data) seems marginally helpful for english (+0.01 point f1) but marginally unhelpful for macedonian (-0.01 point f1) and arabic (-0.03 point f1). if the strength of our system indeed lies in it learning the relationships between the labels of the hierarchy, it is likely that a smaller amount of data was just enough to learn this, and adding more data just makes the model overfit. ## conclusion this paper described the system used to generate the test submissions for subtask 1 of task 4 of se-meval 2024 'multilingual detection of persuasion techniques in memes'. we proposed a system consisting of different neural modules, the most innovative of which was an rnn that was trained to generate sequences that reflect the position of the relevant labels in the hierarchy. our model did not perform particularly well on the english data, but compared to the other teams, it generalized unexpectedly well to other languages in a zero-shot setting. we conducted some additional experiments to find out what might have contributed to this. we found that our preprocessing steps (normalization and ner) did not make the model more generalizable, but we did find some evidence that the rnn module might have played a role as hypothesized. we see plenty of possibilities to improve on our original idea in the future. first of all, we would like to explore the performance of different types of embeddings. we found that our system as a whole had a high recall, but a low precision; however, the rnn module showed the exact opposite pattern, having high precision and low recall. allowing the rnn output in module 3 to be longer (up to 100 tokens) partially alleviated this problem and improved performance. we hypothesize this is because the eos token is generated too easily. somehow raising a barrier for the module to generate the eos token might help to improve its recall. implementing an attention mechanism in the final module could also help aleviate this problem. other options to explore are a ner preprocessing step that better generalizes to other languages than just replacing people with "marks". finally, it would be interesting to explore the capabilities of a hierarchical-path generating rnn with more sophisticated layers (gru or lstm), or combined with a convolutional model.