Computational Meme Understanding: A Survey (2024)

Abstract: Computational Meme Understanding, which concerns the automated comprehension of memes, has garnered interest over the last four years and is facing both substantial opportunities and challenges. We survey this emerging area of research by first introducing a comprehensive taxonomy for memes along three dimensions – forms, functions, and topics. Next, we present three key tasks in Computational Meme Understanding, namely, classification, interpretation, and explanation, and conduct a comprehensive review of existing datasets and models, discussing their limitations. Finally, we highlight the key challenges and recommend avenues for future work.

## introduction in the current age, memes -user-created combinations of pictures and images overlaid with textare widespread due to their nature of being amusing and relatively quick to consume compared to text. memes have become a novel and prevalent means of online communication @xcite . memes can be malicious, such as those that are hateful, harmful, or politically manipulative. for example, during the last two us presidential elections, a large amount of coordinated media content, especially memes, was used to affect public opinion and influence the election results. 2 however, given the vastness of the internet, it is impossible to have human workers inspect every single meme on the social network platforms @xcite . having an automated system to detect malignity in memes hence can be of great help. at the same time, as memes are often used to express thoughts and personal opinions, we could use meme understanding technologies to help improve human communication. for example, teachers may understand their students' thoughts better via the memes the students composed, thus possibly helping to narrow the generation gap between them. a student who just started their study in a foreign country could adapt to the new culture better if they can understand the jokes made by their friends via the memes they composed. such societal motivations have spurred an interest in computational meme understanding (henceforth, cmu), which is an umbrella term we introduce to refer to a collection of tasks involving the automated comprehension of memes. cmu presents several key challenges to researchers. first, cmu systems need to seamlessly recognize and combine textual and visual elements from the meme itself. second, generating a full textual description of the message conveyed in a meme, a key task in cmu, requires both breadth and depth of knowledge about recent news, internet subcultures, meme cultures, and the world. finally, cmu systems often need to read between the lines, such as decoding figurative language, to successfully understand a meme. our goal in this paper is to present a timely and comprehensive review of work on cmu. to our knowledge, there have been no comprehensive surveys in this area of research. related surveys are mostly tangentially related to memes, including those on computational propaganda @xcite , multimodal disinformation and fact-checking @xcite , hate speech @xcite , and humor generation @xcite . the closest review for memes is by @xcite , which was written two years ago and concerns only harmful memes and the associated classification tasks. our review has a broader scope, covering many more types of memes and more technical tasks. ## a taxonomy for memes recognizing the types of memes in a dataset, as well as the distribution of such types in the wild, is important in directing research efforts to align with real-world needs. for example, if a type of memes is popular on the web, but is not collected, trained, and evaluated on, models may suffer from out-of-distribution problems in production settings. before discussing the distribution of meme types, we need a unified language in which a meme can be classified. currently, meme taxonomies in nlp are narrowly defined within a subset of memes such as the harmful ones @xcite @xcite . therefore, we borrow from the social sciences a taxonomy of memes along three dimensions: forms, functions, and topics. ## datasets in this section, we present existing cmu datasets. ## models in this section, we present an overview of the models that have been built for the three cmu tasks. ## ethical considerations memes are an effective means of communication. therefore, any technology that harnesses the power of memes also carries the risk of being misused and other negative side effects to humans. this section highlights two major ethical considerations related to the development of cmu technologies. should we expose annotators to sexual and hateful content? the main motivation for annotating memes so far has been to automatically process malicious contents such as manipulation, including hateful memes @xcite and harmful memes @xcite . however, exposing annotators to such content may negatively affect their mental health. as such, researchers working on meme annotation should impose age restrictions on annotators and have appropriate preannotation screening to ensure the candidate annotators understand the potential harm of the data before joining the annotation team. in a crowdsourcing setting, however, some people may choose to become annotators because the cost of forfeiting the opportunity is too high, thus accepting the risks to their mental health. therefore, if the data is harmful, one should hire annotators instead of perform crowdsourcing so that they can monitor the mental health of the participants. should we create datasets and models on harmful memes? recall that one of our goals is to help identify hateful memes or display warning messages. however, there is a risk that the datasets and models will be misused to generate hateful messages or bypass hate detection models (i.e., adversarial attacks). therefore, the public release of such data and models should be handled with care, ideally after proper consultation from social science researchers. for datasets, one may adopt the licensing approach similar to that employed by the authors of the hateful memes dataset @xcite . for models, red teaming @xcite should be conducted rigorously before deploying them to the real world. ## concluding remarks we conclude this survey by enumerating promising avenues for future research in cmu.