Social Meme-ing: Measuring Linguistic Variation in Memes (2024)

Abstract: Much work in the space of NLP has used computational methods to explore sociolinguistic variation in text. In this paper, we argue that memes, as multimodal forms of language comprised of visual templates and text, also exhibit meaningful social variation. We construct a computational pipeline to cluster individual instances of memes into templates and semantic variables, taking advantage of their multimodal structure in doing so. We apply this method to a large collection of meme images from Reddit and make available the resulting SemanticMemes dataset of 3.8M images clustered by their semantic function. We use these clusters to analyze linguistic variation in memes, discovering not only that socially meaningful variation in meme usage exists between subreddits, but that patterns of meme innovation and acculturation within these communities align with previous findings on written language.

## introduction one objective in variationist sociolinguistics is to study how social factors contribute to differences in the way people use language. work in natural language processing has followed this tradition, offering large-scale analyses of how language use is conditioned on geography, @xcite @xcite , community @xcite @xcite and time @xcite . this work is important not only because language variation often exposes shortcomings in nlp tools, which are primarily developed for standard language varieties @xcite , but also because variation often embeds social meaning. we make inferences about people's social class, regionality, gender, and much more based on the way they talk @xcite , and we strategically use language to actively construct and perform identities @xcite . most of this work has focused on lexical or morphosyntactic variation in written texts. however, language exists beyond text or speech. in face-to-face interaction, multimodality in language has been construed as features like co-speech gesture, facial expression or body movement @xcite . in online communication, previous work has extended the term to include the interplay between images and text @xcite @xcite . understanding text in isolation is insufficient to understanding how we communicate online. in the space of multimodal online language, memes are interesting for their compositionality. they consist of a base image (the template) as well as superimposed text (which we refer to as the fill). for example, the "drake" template depicted in figure 2 serves the semantic function of expressing a preference relation between the fills. this same drake template can be used to express preference relations between a range of fills; at the same time, multiple different templates can share the same or similar semantic function, as illustrated in fig. 1 for the function of "scalar increase." we refer to this set of functionally equivalent templates as a semantic cluster. in this work, we follow the variationist sociolinguistics tradition by treating templates as variants and semantic clusters as variables, observing how social factors might contribute to the distribution among these variants. to conduct this analysis, we develop a method for identifying semantic clusters by exploiting the visual structure of meme templates and the linguistic structure of meme fills. we use this to create the semanticmemes dataset of 3.8m reddit memes foot_0 grouped into semantic clusters and validated with a human evaluation. finally, we use these semantic clusters to perform a series of case studies demonstrating their use in studying linguistic variation, linguistic innovation, and linguistic acculturation. we find that: these findings illustrate the ways in which memes function as multimodal acts of communication, and how methods from computational sociolinguistics can shed light on meaningful variation within them. ## methods to study variation in meme use, we need to identify the meme variables that organize a collection of meme instances-the individual memes that are created and posted online by specific people at specific moments in time. we create a pipeline that visually clusters meme instances into templates (i.e., the same memes that differ by variation in fills) by exploiting the visual similarity between them; and linguistically clustering meme templates into semantic clusters by exploiting the similarity among the fills used in different templates. fig. 3 provides an overview of the process, which involves first clustering instances into templates ( §2.1), and then clustering templates into variables ( §2.2). ## semanticmemes dataset we used the pipeline described above to generate semantic clusters from a dataset of 27.9m images collected from reddit @xcite . we scraped image posts from the top 1000 most active subreddits with "meme" in name (e.g. r/historymemes). temporally, the dataset spans the decade between 2011 and mid-2021. we fine-tuned both the roberta and clip models for three epochs on memes whose template appeared at least 100 times in the dataset. we used an 80/10/10 split of train, dev and test data, ensuring there was no leakage of fill text between splits. using the pipeline with the roberta model results in 784 semantic clusters spanning 6,384 templates and over 3.8m meme instances. figure 4 shows some templates that appear in the same semantic cluster. the dataset includes posts to 655 subreddits by 908,917 users. we include examples and descriptive statistics for clusters generated with each of the embedding models in the appendix (figures 17 18 19 20 21 ). a qualitative examination of the clusters showed that the largest, most commonly occurring clusters were highly interpretable with clear semantics. for some of the less common clusters, it was more difficult to assign a clear semantic meaning, but even these clusters often had a coherent quality or affect. a potential line of qualitative future work would be to better understand and identify the unifying features of these clusters. ## evaluation we evaluate the coherence and visual diversity of semantic clusters derived from each model using human judgment. we design an evaluation task in which annotators are presented with a pair of templates, and randomly vary if the templates are drawn from the same or different semantic clusters. they are asked to evaluate whether the two templates are 1) semantically similar and 2) visually similar. we define semantic similarity as being able to reasonably substitute the text from one template into the other with minor changes. we define visual similarity to include sharing a similar art style or source (e.g., two different templates featuring spongebob). we include example pairs in the appendix; one strong source of visual similarity (cf. appendix fig. 19 ) are sets of templates that are largely identical in their form but that exhibit slight variation in size, crop, and margins. we collect judgments for the top ten semantic clusters from each model most commonly represented in our dataset as well as a random selection of ten clusters from each model. for each cluster, we sample 10 pairs, and the same human evaluators provided judgment across all the models. we find strong interannotator agreement (krippendorff's α = 0.75, calculated across all models). we provide more details on this process, including the agreement scores for each individual model, in section c.1 of the appendix. from the human judgments, we calculate p s (the probability that a pair of templates are semantically similar if they appear in the same cluster) and p v (the equivalent measurement for visual similarity) for each model. to measure variation, it is more important each semantic cluster is semantically coherent and visually diverse, but less important that all relevant templates are surfaced within the same cluster. following this reasoning, we focus on evaluating the semantic precision p s . our goal in this work is to explore meaningful semantic variation across visually diverse memes, since memes that are visually similar (e.g., slight variations on the same template) have trivially similar semantics. accordingly, we design a measure of visually adjusted precision based on cohen's κ: intuitively, this metric represents the extent to which the semantic clusters agree with annotator judgments of semantic similarity while controlling for correlations with visual similarity. a negative score means the model clusters based on visual similarity instead of semantic coherence.we calculate metrics on the model judgments over the set of all annotated pairs across models. this not only allows us to evaluate on a larger set of annotations, but also helps highlight differences between models. semantic clusters provide a strong separation between content (the semantic cluster) and style (the choice of template within a semantic cluster). in other words, the choice of semantic cluster is what a user is trying to say, and the choice of a template within that cluster is how they are saying it. in the remainder of the paper, we use the clusters generated from the roberta embeddings, which have the highest visual-adjusted precision, for our case studies on linguistic variation and change. ## linguistic variation the sociolinguistic study of variation centers around the linguistic variable, which captures different ways of saying the same thing. the specific choice a speaker make varies systematically based on information such as the speaker's identity, their relationship to interlocutors, sociopragmatic context, among many other factors @xcite . through variation, language conveys social meaning @xcite . there is a rich body of work that aims to analyze linguistic variation computationally. often, the focus is on lexical variation @xcite ; zhang et al., 2017; zhu and jurgens, 2021a); semantic variation in online communities @xcite ; or orthographic variation in online text @xcite . in our view of memes as language, we ask the same kind of question: rq1: does the template choice within a semantic cluster vary systematically between communities? methods. the semantic clusters form our variable context, and set of templates within any given semantic cluster form a discrete set of choices with the same semantic value. we use the weighted log odds-ratio to compute the extent to which a template is specific to a given subreddit compared to all other subreddits, relative to the other templates in a semantic cluster @xcite . we find the templates that have a statistically significant association with a subreddit (z-@xmath0.96); the semantic clusters these templates belong to are in variation: a community prefers one variant over the others in this cluster. results. we find 94 out of 784 semantic clusters exhibit statistically significant variation, spanning 391 different templates. figure 5 shows how functionally similar memes take different forms in different communities. speakers use language to construct their social identities @xcite . we find that, not only do subreddits prefer certain variants of a template over others, but they choose templates that index into a localized cultural knowledge, making cultural allusions to characters or celebrities. for example, the orange drake template (fig. 5c , left) is used frequently in general purpose meme subreddits like r/memes, but alternatives are used in other subreddits. one version that is specific to r/dndmemes (which discusses the role playing game dungeons and dragons) replaces drake with matthew mercer, a voice actor who stars in a popular dungeons and dragons web series (fig. 5c , middle). linguistic variants usually become associated with identities through a gradual process in which the association slowly permeates public awareness @xcite . in general, a phonological variable does not inherently index any given identity. however, the multimodality of memes permits greater expressiveness-a meme in r/animemes might use the anime art style, indexing into the aesthetic of that community explicitly. ## linguistic innovation equally as important as the study of synchronic linguistic variation is the study of diachronic linguistic change. language change has been heavily studied in natural language processing @xcite @xcite . we focus on understanding the innovation of meme templates within a semantic cluster. rq2: do new meme templates co-exist with preexisting templates in the semantic cluster, or does the most popular template monopolize the cluster? when multiple templates that fulfill the same function appear, we expect there to be competition. prior work has observed this competition between lexical choices, with two outcomes: new words replace old ones that serve the same function, but if similar words have discourse-relevant differences in meaning, they can coexist @xcite . ## linguistic acculturation finally, we study how users alter their meme posting habits as they spend more time in a subreddit. previous work on linguistic acculturation show that users adopt more community-specific language as they become enculturated within a community @xcite . we can ask a similar question here: rq4: do veteran users in a subreddit use more community-specific templates? methods to answer this question, we measure the average specificity of a user's posts in successive months after they enter a community. we once again calculate the ppmi of templates as a measure of specificity; this time, we calculate the value over the full range of the dataset. for each user in a subreddit, we bin their posts by 30-day windows starting with their first post in the subreddit (i.e., for each month after they joined), and compute the average ppmi of their posts for that time period. we filter the dataset to users with at least 10 lifetime posts and subreddits with at least 30 such users. to prevent extremely popular subreddits from unduly influencing the results, we sample up to 100 users from each subreddit to compute the average across all subreddits. this yields a total of 3,174 users in 130 subreddits. ## related work prior work on memes in nlp and social computing has largely focused on two tasks: meme understanding and modeling how memes originate and spread. our work offers novel methods and perspectives at the intersection of these areas of research. meme understanding encompasses a number of discrete tasks, including classifying if memes convey harmful messages @xcite , labeling emotion @xcite , and detecting humor @xcite or figurative speech @xcite within them. while these can generally be framed as classification tasks, other work generates open-ended explanations of visual humor using large multimodal language models @xcite . our work complements this existing body of research by inferring semantic variables in an unsupervised approach, leveraging the implicit structure within memes by modeling template semantics separately from the fills. in modeling the internal structure of memes, our work draws on existing research examining the relationship between fills and templates to match semantic roles to entities within harmful memes @xcite and mapping fill text to explanatory background information @xcite . we hope that our method of construing templates as semantic predicates can contribute to this body of work. in the social computing space, another line of research focuses on understanding how memes originate @xcite and spread across platforms @xcite . these treat meme templates as discrete tokens. we model template semantics, which have the granularity to enable analysis of variation and social meaning. qu et al. uses clip to understand how memes evolve as they spread. while they use the text in comments to model the high-level concepts indexed by particular variants, we use the fill text of memes to model low-level template semantics. ## conclusion in this paper, we analyze memes as a form of language subject to the same kinds of sociolinguistic variation as other modalities, such as written language and speech. we propose a new approach to understanding meme semantics, taking advantage of the multimodal structure of memes to learn semantic representations of templates from an unlabeled dataset. we use this method on a large dataset of memes scraped from reddit, and demonstrate that it yields coherent, visually diverse clusters of semantically similar memes. we make these clusters and the code publicly available for future research. finally, we use these clusters to study language variation and change in subreddits. we show that variations between meme template are socially meaningful and memes often share usage patterns with the textual language that has been studied in the past. we find that memes can be rich resources for understanding social language use.