Semantic Congruency Facilitates Memory for Emojis (2022)

Abstract: Emojis can assume different relations with the sentence context in which they occur. While affective elaboration and emoji-word redundancy are frequently investigated in laboratory experiments, the role of emojis in inferential processes has received much less attention. Here, we used an online ratings task and a recognition memory task to investigate whether differences in emoji function within a sentence affect judgments of emoji-text coherence and subsequent recognition accuracy. Emojis that function as synonyms of a target word from the passages were rated as better fitting with the passage (more coherent) than emojis consistent with an inference from the passage, and both types of emojis were rated as more coherent than incongruent (unrelated) emojis. In a recognition test, emojis consistent with the semantic content of passages (synonym and inference emojis) were better recognized than incongruent emojis. Findings of the present study provide corroborating evidence that readers extract semantic information from emojis and then integrate it with surrounding passage content.

## emojis and word processing recent research using both ratings tasks and online processing measures has shown that emojis that are redundant (i.e., synonymous) with a target word can facilitate text comprehension (daniel and camp, 2020). barach and colleagues @xcite used eye tracking measures to examine how readers benefit from the presence of non-face emojis that were positioned at the end of sentences and were synonymous with the target word (e.g., coffee). they compared sentences with emojis that were either semantically congruent with the target word (e.g., "my tall coffee is just the right temperature "), semantically incongruent (e.g., "my tall coffee is just the right temperature "), and sentences without an emoji ("my tall coffee is just the right temperature"). participants in their experiment read the sentences for comprehension while their eye movements were recorded. the congruent emojis were skipped more often and fixated for less time than the incongruent emojis, and the overall sentence reading times were shorter when the emojis were congruent compared to incongruent with the proceeding text. these effects of semantic congruency reported by @xcite suggest that, similar to semantic congruency effects with words, readers extract semantic information from emojis and integrate it with the surrounding text. similar findings were shown when text includes face emojis, which can convey more subtle and less literal meanings than non-face emojis @xcite , and when emojis were used to replace words in sentences @xcite @xcite . ## emojis and higher-level language processing in contrast to the research summarized above that examined linguistic processing of emojis associated with the meaning of isolated words, other studies have examined how readers process sentences with emojis that cannot be mapped to a single word, such as the detection of sarcastic intent @xcite and other types of indirect messages that go beyond the literal meaning of a statement @xcite . both behavioral and neurophysiological measures indicate that a winking face emoji invites sarcastic and ironic interpretations during reading @xcite . for example, accuracy in detecting sarcastic intent increased when a winking face emoji was present, demonstrating that the presence of an emoji can promote sarcastic relative to more literal sentence interpretations @xcite . in addition to sarcasm, the presence of an emoji can disambiguate other indirect aspects of sentence meaning. holtgraves and robinson @xcite had participants read a series of questions followed by indirect replies that either contained an emoji (e.g., a worried face) or did not contain an emoji. for disclosures (e.g., "how did you do in chemistry?") and opinions (e.g., "what did you think of my presentation?"), but not requests for actions (e.g., "can you type my term paper for me?"), judgements about whether a reply was congruent with the intent of the question were more accurate and faster when the reply contained an emoji than when it did not. collectively, these findings provide evidence that emojis can help to offset a discrepancy between literal and nonliteral aspects of meaning, often associated with grasping a speaker's intent. ## memory for emojis in text in contrast to the comprehension effects delineated above, only two studies have examined effects of emojis on memory @xcite . chatzichristos and colleagues @xcite provided evidence that emojis can influence memory for autobiographical events during reading. participants in this study completed a retrieval task for autobiographic memories cued by a word paired either with a positive or a negative face emoji. emotional incongruity of word-emoji pairs led to longer reaction times for retrieval as well as enhanced activation in brain areas associated with languageinduced semantic conflict, suggesting that emoji affect influenced memory retrieval. to examine how memory for emojis differs from memory for words, homann and colleagues @xcite compared isolated words or emojis under full attention and under divided attention conditions. in the divided attention condition, participants recalled the previously studied stimuli while completing a distractor 1-back recall task with one of three types of materials (i.e., words, emojis, and shapes). in that study, recall performance was better for isolated emojis than for words. in addition, recall memory for words was more disrupted when the simultaneous distractor task involved words, whereas declines in recall accuracy were smaller when the distractor task involved emojis. however, emoji recall was equally good when a distractor task involved emo-jis, words, or shapes. the authors concluded that because emojis have not only verbal but also distinctive visuo-spatial attributes, they interfere less with memory for single words and for each other than do words. the findings by @xcite suggest that emojis impact memory retrieval, and the findings by @xcite suggest that the combination of both verbal and visuo-spatial attributes of emojis makes them easier to recall in memory than words. the levels of processing framework proposes that semantic elaboration results in stronger memory traces than shallower processing @xcite , therefore, memory can be enhanced for content that is semantically congruent with surrounding context (e.g., @xcite ). the classic finding is that when participants are instructed to process single words for form (shallow processing) as compared with for meaning (deep processing), encoding for deep level processing takes longer but, importantly, recall accuracy is higher @xcite . typically, comparisons of processing depth for recognition memory have been demonstrated for single words @xcite . researchers have yet to examine memory for emojis whose integration with text requires different degrees of semantic processing. central to the present study, we understand inferential processing to entail a deeper level of semantic analysis than the semantic processing of isolated words @xcite and we ask whether memory can be modulated by emoji-induced elaborations on critical regions of a sentence @xcite . precisely aligned stress on critical words in speech @xcite , or italicizing or bolding in text @xcite , can signal focus. here we assume that emojis can provide a similar marker of focus and we ask whether memory for emojis may differ depending on whether the emoji-text relation supports shallower (word substitution) or deeper (inference) processing. emoji condition passage congruent inference bobby brought home a new pet to live in the tank. he hoped he would remember to buy some mice to feed it on his way home from basketball practice the next night. congruent synonym bobby brought home a new pet to live in the tank. he hoped he would remember to buy some mice to feed it on his way home from basketball practice the next night. incongruent bobby brought home a new pet to live in the tank. he hoped he would remember to buy some mice to feed it on his way home from basketball practice the next night. based on the levels of processing framework and prior work showing that readers show semantic congruency effects for emojis during reading @xcite , we predicted that readers would show higher recognition accuracy for emojis that are congruent with the surrounding passage content (synonym and inference emojis) than incongruent emojis. extrapolating from the levels of processing framework (craik and lockhart, 1972), we also hypothesized a memory advantage for emojis consistent with passage inferences, such that readers would show higher recognition accuracy for inference emojis than for synonym emojis. ## results data from 23 participants who had low recognition accuracy (i.e., below 60% accuracy) were removed from the dataset. after removal, the dataset consists of data from 66 participants. ## discussion the goal of the present study was to examine emojitext coherence and recognition memory for emojis that appear in passage-final position and relate to passages in one of three ways. with respect to judgments of coherence, readers judged congruent emojis (synonym and inference emojis) as better fitting with the passages than incongruent emojis. in addition, readers judged synonym emojis as better fitting with the passages than inference emojis. with respect to recognition accuracy, results show that, consistent with the levels of processing framework, readers had more accurate recognition memory for emojis that were semantically relevant to the paired text compared to emojis that were not related to the text, suggesting that readers encoded the semantic content of the emojis and integrated it with the surrounding text. the present study extends previous findings suggesting that readers show semantic congruency effects in online processing measures @xcite by providing insight about what is retained in memory shortly after reading passages with emojis. specifically, our findings suggest that readers encode the semantic properties of emojis in memory and show better recognition of semantically congruent emojis compared to semanticallyincongruent emojis after a short delay. contrary to the degree of semantic elaboration within the levels of processing framework (craik and lockhart, 1972), readers showed a numerical, but not a statistically reliable benefit for inference emojis compared to synonym emojis.