Determining Event Outcomes: The Case of #fail (2020)

Abstract: This paper targets the task of determining event outcomes in social media. We work with tweets containing either #cookingFail or #bakingFail, and show that many of the events described in them resulted in something edible. Tweets that contain images are more likely to result in edible albeit imperfect outcomes. Experimental results show that edibility is easier to predict than outcome quality.

## introduction while the definition of event is controversial @xcite , there is general consensus that events occur (or happen, or take place) at a time and location. people share in social media a deluge of information including events they care about. these events range from mundane events such as eating or watching tv to important life events such as getting married and graduating from college @xcite . twitter is one of the most popular social networks with 166 million daily active users @xcite . an important property of events is whether they actually occurred. the literature has studied this property under different terms, e.g., factuality @xcite and veridicality @xcite . other related tasks have studied the level of commitment a speaker or writer has towards a proposition @xcite . assessing the degree to which an event occurred or is believed to be true is critical to make inferences and information extraction. even when an event is guaranteed to have occurred, however, it is not necessarily the case that the desired outcome came to fruition. for example, people make phone calls (presumably) to communicate with whoever they are calling. making the call, however, does not guarantee that the communication took placethe recipient could have not picked up the phone. some events have fairly clear desired outcomes even if they are not explicitly stated: people make phone calls to communicate, run in elections so that they are elected, etc. the desired outcomes of other events, however, are not so clear: people may plant a tree to help the environment, to provide privacy or shade, or so that it bears fruit. like factuality, determining whether an event resulted in the desired outcome is a matter or degree and not a binary decision. in other words, events often do not result in perfect outcomes or complete failures. for example, a phone call may result in communication that is far from perfect because there is background noise or because the call suddenly drops. consider the tweet in figure 1 . despite the hashtag #bakingfail, the baking was partially successful: something edible came out of the baking, although it was not visually appealing. in this paper, we target cooking and baking events that include some form of the hashtag #fail, and study the degree to which they resulted in their desired outcomes in terms of edibility and quality. the main contributions are: (a) a corpus of 4,000 tweets annotated with event outcome information in two stages: edibility and quality; foot_0 (b) analysis showing that more information can be extracted from tweets including an image; (c) experimental results showing that determining outcome quality remains a challenge; and (d) error analysis shedding light into the difficulty of the task. ## previous work the language of social media has been studied from many angles, including applications in the social sciences @xcite and public health @xcite . in the context of emergencies, detecting the first message about a new disaster and information aggregation are important problems @xcite . in this paper, we work with mundane events (cooking and baking) described in one tweet, and study the degree to which they resulted in their desired outcomes. event detection from social media has received considerable attention, in particular, pinpointing important life events @xcite . previous research shows that people often tweet about events they do not participate in @xcite , targets recurring events @xcite , and summarizes tweet streams about tv shows @xcite . the work presented here is not concerned with event detection, our selection criteria virtually guarantees that we only work with tweets about cooking or baking (section 3). determining the degree to which an event results in its desired outcome is distantly related to assessing factuality and other event properties. previous efforts working with social media target event factuality @xcite , identify controversial events @xcite and credible eyewitnesses @xcite , and work with arguably more challenging properties such as rumors @xcite and credibility @xcite . in the work presented here, we work with factual mundane events whose credibility is undisputed. lack of factuality or credibility indicates that an event did not occur thus also that the desired outcomes were not achieved. we note, however, that fac-tual and credible events did not necessarily result in their desired outcomes, as the examples in this paper illustrate with cooking and baking events. to our knowledge, there are only a few previous works investigating event outcomes from a computational perspective. outside the social media domain, @xcite investigate models to predict the outcome of sports events from interviews conducted shortly before the event. within social media, @xcite present models to determine whether people evacuate during a hurricane event from their tweets. finally, @xcite present a framework to forecast winners of events (e.g., sports events, elections, awards) by aggregating predictions made by individual users. our work differs in many respects. first, we work with mundane events (cooking and baking). second, we investigate a finer-grained characterization of event outcomes beyond binary decisions: edibility and quality. third, we work with tweets consisting of only text as well as both text and images, and show that the outcomes are easier to determine in the latter-in particular, edibility, both by humans and computational models. ## annotating event outcomes: cooking and baking events we create a new corpus of tweets annotated with event outcome information. initially we set to work with mundane events carried out by regular people and requiring some degree of skill. we explored the following events: driving, gardening, playing sports, singing, playing musical instruments, cooking, and sewing. after manually observing many tweets discussing these events, it became clear that event outcomes are often unknown for events that do not result in concrete outcomes. additionally, people barely discuss some of the events above unless they result in the expected outcome (e.g., most people talking about driving appear to be good drivers and reach their destinations). we decided to focus on cooking and baking events because (a) they require minimal expertise (i.e., most people can do some cooking); (b) are frequently discussed in social media; and (c) people often discuss the outcome of their cooking and baking in social media, including less than perfect outcomes. annotation guidelines. dictionaries define cooking as "prepare food for eating [. . . ]", and baking as "cook by dry heat especially in an oven" @xcite . thus the desired outcome of cooking or baking events is to create something edible. our event outcome annotation guidelines for cooking and baking events go beyond this binary distinction and include three steps. the first step is to identify relevant tweets, which we define as tweets that describe a cooking or baking event involving the author. annotators choose from the following labels for relevancy: the second step is to identify whether the cooking or baking event resulted in something edible. annotators choose from the following labels: the third step is to identify the quality of edible outcomes. after pilot annotations, we decided to let annotators choose among the following labels: all annotations were made with respect to the cooking or baking event up to the point the tweet was published. for example, the outcome of a tweet describing a baking cake event and mentioning that the oven tripped a circuit breaker would be annotated not edible despite it is possible that the baking was successful after resetting the breaker. annotation process and agreements. annotations were done in-house by two graduate students. both of them annotated 15% of tweets in each group (#cookingfail or #bakingfail, only text or text and image). we note that (a)  coefficients for both edibility and quality are slightly higher when tweets consist tweet with only text annotations relevant? edible? quality? 1: eating crumpets and watching master chef #cookingfail no n/a n/a 2: oh bugger. my oklava won't fit in my suitcase... #bakingfail no n/a n/a 3: right, if i wanna cook the appliances need to be plugged in #cookingfail yes no n/a 4: it's been so long since i've made cupcakes i forgot how to load my frosting gun. :x #bakingfail it's all good now. cupcakes are frosted... yes yes as expected 5: so the plan was to make oreo brownies... i wouldn't quite call it that but still taste pretty good #bakingfail yes yes partial success 6: i tried to make an omelet. it turned into scrambled eggs. #cookingfail yes yes alternative 7: today's dinner so did not go as planned but i guess the important thing is the kids are fed. #cookingfail yes yes unknown ## corpus analysis regarding quality, most cooking and baking events resulted in the expected dish with some flaw (partial success: 57.8% and 72.6%). additionally, people are more likely to share a picture if the cooking or baking event was a partial success rather than resulted in an alternative outcome. ## experiments and results we experiment with models to predict outcome edibility (yes or no) and outcome quality (as expected, partial success, alternative or unknown). we split the tweets into train (80%) and test (20%) splits, and report results evaluating in the test split with (a) the tweets consisting of only text and (b) tweets consisting of both text and an image. baselines. we work with the majority baseline (edibility: always no (only text) or yes (text and image), quality: always partial success for all tweets) and a supervised baseline using logistic regression. the logistic regression model uses bag-ofwords features and only considers the text in tweets as input-it disregards the image if tweets contain one. we use the implementation in the scikit-learn machine learning python package @xcite with default parameters, which in turn uses the liblinear library @xcite . ## error analysis we identify the most common error types made by the best model (nn, only text and nn, text + imgs) after manually analyzing 100 errors. tweets with only text. regarding outcome quality, we identify two major error types. the most common (41%) is also world knowledge. in the example, one must know that potatoes and beans have different cooking times; note that the text does not give any explicit cue about the quality of the resulting dish. a substantial amount of errors occur with tweets whose text lacks information to establish the outcome quality (gold: unknown). in this case, the model tends to predict the majority label, partial success. finally, the remaining errors (26%) are due to other reasons. in the example, the #cookingfail refers to a past cooking (last week), not the one that occurred shortly before tweeting. tweets with text and images. image is key (25%) human error (20%) alternative outcome (20%) gold edible: no gold edible: no gold edible: yes pred. edible: yes pred. edible: yes pred. edible: no lacks information (28%) image is key (26%) human error (7%) gold quality: unknown gold quality: partial success gold quality: partial success pred. quality; partial success pred. quality: unknown pred. quality: unknown regarding outcome edibility (top three examples in regarding outcome quality, we observe two error types covering over half of the errors and a long tail of additional types. first, some tweets lack information in the text and image (28% of errors) to determine the outcome quality (gold: unknown), and the model tends to predict the majority label (partial success). second, the image is key in 26% of errors, as illustrated with in the second example. in this example, the event outcome (edibility and quality) is very ambiguous without looking at the picture. finally, we also identified that the model struggles to identify partial success when cooks make some mistake (human error, 7% of all errors). in the third example, the cook forgot an ingredient but doing so did not result in a complete failure. ## conclusions factual and credible events do not necessarily result in their expected outcomes. in this paper, we target outcomes of cooking and baking events from social media. specifically, we determine whether something edible resulted from them, and also the outcome quality (as expected, partial success or alternative). an annotation effort with 4,000 tweets consisting of either only text or text and an image shows that people often use the hashtag #cookingfail or #bakingfail when the cooking did not result in a complete failure. indeed, the outcome is often edible albeit not perfect, especially if the tweet includes an image (59.7 vs. 31.8%). we believe that a similar approach could be used to assess outcomes of other events. for example, taking exams and going to the grocery store usually have clear expected outcomes: to pass the exam and to buy something. taking an exam or going shopping (factuality is not in question here), however, does not guarantee that the expected outcomes become a reality (e.g., people take exams and fail them).one may be able to determine not only whether instances of these events occurred, but also if they resulted in the desired outcomes.