Help! Need Advice on Identifying Advice (2020)

Abstract: Humans use language to accomplish a wide variety of tasks - asking for and giving advice being one of them. In online advice forums, advice is mixed in with non-advice, like emotional support, and is sometimes stated explicitly, sometimes implicitly. Understanding the language of advice would equip systems with a better grasp of language pragmatics; practically, the ability to identify advice would drastically increase the efficiency of advice-seeking online, as well as advice-giving in natural language generation systems. We present a dataset in English from two Reddit advice forums - r/AskParents and r/needadvice - annotated for whether sentences in posts contain advice or not. Our analysis reveals rich linguistic phenomena in advice discourse. We present preliminary models showing that while pre-trained language models are able to capture advice better than rule-based systems, advice identification is challenging, and we identify directions for future research.

## introduction humans use language in the real world to achieve many goals -communicate intents and desires, to argue and convince, and to ask for and give advice. in recent years, people have increasingly looked to the internet to find advice; advice forums like babycenter and r/needadvice have hundreds of thousands of members; studies also showed that people increasingly seek health advice online @xcite . however, finding the right solution to a problem is difficult, since advice may be spread over multiple posts and pages online. even within the same post, not all sentences contain relevant advice, like in the following (truncated) reply to a question titled is it too late to start a hobby/activity at 12?: only the italicised sentences are advice to the question asked. both sentences that follow the advice sentences lend support to the advice, rather than containing advice towards a course of action themselves. people also give advice in different ways @xcite , often implicitly like in the following reply to a question titled parenting with a history of depression?, where advice is implicitly conveyed via personal experience: automatic identification of advice in text would thus be extremely useful. yet, as we see above, it would also require a deep understanding of semantics and discourse pragmatics. in recent years, nlp systems based on large-scale pre-trained language models have shown impressive gains on several linguistic benchmarks @xcite @xcite . however, these same models have been found to struggle at tasks that require higher-level processing @xcite , including giving advice @xcite . this work aims to advance both our understanding of how people give advice, as well as to provide resources for learning to identify advice. first, we construct a dataset of annotations of advice in english from two advice-focused reddit commu-nities -r/askparents and r/needadvice, totalling 18456 sentences across 684 posts ( §3). these two subreddits are different in a number of respects. r/needadvice is a general advice forum, while r/askparents targets a specific audienceparents-who are often active seekers of advice. r/needadvice is more strongly moderated than r/askparents. in addition, our analysis shows that r/askparents contains more implicit, narrative advice than r/needadvice ( §4). through this dataset we provide the first-of-its-kind resources to explore the breadth of advice-giving strategies, and testbeds for modeling advice. we establish benchmarks for this task with bert @xcite , a large pre-trained language model, to identify sentences that constitute advice. we find that it is substantially better than a rule-based approach ( §5). in an in-depth analysis, we find that bert re-discovers some linguistic rules that have been previously proposed for identifying advice, but struggles with advice that is more implicit, for example in the form of a narrative, like in (2) ( §7). our results also show that r/askparents is more challenging for advice identification, despite the fact that r/needadvice has a wider range of topics. we make all of our data and code available online foot_0 . ## related work advice strategies there has been sociological and pragmatic work analysing how people navigate the task of engaging in advice discourse. people weigh interactional costs when giving and asking for advice @xcite , and they engage in various strategies to persuade their interlocutor and achieve their goals. effective advice givers were found to engage in roles that extended beyond giving advice -they help advice seekers clarify their problem, list possible solutions and sort through them, offer support and reassurance, and more @xcite . while there has been work by @xcite looking at how people use personal narratives to ask for advice online, no work thus far has looked at the discourse of advice giving online. semeval semeval-2019 introduced a pilot task on suggestion mining @xcite , recognizing the growing importance of identifying whether a text contains a suggestion towards a course of action or not. the dataset only considers sentences that explicitly include suggestions -that is, where one can infer without context that a sentence is a suggestion -while we always give the annotators the wider context of the entire post and question, and ask them to evaluate which sentences are advice based on this wider context. for instance, (2) is advice in the context of the question, but that same narrative could also be support for advice, given a different question. additionally, suggestions are not synonymous with advice, and can include tips and recommendations (although none of these terms are mutually exclusive). for example, you should try the food at italian restaurant might be construed as a tip or a recommendation, rather than advice. semeval-2019 task 9 provides two datasetsone from a software suggestions forum and another from a hotel reviews website. while the dataset and the suggestion mining models are useful for understanding suggestions, we find that the definition of suggestion is too constrained -explicit suggestions will not include many implicit instances of advice, which we are interested in studying. secondly, we find the domain of their datasets to be somewhat restricted, and not representative of the wide range of online advice-seeking behavior. we chose to construct datasets based on subreddits devoted to asking for advice related to parenting and general issues, since we want to understand how to model general human advice-seeking interactions. we target parenting as parents frequently seek and give advice online, and express it in linguistically diverse forms. for general advice, r/needadvice has clear grouping mechanisms ("flairs") that inform us with the topic of advice, which we use during analysis. turingadvice contemporaneous work from @xcite introduces a new framework to evaluate the performance of language models. turingadvice challenges models to generate advice that is at least as helpful to the advice seeker as human generated advice. they introduce a new dataset called redditadvice, which scrapes posts from a wide variety of advice subreddits. annotators on mechanical turk were presented with a reddit post seeking advice, along with two replies to the post, and were asked to choose which reply constitutes the more helpful advice. however, as (1) shows, the entirety of a response to a question rarely constitutes advice. in contrast, our work annotates and identifies explicit and implicit advice within a reply to an advice-seeking posts and finds that less than 40% of sentences in a reply are actually advice ( ## preliminary analysis as noted previously, r/askparents and r/needadvice differ with respect to their styles of moderation, but they are also different communities that may engage in giving advice differently. to understand how this impacts the structure of replies to posts, we manually analyzed 10 different posts from r/askparents, and 4 different posts each from the flairs of r/needadvice. we observed that people often give advice by alluding to their personal experience, for example: otherwise, a range of pragmatic strategies are adopted as noted by @xcite , including the use of questions, imperatives, conditionals, etc.: personal narratives are particularly interesting because it can be used to express advice indirectly, as in example (2). the discourse marker "though" is frequently used for signalling concession and contrast @xcite . this intuition is confirmed by an analysis of the discourse connective "though" among all posts we collected, which revealed a clear tendency towards the end of a reply, as illustrated in figure 1 .the lexical discourse marker "though" was found by splitting a large collection of posts and replies from r/askparents into elementary discourse units @xcite , using a neural discourse segmenter @xcite . ## models task setup we have constructed a dataset from the subreddits r/needadvice and r/askparents as a general purpose resource for studying the breadth of advice-giving strategies. our modelling experiments aim to establish baseline performance for rule-based models and language models at identifying advice, as well as explore how their performance varies with domain and provided context. we model advice identification as a binary classification task -given a sentence, predict whether the sentence is advice or not. however, some of these rules are naive and not intepretable -such as classifying a sentence as a suggestion if it contains a modal or the base form of a verb. @xcite improve upon this baseline with more keywords and phrases, searching for more rigorous pos patterns within clauses rather than sentences, and assigning different confidence scores for keyword and pos matches foot_4 . a sentence is classified as a suggestion if it exceeds a preset confidence score. ## results baseline the performance of the baseline models and the finetuned language models are given in ## analysis we chose the bert sent model -the best performing model on both datasets, and analyzed the attention weights to see if they show some of the patterns we used in the baseline models. the attention weights were visualized using bertviz @xcite . ## conclusion we introduce a new dataset on advice given on the online platform reddit, specifically r/askparents and r/needadvice that differ in audience and level of moderation. we find that advice language consists of various pragmatic strategies and discourse structures. we find that fine-tuned bert discovers certain surface-level features indicative of advice, but struggles to disambiguate instances of implicit advice conveyed through personal narrative. future work needs to look into how question and reply context can improve automatic identification of advice.