What is the Real Intention behind this Question? Dataset Collection and Intention Classification (2023)

Abstract: Asking and answering questions are inseparable parts of human social life. The primary purposes of asking questions are to gain knowledge or request help which has been the subject of question-answering studies. However, questions can also reflect negative intentions and include implicit offenses, such as highlighting one’s lack of knowledge or bolstering an alleged superior knowledge, which can lead to conflict in conversations; yet has been scarcely researched. This paper is the first study to introduce a dataset (Question Intention Dataset) that includes questions with positive/neutral and negative intentions and the underlying intention categories within each group. We further conduct a meta-analysis to highlight tacit and apparent intents. We also propose a classification method using Transformers augmented by TF-IDF-based features and report the results of several models for classifying the main intention categories. We aim to highlight the importance of taking intentions into account, especially implicit and negative ones, to gain insight into conflict-evoking questions and better understand human-human communication on the web for NLP applications.

## introduction the essence of conversation is to communicate intentions; however, the uptake of what has been communicated entails more than merely decoding the words in the message @xcite . many layers underlie a communicative message, and as we interact, we try to decode the surface meaning as well as the tacit aspects @xcite . we further use established codes to interpret the meanings. for instance, a question is known to be a means of asking for information or requesting help. however, as humans, we also apply meta-knowledge to override the established rules; thus, we may interpret a question as deceitful, then consider it as a means to criticize. our question have you been invited to edit this article? positive/ neutral intention clarification/confirmation: inquiry to disambiguate whether an invitation was involved. ## background research studies that focus on intentions can be divided into those considering intentions from a linguistics viewpoint and those that consider the psycholinguistic view. the former is in respect to language itself, as in nlp studies on detecting intentions in dialogue systems @xcite , analyzing goals and purposes such as intent to purchase something or to travel @xcite and those focusing on open-domain question answering @xcite . the latter focuses on the speaker's meaning, belief, desire, and mental states, hence involving a wider scope. intentions within nlp area: research on intention analysis mainly draws on nlp and deep neural networks to detect the goal of the message and fulfill a task such as realizing a smooth conversation in chatbots @xcite , retrieving information in search engines @xcite , detecting a user's personal need or classifying feedback for marketing purposes and developing recommender systems @xcite @xcite . these studies mainly focus on affirmative or neutral intentions with the aim of associating users' intentions with pre-defined categories. they handle emerging intents via knowledge transfer from existing intents and group the utterances with similar intents (topics) to find the best response or strategy @xcite . hence they rarely deal with finding implicitly negative intentions, even though it happens in real-world conversations. research is often concerned with explicit attacks and hate speech, aiming to detect toxic behavior @xcite , such as hatred toward religious groups @xcite , racism (park and fung, 2017), sexism @xcite , cyberbullying @xcite , abusive @xcite , and offensive language @xcite . while explicit attacks have high priority @xcite , implicit instances of offensive language use are also important since, in many cases, offensive behavior is not explicitly demonstrated @xcite . thus, more recently studies have explored the use of implicitly abusive language @xcite , latent and indirect hatred on social media @xcite , abusive remarks on identity groups @xcite , stereotypes @xcite , disguised and implicit attacks @xcite and implicit hate speech detection on machine-generated dataset @xcite . interpretation and perceived meaning in conversation: intention and perceived meaning have been investigated from different perspectives @xcite including associating intentions with the speaker meaning @xcite , intention as the characteristics of the message @xcite , or as perceived by the addressee. other studies consider the notion of perspectivetaking and intention perceived as a product of joint communication between the speaker and listener @xcite , the speech acts @xcite , and cognitive processes involved in interpreting the meaning and action @xcite . meaning is not always perceived by the listener as intended by the speaker @xcite . considering both speaker's and addressee's perspectives is optimal for accurate interpretation @xcite ), yet it is not always feasible. thus, studies collect annotations from the readers but provide clear guidelines for higher annotation agreement @xcite , yet ascertain that a certain level of disagreement should be allowed in annotation @xcite . intentions behind questions: before answering a question, the meaning and intention of it need to be decoded. this is a necessary step for many nlp applications that deal with questions @xcite @xcite . most research on detecting question intention centers on finding the mapping between the user's query and the knowledge base to provide a user-satisfying response @xcite . the candidate answer is selected based on sentences ranked by the model score of its suitability @xcite . thus in these studies, a question is considered a query, and its intention is associated with the user's purpose within a specific or open domain @xcite . other studies investigated the form and function of questions @xcite @xcite , inferring appropriate questions for a given personal narrative such as advice-seeking @xcite , and the questions' semantic and pragmatic properties, such as rhetorical questions @xcite @xcite . in this research, we investigate the types of negative and implicit intentions behind questions that can be used as a means of attacking another person, as well as the positive/neutral questions that serve the primary purpose of asking to receive an answer. ## dataset collection our dataset is built on the conversation gone awry dataset @xcite , which encompasses the conversations on wikipedia talk pages @xcite . a combination of machine learning and crowdsourced filtering was used to gather these conversations that begin with civil comments and either remain civil or end with a personal attack (4188 conversations, >30k comments). wikipedia's talk page discussions are similar to public forums where contributors convene to deliberate on issues related to editing a page, including quality evaluation @xcite . wikipedia comments are known to contain a small number of antisocial behaviors-around one percent @xcite , but it includes many cases of negative attitudes @xcite , hence a good resource for our analysis. such cases may interfere with the original goal of improving articles and are disruptive to those seeking to contribute to improving the article in peace by collecting, sharing, and disseminating knowledge. from this dataset, we extracted 2,084 questions and annotated their underlying intentions. ## meta-analysis of the data ## classification of intention categories to address rq4, we classify the intention categories based on their polarity. for positive/neutral categories, we integrated the "true guidance" category with "clarification/confirmation" cases as these two categories were most often selected together (>80% overlap) when annotators could choose up to two categories. these cases were found to be complementary w.r.t our dataset. we also excluded the "show hostility" class from our classification to only focus on implicit cases. pre-processing: our pre-processing included replacing usernames, email addresses, urls, hashtags, and special symbols with assigned tokens and handling misspellings with textblob @xcite . context: we added the sentences preceding and succeeding the question, to provide the context to the classifier. if the question started/ended the comment, we used the remaining adjacent sentences. note that this procedure is done within the comment containing the question. while we recognize that including subsequent comments can enhance accuracy and may even be necessary to understand the question fully, we have restricted the context to only the comment containing the question in order to generate predictions for each individual comment as it is posted. classification method: we classify the intention categories using binary and multi-class classification methods. for binary classification, we target each category of intention individually and finetune a transformer model as a proof-of-concept. for the main task of multi-class classification, we propose an architecture to fine-tune transformers augmented by a tf-idf-based dictionary, depicted in figure 1 . the use of dictionary plus transformer has led to improvement in previous studies on relatively similar tasks @xcite . in the left branch, the question, together with its context, is given to a transformer, which is pre-trained and fine-tuned on the polarity labels of our dataset (positive/neutral vs. negative intentions). the transformer is then trained on our intention categories, and its outputs are fed into global max pooling. on the right side, we applied tf-idf to our proposed intention categories to find the most specific words within each class. we trim these vocabularies so that each word appears in only one category (uniqueness trimming) and discard words with higher frequency in spoken/written texts to build our tailored dictionary (see appendix b). the tf-idf dictionary was populated with highly specific words from the training portion of the dataset. we did this not only to avoid the risk of target validation leakage but also to enhance the transferability of the model to unseen conversations. words in each question are lemmatized and matched to dictionary vocabulary to make the feature list. we concatenate the max pooling output with dictionarybased features to be processed by three fc layers and output the label of the question's intention. competitive models: we chose svm, bert @xcite , roberta @xcite , and xlnet @xcite to build our baseline models for intention category classification and compared their results. these models were selected to yield competitive results in nlp tasks based on previous studies @xcite @xcite . we also used svm as a strong competitor for being fast and working well with fewer data. implementation details are provided in appendix c. we stratified the annotated data and randomly split it into training, val, and test sets (70:10:20). results are an average of 3 runs. ## results and analysis we conducted two experiments: binary and multiclass classification and reported the results based on precision (p), recall (r), and f1 score. ## conclusion this paper proposes the new problem of investigating how humans use questions as a means to attack others and disguise their intentions rather than asking sincere questions to get information. the goal is to incorporate such knowledge into the nlp area. we used the wikipedia discussions where the editors actively collaborate with the goal of improving wiki pages. we gathered and annotated questions from discussions to distinguish positive/neutral and negative intentions, plus the intention types. it is only after considering such information that we can learn why a question is perceived negatively. we did a meta-analysis to explore each class's characteristics and the role of thresholds and perspectives in interpreting questions. we also built a tf-idf dictionary-based transformer and benchmarked several classifiers on intention detection. questions are frequently used in conversations, and finding their true intentions is a non-negligible task for ai to understand human communications. the type of intention pursued and how it is perceived by people of different cultures/backgrounds need illumination through the inclusion of diverse perspectives. this future task will enrich research on human reasoning, thereby largely impacting the nlp area on understating human interactions.