z-uo/qasper-squad huggingface dataset:
Paper: Assessing the Applicability of Authorship Verification Methods
Digital text forensics aims at examining the originality and credibility of information in electronic documents and, in this regard, at extracting and analyzing information about the authors of the respective texts BIBREF0 . Among the most important tasks of this field are authorship attribution (AA) and authorship verification (AV), where the former deals with the problem of identifying the most likely author of a document INLINEFORM0 with unknown authorship, given a set of texts of candidate authors. AV, on the other hand, focuses on the question whether INLINEFORM1 was in fact written by a known author INLINEFORM2 , where only a set of reference texts INLINEFORM3 of this author is given. Both disciplines are strongly related to each other, as any AA problem can be broken down into a series of AV problems BIBREF1 . Breaking down an AA problem into multiple AV problems is especially important in such scenarios, where the presence of the true author of INLINEFORM4 in the candidate set cannot be guaranteed.
Question: Do they report results only on English data?
Paper: Assessing the Applicability of Authorship Verification Methods
Over the years, researchers in the field of authorship analysis identified a number of challenges and limitations regarding existing studies and approaches. Azarbonyad et al. BIBREF8 , for example, focused on the questions if the writing styles of authors of short texts change over time and how this affects AA. To answer these questions, the authors proposed an AA approach based on time-aware language models that incorporate the temporal changes of the writing style of authors. In one of our experiments, we focus on a similar question, namely, whether it is possible to recognize the writing style of authors, despite of large time spans between their documents. However, there are several differences between our experiment and the study of Azarbonyad et al. First, the authors consider an AA task, where one anonymous document INLINEFORM0 has to be attributed to one of INLINEFORM1 possible candidate authors, while we focus on an AV task, where INLINEFORM2 is compared against one document INLINEFORM3 of a known author. Second, the authors focus on texts with informal language (emails and tweets) in their study, while in our experiment we consider documents written in a formal language (scientific works). Third, Azarbonyad et al. analyzed texts with a time span of four years, while in our experiment the average time span is 15.6 years. Fourth, in contrast to the approach of the authors, none of the 12 examined AV approaches in our experiment considers a special handling of temporal stylistic changes.
Question: Which is the best performing method?
Paper: Assessing the Applicability of Authorship Verification Methods
Before we can assess the applicability of AV methods, it is important to understand their fundamental characteristics. Due to the increasing number of proposed AV approaches in the last two decades, the need arose to develop a systematization including the conception, implementation and evaluation of authorship verification methods. In regard to this, only a few attempts have been made so far. In 2004, for example, Koppel and Schler BIBREF13 described for the first time the connection between AV and unary classification, also known as one-class classification. In 2008, Stein et al. BIBREF14 compiled an overview of important algorithmic building blocks for AV where, among other things, they also formulated three AV problems as decision problems. In 2009, Stamatatos BIBREF15 coined the phrases profile- and instance-based approaches that initially were used in the field of AA, but later found their way also into AV. In 2013 and 2014, Stamatatos et al. BIBREF11 , BIBREF16 introduced the terms intrinsic- and extrinsic models that aim to further distinguish between AV methods. However, a closer look at previous attempts to characterize authorship verification approaches reveals a number of misunderstandings, for instance, when it comes to draw the borders between their underlying classification models. In the following subsections, we clarify these misunderstandings, where we redefine previous definitions and propose new properties that enable a better comparison between AV methods.
Question: What size are the corpora?
Paper: Assessing the Applicability of Authorship Verification Methods
Reliability is a fundamental property any AV method must fulfill in order to be applicable in real-world forensic settings. However, since there is no consistent concept nor a uniform definition of the term “reliability” in the context of authorship verification according to the screened literature, we decided to reuse a definition from applied statistics, and adapt it carefully to AV.
Question: What is a self-compiled corpus?
Paper: Assessing the Applicability of Authorship Verification Methods
Another important property of an AV method is optimizability. We define an AV method as optimizable, if it is designed in such a way that it offers adjustable hyperparameters that can be tuned against a training/validation corpus, given an optimization method such as grid or random search. Hyperparameters might be, for instance, the selected distance/similarity function, the number of layers and neurons in a neural network or the choice of a kernel method. The majority of existing AV approaches in the literature (for example, BIBREF13 , BIBREF23 , BIBREF24 , BIBREF22 , BIBREF31 , BIBREF4 , BIBREF32 , BIBREF16 ) belong to this category. On the other hand, if a published AV approach involves hyperparameters that have been entirely fixed such that there is no further possibility to improve its performance from outside (without deviating from the definitions in the publication of the method), the method is considered to be non-optimizable. Non-optimizable AV methods are preferable in forensic settings as, here, the existence of a training/validation corpus is not always self-evident. Among the proposed AV approaches in the respective literature, we identified only a small fraction BIBREF21 , BIBREF2 , BIBREF30 that fall into this category.
Question: What are the 12 AV approaches which are examined?
Answer: MOCC, OCCAV, COAV, AVeer, GLAD, DistAV, Unmasking, Caravel, GenIM, ImpGI, SPATIUM and NNCD
Paper: A Trolling Hierarchy in Social Media and A Conditional Random Field For Trolling Detection
In contrast to traditional content distribution channels like television, radio and newspapers, Internet opened the door for direct interaction between the content creator and its audience. One of these forms of interaction is the presence of comments sections that are found in many websites. The comments section allows visitors, authenticated in some cases and unauthenticated in others, to leave a message for others to read. This is a type of multi-party asynchronous conversation that offers interesting insights: one can learn what is the commenting community thinking about the topic being discussed, their sentiment, recommendations among many other. There are some comment sections in which commentators are allowed to directly respond to others, creating a comment hierarchy. These kind of written conversations are interesting because they bring light to the types interaction between participants with minimal supervision. This lack of supervision and in some forums, anonymity, give place to interactions that may not be necessarily related with the original topic being discussed, and as in regular conversations, there are participants with not the best intentions. Such participants are called trolls in some communities.
Question: how was annotation done?
Answer: Annotation was done with the help of annotators from Amazon Mechanical Turk on snippets of conversations
Paper: A Trolling Hierarchy in Social Media and A Conditional Random Field For Trolling Detection
Based on the previous definitions we identify four aspects that uniquely define a trolling event-response pair: 1) Intention: what is the author of the comment in consideration purpose, a) trolling, the comment is malicious in nature, aims to disrupt, annoy, offend, harm or spread purposely false information, b) playing the comment is playful, joking, teasing others without the malicious intentions as in a), or c) none, the comment has no malicious intentions nor is playful, it is a simple comment. 2) Intention Disclosure: this aspect is meant to indicate weather a trolling comment is trying to deceive its readers, the possible values for this aspect are a) the comment's author is a troll and is trying to hide its real intentions, and pretends to convey a different meaning, at least temporarily, b) the comment's author is a troll but is clearly exposing its malicious intentions and c) the comment's author is not a troll, therefore there are not hidden or exposed malicious or playful intentions. There are two aspects defined on the comments that direct address the comment in consideration, 3) Intentions Interpretation: this aspect refers to the responder's understanding of the parent's comment intentions. There possible interpretations are the same as the intentions aspect: trolling, playing or none. The last element, is the 4) Response strategy employed by the commentators directly replaying to a comment, which can be a trolling event. The response strategy is influenced directly by the responder's interpretation of the parent's comment intention. We identify 14 possible response strategies. Some of these strategies are tied with combinations of the three other aspects. We briefly define each of them in the appendix.
Question: what is the source of the new dataset?
Paper: Word frequency and sentiment analysis of twitter messages during Coronavirus pandemic
Catastrophic global circumstances have a pronounced effect on the lives of human beings across the world. The ramifications of such a scenario are experienced in diverse and multiplicative ways spanning routine tasks, media and news reports, detrimental physical and mental health, and also routine conversations. A similar footprint has been left by the global pandemic Coronavirus particularly since February 2020. The outbreak has not only created havoc in the economic conditions, physical health, working conditions, and manufacturing sector to name a few but has also created a niche in the minds of the people worldwide. It has had serious repercussions on the psychological state of the humans that is most evident now.
Question: Do the authors give examples of positive and negative sentiment with regard to the virus?
Paper: Word frequency and sentiment analysis of twitter messages during Coronavirus pandemic
Several researchers have devised statistical and mathematical techniques to analyze literary artifacts. A substantially significant approach among these is inferring the pattern of frequency distributions of the words in the text BIBREF2. Zipf's law is mostly immanent in word frequency distributions BIBREF3, BIBREF4. The law essentially proclaims that for the words' vector $x$, the word frequency distribution $\nu $ varies as an inverse power of $x$. Some other distributions that are prevalent include Zipf–Mandelbrot BIBREF5, lognormal BIBREF6, BIBREF7, and Gauss–Poisson BIBREF6. Such studies have been conducted for several languages such as Chinese BIBREF8, Japanese BIBREF9, Hindi BIBREF10 and many others BIBREF2. Not only single word frequencies, but also multi-word frequencies have been exhaustively explored. One of the examples is BIBREF11 wherein bigram and trigram frequencies and versatilities were analyzed and 577 different bigrams and 6,140 different trigrams were reported.
Question: Which word frequencies reflect on the psychology of the twitter users, according to the authors?
Paper: Word frequency and sentiment analysis of twitter messages during Coronavirus pandemic
In this section, we present the details of the analysis performed on the data obtained pertaining to Twitter messages from January 2020 upto now, that is the time since the news of the Coronavirus outbreak in China was spread across nations. The word frequency data corresponding to the twitter messages has been taken from BIBREF16. The data source indicates that during March 11th to March 30th there were over 4 million tweets a day with the surge in the awareness. Also, the data prominently captures the tweets in English, Spanish, and French languages. A total of four datasets have been used to carry out the study.
Question: Do they specify which countries they collected twitter data from?
Paper: Word frequency and sentiment analysis of twitter messages during Coronavirus pandemic
First, we consider the data corresponding to the number of twitter ids tweeting about coronavirus at a particular time. Fig. FIGREF1 depicts the pattern of the twitter id evolution. A couple of peaks can be observed in its evolution in the months of February and March.
Question: Do they collect only English data?
Paper: e-QRAQ: A Multi-turn Reasoning Dataset and Simulator with Explanations
In recent years deep neural network models have been successfully applied in a variety of applications such as machine translation BIBREF0 , object recognition BIBREF1 , BIBREF2 , game playing BIBREF3 , dialog BIBREF4 and more. However, their lack of interpretability makes them a less attractive choice when stakeholders must be able to understand and validate the inference process. Examples include medical diagnosis, business decision-making and reasoning, legal and safety compliance, etc. This opacity also presents a challenge simply for debugging and improving model performance. For neural systems to move into realms where more transparent, symbolic models are currently employed, we must find mechanisms to ground neural computation in meaningful human concepts, inferences, and explanations. One approach to this problem is to treat the explanation problem itself as a learning problem and train a network to explain the results of a neural computation. This can be done either with a single network learning jointly to explain its own predictions or with separate networks for prediction and explanation. Regardless, the availability of sufficient labelled training data is a key impediment. In previous work BIBREF5 we developed a synthetic conversational reasoning dataset in which the User presents the Agent with a simple, ambiguous story and a challenge question about that story. Ambiguities arise because some of the entities in the story have been replaced by variables, some of which may need to be known to answer the challenge question. A successful Agent must reason about what the answers might be, given the ambiguity, and, if there is more than one possible answer, ask for the value of a relevant variable to reduce the possible answer set. In this paper we present a new dataset e-QRAQ constructed by augmenting the QRAQ simulator with the ability to provide detailed explanations about whether the Agent's response was correct and why. Using this dataset we perform some preliminary experiments, training an extended End-to-End Memory Network architecture BIBREF6 to jointly predict a response and a partial explanation of its reasoning. We consider two types of partial explanation in these experiments: the set of relevant variables, which the Agent must know to ask a relevant, reasoned question; and the set of possible answers, which the Agent must know to answer correctly. We demonstrate a strong correlation between the qualities of the prediction and explanation.
Question: How do they measure correlation between the prediction and explanation quality?
Answer: They look at the performance accuracy of explanation and the prediction performance
Paper: e-QRAQ: A Multi-turn Reasoning Dataset and Simulator with Explanations
Current interpretable machine learning algorithms for deep learning can be divided into two approaches: one approach aims to explain black box models in a model-agnostic fashion BIBREF7 , BIBREF8 ; another studies learning models, in particular deep neural networks, by visualizing for example the activations or gradients inside the networks BIBREF9 , BIBREF10 , BIBREF11 . Other work has studied the interpretability of traditional machine learning algorithms, such as decision trees BIBREF12 , graphical models BIBREF13 , and learned rule-based systems BIBREF14 . Notably, none of these algorithms produces natural language explanations, although the rule-based system is close to a human-understandable form if the features are interpretable. We believe one of the major impediments to getting NL explanations is the lack of datasets containing supervised explanations.
Question: Does the Agent ask for a value of a variable using natural language generated text?
Paper: An overview of embedding models of entities and relationships for knowledge base completion
Before introducing the KB completion task in details, let us return to the classic Word2Vec example of a “royal” relationship between “ $\mathsf {king}$ ” and “ $\mathsf {man}$ ”, and between “ $\mathsf {queen}$ ” and “ $\mathsf {woman}$ .” As illustrated in this example: $v_{king} - v_{man} \approx v_{queen} - v_{woman}$ , word vectors learned from a large corpus can model relational similarities or linguistic regularities between pairs of words as translations in the projected vector space BIBREF0 , BIBREF1 . Figure 1 shows another example of a relational similarity between word pairs of countries and capital cities: $
v_{Japan} - v_{Tokyo} &\approx & v_{Germany} - v_{Berlin}\\
v_{Germany} - v_{Berlin} &\approx & v_{Italy} - v_{Rome} \\
v_{Italy} - v_{Rome} &\approx & v_{Portugal} - v_{Lisbon}
$ 
Question: What models does this overview cover?
Paper: Lingke: A Fine-grained Multi-turn Chatbot for Customer Service
 $\dagger $ Corresponding author. This paper was partially supported by National Key Research and Development Program of China (No. 2017YFB0304100), National Natural Science Foundation of China (No. 61672343 and No. 61733011), Key Project of National Society Science Foundation of China (No. 15-ZDA041), The Art and Science Interdisciplinary Funds of Shanghai Jiao Tong University (No. 14JCRZ04). This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/.
Question: What datasets are used to evaluate the introduced method?
Answer: They used a dataset from Taobao which contained a collection of conversation records between customers and customer service staffs. It contains over five kinds of conversations,
including chit-chat, product and discount consultation, querying delivery progress and after-sales feedback.
Paper: Lingke: A Fine-grained Multi-turn Chatbot for Customer Service
This section presents the architecture of Lingke, which is overall shown in Figure 1 .
Question: What are the results achieved from the introduced method?
Answer: Their model resulted in values of 0.476, 0.672 and 0.893 for recall at position 1,2 and 5 respectively in 10 candidates.
Paper: Learning Relational Dependency Networks for Relation Extraction
The problem of knowledge base population (KBP) – constructing a knowledge base (KB) of facts gleaned from a large corpus of unstructured data – poses several challenges for the NLP community. Commonly, this relation extraction task is decomposed into two subtasks – entity linking, in which entities are linked to already identified identities within the document or to entities in the existing KB, and slot filling, which identifies certain attributes about a target entity.
Question: How do they incorporate human advice?
Answer: by converting human advice to first-order logic format and use as an input to calculate gradient
Paper: Learning Relational Dependency Networks for Relation Extraction
We present the different aspects of our pipeline, depicted in Figure FIGREF1 . We will first describe our approach to generating features and training examples from the KBP corpus, before describing the core of our framework – the RDN Boost algorithm.
Question: What do they learn jointly?
Paper: Stance Classification for Rumour Analysis in Twitter: Exploiting Affective Information and Conversation Structure
Nowadays, people increasingly tend to use social media like Facebook and Twitter as their primary source of information and news consumption. There are several reasons behind this tendency, such as the simplicity to gather and share the news and the possibility of staying abreast of the latest news and updated faster than with traditional media. An important factor is also that people can be engaged in conversations on the latest breaking news with their contacts by using these platforms. Pew Research Center's newest report shows that two-thirds of U.S. adults gather their news from social media, where Twitter is the most used platform. However, the absence of a systematic approach to do some form of fact and veracity checking may also encourage the spread of rumourous stories and misinformation BIBREF0 . Indeed, in social media, unverified information can spread very quickly and becomes viral easily, enabling the diffusion of false rumours and fake information.
Question: Is this an English-language dataset?
Paper: Stance Classification for Rumour Analysis in Twitter: Exploiting Affective Information and Conversation Structure
The SemEval-2017 Task 8 Task A BIBREF2 has as its main objective to determine the stance of the users in a Twitter thread towards a given rumour, in terms of support, denying, querying or commenting (SDQC) on the original rumour. Rumour is defined as a “circulating story of questionable veracity, which is apparently credible but hard to verify, and produces sufficient skepticism and/or anxiety so as to motivate finding out the actual truth” BIBREF7 . The task was very timing due to the growing importance of rumour resolution in the breaking news and to the urgency of preventing the spreading of misinformation.
Question: What affective-based features are used?
Answer: affective features provided by different emotion models such as Emolex, EmoSenticNet, Dictionary of Affect in Language, Affective Norms for English Words and Linguistics Inquiry and Word Count
Paper: Stance Classification for Rumour Analysis in Twitter: Exploiting Affective Information and Conversation Structure
We developed a new model by exploiting several stylistic and structural features characterizing Twitter language. In addition, we propose to utilize conversational-based features by exploiting the peculiar tree structure of the dataset. We also explored the use of affective based feature by extracting information from several affective resources including dialogue-act inspired features.
Question: What conversation-based features are used?
Paper: Bleaching Text: Abstract Features for Cross-lingual Gender Prediction
Author profiling is the task of discovering latent user attributes disclosed through text, such as gender, age, personality, income, location and occupation BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 . It is of interest to several applications including personalized machine translation, forensics, and marketing BIBREF7 , BIBREF8 .
Question: What are the evaluation metrics used?