qinchuanhui/UDA-QA huggingface dataset:
Question: What is the task of slot filling?

Evidence:
raw_evidence: Slot Filling is an information extraction task which has become popular in the last years BIBREF3 . It is a shared task organized by the Text Analysis Conference (TAC). The task aims at extracting information about persons, organizations or geo-political entities from a large collection of news, web and discussion forum documents. An example is “Steve Jobs” for the slot “X founded Apple”. Thinking of a text passage like “Steve Jobs was an American businessman. In 1976, he co-founded Apple”, it is clear that coreference resolution can play an important role for finding the correct slot filler value.

Answers:
The task aims at extracting information about persons, organizations or geo-political entities from a large collection of news, web and discussion forum documents.
Question: How are the potentially relevant text fragments identified?

Evidence:
raw_evidence: This step consists of generating a query out of the claim and querying a search engine (here, we experiment with Google and Bing) in order to retrieve supporting documents. Rather than querying the search engine with the full claim (as on average, a claim is two sentences long), we generate a shorter query following the lessons highlighted in BIBREF0 .
We rank the words by means of tf-idf. We compute the idf values on a 2015 Wikipedia dump and the English Gigaword. BIBREF0 suggested that a good way to perform high-quality search is to only consider the verbs, the nouns and the adjectives in the claim; thus, we exclude all words in the claim that belong to other parts of speech. Moreover, claims often contain named entities (e.g., names of persons, locations, and organizations); hence, we augment the initial query with all the named entities from the claim's text. We use IBM's AlchemyAPI to identify named entities. Ultimately, we generate queries of 5–10 tokens, which we execute against a search engine. We then collect the snippets and the URLs in the results, skipping any result that points to a domain that is considered unreliable. Finally, if our query has returned no results, we iteratively relax it by dropping the final tokens one at a time.

Answers:
 Generate a query out of the claim and querying a search engine, rank the words by means of TF-IDF, use IBM's AlchemyAPI to identify named entities, generate queries of 5–10 tokens, which execute against a search engine, and collect the snippets and the URLs in the results, skipping any result that points to a domain that is considered unreliable.
Question: What algorithm and embedding dimensions are used to build the task-specific embeddings?

Evidence:
raw_evidence: We further use as features the embeddings of the claim, of the best-scoring snippet, and of the best-scoring sentence triplet from a Web page. We calculate these embeddings (i) as the average of the embeddings of the words in the text, and also (ii) using LSTM encodings, which we train for the task as part of a deep neural network (NN). We also use a task-specific embedding of the claim together with all the above evidence about it, which comes from the last hidden layer of the NN.

Answers:
 task-specific embedding of the claim together with all the above evidence about it, which comes from the last hidden layer of the NN
Question: What data is used to build the task-specific embeddings?

Evidence:
raw_evidence: The system starts with a claim to verify. First, we automatically convert the claim into a query, which we execute against a search engine in order to obtain a list of potentially relevant documents. Then, we take both the snippets and the most relevant sentences in the full text of these Web documents, and we compare them to the claim. The features we use are dense representations of the claim, of the snippets and of related sentences from the Web pages, which we automatically train for the task using Long Short-Term Memory networks (LSTMs). We also use the final hidden layer of the neural network as a task-specific embedding of the claim, together with the Web evidence. We feed all these representations as features, together with pairwise similarities, into a Support Vector Machine (SVM) classifier using an RBF kernel to classify the claim as True or False.

Answers:
embedding of the claim, Web evidence
Question: Do they evaluate the syntactic parses?

Evidence:

Answers:
No
Question: What knowledge bases do they use?

Evidence:
raw_evidence: We present here a step in this direction: a probabilistic semantic parser that uses a large knowledge base (NELL) to form a prior probability distribution on the meanings of sentences it parses, and that "understands" each sentence either by identifying its existing beliefs that correspond to the sentence's meaning, or by creating new beliefs. More precisely, our semantic parser corresponds to a probabilistic generative model that assigns high probability to sentence semantic parses resulting in beliefs it already holds, lower prior probability to parses resulting in beliefs it does not hold but which are consistent with its more abstract knowledge about semantic types of arguments to different relations, and still lower prior probability to parses that contradict its beliefs about which entity types can participate in which relations.

Answers:
NELL
Question: Which dataset do they use?

Evidence:
raw_evidence: We used the annotated dataset reported by degen2015investigating, a dataset of the utterances from the Switchboard corpus of telephone dialogues BIBREF21 that contain the word some. The dataset consists of 1,362 unique utterances with a noun phrase containing some (some-NP). For each example with a some-NP, degen2015investigating collected inference strength ratings from at least 10 participants recruited on Amazon's Mechanical Turk. Participants saw both the target utterance and ten utterances from the preceding discourse context. They then rated the similarity between the original utterance like (UNKREF8) and an utterance in which some was replaced with some, but not all like (UNKREF9), on a 7-point Likert scale with endpoints labeled “very different meaning” (1) and “same meaning” (7). Low similarity ratings thus indicate low inference strength, and high similarity ratings indicate high inference strength.

Answers:
the annotated dataset reported by degen2015investigating, a dataset of the utterances from the Switchboard corpus of telephone dialogues BIBREF21 that contain the word some
Question: What pre-trained models did they compare to?

Evidence:
raw_evidence: To test the applicability of the proposed MWA attention, we choose three publicly available Chinese pre-trained models as the basic encoder: BERT, ERNIE, and BERT-wwm. In order to make a fair comparison, we keep the same hyper-parameters (such maximum length, warm-up steps, initial learning rate, etc) as suggested in BERT-wwm BIBREF13 for both baselines and our method on each dataset. We run the same experiment for five times and report the average score to ensure the reliability of results. For detailed hyper-parameter settings, please see Appendix. Besides, three popular CWS tools thulac BIBREF14, ictclas BIBREF15 and hanlp BIBREF16 are employed to segment the Chinese sentences into words.

Answers:
BERT, ERNIE, and BERT-wwm
Question: How does the fusion method work?

Evidence:
raw_evidence: In this way, an attention vector sequence is segmented into several subsequences and each subsequence represents the attention of one word. Then, motivated by the psycholinguistic finding that readers are likely to pay approximate attention to each character in one Chinese word, we devise an appropriate aggregation module to fuse the inner-word character attention. Concretely, we first transform $\lbrace \textbf {a}_c^s,..., \textbf {a}_c^{s+l-1}\rbrace $ into one attention vector $\textbf {a}_w^i$ for $w_i$ with the mixed pooling strategy BIBREF11. Then we execute the piecewise up- mpling operation over each $\textbf {a}_w^i$ to keep input and output dimensions unchanged for the sake of plug and play. The detailed process can be summarized as follows:

Answers:
ttention vector sequence is segmented into several subsequences and each subsequence represents the attention of one word, we devise an appropriate aggregation module to fuse the inner-word character attention
Question: What benchmarks did they experiment on?

Evidence:
raw_evidence: We carried out experiments on four Chinese NLP tasks, including Emotion Classification (EC), Named Entity Recognition (NER), Sentence Pair Matching (SPM) and Natural Language Inference (NLI). The detail of those tasks and the corresponding datasets are introduced in Appendix.

Answers:
Emotion Classification (EC), Named Entity Recognition (NER), Sentence Pair Matching (SPM), Natural Language Inference (NLI)
Question: What were the evaluation metrics used?

Evidence:
raw_evidence: Our experimental results are shown in Table TABREF21. The first half of the table contains results for task-oriented dialogue with the Sequicity framework with two scenarios for training data preparation. For each experiment, we run our models for 3 times and their scores are averaged as the final score. The mixed training scenario performs the mixing of both the training data, development data and the test data as described in the previous subsection. The non-mixed training scenario performs the mixing only on the development and test data, keeps the training data unmixed as in the original KVRET dataset. As in the Sequicity framework, we report entity match rate, BLEU score and Success F1 score. Entity match rate evaluates task completion, it determines if a system can generate all correct constraints to search the indicated entities of the user. BLEU score evaluates the language quality of generated responses. Success F1 balances the recall and precision rates of slot answers. For further details on these metrics, please refer to BIBREF8.

Answers:
entity match rate, BLEU score, Success F1 score
Question: What is the size of the dataset?

Evidence:
raw_evidence: We use the publicly available dataset KVRET BIBREF5 in our experiments. This dataset is created by the Wizard-of-Oz method BIBREF10 on Amazon Mechanical Turk platform. This dataset includes dialogues in 3 domains: calendar, weather, navigation (POI) which is suitable for our mix-domain dialogue experiments. There are 2,425 dialogues for training, 302 for validation and 302 for testing, as shown in the upper half of Table TABREF12.

Answers:
3029
Question: What multi-domain dataset is used?

Evidence:
raw_evidence: We use the publicly available dataset KVRET BIBREF5 in our experiments. This dataset is created by the Wizard-of-Oz method BIBREF10 on Amazon Mechanical Turk platform. This dataset includes dialogues in 3 domains: calendar, weather, navigation (POI) which is suitable for our mix-domain dialogue experiments. There are 2,425 dialogues for training, 302 for validation and 302 for testing, as shown in the upper half of Table TABREF12.

Answers:
KVRET
Question: Which domains did they explored?

Evidence:
raw_evidence: We use the publicly available dataset KVRET BIBREF5 in our experiments. This dataset is created by the Wizard-of-Oz method BIBREF10 on Amazon Mechanical Turk platform. This dataset includes dialogues in 3 domains: calendar, weather, navigation (POI) which is suitable for our mix-domain dialogue experiments. There are 2,425 dialogues for training, 302 for validation and 302 for testing, as shown in the upper half of Table TABREF12.

Answers:
calendar, weather, navigation
Question: Do they report results only on English data?

Evidence:
raw_evidence: A serious challenge in the field of AV is the lack of publicly available (and suitable) corpora, which are required to train and evaluate AV methods. Among the few publicly available corpora are those that were released by the organizers of the well-known PAN-AV competitions BIBREF11 , BIBREF5 , BIBREF12 . In regard to our experiments, however, we cannot use these corpora, due to the absence of relevant meta-data such as the precise time spans where the documents have been written as well as the topic category of the texts. Therefore, we decided to compile our own corpora based on English documents, which we crawled from different publicly accessible sources. In what follows, we describe our three constructed corpora, which are listed together with their statistics in Table TABREF23 . Note that all corpora are balanced such that verification cases with matching (Y) and non-matching (N) authorships are evenly distributed.

Answers:
Yes
Question: Which is the best performing method?

Evidence:
raw_evidence: The top performing approaches Caravel, COAV and NNCD deserve closer attention. All three are based on character-level language models that capture low-level features similar to character INLINEFORM0 -grams, which have been shown in numerous AA and AV studies (for instance, BIBREF39 , BIBREF26 ) to be highly effective and robust. In BIBREF19 , BIBREF28 , it has been shown that Caravel and COAV were also the two top-performing approaches, where in BIBREF19 they were evaluated on the PAN-2015 AV corpus BIBREF12 , while in BIBREF28 they were applied on texts obtained from Project Gutenberg. Although both approaches perform similarly, they differ in the way how the decision criterion INLINEFORM1 is determined. While COAV requires a training corpus to learn INLINEFORM2 , Caravel assumes that the given test corpus (which provides the impostors) is balanced. Given this assumption, Caravel first computes similarity scores for all verification problems in the corpus and then sets INLINEFORM3 to the median of all similarities (cf. Figure FIGREF49 ). Thus, from a machine learning perspective, there is some undue training on the test set. Moreover, the applicability of Caravel in realistic scenarios is questionable, as a forensic case is not part of a corpus where the Y/N-distribution is known beforehand.

Answers:
Caravel, COAV and NNCD
Question: What is a self-compiled corpus?

Evidence:
raw_evidence: As a first corpus, we compiled INLINEFORM0 that represents a collection of 80 excerpts from scientific works including papers, dissertations, book chapters and technical reports, which we have chosen from the well-known Digital Bibliography & Library Project (DBLP) platform. Overall, the documents were written by 40 researchers, where for each author INLINEFORM1 , there are exactly two documents. Given the 80 documents, we constructed for each author INLINEFORM2 two verification problems INLINEFORM3 (a Y-case) and INLINEFORM4 (an N-case). For INLINEFORM5 we set INLINEFORM6 's first document as INLINEFORM7 and the second document as INLINEFORM8 . For INLINEFORM9 we reuse INLINEFORM10 from INLINEFORM11 as the known document and selected a text from another (random) author as the unknown document. The result of this procedure is a set of 80 verification problems, which we split into a training and test set based on a 40/60% ratio. Where possible, we tried to restrict the content of each text to the abstract and conclusion of the original work. However, since in many cases these sections were too short, we also considered other parts of the original works such as introduction or discussion sections. To ensure that the extracted text portions are appropriate for the AV task, each original work was preprocessed manually. More precisely, we removed tables, formulas, citations, quotes and sentences that include non-language content such as mathematical constructs or specific names of researchers, systems or algorithms. The average time span between both documents of an author is 15.6 years. The minimum and maximum time span are 6 and 40 years, respectively. Besides the temporal aspect of INLINEFORM12 , another challenge of this corpus is the formal (scientific) language, where the usage of stylistic devices is more restricted, in contrast to other genres such as novels or poems.

Answers:
 restrict the content of each text to the abstract and conclusion of the original work, considered other parts of the original works such as introduction or discussion sections, extracted text portions are appropriate for the AV task, each original work was preprocessed manually, removed tables, formulas, citations, quotes and sentences that include non-language content such as mathematical constructs or specific names of researchers, systems or algorithms
Question: how was annotation done?

Evidence:
raw_evidence: With the gathered comments, we reconstructed the original conversation trees, from the original post, the root, to the leaves, when they were available and selected a subset to annotated. For annotation purposes, we created snippets of conversations as the ones shown in Example 1 and Example 2 consisting of the parent of the suspected trolling event, the suspected trolling event comment, and all of the direct responses to the suspected trolling event. We added an extra constraint that the parent of the suspected trolling event should also be part of the direct responses, we hypothesize that if the suspected trolling event is indeed trolling, its parent should be the object of its trolling and would have a say about it. We recognize that this limited amount of information is not always sufficient to recover the original message conveyed by all of the participants in the snippet, and additional context would be beneficial. However, the trade off is that snippets like this allow us to make use of Amazon Mechanical Turk (AMT) to have the dataset annotated, because it is not a big burden for a “turker” to work on an individual snippet in exchange for a small pay, and expedites the annotation process by distributing it over dozens of people. Specifically, for each snippet, we requested three annotators to label the four aspects previously described. Before annotating, we set up a qualification test along with borderline examples to guide them in process and align them with our criteria. The qualification test turned out to be very selective since only 5% of all of the turkers that attempted it passed the exam. Our dataset consists of 1000 conversations with 5868 sentences and 71033 tokens. The distribution over the classes per trolling aspect is shown in the table TABREF24 in the column “Size”.

Answers:
Annotation was done with the help of annotators from Amazon Mechanical Turk on snippets of conversations
Question: what is the source of the new dataset?

Evidence:
raw_evidence: We collected all available comments in the stories from Reddit from August 2015. Reddit is popular website that allows registered users (without identity verification) to participate in forums specific a post or topic. These forums are of they hierarchical type, those that allow nested conversation, where the children of a comment are its direct response. To increase recall and make the annotation process feasible we created an inverted index with Lucene and queried for comments containing the word troll with an edit distance of 1, to include close variations of this word. We do so inspired by the method by BIBREF2 to created a bullying dataset, and because we hypothesize that such comments will be related or involved in a trolling event. As we observed in the dataset, people use the word troll in many different ways, sometimes it is to point out that some used is indeed trolling him or her or is accusing someone else of being a troll. Other times, people use the term, to express their frustration or dislike about a particular user, but there is no trolling event. Other times, people simple discuss about trolling and trolls, without actually participating or observing one directly. Nonetheless, we found that this search produced a dataset in which 44.3 % of the comments directly involved a trolling event. Moreover, as we exposed our trolling definition, it is possible for commentators in a conversation to believe that they are witnessing a trolling event and respond accordingly even where there is none. Therefore, even in the comments that do not involve trolling, we are interested in learning what triggers users interpretation of trolling where it is not present and what kind of response strategies are used. We define as a suspected trolling event in our dataset a comment in which at least one of its children contains the word troll.

Answers:
Reddit
Question: Do the authors give examples of positive and negative sentiment with regard to the virus?

Evidence:

Answers:
No
Question: Which word frequencies reflect on the psychology of the twitter users, according to the authors?

Evidence:
raw_evidence: Statistical analysis of tweets ::: Unigram, Bigram an Trigram word frequency analysis
Three forms of tokens of words have been considered for the study viz. unigram, bigram and trigram. These represent the frequencies of one word, two words together and finally three words coupled. The dataset provides the top 1000 unigrams, top 1000 bigrams and the top 1000 trigrams.

Answers:
unigram, bigram and trigram
Question: Do they specify which countries they collected twitter data from?

Evidence:

Answers:
No
Question: Do they collect only English data?

Evidence:
raw_evidence: In this section, we present the details of the analysis performed on the data obtained pertaining to Twitter messages from January 2020 upto now, that is the time since the news of the Coronavirus outbreak in China was spread across nations. The word frequency data corresponding to the twitter messages has been taken from BIBREF16. The data source indicates that during March 11th to March 30th there were over 4 million tweets a day with the surge in the awareness. Also, the data prominently captures the tweets in English, Spanish, and French languages. A total of four datasets have been used to carry out the study.

Answers:
No
Question: How do they measure correlation between the prediction and explanation quality?

Evidence:
raw_evidence: We would expect that explanation performance should correlate with prediction performance. Since Possible-answer knowledge is primarily needed to decide if the net has enough information to answer the challenge question without guessing and relevant-variable knowledge is needed for the net to know what to query, we analyzed the network's performance on querying and answering separately. The memory network has particular difficulty learning to query relevant variables, reaching only about .5 accuracy when querying. At the same time, it learns to answer very well, reaching over .9 accuracy there. Since these two parts of the interaction are what we ask it to explain in the two modes, we find that the quality of the explanations strongly correlates with the quality of the algorithm executed by the network.

Answers:
They look at the performance accuracy of explanation and the prediction performance
Question: What datasets are used to evaluate the introduced method?

Evidence:

Answers:
They used a dataset from Taobao which contained a collection of conversation records between customers and customer service staffs. It contains over five kinds of conversations,
including chit-chat, product and discount consultation, querying delivery progress and after-sales feedback.