qinchuanhui/UDA-QA huggingface dataset:
Question: What do the models that they compare predict?

Evidence:
raw_evidence: The main set of experiments uses a Linear Support Vector Machine (Joachims, 1998) to classify dialects using CxG features. Parameters are tuned using the development data. Given the general robust performance of SVMs in the literature relative to other similar classifiers on variation tasks (c.f., Dunn, et al., 2016), we forego a systematic evaluation of classifiers.
This paper has used data-driven language mapping to select national dialects of English to be included in a global dialect identification model. The main experiments have focused on a dynamic syntactic feature set, showing that it is possible to predict dialect membership within-domain with only a small loss of performance against lexical models. This work raises two remaining problems:

Answers:
national dialects of English
Question: What SMT models did they look at?

Evidence:
raw_evidence: To evaluate the usefulness of our corpus for SMT purposes, we used it to train an automatic translator with Moses BIBREF8 . We also trained an NMT model using the OpenNMT system BIBREF9 , and used the Google Translate Toolkit to produce state-of-the-art comparison results. The produced translations were evaluated according to the BLEU score BIBREF10 .

Answers:
automatic translator with Moses
Question: Which NMT models did they experiment with?

Evidence:
raw_evidence: Prior to the MT experiments, sentences were randomly split in three disjoint datasets: training, development, and test. Approximately 13,000 sentences were allocated in the development and test sets, while the remaining was used for training. For the SMT experiment, we followed the instructions of Moses baseline system. For the NMT experiment, we used the Torch implementation to train a 2-layer LSTM model with 500 hidden units in both encoder and decoder, with 12 epochs. During translation, the option to replace UNK words by the word in the input language was used, since this is also the default in Moses.

Answers:
2-layer LSTM model with 500 hidden units in both encoder and decoder
Question: How big PIE datasets are obtained from dictionaries?

Evidence:
raw_evidence: We use only the written part of the BNC. From this, we extract a set of documents with the aim of having as much genre variation as possible. To achieve this, we select the first document in each genre, as defined by the classCode attribute (e.g. nonAc, commerce, letters). The resulting set of 46 documents makes up our base corpus. Note that these documents vary greatly in size, which means the resulting corpus is varied, but not balanced in terms of size (Table TABREF43). The documents are split across a development and test set, as specified at the end of Section SECREF46. We exclude documents with IDs starting with A0 from all annotation and evaluation procedures, as these were used during development of the extraction tool and annotation guidelines.

Answers:
46 documents makes up our base corpus
Question: What compleentary PIE extraction methods are used to increase reliability further?

Evidence:
raw_evidence: We experiment with two such combinations, by simply taking the union of the sets of extracted idioms of both systems, and filtering out duplicates. Results are shown in Table TABREF77. Both combinations show the expected effect: a clear gain in recall at a minimal loss in precision. Compared to the in-context-parsing-based system, the combination with exact string matching yields a gain in recall of over 6%, and the combination with inflectional string matching yields an even bigger gain of almost 8%, at precision losses of 0.6% and 0.8%, respectively. This indicates that the systems are very much complementary in the PIEs they extract. It also means that, when used in practice, combining inflectional string matching and parse-based extraction is the most reliable configuration.

Answers:
exact string matching, inflectional string matching
Question: Are PIEs extracted automatically subjected to human evaluation?

Evidence:
raw_evidence: For parser-based extraction, systems with and without in-context parsing, ignoring labels, and ignoring directionality are tested. For the three string-based extraction methods, varying numbers of intervening words and case sensitivity are evaluated. Evaluation is done using the development set, consisting of 22 documents and 1112 PIE candidates, and the test set, which consists of 23 documents and 1127 PIE candidates. For each method the best set of parameters and/or options is determined using the development set, after which the best variant by F1-score of each method is evaluated on the test set.
Since these documents in the corpus are exhaustively annotated for PIEs (see Section SECREF40), we can calculate true and false positives, and false negatives, and thus precision, recall and F1-score. The exact spans are ignored, because the spans annotated in the evaluation corpus are not completely reliable. These were automatically generated during candidate extraction, as described in Section SECREF45. Rather, we count an extraction as a true positive if it finds the correct PIE type in the correct sentence.

Answers:
Yes
Question: What dictionaries are used for automatic extraction of PIEs?

Evidence:
raw_evidence: We evaluate the quality of three idiom dictionaries by comparing them to each other and to three idiom corpora. Before we report on the comparison we first describe why we select and how we prepare these resources. We investigate the following six idiom resources:
Wiktionary;
the Oxford Dictionary of English Idioms (ODEI, BIBREF31);
UsingEnglish.com (UE);
the Sporleder corpus BIBREF10;
the VNC dataset BIBREF9;
There are four sizeable sense-annotated PIE corpora for English: the VNC-Tokens Dataset BIBREF9, the Gigaword dataset BIBREF14, the IDIX Corpus BIBREF10, and the SemEval-2013 Task 5 dataset BIBREF15. An overview of these corpora is presented in Table TABREF7.

Answers:
Wiktionary, Oxford Dictionary of English Idioms, UsingEnglish.com (UE), Sporleder corpus, VNC dataset, SemEval-2013 Task 5 dataset
Question: Are experiments performed with any other pair of languages, how did proposed method perform compared to other models?

Evidence:
raw_evidence: We evaluate the proposed transfer learning techniques in two non-English language pairs of WMT 2019 news translation tasks: French$\rightarrow $German and German$\rightarrow $Czech.

Answers:
No
Question: Is pivot language used in experiments English or some other language?

Evidence:
raw_evidence: Nonetheless, the main caveat of this basic pre-training is that the source encoder is trained to be used by an English decoder, while the target decoder is trained to use the outputs of an English encoder — not of a source encoder. In the following, we propose three techniques to mitigate the inconsistency of source$\rightarrow $pivot and pivot$\rightarrow $target pre-training stages. Note that these techniques are not exclusive and some of them can complement others for a better performance of the final model.

Answers:
Yes
Question: What are multilingual models that were outperformed in performed experiment?

Evidence:
raw_evidence: Baselines We thoroughly compare our approaches to the following baselines:
Direct source$\rightarrow $target: A standard NMT model trained on given source$\rightarrow $target parallel data.
Multilingual: A single, shared NMT model for multiple translation directions BIBREF6.
Many-to-many: Trained for all possible directions among source, target, and pivot languages.
Many-to-one: Trained for only the directions to target language, i.e., source$\rightarrow $target and pivot$\rightarrow $target, which tends to work better than many-to-many systems BIBREF27.

Answers:
Direct source$\rightarrow $target: A standard NMT model trained on given source$\rightarrow $target, Multilingual: A single, shared NMT model for multiple translation directions, Many-to-many: Trained for all possible directions among source, target, and pivot languages, Many-to-one: Trained for only the directions to target language
Question: What are the common captioning metrics?

Evidence:
raw_evidence: We trained and evaluated our algorithm on the Microsoft COCO (MS-COCO) 2014 Captions dataset BIBREF21 . We report results on the Karpathy validation and test splits BIBREF8 , which are commonly used in other image captioning publications. The dataset contains 113K training images with 5 human annotated captions for each image. The Karpathy test and validation sets contain 5K images each. We evaluate our models using the CIDEr-D BIBREF22 , SPICE BIBREF23 , BLEU BIBREF24 , METEOR BIBREF25 , and ROUGE-L BIBREF26 metrics. While it has been shown experimentally that BLEU and ROUGE have lower correlation with human judgments than the other metrics BIBREF23 , BIBREF22 , the common practice in the image captioning literature is to report all the mentioned metrics.

Answers:
the CIDEr-D BIBREF22 , SPICE BIBREF23 , BLEU BIBREF24 , METEOR BIBREF25 , and ROUGE-L BIBREF26 metrics
Question: What is the road exam metric?

Evidence:
raw_evidence: In this paper, we adopt two simple strategies, multi-range reinforcing and multi-entropy sampling to overcome the reward sparseness during training. With the tricks applied, our model demonstrates a significant improvement over competing models. In addition, we propose road exam as a new metric to reveal a model's robustness against exposure bias.

Answers:
a new metric to reveal a model's robustness against exposure bias
Question: How is the input triple translated to a slot-filling task?

Evidence:
raw_evidence: We show that it is possible to reduce relation extraction to the problem of answering simple reading comprehension questions. We map each relation type $R(x,y)$ to at least one parametrized natural-language question $q_x$ whose answer is $y$ . For example, the relation $educated\_at(x,y)$ can be mapped to “Where did $x$ study?” and “Which university did $x$ graduate from?”. Given a particular entity $x$ (“Turing”) and a text that mentions $x$ (“Turing obtained his PhD from Princeton”), a non-null answer to any of these questions (“Princeton”) asserts the fact and also fills the slot $y$ . Figure 1 illustrates a few more examples.

Answers:
The relation R(x,y) is mapped onto a question q whose answer is y
Question: Is model compared against state of the art models on these datasets?

Evidence:
raw_evidence: Deep convolutional neural networks (CNNs) with 2D convolutions and small kernels BIBREF1, have achieved state-of-the-art results for several speech recognition tasks BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6. The accuracy of those models grows with their complexity, leading to redundant latent representations. Several approaches have been proposed in the literature to reduce this redundancy BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, and therefore to improve their efficiency.
Models: Our baseline CNN model BIBREF21 consists of 15 convolutional and one fully-connected layer. We use $3\times 3$ kernels throughout the network. We start with 64 output channels in the first layer and double them after 3 and 9 layers. We use batch normalization in every convolutional layer, and ReLU afterwards (unless a reverse order is noted). The initial learning rate is 0.001. We use early stopping for training.

Answers:
Yes
Question: How is octave convolution concept extended to multiple resolutions and octaves?

Evidence:
raw_evidence: An octave convolutional layer BIBREF0 factorizes the output feature maps of a convolutional layer into two groups. The resolution of the low-frequency feature maps is reduced by an octave – height and width dimensions are divided by 2. In this work, we explore spatial reduction by up to 3 octaves – dividing by $2^t$, where $t=1,2,3$ – and for up to 4 groups. We refer to such a layer as a multi-octave convolutional (MultiOctConv) layer, and an example with three groups and reductions of one and two octaves is depicted in Fig. FIGREF1.

Answers:
The resolution of the low-frequency feature maps is reduced by an octave – height and width dimensions are divided by 2. In this work, we explore spatial reduction by up to 3 octaves – dividing by $2^t$, where $t=1,2,3$ – and for up to 4 groups. We refer to such a layer as a multi-octave convolutional (MultiOctConv) layer,
Question: Does this paper address the variation among English dialects regarding these hedges?

Evidence:

Answers:
No
Question: On which dataset is model trained?

Evidence:
raw_evidence: For evaluating the proposed model on behavior related data, we employ the Couples Therapy Corpus (CoupTher) BIBREF21 and Cancer Couples Interaction Dataset (Cancer) BIBREF22. These are the targeted conditions under which a behavior-gated language model can offer improved performance.
We utilize the Couple's Therapy Corpus as an in-domain experimental corpus since our behavior classification model is also trained on the same. The RNNLM architecture is similar to BIBREF1, but with hyperparameters optimized for the couple's corpus. The results are tabulated in Table TABREF16 in terms of perplexity. We find that the behavior gated language models yield lower perplexity compared to vanilla LSTM language model. A relative improvement of 2.43% is obtained with behavior gating on the couple's data.

Answers:
Couples Therapy Corpus (CoupTher) BIBREF21
Question: How is module that analyzes behavioral state trained?

Evidence:
raw_evidence: The behavior model was implemented using an RNN with LSTM units and trained with the Couples Therapy Corpus. Out of the 33 behavioral codes included in the corpus we applied the behaviors Acceptance, Blame, Negativity, Positivity, and Sadness to train our models. This is motivated from previous works which showed good separability in these behaviors as well as being easy to interpret. The behavior model is pre-trained to identify the presence of each behavior from a sequence of words using a multi-label classification scheme. The pre-trained portion of the behavior model was implemented using a single layer RNN with LSTM units with dimension size 50.

Answers:
pre-trained to identify the presence of behavior from a sequence of word using the Couples Therapy Corpus
Question: Can the model add new relations to the knowledge graph, or just new entities?

Evidence:
raw_evidence: In ConMask, we use a similar idea to select the most related words given some relationship and mask irrelevant words by assigning a relationship-dependent similarity score to words in the given entity description. We formally define relationship-dependent content masking as:
ConMask selects words that are related to the given relationship to mitigate the inclusion of irrelevant and noisy words. From the relevant text, ConMask then uses fully convolutional network (FCN) to extract word-based embeddings. Finally, it compares the extracted embeddings to existing entities in the KG to resolve a ranked list of target entities. The overall structure of ConMask is illustrated in Fig. 1 . Later subsections describe the model in detail.

Answers:
The model does not add new relations to the knowledge graph.
Question: How large is the dataset?

Evidence:
raw_evidence: Table TABREF17 shows our annotated corpus characteristics. Our corpus comprises a total of 6,127 scientific entities, including 2,112 Process, 258 Method, 2,099 Material, and 1,658 Data entities. The number of entities per abstract directly correlates with the length of the abstracts (Pearson's R 0.97). Among the concepts, Process and Material directly correlate with abstract length (R 0.8 and 0.83, respectively), while Data has only a slight correlation (R 0.35) and Method has no correlation (R 0.02).

Answers:
6,127 scientific entities, including 2,112 Process, 258 Method, 2,099 Material, and 1,658 Data entities
Question: Why is a Gaussian process an especially appropriate method for this classification problem?

Evidence:
raw_evidence: We use Gaussian Processes as this probabilistic kernelised framework avoids the need for expensive cross-validation for hyperparameter selection. Instead, the marginal likelihood of the data can be used for hyperparameter selection.

Answers:
avoids the need for expensive cross-validation for hyperparameter selection
Question: Do the authors do manual evaluation?

Evidence:
raw_evidence: We compare the performance of our model (Table 2 ) with traditional Bag of Words (BoW), TF-IDF, and n-grams features based classifiers. We also compare against averaged Skip-Gram BIBREF29 , Doc2Vec BIBREF30 , CNN BIBREF23 , Hierarchical Attention (HN-ATT) BIBREF24 and hierarchical network (HN) models. HN it is similar to our model HN-SA but without any self attention.
Analysis: As is evident from the experiments on both the versions of SWBD, our model (HN-SA) outperforms traditional feature based topic spotting models and deep learning based document classification models. It is interesting to see that simple BoW and n-gram baselines are quite competitive and outperform some of the deep learning based document classification model. Similar observation has also been reported by BIBREF31 ( BIBREF31 ) for the task of sentiment analysis. The task of topic spotting is arguably more challenging than document classification. In the topic spotting task, the number of output classes (66/42 classes) is much more than those in document classification (5/6 classes), which is done mainly on the texts from customer reviews. Dialogues in SWBD have on an average 200 utterances and are much longer texts than customer reviews. Additionally, the number of dialogues available for training the model is significantly lesser than customer reviews. We further investigated the performance on SWBD2 by examining the confusion matrix of the model. Figure 2 shows the heatmap of the normalized confusion matrix of the model on SWBD2. For most of the classes the classifier is able to predict accurately. However, the model gets confused between the classes which are semantically close (w.r.t. terms used) to each other, for example, the model gets confused between pragmatically similar topics e.g. HOBBIES vs GARDENING, MOVIES vs TV PROGRAMSâ, RIGHT TO PRIVACY vs DRUG TESTING.

Answers:
No
Question: What datasets did they use?

Evidence:
raw_evidence: Our experiments on a goal-oriented dialog corpus, the personalized bAbI dialog dataset, show that leveraging personal information can significantly improve the performance of dialog systems. The Personalized MemN2N outperforms current state-of-the-art methods with over 7% improvement in terms of per-response accuracy. A test with real human users also illustrates that the proposed model leads to better outcomes, including higher task completion rate and user satisfaction.

Answers:
the personalized bAbI dialog dataset
Question: Do twitter users tend to tweet about the DOS attack when it occurs? How much data supports this assumption?

Evidence:
raw_evidence: In this subsection we discuss the experiment on the attack tweets found in the whole dataset. As stated in section 3.3, the whole dataset was divided into two parts. $D_a$ contained all of the tweets collected on the attack day of the five attacks mentioned in section 4.2. And $D_b$ contained all of the tweets collected before the five attacks. There are 1180 tweets in $D_a$ and 7979 tweets in $D_b$. The tweets on the attack days ($D_a$) are manually annotated and only 50 percent of those tweets are actually about a DDoS attack.

Answers:
The dataset contains about 590 tweets about DDos attacks.
Question: What is the training and test data used?

Evidence:
raw_evidence: We collected tweets related to five different DDoS attacks on three different American banks. For each attack, all the tweets containing the bank's name posted from one week before the attack until the attack day were collected. There are in total 35214 tweets in the dataset. Then the collected tweets were preprocessed as mentioned in the preprocessing section.
Only the tweets from the Bank of America attack on 09/19/2012 were used in this experiment. The tweets before the attack day and on the attack day were used to train the two LDA models mentioned in the approach section.
In this subsection we evaluate how good the model generalizes. To achieve that, the dataset is divided into two groups, one is about the attacks on Bank of America and the other group is about PNC and Wells Fargo. The only difference between this experiment and the experiment in section 4.4 is the dataset. In this experiment setting $D_a$ contains only the tweets collected on the days of attack on PNC and Wells Fargo. $D_b$ only contains the tweets collected before the Bank of America attack. There are 590 tweets in $D_a$ and 5229 tweets in $D_b$. In this experiment, we want to find out whether a model trained on Bank of America data can make good classification on PNC and Wells Fargo data.

Answers:
Tweets related to a Bank of America DDos attack were used as training data. The test datasets contain tweets related to attacks to Bank of America, PNC and Wells Fargo.
Question: Was performance of the weakly-supervised model compared to the performance of a supervised model?

Evidence:
raw_evidence: The precision when labeling the first x ranked tweets as attack tweet is shown in the figure FIGREF39. The x-axis is the number of ranked tweets treated as attack tweets. And the y-axis is the corresponding precision. The straight line in figures FIGREF39, FIGREF43 and FIGREF51 is the result of a supervised LDA algorithm which is used as a baseline. Supervised LDA achieved 96.44 percent precision with 10 fold cross validation.

Answers:
Yes
Question: Do the tweets come from a specific region?

Evidence:
raw_evidence: Table TABREF14 presents the distribution of the tweets by country before and after the filtering process. A large portion of the samples is from India because the MeToo movement has peaked towards the end of 2018 in India. There are very few samples from Russia likely because of content moderation and regulations on social media usage in the country. Figure FIGREF15 gives a geographical distribution of the curated dataset.

Answers:
No
Question: Did they experiment with the corpus?

Evidence:
raw_evidence: The corpus creation process involved a small number of people that have voluntarily joined the initiative, with the authors of this paper directing the work. Initially, we searched for NER resources in Romanian, and found none. Then we looked at English resources and read the in-depth ACE guide, out of which a 16-class draft evolved. We then identified a copy-right free text from which we hand-picked sentences to maximize the amount of entities while maintaining style balance. The annotation process was a trial-and-error, with cycles composed of annotation, discussing confusing entities, updating the annotation guide schematic and going through the corpus section again to correct entities following guide changes. The annotation process was done online, in BRAT. The actual annotation involved 4 people, has taken about 6 months (as work was volunteer-based, we could not have reached for 100% time commitment from the people involved), and followed the steps:

Answers:
Yes
Question: How did they determine the distinct classes?

Evidence:
raw_evidence: The 16 classes are inspired by the OntoNotes5 corpus BIBREF7 as well as the ACE (Automatic Content Extraction) English Annotation Guidelines for Entities Version 6.6 2008.06.13 BIBREF8. Each class will be presented in detail, with examples, in the section SECREF3 A summary of available classes with word counts for each is available in table TABREF18.

Answers:
inspired by the OntoNotes5 corpus BIBREF7 as well as the ACE (Automatic Content Extraction) English Annotation Guidelines for Entities Version 6.6 2008.06.13 BIBREF8