qinchuanhui/UDA-QA huggingface dataset:
Question: What conclusions do the authors draw from their detailed analyses?

Evidence:
raw_evidence: We established strong baselines for two story narrative understanding datasets: CaTeRS and RED. We have shown that neural network-based models can outperform feature-based models with wide margins, and we conducted an ablation study to show that contextualized representation learning can boost performance of NN models. Further research can focus on more systematic study or build stronger NN models over the same datasets used in this work. Exploring possibilities to directly apply temporal relation extraction to enhance performance of story generation systems is another promising research direction.

Answers:
neural network-based models can outperform feature-based models with wide margins, contextualized representation learning can boost performance of NN models
Question: Do the BERT-based embeddings improve results?

Evidence:
raw_evidence: Table TABREF25 contains the best hyper-parameters and Table TABREF26 contains micro-average F1 scores for both datasets on dev and test sets. We only consider positive pairs, i.e. correct predictions on NONE pairs are excluded for evaluation. In general, the baseline model CAEVO is outperformed by both NN models, and NN model with BERT embedding achieves the greatest performance. We now provide more detailed analysis and discussion for each dataset.

Answers:
Yes
Question: What were the traditional linguistic feature-based models?

Evidence:
raw_evidence: The series of TempEval competitions BIBREF21 , BIBREF22 , BIBREF23 have attracted many research interests in predicting event temporal relations. Early attempts by BIBREF24 , BIBREF21 , BIBREF25 , BIBREF26 only use pair-wise classification models. State-of-the-art local methods, such as ClearTK BIBREF27 , UTTime BIBREF28 , and NavyTime BIBREF29 improve on earlier work by feature engineering with linguistic and syntactic rules. As we mention in the Section 2, CAEVO is the current state-of-the-art system for feature-based temporal event relation extraction BIBREF10 . It's widely used as the baseline for evaluating TB-Dense data. We adopt it as our baseline for evaluating CaTeRS and RED datasets. Additionally, several models BramsenDLB2006, ChambersJ2008, DoLuRo12, NingWuRo18, P18-1212 have successfully incorporated global inference to impose global prediction consistency such as temporal transitivity.

Answers:
CAEVO
Question: What type of baseline are established for the two datasets?

Evidence:
raw_evidence: The series of TempEval competitions BIBREF21 , BIBREF22 , BIBREF23 have attracted many research interests in predicting event temporal relations. Early attempts by BIBREF24 , BIBREF21 , BIBREF25 , BIBREF26 only use pair-wise classification models. State-of-the-art local methods, such as ClearTK BIBREF27 , UTTime BIBREF28 , and NavyTime BIBREF29 improve on earlier work by feature engineering with linguistic and syntactic rules. As we mention in the Section 2, CAEVO is the current state-of-the-art system for feature-based temporal event relation extraction BIBREF10 . It's widely used as the baseline for evaluating TB-Dense data. We adopt it as our baseline for evaluating CaTeRS and RED datasets. Additionally, several models BramsenDLB2006, ChambersJ2008, DoLuRo12, NingWuRo18, P18-1212 have successfully incorporated global inference to impose global prediction consistency such as temporal transitivity.

Answers:
CAEVO
Question: What model achieves state of the art performance on this task?

Evidence:
raw_evidence: Multitask learning, on the other hand, does not appear to have any positive impact on performance. Comparing the two CNN models, the addition of multitask learning actually appears to impair performance, with MultitaskCNN doing worse than BasicCNN in all three metrics. The difference is smaller when comparing BasicDCGAN and MultitaskDCGAN, and may not be enough to decidedly conclude that the use of multitask learning has a net negative impact there, but certainly there is no indication of a net positive impact. The observed performance of both the BasicDCGAN and MultitaskDCGAN using 3-classes is comparable to the state-of-the-art, with 49.80% compared to 49.99% reported in BIBREF16 . It needs to be noted that in BIBREF16 data from the test speaker's session partner was utilized in the training of the model. Our models in contrast are trained on only four of the five sessions as discussed in SECREF5 . Further, the here presented models are trained on the raw spectrograms of the audio and no feature extraction was employed whatsoever. This representation learning approach is employed in order to allow the DCGAN component of the model to train on vast amounts of unsupervised speech data.

Answers:
BIBREF16
Question: Which multitask annotated corpus is used?

Evidence:
raw_evidence: Due to the semi-supervised nature of the proposed Multitask DCGAN model, we utilize both labeled and unlabeled data. For the unlabeled data, we use audio from the AMI BIBREF8 and IEMOCAP BIBREF7 datasets. For the labeled data, we use audio from the IEMOCAP dataset, which comes with labels for activation and valence, both measured on a 5-point Likert scale from three distinct annotators. Although IEMOCAP provides per-word activation and valence labels, in practice these labels do not generally change over time in a given audio file, and so for simplicity we label each audio clip with the average valence and activation. Since valence and activation are both measured on a 5-point scale, the labels are encoded in 5-element one-hot vectors. For instance, a valence of 5 is represented with the vector INLINEFORM0 . The one-hot encoding can be thought of as a probability distribution representing the likelihood of the correct label being some particular value. Thus, in cases where the annotators disagree on the valence or activation label, this can be represented by assigning probabilities to multiple positions in the label vector. For instance, a label of 4.5 conceptually means that the “correct” valence is either 4 or 5 with equal probability, so the corresponding vector would be INLINEFORM1 . These “fuzzy labels” have been shown to improve classification performance in a number of applications BIBREF14 , BIBREF15 . It should be noted here that we had generally greater success with this fuzzy label method than training the neural network model on the valence label directly, i.e. classification task vs. regression.

Answers:
IEMOCAP
Question: What are the tasks in the multitask learning setup?

Evidence:
raw_evidence: Within this work, we particularly target emotional valence as the primary task, as it has been shown to be the most challenging emotional dimension for acoustic analyses in a number of studies BIBREF10 , BIBREF11 . Apart from solely targeting valence classification, we further investigate the principle of multitask learning. In multitask learning, a set of related tasks are learned (e.g., emotional activation), along with a primary task (e.g., emotional valence); both tasks share parts of the network topology and are hence jointly trained, as depicted in Figure FIGREF4 . It is expected that data for the secondary task models information, which would also be discriminative in learning the primary task. In fact, this approach has been shown to improve generalizability across corpora BIBREF12 .

Answers:
set of related tasks are learned (e.g., emotional activation), primary task (e.g., emotional valence)
Question: how are rare words defined?

Evidence:
raw_evidence: In our experiments, the short list is determined according to the word frequency. Concretely, we sort the vocabulary according to the word frequency from high to low. A frequency filter ratio INLINEFORM0 is set to filter out the low-frequency words (rare words) from the lookup table. For example, INLINEFORM1 =0.9 means the least frequent 10% words are replaced with the default UNK notation.

Answers:
low-frequency words
Question: which public datasets were used?

Evidence:
raw_evidence: To verify the effectiveness of our proposed model, we conduct multiple experiments on three Chinese Machine Reading Comprehension datasets, namely CMRC-2017 BIBREF17 , People's Daily (PD) and Children Fairy Tales (CFT) BIBREF2 . In these datasets, a story containing consecutive sentences is formed as the Document and one of the sentences is either automatically or manually selected as the Query where one token is replaced by a placeholder to indicate the answer to fill in. Table TABREF8 gives data statistics. Different from the current cloze-style datasets for English reading comprehension, such as CBT, Daily Mail and CNN BIBREF0 , the three Chinese datasets do not provide candidate answers. Thus, the model has to find the correct answer from the entire document.
Besides, we also use the Children's Book Test (CBT) dataset BIBREF1 to test the generalization ability in multi-lingual case. We only focus on subsets where the answer is either a common noun (CN) or NE which is more challenging since the answer is likely to be rare words. We evaluate all the models in terms of accuracy, which is the standard evaluation metric for this task.

Answers:
CMRC-2017, People's Daily (PD), Children Fairy Tales (CFT) , Children's Book Test (CBT)
Question: What are the results of the experiment?

Evidence:
raw_evidence: The BD-4SK-ASR Dataset ::: The Language Model
We created the language from the transcriptions. The model was created using CMUSphinx in which (fixed) discount mass is 0.5, and backoffs are computed using the ratio method. The model includes 283 unigrams, 5337 bigrams, and 6935 trigrams.

Answers:
They were able to create a language model from the dataset, but did not test.
Question: How was the dataset collected?

Evidence:
raw_evidence: To develop the dataset, we extracted 200 sentences from Sorani Kurdish books of grades one to three of the primary school in the Kurdistan Region of Iraq. We randomly created 2000 sentences from the extracted sentences.

Answers:
extracted text from Sorani Kurdish books of primary school and randomly created sentences
Question: What is the size of the dataset?

Evidence:
raw_evidence: To develop the dataset, we extracted 200 sentences from Sorani Kurdish books of grades one to three of the primary school in the Kurdistan Region of Iraq. We randomly created 2000 sentences from the extracted sentences.

Answers:
2000 sentences
Question: How many annotators participated?

Evidence:
raw_evidence: Two thousand narration files were created. We used Audacity to record the narrations. We used a normal laptop in a quiet room and minimized the background noise. However, we could not manage to avoid the noise of the fan of the laptop. A single speaker narrated the 2000 sentences, which took several days. We set the Audacity software to have a sampling rate of 16, 16-bit bit rate, and a mono (single) channel. The noise reduction db was set to 6, the sensitivity to 4.00, and the frequency smoothing to 0.

Answers:
1
Question: How long is the dataset?

Evidence:
raw_evidence: The corpus includes 2000 sentences. Theses sentence are random renderings of 200 sentences, which we have taken from Sorani Kurdish books of the grades one to three of the primary school in the Kurdistan Region of Iraq. The reason that we have taken only 200 sentences is to have a smaller dictionary and also to increase the repetition of each word in the narrated speech. We transformed the corpus sentences, which are in Persian-Arabic script, into the format which complies with the suggested phones for the related Sorani letters (see Section SECREF6).

Answers:
2000
Question: Do the authors mention any possible confounds in their study?

Evidence:
raw_evidence: On Twitter we can see results that are consistent with the RCV results for the left-to-center political spectrum. The exception, which clearly stands out, is the right-wing groups ENL and EFDD that seem to be the most cohesive ones. This is the direct opposite of what was observed in the RCV data. We speculate that this phenomenon can be attributed to the fact that European right-wing groups, on a European but also on a national level, rely to a large degree on social media to spread their narratives critical of European integration. We observed the same phenomenon recently during the Brexit campaign BIBREF38 . Along our interpretation the Brexit was “won” to some extent due to these social media activities, which are practically non-existent among the pro-EU political groups. The fact that ENL and EFDD are the least cohesive groups in the European Parliament can be attributed to their political focus. It seems more important for the group to agree on its anti-EU stance and to call for independence and sovereignty, and much less important to agree on other issues put forward in the parliament.

Answers:
Yes
Question: What is the relationship between the co-voting and retweeting patterns?

Evidence:
raw_evidence: Overall, we observe a positive correlation between retweeting and co-voting, which is significantly different from zero. The strongest positive correlations are in the areas Area of freedom, security and justice, External relations of the Union, and Internal markets. Weaker, but still positive, correlations are observed in the areas Economic, social and territorial cohesion, European citizenship, and State and evolution of the Union. The only exception, with a significantly negative coefficient, is the area Economic and monetary system. This implies that in the area Economic and monetary system we observe a significant deviation from the usual co-voting patterns. Results from section “sec:coalitionpolicy”, confirm that this is indeed the case. Especially noteworthy are the coalitions between GUE-NGL and Greens-EFA on the left wing, and EFDD and ENL on the right wing. In the section “sec:coalitionpolicy” we interpret these results as a combination of Euroscepticism on both sides, motivated on the left by a skeptical attitude towards the market orientation of the EU, and on the right by a reluctance to give up national sovereignty.

Answers:
we observe a positive correlation between retweeting and co-voting, strongest positive correlations are in the areas Area of freedom, security and justice, External relations of the Union, and Internal markets, Weaker, but still positive, correlations are observed in the areas Economic, social and territorial cohesion, European citizenship, and State and evolution of the Union, significantly negative coefficient, is the area Economic and monetary system
Question: Does the analysis find that coalitions are formed in the same way for different policy areas?

Evidence:
raw_evidence: In the area of Economic and monetary system we see a strong cooperation between EPP and S&D (Fig FIGREF42 A), which is on a par with the cohesion of the most cohesive groups (GUE-NGL, S&D, Greens-EFA, ALDE, and EPP), and is above the cohesion of the other groups. As pointed out in the section “sec:coalitionpolicy”, there is a strong separation in two blocks between supporters and opponents of European integration, which is even more clearly observed in Fig FIGREF42 B. On one hand, we observe cooperation between S&D, ALDE, EPP, and ECR, and on the other, cooperation between GUE-NGL, Greens-EFA, EFDD, ENL, and NI. This division in blocks is seen again in Fig FIGREF42 C, which shows the strongest disagreements. Here, we observe two blocks composed of S&D, EPP, and ALDE on one hand, and GUE-NGL, EFDD, ENL, and NI on the other, which are in strong opposition to each other.
In the area of State and Evolution of the Union we again observe a strong division in two blocks (see Fig FIGREF42 E). This is different to the Economic and monetary system, however, where we observe a far-left and far-right cooperation, where the division is along the traditional left-right axis.
The patterns of coalitions forming on Twitter closely resemble those in the European Parliament. In Fig FIGREF42 J we see that the strongest degrees of cooperation on Twitter are within the groups. The only group with low cohesion is the NI, whose members have only seven retweets between them. The coalitions on Twitter follow the seating order in the European Parliament remarkably well (see Fig FIGREF42 K). What is striking is that the same blocks form on the left, center, and on the center-right, both in the European Parliament and on Twitter. The largest difference between the coalitions in the European Parliament and on Twitter is on the far-right, where we observe ENL and NI as isolated blocks.

Answers:
No
Question: What insights does the analysis give about the cohesion of political groups in the European parliament?

Evidence:
raw_evidence: As with INLINEFORM0 , Greens-EFA, S&D, and EPP exhibit the highest cohesion, even though their ranking is permuted when compared to the ranking obtained with INLINEFORM1 . At the other end of the scale, we observe the same situation as with INLINEFORM2 . The non-aligned members NI have the lowest cohesion, followed by EFDD and ENL.
The only place where the two methods disagree is the level of cohesion of GUE-NGL. The Alpha attributes GUE-NGL a rather high level of cohesion, on a par with ALDE, whereas the ERGM attributes them a much lower cohesion. The reason for this difference is the relatively high abstention rate of GUE-NGL. Whereas the overall fraction of non-attending and abstaining MEPs across all RCVs and all political groups is 25%, the GUE-NGL abstention rate is 34%. This is reflected in an above average cohesion by INLINEFORM0 where only yes/no votes are considered, and in a relatively lower, below average cohesion by ERGM. In the later case, the non-attendance is interpreted as a non-cohesive voting of a political groups as a whole.

Answers:
Greens-EFA, S&D, and EPP exhibit the highest cohesion, non-aligned members NI have the lowest cohesion, followed by EFDD and ENL, two methods disagree is the level of cohesion of GUE-NGL
Question: Do they authors account for differences in usage of Twitter amongst MPs into their model?

Evidence:
raw_evidence: The retweeting behavior of MEPs is captured by their retweet network. Each MEP active on Twitter is a node in this network. An edge in the network between two MEPs exists when one MEP retweeted the other. The weight of the edge is the number of retweets between the two MEPs. The resulting retweet network is an undirected, weighted network.
We measure the cohesion of a political group INLINEFORM0 as the average retweets, i.e., the ratio of the number of retweets between the MEPs in the group INLINEFORM1 to the number of MEPs in the group INLINEFORM2 . The higher the ratio, the more each MEP (on average) retweets the MEPs from the same political group, hence, the higher the cohesion of the political group. The definition of the average retweeets ( INLINEFORM3 ) of a group INLINEFORM4 is: INLINEFORM5

Answers:
No
Question: Did the authors examine if any of the MEPs used the disclaimer that retweeting does not imply endorsement on their twitter profile?

Evidence:

Answers:
No
Question: How do they show their model discovers underlying syntactic structure?

Evidence:
raw_evidence: In Figure FIGREF32 , we visualize the syntactic distance estimated by the Parsing Network, while reading three different sequences from the PTB test set. We observe that the syntactic distance tends to be higher between the last character of a word and a space, which is a reasonable breakpoint to separate between words. In other words, if the model sees a space, it will attend on all previous step. If the model sees a letter, it will attend no further then the last space step. The model autonomously discovered to avoid inter-word attention connection, and use the hidden states of space (separator) tokens to summarize previous information. This is strong proof that the model can understand the latent structure of data. As a result our model achieve state-of-the-art performance and significantly outperform baseline models. It is worth noting that HM-LSTM BIBREF6 also unsupervisedly induce similar structure from data. But discrete operations in HM-LSTM make their training procedure more complicated then ours.

Answers:
By visualizing syntactic distance estimated by the parsing network
Question: Which dataset do they experiment with?

Evidence:
raw_evidence: From a character-level view, natural language is a discrete sequence of data, where discrete symbols form a distinct and shallow tree structure: the sentence is the root, words are children of the root, and characters are leafs. However, compared to word-level language modeling, character-level language modeling requires the model to handle longer-term dependencies. We evaluate a character-level variant of our proposed language model over a preprocessed version of the Penn Treebank (PTB) and Text8 datasets.
The unsupervised constituency parsing task compares hte tree structure inferred by the model with those annotated by human experts. The experiment is performed on WSJ10 dataset. WSJ10 is the 7422 sentences in the Penn Treebank Wall Street Journal section which contained 10 words or less after the removal of punctuation and null elements. Evaluation was done by seeing whether proposed constituent spans are also in the Treebank parse, measuring unlabeled F1 ( INLINEFORM0 ) of unlabeled constituent precision and recall. Constituents which could not be gotten wrong (those of span one and those spanning entire sentences) were discarded. Given the mechanism discussed in Section SECREF14 , our model generates a binary tree. Although standard constituency parsing tree is not limited to binary tree. Previous unsupervised constituency parsing model also generate binary trees BIBREF11 , BIBREF13 . Our model is compared with the several baseline methods, that are explained in Appendix .

Answers:
Penn Treebank, Text8, WSJ10
Question: How are content clusters used to improve the prediction of incident severity?

Evidence:
raw_evidence: As a further application of our work, we have carried out a supervised classification task aimed at predicting the degree of harm of an incident directly from the text and the hand-coded features (e.g., external category, medical specialty, location). A one-hot encoding is applied to turn these categorical values into numerical ones. We also checked if using our unsupervised content-driven cluster labels as additional features can improve the performance of the supervised classification.

Answers:
they are used as additional features in a supervised classification task
Question: What cluster identification method is used in this paper?

Evidence:
raw_evidence: The trained Doc2Vec model is subsequently used to infer high-dimensional vector descriptions for the text of each document in our target analysis set. We then compute a matrix containing all the pairwise (cosine) similarities between the Doc2Vec document vectors. This similarity matrix can be thought of as the adjacency matrix of a full, weighted graph with documents as nodes and edges weighted by their similarity. We sparsify this graph to the union of a minimum spanning tree and a k-Nearest Neighbors (MST-kNN) graph BIBREF14, a geometric construction that removes less important similarities but preserves global connectivity for the graph and, hence, for the dataset. The MST-kNN graph is then analysed with Markov Stability BIBREF15, BIBREF16, BIBREF17, BIBREF18, a multi-resolution graph partitioning method that identifies relevant subgraphs (i.e., clusters of documents) at different levels of granularity. MS uses a diffusive process on the graph to reveal the multiscale organisation at different resolutions without the need to choose a priori the number or type of clusters.

Answers:
A combination of Minimum spanning trees, K-Nearest Neighbors and Markov Stability BIBREF15, BIBREF16, BIBREF17, BIBREF18