qinchuanhui/UDA-QA huggingface dataset:
Question: What additional information is found in the dataset?

Evidence:
raw_evidence: We collected COVID-19 related Arabic tweets from January 1, 2020 until April 15, 2020, using Twitter streaming API and the Tweepy Python library. We have collected more than 3,934,610 million tweets so far. In our dataset, we store the full tweet object including the id of the tweet, username, hashtags, and geolocation of the tweet. We created a list of the most common Arabic keywords associated with COVID-19. Using Twitter’s streaming API, we searched for any tweet containing the keyword(s) in the text of the tweet. Table TABREF1 shows the list of keywords used along with the starting date of tracking each keyword. Furthermore, Table TABREF2 shows the list of hashtags we have been tracking along with the number of tweets collected from each hashtag. Indeed, some tweets were irrelevant, and we kept only those that were relevant to the pandemic.

Answers:
the full tweet object including the id of the tweet, username, hashtags, and geolocation of the tweet
Question: Over what period of time were the tweets collected?

Evidence:
raw_evidence: We collected COVID-19 related Arabic tweets from January 1, 2020 until April 15, 2020, using Twitter streaming API and the Tweepy Python library. We have collected more than 3,934,610 million tweets so far. In our dataset, we store the full tweet object including the id of the tweet, username, hashtags, and geolocation of the tweet. We created a list of the most common Arabic keywords associated with COVID-19. Using Twitter’s streaming API, we searched for any tweet containing the keyword(s) in the text of the tweet. Table TABREF1 shows the list of keywords used along with the starting date of tracking each keyword. Furthermore, Table TABREF2 shows the list of hashtags we have been tracking along with the number of tweets collected from each hashtag. Indeed, some tweets were irrelevant, and we kept only those that were relevant to the pandemic.

Answers:
from January 1, 2020 until April 15, 2020
Question: How big is the dataset?

Evidence:
raw_evidence: We collected COVID-19 related Arabic tweets from January 1, 2020 until April 15, 2020, using Twitter streaming API and the Tweepy Python library. We have collected more than 3,934,610 million tweets so far. In our dataset, we store the full tweet object including the id of the tweet, username, hashtags, and geolocation of the tweet. We created a list of the most common Arabic keywords associated with COVID-19. Using Twitter’s streaming API, we searched for any tweet containing the keyword(s) in the text of the tweet. Table TABREF1 shows the list of keywords used along with the starting date of tracking each keyword. Furthermore, Table TABREF2 shows the list of hashtags we have been tracking along with the number of tweets collected from each hashtag. Indeed, some tweets were irrelevant, and we kept only those that were relevant to the pandemic.

Answers:
more than 3,934,610 million tweets
Question: Do the authors suggest any future extensions to this work?

Evidence:
raw_evidence: The main contributions of this paper are (1) to overcome twitter challenges of acronyms, short text, ambiguity and synonyms, (2) to identify the set of word-pairs to be used as features for live event detection, (3) to build an end-to-end framework that can detect the events lively according to the word counts. This work can be applied to similar problems, where specific tweets can be associated with life events such as disease outbreak or stock market fluctuation. This work can be extended to predict future events with one day in advance, where we will use the same method for feature selection in addition to to time series analysis of the historical patterns of the word-pairs.

Answers:
Yes
Question: Which of the classifiers showed the best performance?

Evidence:
raw_evidence: FLOAT SELECTED: TABLE II: A comparison of classification AVCs using word-pairs extracted by different feature selection methods

Answers:
Logistic regression
Question: Were any other word similar metrics, besides Jaccard metric, tested?

Evidence:
raw_evidence: FLOAT SELECTED: TABLE II: A comparison of classification AVCs using word-pairs extracted by different feature selection methods

Answers:
Yes
Question: How are the keywords associated with events such as protests selected?

Evidence:
raw_evidence: We approached the first and second challenges by using a Bayesian approach to learn which terms were associated with events, regardless of whether they are standard language, acronyms, or even a made-up word, so long as they match the events of interest. The third and fourth challenges are approached by using word-pairs, where we extract all the pairs of co-occurring words within each tweet. This allows us to recognize the context of the word ('Messi','strike' ) is different than ('labour','strike').
According to the distributional semantic hypothesis, event-related words are likely to be used on the day of an event more frequently than any normal day before or after the event. This will form a spike in the keyword count magnitude along the timeline as illustrated in Figure FIGREF6 . To find the words most associated with events, we search for the words that achieve the highest number of spikes matching the days of events. We use the Jaccard similarity metric as it values the spikes matching events and penalizes spikes with no event and penalizes events without spikes. Separate words can be noisy due to the misuse of the term by people, especially in big data environments. So, we rather used the word-pairs as textual features in order to capture the context of the word. For example, this can differentiate between the multiple usages of the word “strike” within the contexts of “lightning strike”, “football strike” and “labour strike”

Answers:
By using a Bayesian approach  and by using word-pairs, where they extract all the pairs of co-occurring words within each tweet.  They search for the words that achieve the highest number of spikes matching the days of events.
Question: How many speeches are in the dataset?

Evidence:

Answers:
5575 speeches
Question: What classification models were used?

Evidence:
raw_evidence: Text classification is a core task to many applications, like spam detection, sentiment analysis or smart replies. We used fastText and SVM BIBREF16 for preliminary experiments. We have pre-processed the text removing punctuation's and lowering the case. Facebook developers have developed fastText BIBREF17 which is a library for efficient learning of word representations and sentence classification. The reason we have used fastText is because of its promising results in BIBREF18 .

Answers:
fastText and SVM BIBREF16
Question: What is different in BERT-gen from standard BERT?

Evidence:
raw_evidence: As we iteratively concatenate the generated tokens, the BERT bi-directional self-attention mechanism would impact, at every new token, the representations of the previous tokens. To counter that, we use a left-to-right attention mask, similar to the one employed in the original Transformer decoder BIBREF1. For the input tokens in $X$, we apply such mask to all the target tokens $Y$ that were concatenated to $X$, so that input tokens can only attend to the other input tokens. Conversely, for target tokens $y_t$, we put an attention mask on all tokens $y_{>t}$, allowing target tokens $y_t$ to attend only to the input tokens and the already generated target tokens.

Answers:
They use a left-to-right attention mask so that the input tokens can only attend to other input tokens, and the target tokens can only attend to the input tokens and already generated target tokens.
Question: How are multimodal representations combined?

Evidence:
raw_evidence: To investigate whether our BERT-based model can transfer knowledge beyond language, we consider image features as simple visual tokens that can be presented to the model analogously to textual token embeddings. In order to make the $o_j$ vectors (of dimension $2048+4=2052$) comparable to BERT embeddings (of dimension 768), we use a simple linear cross-modal projection layer $W$ of dimensions $2052\hspace{-1.00006pt}\times \hspace{-1.00006pt}768$. The $N$ object regions detected in an image, are thus represented as $X_{img} = (W.o_1,...,W.o_N)$. Once mapped into the BERT embedding space with $W$, the image is seen by the rest of the model as a sequence of units with no explicit indication if it is a text or an image embedding.

Answers:
The image feature vectors are mapped into BERT embedding dimensions and treated like a text sequence afterwards.
Question: What is the problem with existing metrics that they are trying to address?

Evidence:
raw_evidence: Building Extractive CNN/Daily Mail

Answers:
Answer with content missing: (whole introduction) However, recent
studies observe the limits of ROUGE and find in
some cases, it fails to reach consensus with human.
judgment (Paulus et al., 2017; Schluter, 2017).
Question: How do they evaluate their proposed metric?

Evidence:
raw_evidence: As shown in Table TABREF13, there is almost no discrimination among the last four methods under ROUGE-1 F1, and their rankings under ROUGE-1/2/L are quite different. In contrast, FAR shows that UnifiedSum(E) covers the most facets. Although FAR is supposed to be favored as FAMs are already manually labeled and tell exactly if one sentence should be extracted (assuming our annotations are in agreement), to further verify that FAR correlates with human preference, we rank UnifiedSum(E), NeuSum, and Lead-3 in Table TABREF15. The order of the 1st rank in the human evaluation coincides with FAR. FAR also has higher Spearman's coefficient $\rho $ than ROUGE (0.457 vs. 0.44, n=30, threshold=0.362 at 95% significance).

Answers:
manually labeled and tell exactly if one sentence should be extracted (assuming our annotations are in agreement), to further verify that FAR correlates with human preference,
Question: How are discourse embeddings analyzed?

Evidence:
raw_evidence: To further study the information encoded in the discourse embeddings, we perform t-SNE clustering BIBREF20 on them, using the best performing model CNN2-DE (global). We examine the closest neighbors of each embedding, and observe that similar discourse relations tend to go together (e.g., explanation and interpretation; consequence and result). Some examples are given in Table TABREF29 . However, it is unclear how this pattern helps improve classification performance. We intend to investigate this question in future work.

Answers:
They perform t-SNE clustering to analyze discourse embeddings
Question: What was the previous state-of-the-art?

Evidence:
raw_evidence: In our paper, we opt for a state-of-the-art character bigram CNN classifier BIBREF4 , and investigate various ways in which the discourse information can be featurized and integrated into the CNN. Specifically,

Answers:
character bigram CNN classifier
Question: How are discourse features incorporated into the model?

Evidence:
raw_evidence: CNN2-PV. This model (Figure FIGREF10 , left+center) featurizes discourse information into a vector of relation probabilities. In order to derive the discourse features, an entity grid is constructed by feeding the document through an NLP pipeline to identify salient entities. Two flavors of discourse features are created by populating the entity grid with either (i) grammatical relations (GR) or (ii) RST discourse relations (RST). The GR features are represented as grammatical relation transitions derived from the entity grid, e.g., INLINEFORM0 . The RST features are represented as RST discourse relations with their nuclearity, e.g., INLINEFORM1 . The probability vectors are then distributions over relation types. For GR, the vector is a distribution over all the entity role transitions, i.e., INLINEFORM2 (see Table TABREF2 ). For RST, the vector is a distribution over all the RST discourse relations, i.e., INLINEFORM3 Denoting a feature as such with INLINEFORM4 , we construct the pooling vector INLINEFORM5 for the char-bigrams, and concatenate INLINEFORM6 to INLINEFORM7 before feeding the resulting vector to the softmax layer.

Answers:
They derive entity grid with grammatical relations and RST discourse relations and concatenate them with pooling vector for the char-bigrams before feeding to the resulting vector to the softmax layer.
Question: What discourse features are used?

Evidence:
raw_evidence: CNN2-PV. This model (Figure FIGREF10 , left+center) featurizes discourse information into a vector of relation probabilities. In order to derive the discourse features, an entity grid is constructed by feeding the document through an NLP pipeline to identify salient entities. Two flavors of discourse features are created by populating the entity grid with either (i) grammatical relations (GR) or (ii) RST discourse relations (RST). The GR features are represented as grammatical relation transitions derived from the entity grid, e.g., INLINEFORM0 . The RST features are represented as RST discourse relations with their nuclearity, e.g., INLINEFORM1 . The probability vectors are then distributions over relation types. For GR, the vector is a distribution over all the entity role transitions, i.e., INLINEFORM2 (see Table TABREF2 ). For RST, the vector is a distribution over all the RST discourse relations, i.e., INLINEFORM3 Denoting a feature as such with INLINEFORM4 , we construct the pooling vector INLINEFORM5 for the char-bigrams, and concatenate INLINEFORM6 to INLINEFORM7 before feeding the resulting vector to the softmax layer.

Answers:
Entity grid with grammatical relations and RST discourse relations.
Question: How are proof scores calculated?

Evidence:
raw_evidence: Unification Module. In backward chaining, unification between two atoms is used for checking whether they can represent the same structure. In discrete unification, non-variable symbols are checked for equality, and the proof fails if the symbols differ. In NTP, rather than comparing symbols, their embedding representations are compared by means of a RBF kernel. This allows matching different symbols with similar semantics, such as matching relations like ${grandFatherOf}$ and ${grandpaOf}$ . Given a proof state $= (_, _)$ , where $_$ and $_$ denote a substitution set and a proof score, respectively, unification is computed as follows:
The resulting proof score of $g$ is given by:
$$ \begin{aligned} \max _{f \in \mathcal {K}} & \; {unify}_(g, [f_{p}, f_{s}, f_{o}], (\emptyset , )) \\ & = \max _{f \in \mathcal {K}} \; \min \big \lbrace , \operatorname{k}(_{\scriptsize {grandpaOf}:}, _{f_{p}:}),\\ &\qquad \qquad \qquad \operatorname{k}(_{{abe}:}, _{f_{s}:}), \operatorname{k}(_{{bart}:}, _{f_{o}:}) \big \rbrace , \end{aligned}$$ (Eq. 3)
where $f \triangleq [f_{p}, f_{s}, f_{o}]$ is a fact in $\mathcal {K}$ denoting a relationship of type $f_{p}$ between $f_{s}$ and $f_{o}$ , $_{s:}$ is the embedding representation of a symbol $s$ , $$ denotes the initial proof score, and $\operatorname{k}({}\cdot {}, {}\cdot {})$ denotes the RBF kernel. Note that the maximum proof score is given by the fact $f \in \mathcal {K}$ that maximises the similarity between its components and the goal $\mathcal {K}$0 : solving the maximisation problem in eq:inference can be equivalently stated as a nearest neighbour search problem. In this work, we use ANNS during the forward pass for considering only the most promising proof paths during the construction of the neural network.

Answers:
'= ( , { ll k(h:, g:) if hV, gV

1 otherwise } )

where $_{h:}$ and $_{g:}$ denote the embedding representations of $h$ and $g$ , respectively.
Question: What submodules does the model consist of?

Evidence:
raw_evidence: We fill this gap by investigating rich external pretraining for neural segmentation. Following BIBREF4 and BIBREF5 , we adopt a globally optimised beam-search framework for neural structured prediction BIBREF9 , BIBREF17 , BIBREF18 , which allows word information to be modelled explicitly. Different from previous work, we make our model conceptually simple and modular, so that the most important sub module, namely a five-character window context, can be pretrained using external data. We adopt a multi-task learning strategy BIBREF19 , casting each external source of information as a auxiliary classification task, sharing a five-character window network. After pretraining, the character window network is used to initialize the corresponding module in our segmentor.

Answers:
five-character window context
Question: How they add human prefference annotation to fine-tuning process?

Evidence:
raw_evidence: The comparative evaluator is trained with maximum likelihood estimation (MLE) objective, as described in eq DISPLAY_FORM6
where $\mathcal {X}$ is the set of pairwise training examples contructed as described above, $Q(x_1, x_2) \in \lbrace >,<,\approx \rbrace $ is the true label for the pair ($x_1$, $x_2$), $D_\phi ^q(x_1, x_2)$ is the probability of the comparative discriminator's prediction being $q$ ($q \in \lbrace >,<,\approx \rbrace $) for the pair ($x_1$, $x_2$).

Answers:
human preference annotation is available, $Q(x_1, x_2) \in \lbrace >,<,\approx \rbrace $ is the true label for the pair
Question: Do the authors suggest that proposed metric replace human evaluation on this task?

Evidence:

Answers:
No
Question: What is the training objective of their pair-to-sequence model?

Evidence:
raw_evidence: The training objective is to minimize the negative likelihood of the aligned unanswerable question $\tilde{q}$ given the answerable question $q$ and its corresponding paragraph $p$ that contains the answer $a$ : L=-(q,q,p,a)DP(q|q,p,a;) where $\mathcal {D}$ is the training corpus and $\theta $ denotes all the parameters. Sequence-to-sequence and pair-to-sequence models are trained with the same objective.

Answers:
is to minimize the negative likelihood of the aligned unanswerable question $\tilde{q}$ given the answerable question $q$ and its corresponding paragraph $p$ that contains the answer
Question: How do they ensure the generated questions are unanswerable?

Evidence:
raw_evidence: To create training data for unanswerable question generation, we use (plausible) answer spans in paragraphs as pivots to align pairs of answerable questions and unanswerable questions. As shown in Figure 1 , the answerable and unanswerable questions of a paragraph are aligned through the text span “Victoria Department of Education” for being both the answer and plausible answer. These two questions are lexically similar and both asked with the same answer type in mind. In this way, we obtain the data with which the models can learn to ask unanswerable questions by editing answerable ones with word exchanges, negations, etc. Consequently, we can generate a mass of unanswerable questions with existing large-scale machine reading comprehension datasets.

Answers:
learn to ask unanswerable questions by editing answerable ones with word exchanges, negations, etc
Question: Does their approach require a dataset of unanswerable questions mapped to similar answerable questions?

Evidence:
raw_evidence: We attack the problem by automatically generating unanswerable questions for data augmentation to improve question answering models. The generated unanswerable questions should not be too easy for the question answering model so that data augmentation can better help the model. For example, a simple baseline method is randomly choosing a question asked for another paragraph, and using it as an unanswerable question. However, it would be trivial to determine whether the retrieved question is answerable by using word-overlap heuristics, because the question is irrelevant to the context BIBREF6 . In this work, we propose to generate unanswerable questions by editing an answerable question and conditioning on the corresponding paragraph that contains the answer. So the generated unanswerable questions are more lexically similar and relevant to the context. Moreover, by using the answerable question as a prototype and its answer span as a plausible answer, the generated examples can provide more discriminative training signal to the question answering model.

Answers:
Yes
Question: What conclusions are drawn from these experiments?

Evidence:
raw_evidence: The analysis of results from Tables TABREF17 , TABREF18 and TABREF19 show that 12 of 15 best results were obtained using new word embeddings. The evaluation results presented in Table TABREF20 (the chosen best embeddings models from Table TABREF19 ) prove that the best group of word embeddings is EC. The highest type F1-score was obtained for EC1 model, built using binary FastText Skip-gram method utilising subword information, with vector dimension equal to 300 and negative sampling equal to 10. The ability of the model to provide vector representation for the unknown words seems to be the most important. Also, previous models built using KGR10 (EP) are probably less accurate due to an incorrect tokenisation of the corpus. We used WCRFT tagger BIBREF22 , which utilises Toki BIBREF21 to tokenise the input text before the creation of the embeddings model. The comparison of EC1 with previous results obtained using only CRF BIBREF38 show the significant improvement across all the tested metrics: 3.6pp increase in strict F1-score, 1.36pp increase in relaxed precision, 5.61pp increase in relaxed recall and 3.51pp increase in relaxed F1-score.

Answers:
best results were obtained using new word embeddings, best group of word embeddings is EC, The highest type F1-score was obtained for EC1 model, built using binary FastText Skip-gram method utilising subword information, ability of the model to provide vector representation for the unknown words seems to be the most important
Question: What experiments are presented?

Evidence:
raw_evidence: Experiments were carried out by the method proposed in BIBREF27 . The first part is described as Task A, the purpose of which is to identify the boundaries of timexes and assign them to one of the following classes: date, time, duration, set.
We chose the best 3 results from each word embeddings group (EE, EP, EC) from Table TABREF19 presenting F1-scores for all models. Then we evaluated these results using more detailed measures for timexes, presented in BIBREF27 . The following measures were used to evaluate the quality of boundaries and class recognition, so-called strict match: strict precision (Str.P), strict recall (Str.R) and strict F1-score (Str.F1). A relaxed match (Rel.P, Rel.R, Rel.F1) evaluation has also been carried out to determine whether there is an overlap between the system entity and gold entity, e.g. [Sunday] and [Sunday morning] BIBREF27 . If there was an overlap, a relaxed type F1-score (Type.F1) was calculated BIBREF27 . The results are presented in Table TABREF20 .

Answers:
identify the boundaries of timexes and assign them to one of the following classes: date, time, duration, set,  Then we evaluated these results using more detailed measures for timexes