qinchuanhui/UDA-QA huggingface dataset:
Question: How does the model compute the likelihood of executing to the correction semantic denotation?

Evidence:
raw_evidence: Since the training data consists only of utterance-denotation pairs, the ranker is trained to maximize the log-likelihood of the correct answer $z$ by treating logical forms as a latent variable:
It is impractical to rely solely on a neural decoder to find the most likely logical form at run time in the weakly-supervised setting. One reason is that although the decoder utilizes global utterance features for generation, it cannot leverage global features of the logical form since a logical form is conditionally generated following a specific tree-traversal order. To this end, we follow previous work BIBREF21 and introduce a ranker to the system. The role of the ranker is to score the candidate logical forms generated by the parser; at test time, the logical form receiving the highest score will be used for execution. The ranker is a discriminative log-linear model over logical form $y$ given utterance $x$ :

Answers:
By treating logical forms as a latent variable and training a discriminative log-linear model over logical form y given x.
Question: Which conventional alignment models do they use as guidance?

Evidence:
raw_evidence: Inspired by the supervised reordering in conventional SMT, in this paper, we propose a Supervised Attention based NMT (SA-NMT) model. Specifically, similar to conventional SMT, we first run off-the-shelf aligners (GIZA++ BIBREF3 or fast_align BIBREF4 etc.) to obtain the alignment of the bilingual training corpus in advance. Then, treating this alignment result as the supervision of attention, we jointly learn attention and translation, both in supervised manners. Since the conventional aligners delivers higher quality alignment, it is expected that the alignment in the supervised attention NMT will be improved leading to better end-to-end translation performance. One advantage of the proposed SA-NMT is that it implements the supervision of attention as a regularization in the joint training objective (§3.2). Furthermore, since the supervision of attention lies in the middle of the entire network architecture rather than the top ( as in the supervision of translation (see Figure 1(b)), it serves to mitigate the vanishing gradient problem during the back-propagation BIBREF7 .

Answers:
GIZA++ BIBREF3 or fast_align BIBREF4
Question: Which dataset do they use?

Evidence:
raw_evidence: For the low resource translation task, we used the BTEC corpus as the training data, which consists of 30k sentence pairs with 0.27M Chinese words and 0.33M English words. As development and test sets, we used the CSTAR03 and IWSLT04 held out sets, respectively. We trained a 4-gram language model on the target side of training corpus for running Moses. For training all NMT systems, we employed the same settings as those in the large scale task, except that vocabulary size is 6000, batch size is 16, and the hyper-parameter INLINEFORM0 for SA-NMT.
We used the data from the NIST2008 Open Machine Translation Campaign. The training data consisted of 1.8M sentence pairs, the development set was nist02 (878 sentences), and the test sets are were nist05 (1082 sentences), nist06 (1664 sentences) and nist08 (1357 sentences).

Answers:
BTEC corpus, the CSTAR03 and IWSLT04 held out sets, the NIST2008 Open Machine Translation Campaign
Question: Do they compare their algorithm to voting without weights?

Evidence:

Answers:
No
Question: What are state of the art methods authors compare their work with? 

Evidence:
raw_evidence: Table TABREF21 shows the performance of non-static capsule network for fake news detection in comparison to other methods. The accuracy of our model is 7.8% higher than the best result achieved by LSVM.
As mentioned in Section SECREF13, the LIAR dataset is a multi-label dataset with short news statements. In comparison to the ISOT dataset, the classification task for this dataset is more challenging. We evaluate the proposed model while using different metadata, which is considered as speaker profiles. Table TABREF30 shows the performance of the capsule network for fake news detection by adding every metadata. The best result of the model is achieved by using history as metadata. The results show that this model can perform better than state-of-the-art baselines including hybrid CNN BIBREF15 and LSTM with attention BIBREF16 by 3.1% on the validation set and 1% on the test set.

Answers:
ISOT dataset: LLVM
Liar dataset: Hybrid CNN and LSTM with attention
Question: What are the baselines model?

Evidence:
raw_evidence: Results of Predictive Models. For the purpose of evaluation, we report the average results after 10-fold cross-validation. Here we consider five baselines to compare with GraLap: (i) Uniform: assign 3 to all the references assuming equal intensity, (ii) SVR+W: recently proposed Support Vector Regression (SVR) with the feature set mentioned in BIBREF4 , (iii) SVR+O: SVR model with our feature set, (iv) C4.5SSL: C4.5 semi-supervised algorithm with our feature set BIBREF23 , and (v) GLM: the traditional graph-based LP model with our feature set BIBREF9 . Three metrics are used to compare the results of the competing models with the annotated labels: Root Mean Square Error (RMSE), Pearson's correlation coefficient ( INLINEFORM0 ), and coefficient of determination ( INLINEFORM1 ).

Answers:
(i) Uniform, (ii) SVR+W, (iii) SVR+O, (iv) C4.5SSL, (v) GLM
Question: What languages are explored in the work?

Evidence:
raw_evidence: In this section, we present the experimental setting for this task
Corpus: SEAME (South East Asia Mandarin-English), a conversational Mandarin-English code-switching speech corpus consists of spontaneously spoken interviews and conversations BIBREF8 . Our dataset (LDC2015S04) is the most updated version of the Linguistic Data Consortium (LDC) database. However, the statistics are not identical to BIBREF23 . The corpus consists of two phases. In Phase I, only selected audio segments were transcribed. In Phase II, most of the audio segments were transcribed. According to the authors, it was not possible to restore the original dataset. The authors only used Phase I corpus. Few speaker ids are not in the speaker list provided by the authors BIBREF23 . Therefore as a workaround, we added these ids to the train set. As our future reference, the recording lists are included in the supplementary material.

Answers:
Mandarin, English
Question: What is the state-of-the-art neural coreference resolution model?

Evidence:
raw_evidence: We use the English coreference resolution dataset from the CoNLL-2012 shared task BIBREF15 , the benchmark dataset for the training and evaluation of coreference resolution. The training dataset contains 2408 documents with 1.3 million words. We use two state-of-art neural coreference resolution models described by BIBREF2 and BIBREF1 . We report the average F1 value of standard MUC, B INLINEFORM0 and CEAF INLINEFORM1 metrics for the original test set.

Answers:
BIBREF2 , BIBREF1
Question: How large is the dataset?

Evidence:
raw_evidence: There are 8757 news records in our preprocessed data set. We use Jenks natural breaks BIBREF24 to discretize continuous variables $S_{N\!P}$ and $S_{Q\!P}$ both into five categories denoted by nominal values from 0 to 4, where larger values still fall into bins with larger nominal value. Let $D_{N\!P}$ and $D_{Q\!P}$ denote the discretized variables $S_{N\!P}$ and $S_{Q\!P}$, respectively. We derived the information table that only contains discrete features from our original dataset. A fraction of the information table is shown in Table TABREF23.

Answers:
8757 news records
Question: What features do they extract?

Evidence:
raw_evidence: Satirical news is not based on or does not aim to state the fact. Rather, it uses parody or humor to make statement, criticisms, or just amusements. In order to achieve such effect, contradictions are greatly utilized. Therefore, inconsistencies significantly exist in different parts of a satirical news tweet. In addition, there is a lack of entity or inconsistency between entities in news satire. We extracted these features at semantic level from different sub-structures of the news tweet. Different structural parts of the sentence are derived by part-of-speech tagging and named entity recognition by Flair. The inconsistencies in different structures are measured by cosine similarity of word phrases where words are represented by Glove word vectors. We explored three different aspects of inconsistency and designed metrics for their measurements. A word level feature using tf-idf BIBREF22 is added for robustness.

Answers:
Inconsistency in Noun Phrase Structures,  Inconsistency Between Clauses, Inconsistency Between Named Entities and Noun Phrases, Word Level Feature Using TF-IDF
Question: What they use as a metric of finding hot spots in meeting?

Evidence:
raw_evidence: In spite of the windowing approach, the class distribution is still skewed, and an accuracy metric would reflect the particular class distribution in our data set. Therefore, we adopt the unweighted average recall (UAR) metric commonly used in emotion classification research. UAR is a reweighted accuracy where the samples of both classes are weighted equally in aggregate. UAR thus simulates a uniform class distribution. To match the objective, our classifiers are trained on appropriately weighted training data. Note that chance performance for UAR is by definition 50%, making results more comparable across different data sets.

Answers:
unweighted average recall (UAR) metric
Question: How big is ICSI meeting corpus?

Evidence:
raw_evidence: The ICSI Meeting Corpus BIBREF11 is a collection of meeting recordings that has been thoroughly annotated, including annotations for involvement hot spots BIBREF12, linguistic utterance units, and word time boundaries based on forced alignment. The dataset is comprised of 75 meetings and about 70 hours of real-time audio duration, with 6 speakers per meeting on average. Most of the participants are well-acquainted and friendly with each other. Hot spots were originally annotated with 8 levels and degrees, ranging from `not hot' to `luke warm' to `hot +'. Every utterance was labeled with one of these discrete labels by a single annotator. Hightened involvement is rare, being marked on only 1% of utterances.

Answers:
 75 meetings and about 70 hours of real-time audio duration
Question: What annotations are available in ICSI meeting corpus?

Evidence:
raw_evidence: The ICSI Meeting Corpus BIBREF11 is a collection of meeting recordings that has been thoroughly annotated, including annotations for involvement hot spots BIBREF12, linguistic utterance units, and word time boundaries based on forced alignment. The dataset is comprised of 75 meetings and about 70 hours of real-time audio duration, with 6 speakers per meeting on average. Most of the participants are well-acquainted and friendly with each other. Hot spots were originally annotated with 8 levels and degrees, ranging from `not hot' to `luke warm' to `hot +'. Every utterance was labeled with one of these discrete labels by a single annotator. Hightened involvement is rare, being marked on only 1% of utterances.

Answers:
8 levels and degrees, ranging from `not hot' to `luke warm' to `hot +'. Every utterance was labeled with one of these discrete labels by a single annotator
Question: Is such bias caused by bad annotation?

Evidence:
raw_evidence: Natural Language Inference (NLI) is often used to gauge a model's ability to understand a relationship between two texts BIBREF0 , BIBREF1 . In NLI, a model is tasked with determining whether a hypothesis (a woman is sleeping) would likely be inferred from a premise (a woman is talking on the phone). The development of new large-scale datasets has led to a flurry of various neural network architectures for solving NLI. However, recent work has found that many NLI datasets contain biases, or annotation artifacts, i.e., features present in hypotheses that enable models to perform surprisingly well using only the hypothesis, without learning the relationship between two texts BIBREF2 , BIBREF3 , BIBREF4 . For instance, in some datasets, negation words like “not” and “nobody” are often associated with a relationship of contradiction. As a ramification of such biases, models may not generalize well to other datasets that contain different or no such biases.

Answers:
No
Question: How do they determine similar environments for fragments in their data augmentation scheme?

Evidence:
raw_evidence: The only remaining question is what makes two environments similar enough to infer the existence of a common category. There is, again, a large literature on this question (including the aforementioned language modeling, unsupervised parsing, and alignment work), but in the current work we will make use of a very simple criterion: fragments are interchangeable if they occur in at least one lexical environment that is exactly the same. Given a window size INLINEFORM0 , a sequence of INLINEFORM1 words INLINEFORM2 , and a fragment consisting of a set of INLINEFORM3 spans INLINEFORM4 , the environment is given by INLINEFORM5 , i.e. a INLINEFORM6 -word window around each span of the fragment.

Answers:
fragments are interchangeable if they occur in at least one lexical environment that is exactly the same
Question: Do they experiment with language modeling on large datasets?

Evidence:

Answers:
No
Question: Which languages do they test on?

Evidence:
raw_evidence: Space requirements might still be considerable (comparable to those used by n-gram language models), and similar tricks can be used to reduce memory usage BIBREF27 . The above pseudocode is agnostic with respect to the choice of fragmentation and environment functions; task-specific choices are described in more detail for each experiment below.
Discussion

Answers:
Answer with content missing: (Applications section) We use Wikipedia articles
in five languages
(Kinyarwanda, Lao, Pashto, Tok Pisin, and a subset of English) as well as the Na dataset of Adams
et al. (2017).
Select:
Kinyarwanda, Lao, Pashto, Tok Pisin, and a subset of English
Question: What limitations are mentioned?

Evidence:
raw_evidence: We identified some limitations during the process, which we describe in this section.
When deciding publisher partisanship, the number of people from whom we computed the score was small. For example, de Stentor is estimated to reach 275K readers each day on its official website. Deciding the audience leaning from 55 samples was subject to sampling bias. Besides, the scores differ very little between publishers. None of the publishers had an absolute score higher than 1, meaning that even the most partisan publisher was only slightly partisan. Deciding which publishers we consider as partisan and which not is thus not very reliable.
The article-level annotation task was not as well-defined as on a crowdsourcing platform. We included the questions as part of an existing survey and didn't want to create much burden to the annotators. Therefore, we did not provide long descriptive text that explained how a person should annotate an article. We thus run under the risk of annotator bias. This is one of the reasons for a low inter-rater agreement.

Answers:
deciding publisher partisanship, risk annotator bias because of short description text provided to annotators
Question: What examples of applications are mentioned?

Evidence:
raw_evidence: This dataset is aimed to contribute to developing a partisan news detector. There are several ways that the dataset can be used to devise the system. For example, it is possible to train the detector using publisher-level labels and test with article-level labels. It is also possible to use semi-supervised learning and treat the publisher-level part as unsupervised, or use only the article-level part. We also released the raw survey data so that new mechanisms to decide the article-level labels can be devised.

Answers:
partisan news detector
Question: Did they crowdsource the annotations?

Evidence:
raw_evidence: To collect article-level labels, we utilized a platform in the company that has been used by the market research team to collect surveys from the subscribers of different news publishers. The survey works as follows: The user is first presented with a set of selected pages (usually 4 pages and around 20 articles) from the print paper the day before. The user can select an article each time that he or she has read, and answer some questions about it. We added 3 questions to the existing survey that asked the level of partisanship, the polarity of partisanship, and which pro- or anti- entities the article presents. We also asked the political standpoint of the user. The complete survey can be found in Appendices.

Answers:
Yes
Question: Why they conclude that the usage of Gated-Attention provides no competitive advantage against concatenation in this setting?

Evidence:
raw_evidence: We notice that the concatenation consistently outperforms the gated-attention mechanism for both training and testing instructions. We suspect that the gated-attention is useful in the scenarios where objects are described in terms of multiple attributes, but it has no to harming effect when it comes to the order connectors.

Answers:
concatenation consistently outperforms the gated-attention mechanism for both training and testing instructions
Question: What semantic features help in detecting whether a piece of text is genuine or generated? of 

Evidence:
raw_evidence: Many practical fake news detection algorithms use a kind of semantic side information, such as whether the generated text is factually correct, in addition to its statistical properties. Although statistical side information would be straightforward to incorporate in the hypothesis testing framework, it remains to understand how to cast such semantic knowledge in a statistical decision theory framework.

Answers:
No feature is given, only discussion that semantic features are use in practice and yet to be discovered how to embed that knowledge into statistical decision theory framework.
Question: Which language models generate text that can be easier to classify as genuine or generated?

Evidence:
raw_evidence: Suppose we are given a specific language model such as GPT-2 BIBREF6, GROVER BIBREF8, or CTRL BIBREF7, and it is characterized in terms of estimates of either cross-entropy $H(P,Q)$ or perplexity $\mathrm {PPL}(P,Q)$.
We can see directly that the Neyman-Pearson error of detection in the case of i.i.d. tokens is:
and similar results hold for ergodic observations.
Since we think of $H(P)$ as a constant, we observe that the error exponent for the decision problem is precisely an affine shift of the cross-entropy. Outputs from models that are better in the sense of cross-entropy or perplexity are harder to distinguish from authentic text.
Thus we see that intuitive measures of generative text quality match a formal operational measure of indistinguishability that comes from the hypothesis testing limit.

Answers:
Outputs from models that are better in the sense of cross-entropy or perplexity are harder to distinguish from authentic text.
Question: Is the assumption that natural language is stationary and ergodic valid?

Evidence:
raw_evidence: Manning and Schütze argue that, even though not quite correct, language text can be modeled as stationary, ergodic random processes BIBREF29, an assumption that we follow. Moreover, given the diversity of language production, we assume this stationary ergodic random process with finite alphabet $\mathcal {A}$ denoted $X = \lbrace X_i, -\infty < i < \infty \rbrace $ is non-null in the sense that always $P(x_{-m}^{-1}) > 0$ and
This is sometimes called the smoothing requirement.

Answers:
It is not completely valid for natural languages because of diversity of language - this is called smoothing requirement.
Question: What architectural factors were investigated?

Evidence:
raw_evidence: We find that all the factors we tested can qualitatively affect how a model generalizes on the question formation task. These factors are the type of recurrent unit, the type of attention, and the choice of sequential vs. tree-based model structure. Even though all these factors affected the model's decision between move-main and move-first, only the use of a tree-based model can be said to impart a hierarchical bias, since this was the only model type that chose a hierarchical generalization across both of our tasks. Specific findings that support these general conclusions include:

Answers:
type of recurrent unit, type of attention, choice of sequential vs. tree-based model structure
Question: What is the introduced meta-embedding method introduced in this paper?

Evidence:
raw_evidence: To overcome the above-mentioned challenges, we propose a locally-linear meta-embedding learning method that (a) requires only the words in the vocabulary of each source embedding, without having to predict embeddings for missing words, (b) can meta-embed source embeddings with different dimensionalities, (c) is sensitive to the diversity of the neighbourhoods of the source embeddings.
Our proposed method comprises of two steps: a neighbourhood reconstruction step (Section "Nearest Neighbour Reconstruction" ), and a projection step (Section "Projection to Meta-Embedding Space" ). In the reconstruction step, we represent the embedding of a word by the linearly weighted combination of the embeddings of its nearest neighbours in each source embedding space. Although the number of words in the vocabulary of a particular source embedding can be potentially large, the consideration of nearest neighbours enables us to limit the representation to a handful of parameters per each word, not exceeding the neighbourhood size. The weights we learn are shared across different source embeddings, thereby incorporating the information from different source embeddings in the meta-embedding. Interestingly, vector concatenation, which has found to be an accurate meta-embedding method, can be derived as a special case of this reconstruction step.

Answers:
proposed method comprises of two steps: a neighbourhood reconstruction step (Section "Nearest Neighbour Reconstruction" ), and a projection step (Section "Projection to Meta-Embedding Space" ). In the reconstruction step, we represent the embedding of a word by the linearly weighted combination of the embeddings of its nearest neighbours in each source embedding space.
Question: How long are dialogue recordings used for evaluation?

Evidence:
raw_evidence: We conducted our experiments on the CSJ BIBREF25, which is one of the most widely used evaluation sets for Japanese speech recognition. The CSJ consists of more than 600 hrs of Japanese recordings.
While most of the content is lecture recordings by a single speaker, CSJ also contains 11.5 hrs of 54 dialogue recordings (average 12.8 min per recording) with two speakers, which were the main target of ASR and speaker diarization in this study. During the dialogue recordings, two speakers sat in two adjacent sound proof chambers divided by a glass window. They could talk with each other over voice connection through a headset for each speaker. Therefore, speech was recorded separately for each speaker, and we generated mixed monaural recordings by mixing the corresponding speeches of two speakers. When mixing two recordings, we did not apply any normalization of speech volume. Due to this recording procedure, we were able to use non-overlapped speech to evaluate the oracle WERs.

Answers:
average 12.8 min per recording