qinchuanhui/UDA-QA huggingface dataset:
Question: What is result of their Principal Component Analysis?

Evidence:
raw_evidence: Figure FIGREF14 (Top) shows the variance explained by the Top 100 principal components, obtained by applying PCA on the How2 and Multi30k training set visual features. The original feature dimensions are 2048 in both the cases. It is clear from the Figure FIGREF14 that most of the energy of the visual feature space resides in a low-dimensional subspace BIBREF14. In other words, there exist a few directions in the embedding space which disproportionately explain the variance. These "common" directions affect all of the embeddings in the same way, rendering them less discriminative. Figure FIGREF14 also shows the cumulative variance explained by Top 10, 20, 50 and 100 principal components respectively. It is clear that the visual features in the case of How2 dataset are much more dominated by the "common" dimensions, when compared to the Multi30k dataset. Further, this analysis is still at the sentence level, i.e. the visual features are much less discriminative among individual sentences, further aggravating the problem at the token level. This suggests that the existing visual features aren't sufficient enough to expect benefits from the visual modality in NMT, since they won't provide discriminativeness among the vocabulary elements at the token level during prediction. Further, this also indicates that under subword vocabulary such as BPE BIBREF15 or Sentence-Piece BIBREF16, the utility of such visual embeddings will only aggravate.

Answers:
existing visual features aren't sufficient enough to expect benefits from the visual modality in NMT
Question: What are 3 novel fusion techniques that are proposed?

Evidence:
raw_evidence: Proposed Fusion Techniques ::: Step-Wise Decoder Fusion
Our first proposed technique is the step-wise decoder fusion of visual features during every prediction step i.e. we concatenate the visual encoding as context at each step of the decoding process. This differs from the usual practice of passing the visual feature only at the beginning of the decoding process BIBREF5.
Proposed Fusion Techniques ::: Multimodal Attention Modulation
Similar to general attention BIBREF8, wherein a variable-length alignment vector $a_{th}(s)$, whose size equals the number of time steps on the source side, is derived by comparing the current target hidden state $h_{t}$ with each source hidden state $\overline{h_{s}}$; we consider a variant wherein the visual encoding $v_{t}$ is used to calculate an attention distribution $a_{tv}(s)$ over the source encodings as well. Then, the true attention distribution $a_{t}(s)$ is computed as an interpolation between the visual and text based attention scores. The score function is a content based scoring mechanism as usual.
Proposed Fusion Techniques ::: Visual-Semantic (VS) Regularizer
In terms of leveraging the visual modality for supervision, BIBREF1 use multi-task learning to learn grounded representations through image representation prediction. However, to our knowledge, visual-semantic supervision hasn't been much explored for multimodal translation in terms of loss functions.

Answers:
Step-Wise Decoder Fusion, Multimodal Attention Modulation, Visual-Semantic (VS) Regularizer
Question: What are two models' architectures in proposed solution?

Evidence:
raw_evidence: Method ::: Passage Ranking Model
The key component of our framework is the Ranker model, which is provided with a question $q$ and $K$ passages $\mathcal {P} = \lbrace p_1, p_2 ... p_K\rbrace $ from a pool of candidates, and outputs a chain of selected passages.
Method ::: Cooperative Reasoner
To alleviate the noise in the distant supervision signal $\mathcal {C}$, in addition to the conditional selection, we further propose a cooperative Reasoner model, also implemented with the MatchLSTM architecture (see Appendix SECREF6), to predict the linking entity from the selected passages. Intuitively, when the Ranker makes more accurate passage selections, the Reasoner will work with less noisy data and thus is easier to succeed. Specifically, the Reasoner learns to extract the linking entity from chains selected by a well-trained Ranker, and it benefits the Ranker training by providing extra rewards. Taking 2-hop as an example, we train the Ranker and Reasoner alternatively as a cooperative game:

Answers:
Reasoner model, also implemented with the MatchLSTM architecture, Ranker model
Question: How do two models cooperate to select the most confident chains?

Evidence:
raw_evidence: To alleviate the noise in the distant supervision signal $\mathcal {C}$, in addition to the conditional selection, we further propose a cooperative Reasoner model, also implemented with the MatchLSTM architecture (see Appendix SECREF6), to predict the linking entity from the selected passages. Intuitively, when the Ranker makes more accurate passage selections, the Reasoner will work with less noisy data and thus is easier to succeed. Specifically, the Reasoner learns to extract the linking entity from chains selected by a well-trained Ranker, and it benefits the Ranker training by providing extra rewards. Taking 2-hop as an example, we train the Ranker and Reasoner alternatively as a cooperative game:

Answers:
Reasoner learns to extract the linking entity from chains selected by a well-trained Ranker, and it benefits the Ranker training by providing extra rewards
Question: What benchmarks are created?

Evidence:
raw_evidence: In HotpotQA, on average we can find 6 candidate chains (2-hop) in a instance, and the human labeled true reasoning chain is unique. A predicted chain is correct if the chain only contains all supporting passages (exact match of passages).
In MedHop, on average we can find 30 candidate chains (3-hop). For each candidate chain our human annotators labeled whether it is correct or not, and the correct reasoning chain is not unique. A predicted chain is correct if it is one of the chains that human labeled as correct.
The accuracy is defined as the ratio:

Answers:
Answer with content missing: (formula) The accuracy is defined as the ratio # of correct chains predicted to # of evaluation samples
Question: What empricial investigations do they reference?

Evidence:
raw_evidence: Our paper investigates three aspects of human MT evaluation, with a special focus on assessing human–machine parity: the choice of raters, the use of linguistic context, and the creation of reference translations. We focus on the data shared by BIBREF3, and empirically test to what extent changes in the evaluation design affect the outcome of the human evaluation. We find that for all three aspects, human translations are judged more favourably, and significantly better than MT, when we make changes that we believe strengthen the evaluation design. Based on our empirical findings, we formulate a set of recommendations for human MT evaluation in general, and assessing human–machine parity in particular. All of our data are made publicly available for external validation and further analysis.
In this study, we address three aspects that we consider to be particularly relevant for human evaluation of MT, with a special focus on testing human–machine parity: the choice of raters, the use of linguistic context, and the construction of reference translations.
We empirically test and discuss the impact of these factors on human evaluation of MT in Sections SECREF3–SECREF5. Based on our findings, we then distil a set of recommendations for human evaluation of strong MT systems, with a focus on assessing human–machine parity (Section SECREF6).

Answers:
empirically test to what extent changes in the evaluation design affect the outcome of the human evaluation
Question: What languages do they investigate for machine translation?

Evidence:
raw_evidence: We use English translations of the Chinese source texts in the WMT 2017 English–Chinese test set BIBREF18 for all experiments presented in this article:

Answers:
English , Chinese
Question: What recommendations do they offer?

Evidence:
raw_evidence: Our experiments in Sections SECREF3–SECREF5 show that machine translation quality has not yet reached the level of professional human translation, and that human evaluation methods which are currently considered best practice fail to reveal errors in the output of strong NMT systems. In this section, we recommend a set of evaluation design changes that we believe are needed for assessing human–machine parity, and will strengthen the human evaluation of MT in general.
Recommendations ::: (R1) Choose professional translators as raters.
In our blind experiment (Section SECREF3), non-experts assess parity between human and machine translation where professional translators do not, indicating that the former neglect more subtle differences between different translation outputs.
Recommendations ::: (R2) Evaluate documents, not sentences.
When evaluating sentences in random order, professional translators judge machine translation more favourably as they cannot identify errors related to textual coherence and cohesion, such as different translations of the same product name. Our experiments show that using whole documents (i. e., full news articles) as unit of evaluation increases the rating gap between human and machine translation (Section SECREF4).
Recommendations ::: (R3) Evaluate fluency in addition to adequacy.
Raters who judge target language fluency without access to the source texts show a stronger preference for human translation than raters with access to the source texts (Sections SECREF4 and SECREF24). In all of our experiments, raters prefer human translation in terms of fluency while, just as in BIBREF3's BIBREF3 evaluation, they find no significant difference between human and machine translation in sentence-level adequacy (Tables TABREF21 and TABREF30). Our error analysis in Table TABREF34 also indicates that MT still lags behind human translation in fluency, specifically in grammaticality.
Recommendations ::: (R4) Do not heavily edit reference translations for fluency.
In professional translation workflows, texts are typically revised with a focus on target language fluency after an initial translation step. As shown in our experiment in Section SECREF24, aggressive revision can make translations more fluent but less accurate, to the degree that they become indistinguishable from MT in terms of accuracy (Table TABREF30).
Recommendations ::: (R5) Use original source texts.
Raters show a significant preference for human over machine translations of texts that were originally written in the source language, but not for source texts that are translations themselves (Section SECREF35). Our results are further evidence that translated texts tend to be simpler than original texts, and in turn easier to translate with MT.

Answers:
 Choose professional translators as raters,  Evaluate documents, not sentences, Evaluate fluency in addition to adequacy, Do not heavily edit reference translations for fluency, Use original source texts
Question: By how much they improve over the previous state-of-the-art?

Evidence:
raw_evidence: Finally, our combined model which uses both Latent and Explicit structure performs the best with a strong improvement of 1.08 points in ROUGE-L over our base pointer-generator model and 0.6 points in ROUGE-1. It shows that the latent and explicit information are complementary and a model can jointly leverage them to produce better summaries.

Answers:
1.08 points in ROUGE-L over our base pointer-generator model , 0.6 points in ROUGE-1
Question: Is there any evidence that encoders with latent structures work well on other tasks?

Evidence:
raw_evidence: Encoders with induced latent structures have been shown to benefit several tasks including document classification, natural language inference BIBREF12, BIBREF13, and machine translation BIBREF11. Building on this motivation, our latent structure attention module builds upon BIBREF12 to model the dependencies between sentences in a document. It uses a variant of Kirchhoff’s matrix-tree theorem BIBREF14 to model such dependencies as non-projective tree structures(§SECREF3). The explicit attention module is linguistically-motivated and aims to incorporate sentence-level structures from externally annotated document structures. We incorporate a coreference based sentence dependency graph, which is then combined with the output of the latent structure attention module to produce a hybrid structure-aware sentence representation (§SECREF5).

Answers:
Yes
Question: What evidence do they present that the model attends to shallow context clues?

Evidence:
raw_evidence: One way to analyze the model is to compute model gradients with respect to input features BIBREF26, BIBREF25. Figure FIGREF37 shows that in this particular example, the most important model inputs are verbs possibly associated with the entity butter, in addition to the entity's mentions themselves. It further shows that the model learns to extract shallow clues of identifying actions exerted upon only the entity being tracked, regardless of other entities, by leveraging verb semantics.

Answers:
Using model gradients with respect to input features they presented that the most important model inputs are verbs associated with entities which shows that the model attends to shallow context clues
Question: In what way is the input restructured?

Evidence:
raw_evidence: Our approach consists of structuring input to the transformer network to use and guide the self-attention of the transformers, conditioning it on the entity. Our main mode of encoding the input, the entity-first method, is shown in Figure FIGREF4. The input sequence begins with a [START] token, then the entity under consideration, then a [SEP] token. After each sentence, a [CLS] token is used to anchor the prediction for that sentence. In this model, the transformer can always observe the entity it should be primarily “attending to” from the standpoint of building representations. We also have an entity-last variant where the entity is primarily observed just before the classification token to condition the [CLS] token's self-attention accordingly. These variants are naturally more computationally-intensive than post-conditioned models, as we need to rerun the transformer for each distinct entity we want to make a prediction for.
As an additional variation, we can either run the transformer once per document with multiple [CLS] tokens (a document-level model as shown in Figure FIGREF4) or specialize the prediction to a single timestep (a sentence-level model). In a sentence level model, we formulate each pair of entity $e$ and process step $t$ as a separate instance for our classification task. Thus, for a process with $T$ steps and $m$ entities we get $T \times m$ input sequences for fine tuning our classification task.

Answers:
In four entity-centric ways - entity-first, entity-last, document-level and sentence-level
Question: What are their results on the entity recognition task?

Evidence:
raw_evidence: The performances of the NER experiments are reported separately for three different parts of the system proposed.
Table 6 presents the comparison of the various methods while performing NER on the bot-generated corpora and the user-generated corpora. Results shown that, in the first case, in the training set the F1 score is always greater than 97%, with a maximum of 99.65%. With both test sets performances decrease, varying between 94-97%. In the case of UGC, comparing the F1 score we can observe how performances significantly decrease. It can be considered a natural consequence of the complex nature of the users' informal language in comparison to the structured message created by the bot.

Answers:
With both test sets performances decrease, varying between 94-97%
Question: What task-specific features are used?

Evidence:
raw_evidence: In total, we define 26 features for describing each token: 1)POS tag; 2)Chunk tag; 3)Position of the token within the text, normalized between 0 and 1; 4)If the token starts with a capital letter; 5)If the token is a digit. Gazetteers: 6)Contributor first names; 7)Contributor last names; 8)Contributor types ("soprano", "violinist", etc.); 9)Classical work types ("symphony", "overture", etc.); 10)Musical instruments; 11)Opus forms ("op", "opus"); 12)Work number forms ("no", "number"); 13)Work keys ("C", "D", "E", "F" , "G" , "A", "B", "flat", "sharp"); 14)Work Modes ("major", "minor", "m"). Finally, we complete the tokens' description including as token's features the surface form, the POS and the chunk tag of the previous and the following two tokens (12 features).

Answers:
6)Contributor first names, 7)Contributor last names, 8)Contributor types ("soprano", "violinist", etc.), 9)Classical work types ("symphony", "overture", etc.), 10)Musical instruments, 11)Opus forms ("op", "opus"), 12)Work number forms ("no", "number"), 13)Work keys ("C", "D", "E", "F" , "G" , "A", "B", "flat", "sharp"), 14)Work Modes ("major", "minor", "m")
Question: What kind of corpus-based features are taken into account?

Evidence:
raw_evidence: We define a set of features for characterizing the text at the token level. We mix standard linguistic features, such as Part-Of-Speech (POS) and chunk tag, together with several gazetteers specifically built for classical music, and a series of features representing tokens' left and right context. For extracting the POS and the chunk tag we use the Python library twitter_nlp, presented in BIBREF1 .

Answers:
standard linguistic features, such as Part-Of-Speech (POS) and chunk tag, series of features representing tokens' left and right context
Question: Which machine learning algorithms did the explore?

Evidence:
raw_evidence: However, recent advances in Deep Learning techniques have shown that the NER task can benefit from the use of neural architectures, such as biLSTM-networks BIBREF3 , BIBREF4 . We use the implementation proposed in BIBREF24 for conducting three different experiments. In the first, we train the model using only the word embeddings as feature. In the second, together with the word embeddings we use the POS and chunk tag. In the third, all the features previously defined are included, in addition to the word embeddings. For every experiment, we use both the pre-trained embeddings and the ones that we created with our Twitter corpora. In section 4, results obtained from the several experiments are reported.

Answers:
biLSTM-networks
Question: What language is the Twitter content in?

Evidence:
raw_evidence: In this work, we analyze a set of tweets related to a specific classical music radio channel, BBC Radio 3, interested in detecting two types of musical named entities, Contributor and Musical Work.

Answers:
English
Question: What is the architecture of the siamese neural network?

Evidence:
raw_evidence: To further distinguish speech from different Arabic dialects, while making speech from the same dialect more similar, we adopted a Siamese neural network architecture BIBREF24 based on an i-vector feature space. The Siamese neural network has two parallel convolutional networks, INLINEFORM0 , that share the same set of weights, INLINEFORM1 , as shown in Figure FIGREF5 (a). Let INLINEFORM2 and INLINEFORM3 be a pair of i-vectors for which we wish to compute a distance. Let INLINEFORM4 be the label for the pair, where INLINEFORM5 = 1 if the i-vectors INLINEFORM6 and INLINEFORM7 belong to same dialect, and INLINEFORM8 otherwise. To optimize the network, we use a Euclidean distance loss function between the label and the cosine distance, INLINEFORM9 , where INLINEFORM10

Answers:
two parallel convolutional networks, INLINEFORM0 , that share the same set of weights
Question: Which are the four Arabic dialects?

Evidence:
raw_evidence: For the MGB-3 ADI task, the challenge organizers provided 13,825 utterances (53.6 hours) for the training (TRN) set, 1,524 utterances (10 hours) for a development (DEV) set, and 1,492 utterances (10.1 hours) for a test (TST) set. Each dataset consisted of five Arabic dialects: Egyptian (EGY), Levantine (LEV), Gulf (GLF), North African (NOR), and Modern Standard Arabic (MSA). Detailed statistics of the ADI dataset can be found in BIBREF23 . Table TABREF3 shows some facts about the evaluation conditions and data properties. Note that the development set is relatively small compared to the training set. However, it is matched with the test set channel domain. Thus, the development set provides valuable information to adapt or compensate the channel (recording) domain mismatch between the train and test sets.

Answers:
Egyptian (EGY), Levantine (LEV), Gulf (GLF), North African (NOR)
Question: What factors contribute to interpretive biases according to this research?

Evidence:
raw_evidence: While the choice of wording helps to convey bias, just as crucial is the way that the reporters portray the march as being related to other events. Which events authors choose to include in their history, which they leave out, and the way the events chosen relate to the march are crucial factors in conveying bias. Townhall's bias against the March of Science expressed in the argument that it politicizes science cannot be traced back to negative opinion words; it relies on a comparison between the March for Science and the Women's March, which is portrayed as a political, anti-Trump event. Newsbusters takes a different track: the opening paragraph conveys an overall negative perspective on the March for Science, despite its neutral language, but it achieves this by contrasting general interest in the march with a claimed negative view of the march by many “actual scientists.” On the other hand, the New York Times points to an important and presumably positive outcome of the march, despite its controversiality: a renewed look into the role of science in public life and politics. Like Newsbusters, it lacks any explicit evaluative language and relies on the structural relations between events to convey an overall positive perspective; it contrasts the controversy surrounding the march with a claim that the march has triggered an important discussion, which is in turn buttressed by the reporter's mentioning of the responses of the Times' readership.

Answers:
Which events authors choose to include in their history, which they leave out, and the way the events chosen relate to the march
Question: Which interpretative biases are analyzed in this paper?

Evidence:
raw_evidence: An epistemic ME game is an ME game with a Harsanyi type space and a type/history correspondence as we've defined it. By adding types to an ME game, we provide the beginnings of a game theoretic model of interpretive bias that we believe is completely new. Our definition of bias is now: [Interpretive Bias] An interpretive bias in an epistemic ME game is the probability distribution over types given by the belief function of the conversationalists or players, or the Jury. Note that in an ME game there are typically several interpretive biases at work: each player has her own bias, as does the Jury.

Answers:
in an ME game there are typically several interpretive biases at work: each player has her own bias, as does the Jury
Question: How many words are coded in the dictionary?

Evidence:
raw_evidence: We pair 11'248 standard German written words with their phonetical representations in six different Swiss dialects: Zürich, St. Gallen, Basel, Bern, Visp, and Stans (Figure FIGREF1). The phonetic words were written in a modified version of the Speech Assessment Methods Phonetic Alphabet (SAMPA). The Swiss German phonetic words are also paired with Swiss German writings in the latin alphabet. (From here onwards, a phonetic representation of a Swiss German word will be called a SAMPA and a written Swiss German word will be called a GSW.)

Answers:
11'248