qinchuanhui/UDA-QA huggingface dataset:
Question: Do transferring hurt the performance is the corpora are not related?

Evidence:
raw_evidence: We also performe another experiment to examine INIT and MULT method for original WikiQA. The F1-score for this dataset is equal to 33.73; however, the average INIT result for both SQuAD and SelQA as initializers is 30.50. In addition, the average results for MULT and ISS-MULT are 31.76 and 32.65, respectively. The result on original WikiQA indicates that all three transfer learning methods not only do not improve the results but also hurt the F1-score. Therefore, SelQA and SQuAD could not estimate a proper initial point for gradient based optimization method. Moreover, these corpora could not refine the error surface of the original WikiQA dataset during optimization for MULT and ISS-MULT method.
These are because other datasets could not add new information to the original dataset or they apparently add some redundant information which are dissimilar to the target dataset. Although ISS-MULT tries to remove this effect and consequently the result is improved, this method is on top of MULT method, and the result is significantly based on the effectiveness of this method.

Answers:
Yes
Question: Is accuracy the only metric they used to compare systems?

Evidence:
raw_evidence: In this paper, two main question answering tasks such as answer selection and answer triggering have been examined. In the answer triggering task, there is not a guarantee to have the correct answer among the list of answers. However, in answer selection, there is at least one correct answer among the candidates. As a result, answer triggering is a more challenging task. To report the result for answer selection, MAP and MRR are used; however, the answer triggering task is evaluated by F1-score. The result for MULT Method is reported in Table. 1.

Answers:
No
Question: How do they transfer the model?

Evidence:
raw_evidence: Another way to improve this method could be to select the samples which are more relevant to the target dataset. Based on the importance of the similarity between the datasets for transfer learning in the NLP tasks, this paper proposes to use the most relevant samples from the source dataset to train on the target dataset. One way to find the most similar samples is finding the pair-wise distance between all samples of the development set of the target dataset and source dataset.
To solve this problem, we propose using a clustering algorithm on the development set. The clustering algorithm used ihere is a hierarchical clustering algorithm. The cosine similarity is used as a criteria to cluster each question and answer. Therefore, these clusters are representative of the development set of the target dataset and the corresponding center for each cluster is representative of all the samples on that cluster. In the next step, the distance of each center is used to calculate the cosine similarity. Finally, the samples in the source dataset which are far from these centers are ignored. In other words, the outliers do not take part in transfer learning.

Answers:
In the MULT method, two datasets are simultaneously trained, and the weights are tuned based on the inputs which come from both datasets. The hyper-parameter $\lambda \in (0,1)$ is calculated based on a brute-force search or using general global search. This hyper parameter is used to calculate the final cost function which is computed from the combination of the cost function of the source dataset and the target datasets. , this paper proposes to use the most relevant samples from the source dataset to train on the target dataset. One way to find the most similar samples is finding the pair-wise distance between all samples of the development set of the target dataset and source dataset., we propose using a clustering algorithm on the development set. The clustering algorithm used ihere is a hierarchical clustering algorithm. The cosine similarity is used as a criteria to cluster each question and answer. Therefore, these clusters are representative of the development set of the target dataset and the corresponding center for each cluster is representative of all the samples on that cluster. In the next step, the distance of each center is used to calculate the cosine similarity. Finally, the samples in the source dataset which are far from these centers are ignored. In other words, the outliers do not take part in transfer learning.
Question: Will these findings be robust through different datasets and different question answering algorithms?

Evidence:
raw_evidence: We evaluate a multi-task approach and three algorithms that explicitly model the task dependencies. We perform experiments on document-level variants of the SQuAD dataset BIBREF1 . The contributions for our papers are:

Answers:
Yes
Question: What is the underlying question answering algorithm?

Evidence:
raw_evidence: We build on the state-of-the-art publicly available question answering system by docqa. The system extends BiDAF BIBREF4 with self-attention and performs well on document-level QA. We reuse all hyperparameters from docqa with the exception of number of paragraphs sampled in training: 8 instead of 4. Using more negative examples was important when learning from both fine and coarse annotations. The model uses character embeddings with dimension 50, pre-trained Glove embeddings, and hidden units for bi-directional GRU encoders with size 100. Adadelta is used for optimization for all methods. We tune two hyperparameters separately for each condition based on the held-out set: (1) $\alpha \in \lbrace .01, .1, .5, 1, 5, 10, 100 \rbrace $ , the weight of the coarse loss, and (2) the number of steps for early stopping. The training time for all methods using both coarse and fine supervision is comparable. We use Adadelta for optimization for all methods.

Answers:
The system extends BiDAF BIBREF4 with self-attention
Question: What datasets have this method been evaluated on?

Evidence:
raw_evidence: We evaluate a multi-task approach and three algorithms that explicitly model the task dependencies. We perform experiments on document-level variants of the SQuAD dataset BIBREF1 . The contributions for our papers are:

Answers:
document-level variants of the SQuAD dataset
Question: What DCGs are used?

Evidence:
raw_evidence: convertible.pl: implementing DCG rules for 1st and 3rd steps in the three-steps conversion, as well as other rules including lexicon.

Answers:
Author's own DCG rules are defined from scratch.
Question: What else is tried to be solved other than 12 tenses, model verbs and negative form?

Evidence:
raw_evidence: Moreover, this work also handles the cases of singular/plural, subject pronoun/object pronoun, etc. For instance, the pronoun “he" is used for the subject as “he" but is used for the object as “him".

Answers:
cases of singular/plural, subject pronoun/object pronoun, etc.
Question: Is there information about performance of these conversion methods?

Evidence:

Answers:
No
Question: Are there some experiments performed in the paper?

Evidence:

Answers:
No
Question: How much is performance improved by disabling attention in certain heads?

Evidence:
raw_evidence: Our experiments suggest that certain heads have a detrimental effect on the overall performance of BERT, and this trend holds for all the chosen tasks. Unexpectedly, disabling some heads leads not to a drop in accuracy, as one would expect, but to an increase in performance. This is effect is different across tasks and datasets. While disabling some heads improves the results, disabling the others hurts the results. However, it is important to note that across all tasks and datasets, disabling some heads leads to an increase in performance. The gain from disabling a single head is different for different tasks, ranging from the minimum absolute gain of 0.1% for STS-B, to the maximum of 1.2% for MRPC (see fig:disableheadsall). In fact, for some tasks, such as MRPC and RTE, disabling a random head gives, on average, an increase in performance. Furthermore, disabling a whole layer, that is, all 12 heads in a given layer, also improves the results. fig:disablelayers shows the resulting model performance on the target GLUE tasks when different layers are disabled. Notably, disabling the first layer in the RTE task gives a significant boost, resulting in an absolute performance gain of 3.2%. However, effects of this operation vary across tasks, and for QNLI and MNLI, it produces a performance drop of up to -0.2%.

Answers:
disabling the first layer in the RTE task gives a significant boost, resulting in an absolute performance gain of 3.2%,  this operation vary across tasks
Question: In which certain heads was attention disabled in experiments?

Evidence:
raw_evidence: Our experiments suggest that certain heads have a detrimental effect on the overall performance of BERT, and this trend holds for all the chosen tasks. Unexpectedly, disabling some heads leads not to a drop in accuracy, as one would expect, but to an increase in performance. This is effect is different across tasks and datasets. While disabling some heads improves the results, disabling the others hurts the results. However, it is important to note that across all tasks and datasets, disabling some heads leads to an increase in performance. The gain from disabling a single head is different for different tasks, ranging from the minimum absolute gain of 0.1% for STS-B, to the maximum of 1.2% for MRPC (see fig:disableheadsall). In fact, for some tasks, such as MRPC and RTE, disabling a random head gives, on average, an increase in performance. Furthermore, disabling a whole layer, that is, all 12 heads in a given layer, also improves the results. fig:disablelayers shows the resulting model performance on the target GLUE tasks when different layers are disabled. Notably, disabling the first layer in the RTE task gives a significant boost, resulting in an absolute performance gain of 3.2%. However, effects of this operation vary across tasks, and for QNLI and MNLI, it produces a performance drop of up to -0.2%.

Answers:
single head, disabling a whole layer, that is, all 12 heads in a given layer
Question: What handcrafter features-of-interest are used?

Evidence:
raw_evidence: We tested this hypothesis by checking whether there are vertical stripe patterns corresponding to certain linguistically interpretable features, and to what extent such features are relevant for solving a given task. In particular, we investigated attention to nouns, verbs, pronouns, subjects, objects, and negation words, and special BERT tokens across the tasks.

Answers:
nouns, verbs, pronouns, subjects, objects, negation words, special BERT tokens
Question: What subset of GLUE tasks is used?

Evidence:
raw_evidence: We use the following subset of GLUE tasks BIBREF4 for fine-tuning:
MRPC: the Microsoft Research Paraphrase Corpus BIBREF13
STS-B: the Semantic Textual Similarity Benchmark BIBREF14
SST-2: the Stanford Sentiment Treebank, two-way classification BIBREF15
QQP: the Quora Question Pairs dataset
RTE: the Recognizing Textual Entailment datasets
QNLI: Question-answering NLI based on the Stanford Question Answering Dataset BIBREF3
MNLI: the Multi-Genre Natural Language Inference Corpus, matched section BIBREF16

Answers:
MRPC, STS-B, SST-2, QQP, RTE, QNLI, MNLI
Question: Which review dataset do they use?

Evidence:
raw_evidence: We evaluate our proposed model on the SNAP (Stanford Network Analysis Project) Amazon review datasets BIBREF8, which contain not only reviews and ratings, but also golden summaries. In scenarios where there is no user-written summary for a review, we use pointer-generator network BIBREF9 to generate abstractive summaries. Empirical results show that our model significantly outperforms all strong baselines, including joint modeling, separate encoder and joint encoder methods. In addition, our model achieves new state-of-the-art performance, attaining 2.1% (with generated summary) and 4.8% (with golden summary) absolutely improvements compared to the previous best method on SNAP Amazon review benchmark.
We empirically compare different methods using Amazon SNAP Review Dataset BIBREF20, which is a part of Stanford Network Analysis Project. The raw dataset consists of around 34 millions Amazon reviews in different domains, such as books, games, sports and movies. Each review mainly contains a product ID, a piece of user information, a plain text review, a review summary and an overall sentiment rating which ranges from 1 to 5. The statistics of our adopted dataset is shown in Table TABREF20. For fair comparison with previous work, we adopt the same partitions used by previous work BIBREF6, BIBREF7, which is, for each domain, the first 1000 samples are taken as the development set, the following 1000 samples as the test set, and the rest as the training set.

Answers:
SNAP (Stanford Network Analysis Project)
Question: What evaluation metrics did they look at?

Evidence:
raw_evidence: Los resultados de la evaluación se presentan en la Tabla TABREF42, en la forma de promedios normalizados entre [0,1] y de su desviación estándar $\sigma $.

Answers:
accuracy with standard deviation
Question: What datasets are used?

Evidence:
raw_evidence: Corpus utilizados ::: Corpus 5KL
Este corpus fue constituido con aproximadamente 5 000 documentos (en su mayor parte libros) en español. Los documentos originales, en formatos heterogéneos, fueron procesados para crear un único documento codificado en utf8. Las frases fueron segmentadas automáticamente, usando un programa en PERL 5.0 y expresiones regulares, para obtener una frase por línea.
Las características del corpus 5KL se encuentran en la Tabla TABREF4. Este corpus es empleado para el entrenamiento de los modelos de aprendizaje profundo (Deep Learning, Sección SECREF4).
El corpus literario 5KL posee la ventaja de ser muy extenso y adecuado para el aprendizaje automático. Tiene sin embargo, la desventaja de que no todas las frases son necesariamente “frases literarias”. Muchas de ellas son frases de lengua general: estas frases a menudo otorgan una fluidez a la lectura y proporcionan los enlaces necesarios a las ideas expresadas en las frases literarias.
Otra desventaja de este corpus es el ruido que contiene. El proceso de segmentación puede producir errores en la detección de fronteras de frases. También los números de página, capítulos, secciones o índices producen errores. No se realizó ningún proceso manual de verificación, por lo que a veces se introducen informaciones indeseables: copyrights, datos de la edición u otros. Estas son, sin embargo, las condiciones que presenta un corpus literario real.
Corpus utilizados ::: Corpus 8KF
Un corpus heterogéneo de casi 8 000 frases literarias fue constituido manualmente a partir de poemas, discursos, citas, cuentos y otras obras. Se evitaron cuidadosamente las frases de lengua general, y también aquellas demasiado cortas ($N \le 3$ palabras) o demasiado largas ($N \ge 30$ palabras). El vocabulario empleado es complejo y estético, además que el uso de ciertas figuras literarias como la rima, la anáfora, la metáfora y otras pueden ser observadas en estas frases.
Las características del corpus 8KF se muestran en la Tabla TABREF6. Este corpus fue utilizado principalmente en los dos modelos generativos: modelo basado en cadenas de Markov (Sección SECREF13) y modelo basado en la generación de Texto enlatado (Canned Text, Sección SECREF15).

Answers:
Corpus 5KL, Corpus 8KF
Question: What are the datasets used for the task?

Evidence:
raw_evidence: FLOAT SELECTED: TABLE I DATASETS AND LANGUAGES USED FOR EVALUATION. AVERAGE PHONE AND SYLLABLE COUNTS ARE PER WORD.
To produce a language-agnostic syllabifier, it is crucial to test syllabification accuracy across different language families and language groupings within families. We selected six evaluation languages: English, Dutch, Italian, French, Basque, and Manipuri. These represent two language families (Indo-European, Sino-Tibetan), a language isolate thought to be unrelated to any existing language (Basque), and two different subfamilies within the Indo-European family (West Germanic, Romance). The primary constraint was the availability of syllabified datasets for training and testing. Table TABREF17 presents details of each dataset.
Among the six languages we evaluate with, both English and Dutch are notable for the availability of rich datasets of phonetic and syllabic transcriptions. These are found in the CELEX (Dutch Centre for Lexical Information) database BIBREF16. CELEX was built jointly by the University of Nijmegen, the Institute for Dutch Lexicology in Leiden, the Max Planck Institute for Psycholinguistics in Nijmegen, and the Institute for Perception Research in Eindhoven. CELEX is maintained by the Max Planck Institute for Psycholinguistics. The CELEX database contains information on orthography, phonology, morphology, syntax and word frequency. It also contains syllabified words in Dutch and English transcribed using SAM-PA, CELEX, CPA, and DISC notations. The first three are variations of the International Phonetic Alphabet (IPA), in that each uses a standard ASCII character to represent each IPA character. DISC is different than the other three in that it maps a distinct ASCII character to each phone in the sound systems of Dutch, English, and German BIBREF38. Different phonetic transcriptions are used in different datasets. Part of the strength of our proposed syllabifier is that every transcription can be used as-is without any additional modification to the syllabifier or the input sequences. The other datasets were hand-syllabified by linguists with the exception of the IIT-Guwahat dataset and the Festival dataset. Both IIT-Guwahat and Festival were initially syllabified with a naive algorithm and then each entry was confirmed or corrected by hand.

Answers:
Datasets used are Celex (English, Dutch), Festival (Italian), OpenLexuque (French), IIT-Guwahati (Manipuri), E-Hitz (Basque)
Question: What is the accuracy of the model for the six languages tested?

Evidence:
raw_evidence: We tested three model versions against all datasets. The model we call Base is the BiLSTM-CNN-CRF model described in Section SECREF2 with the associated hyperparameters. Another model, Small, uses the same architecture as Base but reduces the number of convolutional layers to 1, the convolutional filters to 40, the LSTM dimension $l$ to 50, and the phone embedding size $d$ to 100. We also tested a Base-Softmax model, which replaces the CRF output of the Base model with a softmax. A comparison of the results of these three models can be seen in Table TABREF25. This comparison empirically motivates the CRF output because Base almost always outperforms Base-Softmax. Of these three models, the Base model performed the best with the exception of the French and Manipuri datasets. The differences in the French results can be considered negligible because the accuracies are all near $100\%$. The Small model performed best on Manipuri, which may suggest that reducing the number of parameters of the Base model leads to better accuracy on smaller datasets.
FLOAT SELECTED: TABLE III THE ACCURACY OF OUR PROPOSED MODEL ON EACH EVALUATION DATASET. MODEL ACCURACY (%± σ) IS REPORTED ON A WORD LEVEL WHICH MEANS THE ENTIRE WORD MUST BE SYLLABIFIED CORRECTLY.

Answers:
Authors report their best models have following accuracy: English CELEX (98.5%), Dutch CELEX (99.47%), Festival (99.990%), OpenLexique (100%), IIT-Guwahat (95.4%), E-Hitz (99.83%)
Question: Which models achieve state-of-the-art performances?

Evidence:
raw_evidence: FLOAT SELECTED: TABLE II REPORTED ACCURACIES OF STATE OF THE ART AND SELECTED HIGH PERFORMING SYLLABIFIERS ON EACH EVALUATION DATASET.
For each dataset used to evaluate the proposed model, we compare our results with published accuracies of existing syllabification systems. Table TABREF21 shows the performance of well known and state of the art syllabifiers for each dataset. Liang's hyphenation algorithm is commonly known for its usage in . The patgen program was used to learn the rules of syllable boundaries BIBREF39. What we call Entropy CRF is a method particular to Manipuri; a rule-based component estimates the entropy of phones and phone clusters while a data-driven CRF component treats syllabification as a sequence modeling task BIBREF35.

Answers:
CELEX (Dutch and English) - SVM-HMM
Festival, E-Hitz and OpenLexique - Liang hyphenation
IIT-Guwahat - Entropy CRF
Question: Is the LSTM bidirectional?

Evidence:
raw_evidence: We use the LSTM network as follows. The $x$ vector is fed through the LSTM network which outputs a vector $\overrightarrow{h_i}$ for each time step $i$ from 0 to $n-1$. This is the forward LSTM. As we have access to the complete vector $x$, we can process a backward LSTM as well. This is done by computing a vector $\overleftarrow{h_i}$ for each time step $i$ from $n-1$ to 0. Finally, we concatenate the backward LSTM with the forward LSTM:
Both $\overrightarrow{h_i}$ and $\overleftarrow{h_i}$ have a dimension of $l$, which is an optimized hyperparameter. The BiLSTM output $h$ thus has dimension $2l\times n$.

Answers:
Yes
Question: What are the three languages studied in the paper?

Evidence:
raw_evidence: In this paper, we explore the benefit of explicitly modeling variations in the surface forms of words using methods from deep latent variable modeling in order to improve the translation accuracy in low-resource and morphologically-rich languages. Latent variable models allow us to inject inductive biases relevant to the task, which, in our case, is word formation, and we believe that follows a certain hierarchical procedure. Our model translates words one character at a time based on word representations learned compositionally from sub-lexical components, which are parameterized by a hierarchical latent variable model mimicking the process of morphological inflection, consisting of a continuous-space dense vector capturing the lexical semantics, and a set of (approximately) discrete features, representing the morphosyntactic role of the word in a given sentence. Each word representation during decoding is reformulated based on the shared latent morphological features, aiding in learning more reliable representations of words under sparse settings by generalizing across their different surface forms. We evaluate our method in translating English into three morphologically-rich languages each with a distinct morphological typology: Arabic, Czech and Turkish, and show that our model is able to obtain better translation accuracy and generalization capacity than conventional approaches to open-vocabulary NMT.

Answers:
Arabic, Czech and Turkish
Question: Do they use pretrained models as part of their parser?

Evidence:
raw_evidence: As a classifier, we use a feed-forward neural network with two hidden layers of 200 tanh units and learning rate set to 0.1, with linear decaying. The input to the network consists of the concatenation of embeddings for words, POS tags and Stanford parser dependencies, one-hot vectors for named entities and additional sparse features, extracted from the current configuration of the transition system; this is reported in more details in Table TABREF27 . The embeddings for words and POS tags were pre-trained on a large unannotated corpus consisting of the first 1 billion characters from Wikipedia. For lexical information, we also extract the leftmost (in the order of the aligned words) child (c), leftmost parent (p) and leftmost grandchild (cc). Leftmost and rightmost items are common features for transition-based parsers BIBREF17 , BIBREF18 but we found only leftmost to be helpful in our case. All POS tags, dependencies and named entities are generated using Stanford CoreNLP BIBREF19 . The accuracy of this classifier on the development set is 84%.

Answers:
Yes