qinchuanhui/UDA-QA huggingface dataset:
Question: What formalism does Grail use?

Evidence:
raw_evidence: This chapter describes the underlying formalism of the theorem provers, as it is visible during an interactive proof trace, and present the general strategy followed by the theorem provers. The presentation in this chapter is somewhat informal, referring the reader elsewhere for full proofs.
The rest of this chapter is structured as follows. Section "Type-logical grammars" presents a general introduction to type-logical grammars and illustrates its basic concepts using the Lambek calculus, ending the section with some problems at the syntax-semantics interface for the Lambek calculus. Section "Modern type-logical grammars" looks at recent developments in type-logical grammars and how they solve some of the problems at the syntax-semantics interface. Section "Theorem proving" looks at two general frameworks for automated theorem proving for type-logical grammars, describing the internal representation of partial proofs and giving a high-level overview of the proof search mechanism.
Type-logical grammars are a family of grammar formalisms built on a foundation of logic and type theory. Type-logical grammars originated when BIBREF4 introduced his Syntactic Calculus (called the Lambek calculus, L, by later authors). Though Lambek built on the work of BIBREF5 , BIBREF6 and others, Lambek's main innovation was to cast the calculus as a logic, giving a sequent calculus and showing decidability by means of cut elimination. This combination of linguistic and computational applications has proved very influential.

Answers:
a family of grammar formalisms built on a foundation of logic and type theory. Type-logical grammars originated when BIBREF4 introduced his Syntactic Calculus (called the Lambek calculus, L, by later authors).
Question: Which components of QA and QG models are shared during training?

Evidence:
raw_evidence: We proposed a neural machine comprehension model that can jointly ask and answer questions given a document. We hypothesized that question answering can benefit from synergistic interaction between the two tasks through parameter sharing and joint training under this multitask setting. Our proposed model adopts an attention-based sequence-to-sequence architecture that learns to dynamically switch between copying words from the document and generating words from a vocabulary. Experiments with the model confirm our hypothesis: the joint model outperforms its QA-only counterpart by a significant margin on the SQuAD dataset.

Answers:
parameter sharing
Question: How much improvement does jointly learning QA and QG give, compared to only training QA?

Evidence:
raw_evidence: Evaluation results are provided in Table 1 . We see that A-gen performance improves significantly with the joint model: both F1 and EM increase by about 10 percentage points. Performance of q-gen worsens after joint training, but the decrease is relatively small. Furthermore, as pointed out by earlier studies, automatic metrics often do not correlate well with the generation quality assessed by humans BIBREF9 . We thus consider the overall outcome to be positive.

Answers:
We see that A-gen performance improves significantly with the joint model: both F1 and EM increase by about 10 percentage points.
Question: Do they test their word embeddings on downstream tasks?

Evidence:
raw_evidence: Outlier Detection. The Outlier Detection task BIBREF0 is to determine which word in a list INLINEFORM0 of INLINEFORM1 words is unrelated to the other INLINEFORM2 which were chosen to be related. For each INLINEFORM3 , one can compute its compactness score INLINEFORM4 , which is the compactness of INLINEFORM5 . INLINEFORM6 is explicitly computed as the mean similarity of all word pairs INLINEFORM7 . The predicted outlier is INLINEFORM8 , as the INLINEFORM9 related words should form a compact cluster with high mean similarity.
Sentiment analysis. We also consider sentiment analysis as described by BIBREF31 . We use the suggested Large Movie Review dataset BIBREF32 , containing 50,000 movie reviews.

Answers:
Yes
Question: What dimensions of word embeddings do they produce using factorization?

Evidence:
raw_evidence: As is common in the literature BIBREF4 , BIBREF8 , we use 300-dimensional vectors for our embeddings and all word vectors are normalized to unit length prior to evaluation.

Answers:
300-dimensional vectors
Question: On which dataset(s) do they compute their word embeddings?

Evidence:
raw_evidence: For a fair comparison, we trained each model on the same corpus of 10 million sentences gathered from Wikipedia. We removed stopwords and words appearing fewer than 2,000 times (130 million tokens total) to reduce noise and uninformative words. Our word2vec and NNSE baselines were trained using the recommended hyperparameters from their original publications, and all optimizers were using using the default settings. Hyperparameters are always consistent across evaluations.

Answers:
10 million sentences gathered from Wikipedia
Question: Do they measure computation time of their factorizations compared to other word embeddings?

Evidence:
raw_evidence: When considering going from two dimensions to three, it is perhaps necessary to discuss the computational issues in such a problem size increase. However, it should be noted that the creation of pre-trained embeddings can be seen as a pre-processing step for many future NLP tasks, so if the training can be completed once, it can be used forever thereafter without having to take training time into account. Despite this, we found that the training of our embeddings was not considerably slower than the training of order-2 equivalents such as SGNS. Explicitly, our GPU trained CBOW vectors (using the experimental settings found below) in 3568 seconds, whereas training CP-S and JCP-S took 6786 and 8686 seconds respectively.

Answers:
Yes
Question: What datasets are experimented with?

Evidence:
raw_evidence: We conducted our experiments on the CMU ARCTIC database BIBREF33, which contains parallel recordings of professional US English speakers sampled at 16 kHz. One female (slt) was chosen as the target speaker and one male (bdl) and one female (clb) were chosen as sources. We selected 100 utterances each for validation and evaluation, and the other 932 utterances were used as training data. For the TTS corpus, we chose a US female English speaker (judy bieber) from the M-AILABS speech dataset BIBREF34 to train a single-speaker Transformer-TTS model. With the sampling rate also at 16 kHz, the training set contained 15,200 utterances, which were roughly 32 hours long.

Answers:
the CMU ARCTIC database BIBREF33,  the M-AILABS speech dataset BIBREF34
Question: What is the baseline model?

Evidence:
raw_evidence: Next, we compared our VTN model with an RNN-based seq2seq VC model called ATTS2S BIBREF8. This model is based on the Tacotron model BIBREF32 with the help of context preservation loss and guided attention loss to stabilize training and maintain linguistic consistency after conversion. We followed the configurations in BIBREF8 but used mel spectrograms instead of WORLD features.

Answers:
a RNN-based seq2seq VC model called ATTS2S based on the Tacotron model
Question: What model do they train?

Evidence:
raw_evidence: We carried out the experimentation with a range of classifiers of different types: Support Vector Machines (SVM), Gaussian Naive Bayes, Multinomial Naive Bayes, Decision Trees, Random Forests and a Maximum Entropy classifier. They were tested in two different settings, one without balancing the weights of the different classes and the other by weighing the classes as the inverse of their frequency in the training set; the latter was tested as a means for dealing with the highly imbalanced data. The selection of these classifiers is in line with those used in the literature, especially with those tested by Han et al. BIBREF41 . This experimentation led to the selection of the weighed Maximum Entropy (MaxEnt) classifier as the most accurate. In the interest of space and focus, we only present results for this classifier.

Answers:
Support Vector Machines (SVM), Gaussian Naive Bayes, Multinomial Naive Bayes, Decision Trees, Random Forests and a Maximum Entropy classifier
Question: How do slot binary classifiers improve performance?

Evidence:
raw_evidence: This paper proposes the Flexibly-Structured Dialogue Model (FSDM) as a new end-to-end task-oriented dialogue system. The state tracking component of FSDM has the advantages of both fully structured and free-form approaches while addressing their shortcomings. On one hand, it is still structured, as it incorporates information about slots in KB schema; on the other hand, it is flexible, as it does not use information about the values contained in the KB records. This makes it easily adaptable to new values. These desirable properties are achieved by a separate decoder for each informable slot and a multi-label classifier for the requestable slots. Those components explicitly assign values to slots like the fully structured approach, while also preserving the capability of dealing with out-of-vocabulary words like the free-form approach. By using these two types of decoders, FSDM produces only valid belief states, overcoming the limitations of the free-form approach. Further, FSDM has a new module called response slot binary classifier that adds extra supervision to generate the slots that will be present in the response more precisely before generating the final textual agent response (see Section "Methodology" for details).

Answers:
by adding extra supervision to generate the slots that will be present in the response
Question: What baselines have been used in this work?

Evidence:
raw_evidence: We compare FSDM with four baseline methods and two ablations.
NDM BIBREF7 proposes a modular end-to-end trainable network. It applies de-lexicalization on user utterances and responses.
LIDM BIBREF9 improves over NDM by employing a discrete latent variable to learn underlying dialogue acts. This allows the system to be refined by reinforcement learning.
KVRN BIBREF13 adopts a copy-augmented Seq2Seq model for agent response generation and uses an attention mechanism on the KB. It does not perform belief state tracking.
TSCP/RL BIBREF10 is a two-stage CopyNet which consists of one encoder and two copy-mechanism-augmented decoders for belief state and response generation. TSCP includes further parameter tuning with reinforcement learning to increase the appearance of response slots in the generated response. We were unable to replicate the reported results using the provided code, hyperparameters, and random seed, so we report both the results from the paper and the average of 5 runs on the code with different random seeds (marked with $^\dagger $ ).

Answers:
NDM, LIDM, KVRN, and TSCP/RL
Question: How successful are they at matching names of authors in Japanese and English?

Evidence:
raw_evidence: Fortunately, 180221 of 231162 author names could be matched successfully. There are many reasons for the remaining uncovered cases. 9073 Latin names could not be found in the name dictionary ENAMDICT and 14827 name matchings between the names' Latin and kanji representations did not succeed. These names might be missing at all in the dictionary, delivered in a very unusual format that the tool does not cover, or might not be Japanese or human names at all. Of course, Japanese computer scientists sometimes also cooperate with foreign colleagues but our tool expects Japanese names and is optimized for them. Both IPSJ DL and ENAMDICT provide katakana representations for some Western names. However, katakana representations for Western names are irrelevant for projects like DBLP. But for instance, Chinese names in Chinese characters are relevant. Understandably, our tool does not support any special Personal Name Matching for Chinese names yet because our work is focused on Japanese names. The tool does not take account of the unclassified names of ENAMDICT by default. We can increase the general success rate of the Name Matching process by enabling the inclusion of unclassified names in the configuration file but the quality of the Name Matching process will decrease because the correct differentiation between given and family name cannot be guaranteed anymore. An unclassified name may substitute a given or a family name.

Answers:
180221 of 231162 author names could be matched successfully
Question: Do they translate metadata from Japanese papers to English?

Evidence:

Answers:
No
Question: what bottlenecks were identified?

Evidence:
raw_evidence: In this paper we proposed a methodology to identify words that could lead to confusion at any given node of a speech recognition based system. We used edit distance as the metric to identifying the possible confusion between the active words. We showed that this metric can be used effectively to enhance the performance of a speech solution without actually putting it to people test. There is a significant saving in terms of being able to identify recognition bottlenecks in a menu based speech solution through this analysis because it does not require actual people testing the system. This methodology was adopted to restructuring the set of active words at each node for better speech recognition in an actual menu based speech recognition system that caters to masses.
We hypothesize that we can identify the performance of a menu based speech system by identifying the possible confusion among all the words that are active at a given node. If active words at a given node are phonetically similar it becomes difficult for the speech recognition system to distinguish them which in turn leads to recognition errors. We used Levenshtein distance BIBREF4 , BIBREF5 a well known measure to analyze and identify the confusion among the active words at a given node. This analysis gives a list of all set of words that have a high degree of confusability among them; this understanding can be then used to (a) restructure the set of active words at that node and/or (b) train the words that can be confused by using a larger corpus of speech data. This allows the speech recognition engine to be equipped to be able to distinguish the confusing words better. Actual use of this analysis was carried out for a speech solution developed for Indian Railway Inquiry System to identify bottlenecks in the system before its actual launch.

Answers:
Confusion in recognizing the words that are active at a given node by a speech recognition solution developed for Indian Railway Inquiry System.
Question: Does the paper report the performance on the task of a Neural Machine Translation model?

Evidence:

Answers:
No
Question: Is the RNN model evaluated against any baseline?

Evidence:
raw_evidence: For this reason as a baseline algorithm for English dataset we refer to results from BIBREF0, and as for Russian dataset, we used the probabilistic language model, described in BIBREF8. The probability of a sequence of words is the product of the probabilities of each word, given the word’s context: the preceding word. As in the following equation:

Answers:
Yes
Question: Which languages are used in the paper?

Evidence:
raw_evidence: For this reason as a baseline algorithm for English dataset we refer to results from BIBREF0, and as for Russian dataset, we used the probabilistic language model, described in BIBREF8. The probability of a sequence of words is the product of the probabilities of each word, given the word’s context: the preceding word. As in the following equation:

Answers:
English, Russian
Question: What metrics are used for evaluation?

Evidence:
raw_evidence: We automatically evaluate on four different types of interpolations (where different combinations of sentences are removed and the model is forced to regenerate them), We evaluate the generations with the ROUGE BIBREF29 and METEOR BIBREF30 metrics using the true sentences as targets. Table TABREF33 shows the automatic evaluation results from interpolations using our proposed models and baselines. The #Sent(s) column indicates which sentence(s) were removed, and then regenerated by the model. We gave the baselines a slight edge over SLDS because they pick the best out of 1000 samples while SLDS is only out of 50. The SLDS models see their largest gain over the baseline models when at least the first sentence is given as an input. The baseline models do better when the first and second sentence need to be imputed. This is likely due to the fact that having access to the earlier sentences allows a better initialization for the Gibbs sampler. Surprisingly, the semi-supervised variants of the SLDS models achieve higher scores. The reasons for this is discussed below in the Perplexity section.

Answers:
ROUGE BIBREF29 and METEOR BIBREF30
Question: What baselines are used?

Evidence:
raw_evidence: Language Model (LM): We train a two layer recurrent neural language model with GRU cells of hidden size 512.
Sequence-to-Sequence Attention Model (S2S): We train a two layer neural sequence to sequence model equipped with bi-linear attention function with GRU cells of hidden size 512. Sentiments tags for a narrative (1 for each sentence) are given as input to the model and the corresponding sentences are concatenated together as the output with only one <eos> tag at the end. This model is trained with a 0.1 dropout. This model is comparable to the static model of BIBREF7, and other recent works employing a notion of scaffolding into neural generation (albeit adapted for our setting).
Linear Dynamical System (LDS): We also train a linear dynamical system as discussed in Section SECREF1 as one of our baselines for fair comparisons. Apart from having just a single transition matrix this model has the same architectural details as SLDS.
Semi-Supervised SLDS (SLDS-X%): To gauge the usability of semi-supervision, we also train semi-supervised SLDS models with varying amount of labelled sentiment tags unlike the original model which uses 100% tagged data. We refer to these as SLDS-X%, where X is the % labelled data used for training: 1%, 10%, 25%, and 50%.

Answers:
a two layer recurrent neural language model with GRU cells of hidden size 512, a two layer neural sequence to sequence model equipped with bi-linear attention function with GRU cells of hidden size 512, a linear dynamical system, semi-supervised SLDS models with varying amount of labelled sentiment tags
Question: How is the robustness of the model evaluated?

Evidence:
raw_evidence: We train our model for a maximal of 6 epochs. We select the best model parameters based on the best $F_1$ score on the development data after each epoch. Note that we split $10\%$ of data from the training data as the development data. The selected model is then applied to the test data for evaluation. During testing, we map words not appearing in the training data to the UNK token. Following the previous works, we perform 10-fold cross validation and report the average results. Our models and variants are implemented using PyTorch BIBREF26.

Answers:
10-fold cross validation
Question: How is the effectiveness of the model evaluated?

Evidence:
raw_evidence: Following the previous works, we report the precision ($P.$), recall ($R.$) and $F_1$ scores for target recognition and targeted sentiment. Note that a correct target prediction requires the boundary of the target to be correct, and a correct targeted sentiment prediction requires both target boundary and sentiment polarity to be correct.

Answers:
precision ($P.$), recall ($R.$) and $F_1$ scores for target recognition and targeted sentiment
Question: Do they assume sentence-level supervision?

Evidence:
raw_evidence: To better exploit such existing data sources, we propose an end-to-end (E2E) model based on pointer networks with attention, which can be trained end-to-end on the input/output pairs of human IE tasks, without requiring token-level annotations.
Since our model does not need token-level labels, we create an E2E version of each data set without token-level labels by chunking the BIO-labeled words and using the labels as fields to extract. If there are multiple outputs for a single field, e.g. multiple destination cities, we join them with a comma. For the ATIS data set, we choose the 10 most common labels, and we use all the labels for the movie and restaurant corpus. The movie data set has 12 fields and the restaurant has 8. See Table 2 for an example of the E2E ATIS data set.

Answers:
No
Question: By how much do they outperform BiLSTMs in Sentiment Analysis?

Evidence:
raw_evidence: On the 16 review datasets (Table TABREF22 ) from BIBREF32 , BIBREF31 , our proposed RCRN architecture achieves the highest score on all 16 datasets, outperforming the existing state-of-the-art model - sentence state LSTMs (SLSTM) BIBREF31 . The macro average performance gain over BiLSTMs ( INLINEFORM0 ) and Stacked (2 X BiLSTM) ( INLINEFORM1 ) is also notable. On the same architecture, our RCRN outperforms ablative baselines BiLSTM by INLINEFORM2 and 3L-BiLSTM by INLINEFORM3 on average across 16 datasets.

Answers:
Proposed RCRN outperforms ablative baselines BiLSTM by +2.9% and 3L-BiLSTM by +1.1% on average across 16 datasets.
Question: Does their model have more parameters than other models?

Evidence:
raw_evidence: Across all 26 datasets, RCRN outperforms not only standard BiLSTMs but also 3L-BiLSTMs which have approximately equal parameterization. 3L-BiLSTMs were overall better than BiLSTMs but lose out on a minority of datasets. RCRN outperforms a wide range of competitive baselines such as DiSAN, Bi-SRUs, BCN and LSTM-CNN, etc. We achieve (close to) state-of-the-art performance on SST, TREC question classification and 16 Amazon review datasets.

Answers:
approximately equal parameterization
Question: what models did they compare to?

Evidence:
raw_evidence: The test-set accuracies obtained by different learning methods, including the current state-of-the-art results, are presented in Table TABREF11 . The results indicate that the bag-of-words MVN outperforms most methods, but obtains lower accuracy than the state-of-the-art results achieved by the tree-LSTM BIBREF21 , BIBREF22 and the high-order CNN BIBREF16 . However, when augmented with 4 convolutional features as described in Section SECREF9 , the MVN strategy surpasses both of these, establishing a new state-of-the-art on this benchmark.

Answers:
High-order CNN, Tree-LSTM, DRNN, DCNN, CNN-MC, NBoW and SVM
Question: which benchmark tasks did they experiment on?

Evidence:
raw_evidence: Experiments on two benchmark data sets, the Stanford Sentiment Treebank BIBREF7 and the AG English news corpus BIBREF3 , show that 1) our method achieves very competitive accuracy, 2) some views distinguish themselves from others by better categorizing specific classes, and 3) when our base bag-of-words feature set is augmented with convolutional features, the method establishes a new state-of-the-art for both data sets.

Answers:
 They used Stanford Sentiment Treebank benchmark for sentiment classification task and   AG English news corpus for the text classification task.
Question: Are recurrent neural networks trained on perturbed data?

Evidence:
raw_evidence: For evaluation, we used the widely popular dialog act and intent prediction datasets. MRDA BIBREF12 is a dialog corpus of multi-party meetings with 6 classes, 78K training and 15K test data; ATIS BIBREF13 is intent prediction dataset for flight reservations with 21 classes, 4.4K training and 893 test examples; and SWDA BIBREF14, BIBREF15 is an open domain dialog corpus between two speakers with 42 classes, 193K training and 5K test examples. For fair comparison, we train LSTM baseline with sub-words and 240 vocabulary size on MRDA, ATIS and SWDA. We uniformly randomly initialized the input word embeddings. We also trained the on-device SGNN model BIBREF6. Then, we created test sets with varying levels of perturbation operations - $\lbrace 20\%,40\%,60\%\rbrace $.

Answers:
No