qinchuanhui/UDA-QA huggingface dataset:
Question: Do they use pretrained word representations in their neural network models?

Evidence:
raw_evidence: Grammatical error correction (GEC) is a challenging task due to the variability of the type of errors and the syntactic and semantic dependencies of the errors on the surrounding context. Most of the grammatical error correction systems use classification and rule-based approaches for correcting specific error types. However, these systems use several linguistic cues as features. The standard linguistic analysis tools like part-of-speech (POS) taggers and parsers are often trained on well-formed text and perform poorly on ungrammatical text. This introduces further errors and limits the performance of rule-based and classification approaches to GEC. As a consequence, the phrase-based statistical machine translation (SMT) approach to GEC has gained popularity because of its ability to learn text transformations from erroneous text to correct text from error-corrected parallel corpora without any additional linguistic information. They are also not limited to specific error types. Currently, many state-of-the-art GEC systems are based on SMT or use SMT components for error correction BIBREF0 , BIBREF1 , BIBREF2 . In this paper, grammatical error correction includes correcting errors of all types, including word choice errors and collocation errors which constitute a large class of learners' errors.
We model our GEC system based on the phrase-based SMT approach. However, traditional phrase-based SMT systems treat words and phrases as discrete entities. We take advantage of continuous space representation by adding two neural network components that have been shown to improve SMT systems BIBREF3 , BIBREF4 . These neural networks are able to capture non-linear relationships between source and target sentences and can encode contextual information more effectively. Our experiments show that the addition of these two neural networks leads to significant improvements over a strong baseline and outperforms the current state of the art.
To train NNJM, we use the publicly available implementation, Neural Probabilistic Language Model (NPLM) BIBREF14 . The latest version of Moses can incorporate NNJM trained using NPLM as a feature while decoding. Similar to NNGLM, we use the parallel text used for training the translation model in order to train NNJM. We use a source context window size of 5 and a target context window size of 4. We select a source context vocabulary of 16,000 most frequent words from the source side. The target context vocabulary and output vocabulary is set to the 32,000 most frequent words. We use a single hidden layer to speed up training and decoding with an input embedding dimension of 192 and 512 hidden layer nodes. We use rectified linear units (ReLU) as the activation function. We train NNJM with noise contrastive estimation with 100 noise samples per training instance, which are obtained from a unigram distribution. The neural network is trained for 30 epochs using stochastic gradient descent optimization with a mini-batch size of 128 and learning rate of 0.1.

Answers:
No
Question: what was their baseline comparison?

Evidence:
raw_evidence: To compare our neural models with the traditional approaches, we experimented with a number of existing models including: Support Vector Machine (SVM), a discriminative max-margin model; Logistic Regression (LR), a discriminative probabilistic model; and Random Forest (RF), an ensemble model of decision trees. We use the implementation from the scikit-learn toolkit BIBREF19 . All algorithms use the default value of their parameters.

Answers:
Support Vector Machine (SVM), Logistic Regression (LR), Random Forest (RF)
Question: Was any variation in results observed based on language typology?

Evidence:
raw_evidence: On another note, we apply our formalization to evaluate multilingual $\textsc {bert} $'s syntax knowledge on a set of six typologically diverse languages. Although it does encode a large amount of information about syntax (more than $81\%$ in all languages), it only encodes at most $5\%$ more information than some trivial baseline knowledge (a type-level representation). This indicates that the task of POS labeling (word-level POS tagging) is not an ideal task for contemplating the syntactic understanding of contextual word embeddings.
We know $\textsc {bert} $ can generate text in many languages, here we assess how much does it actually know about syntax in those languages. And how much more does it know than simple type-level baselines. tab:results-full presents this results, showing how much information $\textsc {bert} $, fastText and onehot embeddings encode about POS tagging. We see that—in all analysed languages—type level embeddings can already capture most of the uncertainty in POS tagging. We also see that BERT only shares a small amount of extra information with the task, having small (or even negative) gains in all languages.
Finally, when put into perspective, multilingual $\textsc {bert} $'s representations do not seem to encode much more information about syntax than a trivial baseline. $\textsc {bert} $ only improves upon fastText in three of the six analysed languages—and even in those, it encodes at most (in English) $5\%$ additional information.

Answers:
It is observed some variability - but not significant. Bert does not seem to gain much more syntax information than with type level information.
Question: Does the work explicitly study the relationship between model complexity and linguistic structure encoding?

Evidence:

Answers:
No
Question: Which datasets are used in this work?

Evidence:
raw_evidence: There have been several workshops on biomedical natural language processing. We focus on the BioNLP Shared Tasks in recent years that had competitions on event extraction. There have been three BioNLP Shared Task competitions so far: 2009, 2011, and 2013. The BioNLP 2009 Shared Task BIBREF195 was based on the GENIA corpus BIBREF196 which contains PubMed abstracts of articles on transcription factors in human blood cells. There was a second BioNLP Shared Task competition organized in 2011 to measure the advances in approaches and associated results BIBREF197 . The third BioNLP ST was held in 2013. We discuss some notable systems from BioNLP ST 2011 and 2013.

Answers:
GENIA corpus
Question: Does the training dataset provide logical form supervision?

Evidence:
raw_evidence: The dataset MSParS is published by NLPCC 2019 evaluation task. The whole dataset consists of 81,826 samples annotated by native English speakers. 80% of them are used as training set. 10% of them are used as validation set while the rest is used as test set. 3000 hard samples are selected from the test set. Metric for this dataset is the exactly matching accuracy on both full test set and hard test subset. Each sample is composed of the question, the logical form, the parameters(entity/value/type) and question type as the Table TABREF3 demonstrates.

Answers:
Yes
Question: What is the difference between the full test set and the hard test set?

Evidence:
raw_evidence: The dataset MSParS is published by NLPCC 2019 evaluation task. The whole dataset consists of 81,826 samples annotated by native English speakers. 80% of them are used as training set. 10% of them are used as validation set while the rest is used as test set. 3000 hard samples are selected from the test set. Metric for this dataset is the exactly matching accuracy on both full test set and hard test subset. Each sample is composed of the question, the logical form, the parameters(entity/value/type) and question type as the Table TABREF3 demonstrates.

Answers:
3000 hard samples are selected from the test set
Question: How is the discriminative training formulation different from the standard ones?

Evidence:
raw_evidence: For the overlapped speech recognition problem, the conditional independence assumption in the output label streams is still made as in Equation ( 5 ). Then the cross-entropy based PIT can be transformed to sequence discriminative criterion based PIT as below,
$$\begin{split} \mathcal {J}_{\text{SEQ-PIT}}=\sum _u \min _{s^{\prime }\in \mathbf {S}} \frac{1}{N} \sum _{n\in [1,N]}-\mathcal {J}_{\text{SEQ}}(\mathbf {L}_{un}^{(s^{\prime })},\mathbf {L}_{un}^{(r)}) \end{split}$$ (Eq. 44)
Different from Equation ( 7 ), the best permutation is decided by $\mathcal {J}_{\text{SEQ}}(\mathbf {L}_{un}^{(s^{\prime })},\mathbf {L}_{un}^{(r)})$ , which is the sequence discriminative criterion of taking the $s^{\prime }$ -th permutation in $n$ -th output inference stream at utterance $u$ . Similar to CE-PIT, $\mathcal {J}_{\text{SEQ}}$ of all the permutations are calculated and the minimum permutation is taken to do the optimization.

Answers:
the best permutation is decided by $\mathcal {J}_{\text{SEQ}}(\mathbf {L}_{un}^{(s^{\prime })},\mathbf {L}_{un}^{(r)})$ , which is the sequence discriminative criterion of taking the $s^{\prime }$ -th permutation in $n$ -th output inference stream at utterance $u$
Question: How are the two datasets artificially overlapped?

Evidence:
raw_evidence: Two-talker overlapped speech is artificially generated by mixing these waveform segments. To maximize the speech overlap, we developed a procedure to mix similarly sized segments at around 0dB. First, we sort the speech segments by length. Then, we take segments in pairs, zero-padding the shorter segment so both have the same length. These pairs are then mixed together to create the overlapped speech data. The overlapping procedure is similar to BIBREF13 except that we make no modification to the signal levels before mixing . After overlapping, there's 150 hours data in the training, called 150 hours dataset, and 915 utterances in the test set. After decoding, there are 1830 utterances for evaluation, and the shortest utterance in the hub5e-swb dataset is discarded. Additionally, we define a small training set, the 50 hours dataset, as a random 50 hour subset of the 150 hours dataset. Results are reported using both datasets.

Answers:
we sort the speech segments by length, we take segments in pairs, zero-padding the shorter segment so both have the same length, These pairs are then mixed together
Question: What type of lexical, syntactic, semantic and polarity features are used?

Evidence:
raw_evidence: Our lexical features include 1-, 2-, and 3-grams in both word and character levels. For each type of INLINEFORM0 -grams, we utilize only the top 1,000 INLINEFORM1 -grams based on the term frequency-inverse document frequency (tf-idf) values. That is, each INLINEFORM2 -gram appearing in a tweet becomes an entry in the feature vector with the corresponding feature value tf-idf. We also use the number of characters and the number of words as features.
We use the NLTK toolkit to tokenize and annotate part-of-speech tags (POS tags) for all tweets in the dataset. We then use all the POS tags with their corresponding tf-idf values as our syntactic features and feature values, respectively.
Firstly, we employ 300-dimensional pre-trained word embeddings from GloVe BIBREF29 to compute a tweet embedding as the average of the embeddings of words in the tweet.
Secondly, we apply the latent semantic indexing BIBREF30 to capture the underlying semantics of the dataset. Here, each tweet is represented as a vector of 100 dimensions.
Thirdly, we also extract tweet representation by applying the Brown clustering algorithm BIBREF31 , BIBREF32 —a hierarchical clustering algorithm which groups the words with similar meaning and syntactical function together. Applying the Brown clustering algorithm, we obtain a set of clusters, where each word belongs to only one cluster. For example in Table TABREF13 , words that indicate the members of a family (e.g., “mum”, “dad”) or positive sentiment (e.g., “interesting”, “awesome”) are grouped into the same cluster. We run the algorithm with different number of clustering settings (i.e., 80, 100, 120) to capture multiple semantic and syntactic aspects. For each clustering setting, we use the number of tweet words in each cluster as a feature. After that, for each tweet, we concatenate the features from all the clustering settings to form a cluster-based tweet embedding.
Motivated by the verbal irony by means of polarity contrast, such as “I really love this year's summer; weeks and weeks of awful weather”, we use the number of polarity signals appearing in a tweet as the polarity features. The signals include positive words (e.g., love), negative words (e.g., awful), positive emoji icon and negative emoji icon. We use the sentiment dictionaries provided by BIBREF33 to identify positive and negative words in a tweet. We further use boolean features that check whether or not a negation word is in a tweet (e.g., not, n't).

Answers:
Our lexical features include 1-, 2-, and 3-grams in both word and character levels., number of characters and the number of words, POS tags, 300-dimensional pre-trained word embeddings from GloVe, latent semantic indexing, tweet representation by applying the Brown clustering algorithm, positive words (e.g., love), negative words (e.g., awful), positive emoji icon and negative emoji icon, boolean features that check whether or not a negation word is in a tweet
Question: How does nextsum work?

Evidence:
raw_evidence: This work proposes an extractive summarization system that focuses on capturing rich summary-internal structure. Our key idea is that since summaries in a domain often follow some predictable structure, a partial summary or set of summary sentences should help predict other summary sentences. We formalize this intuition in a model called NextSum, which selects the next summary sentence based not only on properties of the source text, but also on the previously selected sentences in the summary. An example choice is shown in Table 1 . This setup allows our model to capture summary-specific discourse and topic transitions. For example, it can learn to expand on a topic that is already mentioned in the summary, or to introduce a new topic. It can learn to follow a script or discourse relations that are expected for that domain's summaries. It can even learn to predict the end of the summary, avoiding the need to explicitly define a length cutoff.

Answers:
selects the next summary sentence based not only on properties of the source text, but also on the previously selected sentences in the summary
Question: How many tweets were manually labelled? 

Evidence:
raw_evidence: Tweets related to Forex, specifically to EUR and USD, were acquired through the Twitter search API with the following query: “EURUSD”, “USDEUR”, “EUR”, or “USD”. In the period of three years (January 2014 to December 2016) almost 15 million tweets were collected. A subset of them (44,000 tweets) was manually labeled by knowledgeable students of finance. The label captures the leaning or stance of the Twitter user with respect to the anticipated move of one currency w.r.t. the other. The stance is represented by three values: buy (EUR vs. USD), hold, or sell. The tweets were collected, labeled and provided to us by the Sowa Labs company (http://www.sowalabs.com).

Answers:
44,000 tweets
Question: What dataset they use for evaluation?

Evidence:
raw_evidence: We validate our approach on the Gigaword corpus, which comprises of a training set of 3.8M article headlines (considered to be the full text) and titles (summaries), along with 200K validation pairs, and we report test performance on the same 2K set used in BIBREF7. Since we want to learn systems from fully unaligned data without giving the model an opportunity to learn an implicit mapping, we also further split the training set into 2M examples for which we only use titles, and 1.8M for headlines. All models after the initialization step are implemented as convolutional seq2seq architectures using Fairseq BIBREF20. Artificial data generation uses top-15 sampling, with a minimum length of 16 for full text and a maximum length of 12 for summaries. rouge scores are obtained with an output vocabulary of size 15K and a beam search of size 5 to match BIBREF11.

Answers:
The same 2K set from Gigaword used in BIBREF7
Question: What is the source of the tables?

Evidence:
raw_evidence: The Online Retail Data Set consists of a clean list of 25873 invoices, totaling 541909 rows and 8 columns. InvoiceNo, CustomerID and StockCode are mostly 5 or 6-digit integers with occasional letters. Quantity is mostly 1 to 3-digit integers, a part of them being negative, and UnitPrice is composed of 1 to 6 digits floating values. InvoiceDate are dates all in the same format, Country contains strings representing 38 countries and Description is 4224 strings representing names of products. We reconstruct text mails from this data, by separating each token with a blank space and stacking the lines for a given invoice, grouped by InvoiceNo. We will use the column label as ground truth for the tokens in the dataset. For simplicity reasons we add underscores between words in Country and Description to ease the tokenization. Another slight modification has to be done: $25\%$ of the CustomerId values are missing, and we replace them by '00000'. A sample can be found in Fig. 4 .

Answers:
The Online Retail Data Set consists of a clean list of 25873 invoices, totaling 541909 rows and 8 columns.
Question: Which regions of the United States do they consider?

Evidence:
raw_evidence: FLOAT SELECTED: Fig 5. (A) The average verb regularization fraction by county for the lower 48 states, along with (B) residuals and (C) Gi� z-score. A higher Gi� z-score means a county has a greater regularization fraction than expected. Counties colored black did not have enough data. We used the dataset in row (IV) of Table 1.

Answers:
all regions except those that are colored black
Question: What state-of-the-art general-purpose pretrained models are made available under the unified API? 

Evidence:
raw_evidence: Here is a list of architectures for which reference implementations and pretrained weights are currently provided in Transformers. These models fall into two main categories: generative models (GPT, GPT-2, Transformer-XL, XLNet, XLM) and models for language understanding (Bert, DistilBert, RoBERTa, XLM).
BERT (BIBREF13) is a bi-directional Transformer-based encoder pretrained with a linear combination of masked language modeling and next sentence prediction objectives.
RoBERTa (BIBREF5) is a replication study of BERT which showed that carefully tuning hyper-parameters and training data size lead to significantly improved results on language understanding.
DistilBERT (BIBREF32) is a smaller, faster, cheaper and lighter version BERT pretrained with knowledge distillation.
GPT (BIBREF34) and GPT2 (BIBREF9) are two large auto-regressive language models pretrained with language modeling. GPT2 showcased zero-shot task transfer capabilities on various tasks such as machine translation or reading comprehension.
Transformer-XL (BIBREF35) introduces architectural modifications enabling Transformers to learn dependency beyond a fixed length without disrupting temporal coherence via segment-level recurrence and relative positional encoding schemes.
XLNet (BIBREF4) builds upon Transformer-XL and proposes an auto-regressive pretraining scheme combining BERT's bi-directional context flow with auto-regressive language modeling by maximizing the expected likelihood over permutations of the word sequence.
XLM (BIBREF8) shows the effectiveness of pretrained representations for cross-lingual language modeling (both on monolingual data and parallel data) and cross-lingual language understanding.
We systematically release the model with the corresponding pretraining heads (language modeling, next sentence prediction for BERT) for adaptation using the pretraining objectives. Some models fine-tuned on downstream tasks such as SQuAD1.1 are also available. Overall, more than 30 pretrained weights are provided through the library including more than 10 models pretrained in languages other than English. Some of these non-English pretrained models are multi-lingual models (with two of them being trained on more than 100 languages) .

Answers:
BERT, RoBERTa, DistilBERT, GPT, GPT2, Transformer-XL, XLNet, XLM
Question: How is performance measured?

Evidence:
raw_evidence: To assess the predictive capability of this and other models, we require some method by which we can compare the models. For that purpose, we use receiver operating characteristic (ROC) curves as a visual representation of predictive effectiveness. ROC curves compare the true positive rate (TPR) and false positive rate (FPR) of a model's predictions at different threshold levels. The area under the curve (AUC) (between 0 and 1) is a numerical measure, where the higher the AUC is, the better the model performs.
We cross-validate our model by first randomly splitting the corpus into a training set (95% of the corpus) and test set (5% of the corpus). We then fit the model to the training set, and use it to predict the response of the documents in the test set. We repeat this process 100 times. The threshold-averaged ROC curve BIBREF13 is found from these predictions, and shown in Figure 3 . Table 1 shows the AUC for each model considered.

Answers:
they use ROC curves and cross-validation
Question: What models are included in the toolkit?

Evidence:
raw_evidence: Modules: Modules are the basic building blocks of different models. In LeafNATS, we provide ready-to-use modules for constructing recurrent neural network (RNN)-based sequence-to-sequence (Seq2Seq) models for NATS, e.g., pointer-generator network BIBREF1 . These modules include embedder, RNN encoder, attention BIBREF24 , temporal attention BIBREF6 , attention on decoder BIBREF2 and others. We also use these basic modules to assemble a pointer-generator decoder module and the corresponding beam search algorithms. The embedder can also be used to realize the embedding-weights sharing mechanism BIBREF2 .

Answers:
 recurrent neural network (RNN)-based sequence-to-sequence (Seq2Seq) models for NATS
Question: Is there any human evaluation involved in evaluating this famework?

Evidence:
raw_evidence: Due to the lack of available models for the task, we compare our framework with a previous model developed for image-to-image translation as baseline, which colorizes images without text descriptions. We carried out two human evaluations using Mechanical Turk to compare the performance of our model and the baseline. For each experiment, we randomly sampled 1,000 images from the test set and then turned these images into black and white. For each image, we generated a pair of two images using our model and the baseline, respectively. Our model took into account the caption in generation while the baseline did not. Then we randomly permuted the 2,000 generated images. In the first experiment, we presented to human annotators the 2,000 images, together with their original captions, and asked humans to rate the consistency between the generated images and the captions in a scale of 0 and 1, with 0 indicating no consistency and 1 indicating consistency. In the second experiment, we presented to human annotators the same 2,000 images without captions, but asked human annotators to rate the quality of each image without providing its original caption. The quality was rated in a scale of 0 and 1, with 0 indicating low quality and 1 indicating high quality.

Answers:
Yes
Question: How big is dataset used for fine-tuning BERT?

Evidence:
raw_evidence: To fine-tune BERT, we used a proprietary corpus that consists of hundreds of thousands of legal agreements. We extracted text from the agreements, tokenized it into sentences, and removed sentences without alphanumeric text. We selected the BERT-Base uncased pre-trained model for fine-tuning. To avoid including repetitive content found at the beginning of each agreement we selected the 31st to 50th sentence of each agreement. We ran unsupervised fine-tuning of BERT using sequence lengths of 128, 256 and 512. The loss function over epochs is shown in Figure FIGREF3.

Answers:
hundreds of thousands of legal agreements