qinchuanhui/UDA-QA huggingface dataset:
Question: What is dialogue act recognition?

Evidence:
raw_evidence: Dialogue act (DA) characterizes the type of a speaker's intention in the course of producing an utterance and is approximately equivalent to the illocutionary act of BIBREF0 or the speech act of BIBREF1. The recognition of DA is essential for modeling and automatically detecting discourse structure, especially in developing a human-machine dialogue system. It is natural to predict the Answer acts following an utterance of type Question, and then match the Question utterance to each QA-pair in the knowledge base. The predicted DA can also guide the response generation process BIBREF2. For instance, system generates a Greeting type response to former Greeting type utterance. Moreover, DA is beneficial to other online dialogue strategies, such as conflict avoidance BIBREF3. In the offline system, DA also plays a significant role in summarizing and analyzing the collected utterances. For instance, recognizing DAs of a wholly online service record between customer and agent is beneficial to mine QA-pairs, which are selected and clustered then to expand the knowledge base. DA recognition is challenging due to the same utterance may have a different meaning in a different context. Table TABREF1 shows an example of some utterances together with their DAs from Switchboard dataset. In this example, utterance “Okay.” corresponds to two different DA labels within different semantic context.
DA recognition is aimed to assign a label to each utterance in a conversation. It can be formulated as a supervised classification problem. There are two trends to solve this problem: 1) as a sequence labeling problem, it will predict the labels for all utterances in the whole dialogue history BIBREF13, BIBREF14, BIBREF9; 2) as a sentence classification problem, it will treat utterance independently without any context history BIBREF5, BIBREF15. Early studies rely heavily on handcrafted features such as lexical, syntactic, contextual, prosodic and speaker information and achieve good results BIBREF13, BIBREF4, BIBREF16.

Answers:
DA recognition is aimed to assign a label to each utterance in a conversation. It can be formulated as a supervised classification problem.
Question: Does the performance necessarily drop when more control is desired?

Evidence:
raw_evidence: Performance INLINEFORM0 Trade-off: To see if the selector affects performance, we also ask human annotators to judge the text fluency. The fluency score is computed as the average number of text being judged as fluent. We include generations from the standard Enc-Dec model. Table TABREF32 shows the best fluency is achieved for Enc-Dec. Imposing a content selector always affects the fluency a bit. The main reason is that when the controllability is strong, the change of selection will directly affect the text realization so that a tiny error of content selection might lead to unrealistic text. If the selector is not perfectly trained, the fluency will inevitably be influenced. When the controllability is weaker, like in RS, the fluency is more stable because it will not be affected much by the selection mask. For SS and Bo.Up, the drop of fluency is significant because of the gap of soft approximation and the independent training procedure. In general, VRS does properly decouple content selection from the enc-dec architecture, with only tiny degrade on the fluency.

Answers:
Yes
Question: What is the baseline model used?

Evidence:
raw_evidence: DrQA is a CRC baseline coming with the CoQA dataset. Note that this implementation of DrQA is different from DrQA for SQuAD BIBREF8 in that it is modified to support answering no answer questions by having a special token unknown at the end of the document. So having a span with unknown indicates NO ANSWER. This baseline answers the research question RQ1.
DrQA+CoQA is the above baseline pre-tuned on CoQA dataset and then fine-tuned on $(\text{RC})_2$ . We use this baseline to show that even DrQA pre-trained on CoQA is sub-optimal for RCRC. This baseline is used to answer RQ1 and RQ3.
BERT is the vanilla BERT model directly fine-tuned on $(\text{RC})_2$ . We use this baseline for ablation study on the effectiveness of pre-tuning. All these BERT's variants are used to answer RQ2.
BERT+review first tunes BERT on domain reviews using the same objectives as BERT pre-training and then fine-tunes on $(\text{RC})_2$ . We use this baseline to show that a simple domain-adaptation of BERT is not good.
BERT+CoQA first fine-tunes BERT on the supervised CoQA data and then fine-tunes on $(\text{RC})_2$ . We use this baseline to show that pre-tuning is very competitive even compared with models trained from large-scale supervised data. This also answers RQ3.

Answers:
The baseline models used are DrQA modified to support answering no answer questions, DrQA+CoQA which is pre-tuned on CoQA dataset, vanilla BERT, BERT+review tuned on domain reviews, BERT+CoQA tuned on the supervised CoQA data
Question: Is this auto translation tool based on neural networks?

Evidence:
raw_evidence: CodeInternational: A tool which can translate code between human languages, powered by Google Translate.

Answers:
Yes
Question: What are results of public code repository study?

Evidence:
raw_evidence: How do non-English speakers program in a language like Java, where the keywords and core libraries are written in English? We employ a data driven approach to tell the story of non-English code and inform the decisions we made in our auto-translator. We analyzed Java repositories on GitHub, the largest host of source code in the world, where 1.1 million unique users host 2.9 million public Java projects. We downloaded and analyzed the human language used for writing comments (in Java code), naming identifiers (method and variable names), and writing git commit messages. We focused on Java code as it is both one of the most popular source-code languages on GitHub and in the classroom. A selection of results from this study are that:
Non-English code is a large-scale phenomena.
Transliteration is common in identifiers for all languages.
Languages clusters into three distinct groups based on how speakers use identifiers/comments/transliteration.
Non-latin script users write comments in their L1 script but write identifiers in English.
Right-to-left (RTL) language scripts, such as Arabic, have no observed prevalence on GitHub identifiers, implying that existing coders who speak RTL languages have substantial barriers in using their native script in code.

Answers:
Non-English code is a large-scale phenomena., Transliteration is common in identifiers for all languages., Languages clusters into three distinct groups based on how speakers use identifiers/comments/transliteration., Non-latin script users write comments in their L1 script but write identifiers in English., Right-to-left (RTL) language scripts, such as Arabic, have no observed prevalence on GitHub identifiers
Question: Where is the dataset from?

Evidence:
raw_evidence: Our data collection setup uses a dialogue simulator to generate dialogue outlines first and then paraphrase them to obtain natural utterances. Using a dialogue simulator offers us multiple advantages. First, it ensures the coverage of a large variety of dialogue flows by filtering out similar flows in the simulation phase, thus creating a much diverse dataset. Second, simulated dialogues do not require manual annotation, as opposed to a Wizard-of-Oz setup BIBREF17, which is a common approach utilized in other datasets BIBREF0. It has been shown that such datasets suffer from substantial annotation errors BIBREF18. Thirdly, using a simulator greatly simplifies the data collection task and instructions as only paraphrasing is needed to achieve a natural dialogue. This is particularly important for creating a large dataset spanning multiple domains.

Answers:
dialogue simulator
Question: What data augmentation techniques are used?

Evidence:
raw_evidence: Team 9 BIBREF24: This team submitted the winning entry, beating the second-placed team by around 9% in terms of joint goal accuracy. They use two separate models for categorical and non-categorical slots, and treat numerical categorical slots as non-categorical. They also use the entire dialogue history as input. They perform data augmentation by back translation between English and Chinese, which seems to be one of the distinguishing factors resulting in a much higher accuracy.

Answers:
back translation between English and Chinese
Question: How are the models evaluated?

Evidence:
raw_evidence: We consider the following metrics for automatic evaluation of different submissions. Joint goal accuracy has been used as the primary metric to rank the submissions.
Active Intent Accuracy: The fraction of user turns for which the active intent has been correctly predicted.
Requested Slot F1: The macro-averaged F1 score for requested slots over all eligible turns. Turns with no requested slots in ground truth and predictions are skipped.
Average Goal Accuracy: For each turn, we predict a single value for each slot present in the dialogue state. This is the average accuracy of predicting the value of a slot correctly.
Joint Goal Accuracy: This is the average accuracy of predicting all slot assignments for a given service in a turn correctly.

Answers:
Active Intent Accuracy, Requested Slot F1, Average Goal Accuracy, Joint Goal Accuracy
Question: What eight language are reported on?

Evidence:
raw_evidence: In this context, we propose a robust multilingual sentiment analysis method, tested in eight different languages: Spanish, English, Italian, Arabic, German, Portuguese, Russian and Swedish. We compare our approach ranking in three international contests: TASS'15, SemEval'15-16 and SENTIPOLC'14, for Spanish, English and Italian respectively; the remaining languages are compared directly with the results reported in the literature. The experimental results locate our approach in good positions for all considered competitions; and excellent results in the other five languages tested. Finally, even when our method is almost cross-language, it can be extended to take advantage of language dependencies; we also provide experimental evidence of the advantages of using these language-dependent techniques.

Answers:
Spanish, English, Italian, Arabic, German, Portuguese, Russian and Swedish
Question: What are the components of the multilingual framework?

Evidence:
raw_evidence: In a nutshell, B4MSA starts by applying text-transformations to the messages, then transformed text is represented in a vector space model (see Subsection SECREF13 ), and finally, a Support Vector Machine (with linear kernel) is used as the classifier. B4MSA uses a number of text transformations that are categorized in cross-language features (see Subsection SECREF3 ) and language dependent features (see Subsection SECREF9 ). It is important to note that, all the text-transformations considered are either simple to implement or there is a well-known library (e.g. BIBREF6 , BIBREF7 ) to use them. It is important to note that to maintain the cross-language property, we limit ourselves to not use additional knowledge, this include knowledge from affective lexicons or models based on distributional semantics.

Answers:
text-transformations to the messages, vector space model, Support Vector Machine
Question: Is the proposed method compared to previous methods?

Evidence:
raw_evidence: Following the algorithms above, with the consideration of both the advantages and disadvantages of them, in this project, I am going to use a modified method: sound-class based skip-grams with bipartite networks (BipSkip). The whole procedure is quite straightforward and could be divided into three steps. First step: the word pair and their skip-grams are two sets of the bipartite networks. The second step is optional, which is to refine the bipartite network. Before I run the program, I will be asked to input a threshold, which determines if the program should delete the skip-gram nodes linked to fewer word nodes than the threshold itself. According to the experiment, even though I did not input any threshold as one of the parameters, the algorithm could still give the same answer but with more executing time. In the last step, the final generated bipartite graph would be connected to a monopartite graph and partitioned into cognate sets with the help of graph partitioning algorithms. Here I would use Informap algorithm BIBREF12. To make a comparison to this method, I am using CCM and SCA for distance measurement in this experiment, too. UPGMA algorithm would be used accordingly in these two cases.

Answers:
Yes
Question: Which is the baseline model?

Evidence:
raw_evidence: As the first step, we build three baseline LID systems, one based on the i-vector model, and the other two based on LSTM-RNN, using the speech data of two languages from Babel: Assamese and Georgian (AG).
The two RNN LID baselines are: a standard RNN LID system (AG-RNN-LID) that discriminates between the two languages in its output, and a multi-task system (AG-RNN-MLT) that was trained to discriminate between the two languages as well as the phones. More precisely, the output units of the AG-RNN-MLT are separated into two groups: an LID group that involves two units corresponding to Assamese and Georgian respectively, and an ASR group that involves $3,349$ bilingual senones that are inherited from an HMM/GMM ASR system trained with the speech data of Assamese and Georgian, following the standard WSJ s5 HMM/GMM recipe of Kaldi. The WSJ s5 nnet3 recipe of Kaldi is then used to train the AG-RNN-LID and AG-RNN-MLT systems.

Answers:
The three baseline models are the i-vector model, a standard RNN LID system and a multi-task RNN LID system.
Question: What is the main contribution of the paper? 

Evidence:
raw_evidence: All the present neural LID methods are based on acoustic features, e.g., Mel filter banks (Fbanks) or Mel frequency cepstral coefficients (MFCCs), with phonetic information largely overlooked. This may have significantly hindered the performance of neural LID. Intuitively, it is a long-standing hypothesis that languages can be discriminated between by phonetic properties, either distributional or temporal; additionally, phonetic features represent information at a higher level than acoustic features, and so are more invariant with respect to noise and channels. Pragmatically, it has been demonstrated that phonetic information, either in the form of phone sequences, phone posteriors, or phonetic bottleneck features, can significantly improve LID accuracy in both the conventional PRLM approach BIBREF11 and the more modern i-vector system BIBREF34 , BIBREF35 , BIBREF36 . In this paper, we will investigate the utilization of phonetic information to improve neural LID. The basic concept is to use a phone-discriminative model to produce frame-level phonetic features, and then use these features to enhance RNN LID systems that were originally built with raw acoustic features. The initial step is therefore feature combination, with the phonetic feature used as auxiliary information to assist acoustic RNN LID. This is improved further, as additional research identified that a simpler model using only the phonetic feature as the RNN LID input provides even better performance. We call this RNN model based on phonetic features the phonetic temporal neural LID approach, or PTN LID. As well as having a simplified model structure, the PTN offers deeper insight into the LID task by rediscovering the value of the phonetic temporal property in language discrimination. This property was historically widely and successfully applied in token-based approaches, e.g., PRLM BIBREF11 , but has been largely overlooked due to the popularity of the i-vector approach.

Answers:
Proposing an improved RNN model, the phonetic temporal neural LID approach, based on phonetic features that results in better performance
Question: How do they get the formal languages?

Evidence:
raw_evidence: BIBREF7 investigated the learning capabilities of simple RNNs to process and formalize a context-free grammar containing hierarchical (recursively embedded) dependencies: He observed that distinct parts of the networks were able to learn some complex representations to encode certain grammatical structures and dependencies of the context-free grammar. Later, BIBREF8 introduced an RNN with an external stack memory to learn simple context-free languages, such as $a^n b^m$ , $a^nb^ncb^ma^m$ , and $a^{n+m} b^n c^m$ . Similar studies BIBREF15 , BIBREF16 , BIBREF17 , BIBREF10 , BIBREF11 have explored the existence of stable counting mechanisms in simple RNNs, which would enable them to learn various context-free and context-sensitive languages, but none of the RNN architectures proposed in the early days were able to generalize the training set to longer (or more complex) test samples with substantially high accuracy.
BIBREF9 , on the other hand, proposed a variant of Long Short-Term Memory (LSTM) networks to learn two context-free languages, $a^n b^n$ , $a^n b^m B^m A^n$ , and one strictly context-sensitive language, $a^n b^n c^n$ . Given only a small fraction of samples in a formal language, with values of $n$ (and $m$ ) ranging from 1 to a certain training threshold $N$ , they trained an LSTM model until its full convergence on the training set and then tested it on a more generalized set. They showed that their LSTM model outperformed the previous approaches in capturing and generalizing the aforementioned formal languages. By analyzing the cell states and the activations of the gates in their LSTM model, they further demonstrated that the network learns how to count up and down at certain places in the sample sequences to encode information about the underlying structure of each of these formal languages.

Answers:
These are well-known formal languages some of which was used in the literature to evaluate the learning capabilities of RNNs.
Question: Are the unobserved samples from the same distribution as the training data?

Evidence:
raw_evidence: In the present work, we address these limitations by providing a more nuanced evaluation of the learning capabilities of RNNs. In particular, we investigate the effects of three different aspects of a network's generalization: data distribution, length-window, and network capacity. We define an informative protocol for assessing the performance of RNNs: Instead of training a single network until it has learned its training set and then evaluating it on its test set, as BIBREF9 do in their study, we monitor and test the network's performance at each epoch during the entire course of training. This approach allows us to study the stability of the solutions reached by the network. Furthermore, we do not restrict ourselves to a test set of sequences of fixed lengths during testing. Rather, we exhaustively enumerate all the sequences in a language by their lengths and then go through the sequences in the test set one by one until our network errs $k$ times, thereby providing a more fine-grained evaluation criterion of its generalization capabilities.
Previous studies have examined various length distribution models to generate appropriate training sets for each formal language: BIBREF16 , BIBREF11 , BIBREF12 , for instance, used length distributions that were skewed towards having more short sequences than long sequences given a training length-window, whereas BIBREF9 used a uniform distribution scheme to generate their training sets. The latter briefly comment that the distribution of lengths of sequences in the training set does influence the generalization ability and convergence speed of neural networks, and mention that training sets containing abundant numbers of both short and long sequences are learned by networks much more quickly than uniformly distributed regimes. Nevertheless, they do not systematically compare or explicitly report their findings. To study the effect of various length distributions on the learning capability and speed of LSTM models, we experimented with four discrete probability distributions supported on bounded intervals (Figure 2 ) to sample the lengths of sequences for the languages. We briefly recall the probability distribution functions for discrete uniform and Beta-Binomial distributions used in our data generation procedure.

Answers:
No
Question: By how much does their model outperform the baseline in the cross-domain evaluation?

Evidence:
raw_evidence: To answer RQ3, which is concerned with domain adaptation, we can see in the grayed out cells in tab:results, which correspond to the cross-domain adaption case where the BERT language model is trained on the target domain, that domain adaptation works well with $2.2\%$ absolute accuracy improvement on the laptops test set and even $3.6\%$ accuracy improvement on the restaurants test set compared to BERT-base.

Answers:
$2.2\%$ absolute accuracy improvement on the laptops test set, $3.6\%$ accuracy improvement on the restaurants test set
Question: What are the performance results?

Evidence:
raw_evidence: To answer RQ2, which is concerned with in-domain ATSC performance, we see in tab:results that for the in-domain training case, our models BERT-ADA Lapt and BERT-ADA Rest achieve performance close to state-of-the-art on the laptops dataset and new state-of-the-art on the restaurants dataset with accuracies of $79.19\%$ and $87.14\%$, respectively. On the restaurants dataset, this corresponds to an absolute improvement of $2.2\%$ compared to the previous state-of-the-art method BERT-PT. Language model finetuning produces a larger improvement on the restaurants dataset. We think that one reason for that might be that the restaurants domain is underrepresented in the pre-training corpora of BERTBASE. Generally, we find that language model finetuning helps even if the finetuning domain does not match the evaluation domain. We think the reason for this might be that the BERT-base model is pre-trained more on knowledge-based corpora like Wikipedia than on text containing opinions. Another finding is that BERT-ADA Joint performs better on the laptops dataset than BERT-ADA Rest, although the unique amount of laptop reviews are the same in laptops- and joint-corpora. We think that confusion can be created when mixing the domains, but this needs to be investigated further. We also find that the XLNet-base baseline performs generally stronger than BERT-base and even outperforms BERT-ADA Lapt with an accuracy of $79.89\%$ on the laptops dataset.
In general, the ATSC task generalizes well cross-domain, with about 2-$3\%$ drop in accuracy compared to in-domain training. We think the reason for this might be that syntactical relationships between the aspect-target and the phrase expressing sentiment polarity as well as knowing the sentiment-polarity itself are sufficient to solve the ATSC task in many cases.

Answers:
results that for the in-domain training case, our models BERT-ADA Lapt and BERT-ADA Rest achieve performance close to state-of-the-art on the laptops dataset, new state-of-the-art on the restaurants dataset with accuracies of $79.19\%$ and $87.14\%$, respectively.
Question: What is a confusion network or lattice?

Evidence:
raw_evidence: FLOAT SELECTED: Fig. 2: Standard ASR outputs
A number of important downstream and upstream applications rely on accurate confidence scores in graph-like structures, such as confusion networks (CN) in Fig. 2 and lattices in Fig. 2 , where arcs connected by nodes represent hypothesised words. This section describes an extension of BiRNNs to CNs and lattices.

Answers:
graph-like structures where arcs connect nodes representing multiple hypothesized words, thus allowing multiple incoming arcs unlike 1-best sequences