qinchuanhui/UDA-QA huggingface dataset:
Question: What is novel in author's approach?

Evidence:
raw_evidence: Our novelties include:
Using self-play learning for the neural response ranker (described in detail below).
Optimizing neural models for specific metrics (e.g. diversity, coherence) in our ensemble setup.
Training a separate dialog model for each user, personalizing our socialbot and making it more consistent.
Using a response classification predictor and a response classifier to predict and control aspects of responses such as sentiment, topic, offensiveness, diversity etc.
Using a model predictor to predict the best responding model, before the response candidates are generated, reducing computational expenses.
Using our entropy-based filtering technique to filter all dialog datasets, obtaining higher quality training data BIBREF3.
Building big, pre-trained, hierarchical BERT and GPT dialog models BIBREF6, BIBREF7, BIBREF8.
Constantly monitoring the user input through our automatic metrics, ensuring that the user stays engaged.

Answers:
They use self-play learning , optimize the model for specific metrics, train separate models per user, use model  and response classification predictors, and filter the dataset to obtain higher quality training data.
Question: What dataset is used for train/test of this method?

Evidence:
raw_evidence: Experimental Protocol ::: Datasets ::: Training Dataset
(i) TTS System dataset: We trained our TTS system with a mixture of neutral and newscaster style speech. For a total of 24 hours of training data, split in 20 hours of neutral (22000 utterances) and 4 hours of newscaster styled speech (3000 utterances).
(ii) Embedding selection dataset: As the evaluation was carried out only on the newscaster speaking style, we restrict our linguistic search space to the utterances associated to the newscaster style: 3000 sentences.
Experimental Protocol ::: Datasets ::: Evaluation Dataset
The systems were evaluated on two datasets:
(i) Common Prosody Errors (CPE): The dataset on which the baseline Prostron model fails to generate appropriate prosody. This dataset consists of complex utterances like compound nouns (22%), “or" questions (9%), “wh" questions (18%). This set is further enhanced by sourcing complex utterances (51%) from BIBREF24.
(ii) LFR: As demonstrated in BIBREF25, evaluating sentences in isolation does not suffice if we want to evaluate the quality of long-form speech. Thus, for evaluations on LFR we curated a dataset of news samples. The news style sentences were concatenated into full news stories, to capture the overall experience of our intended use case.

Answers:
Training datasets: TTS System dataset and embedding selection dataset. Evaluation datasets: Common Prosody Errors dataset and LFR dataset.
Question: How many domains of ontologies do they gather data from?

Evidence:
raw_evidence: The Software Ontology (SWO) BIBREF5 is included because its set of CQs is of substantial size and it was part of Ren et al.'s set of analysed CQs. The CQ sets of Dem@Care BIBREF8 and OntoDT BIBREF9 were included because they were available. CQs for the Stuff BIBREF6 and African Wildlife (AWO) BIBREF7 ontologies were added to the set, because the ontologies were developed by one of the authors (therewith facilitating in-depth domain analysis, if needed), they cover other topics, and are of a different `type' (a tutorial ontology (AWO) and a core ontology (Stuff)), thus contributing to maximising diversity in source selection.

Answers:
5 domains: software, stuff, african wildlife, healthcare, datatypes
Question: How is the semi-structured knowledge base created?

Evidence:
raw_evidence: Although techniques for constructing this knowledge base are outside the scope of this paper, we briefly mention them. Tables were constructed using a mixture of manual and semi-automatic techniques. First, the table schemas were manually defined based on the syllabus, study guides, and training questions. Tables were then populated both manually and semi-automatically using IKE BIBREF29 , a table-building tool that performs interactive, bootstrapped relation extraction over a corpus of science text. In addition, to augment these tables with the broad knowledge present in study guides that doesn't always fit the manually defined table schemas, we ran an Open IE BIBREF30 pattern-based subject-verb-object (SVO) extractor from BIBREF31 clark2014:akbc over several science texts to populate three-column Open IE tables. Methods for further automating table construction are under development.

Answers:
using a mixture of manual and semi-automatic techniques
Question: what is the practical application for this paper?

Evidence:
raw_evidence: Morphology deals with the internal structure of words BIBREF0 , BIBREF1 . Languages of the world have different word production processes. Morphological richness vary from language to language, depending on their linguistic typology. In natural language processing (NLP), taking into account the morphological complexity inherent to each language could be important for improving or adapting the existing methods, since the amount of semantic and grammatical information encoded at the word level, may vary significantly from language to language.
Additionally, most of the previous works do not analyze how the complexity changes when different types of morphological normalization procedures are applied to a language, e.g., lemmatization, stemming, morphological segmentation. This information could be useful for linguistic analysis and for measuring the impact of different word form normalization tools depending of the language. In this work, we analyze how the type-token relationship changes using different types of morphological normalization techniques.

Answers:
Improve existing NLP methods. Improve linguistic analysis. Measure impact of word normalization tools.
Question: Do they use a neural model for their task?

Evidence:
raw_evidence: Figure 1 presents architecture of the WSD system. As one may observe, no human labor is used to learn interpretable sense representations and the corresponding disambiguation models. Instead, these are induced from the input text corpus using the JoBimText approach BIBREF8 implemented using the Apache Spark framework, enabling seamless processing of large text collections. Induction of a WSD model consists of several steps. First, a graph of semantically related words, i.e. a distributional thesaurus, is extracted. Second, word senses are induced by clustering of an ego-network of related words BIBREF9 . Each discovered word sense is represented as a cluster of words. Next, the induced sense inventory is used as a pivot to generate sense representations by aggregation of the context clues of cluster words. To improve interpretability of the sense clusters they are labeled with hypernyms, which are in turn extracted from the input corpus using Hearst:92 patterns. Finally, the obtained WSD model is used to retrieve a list of sentences that characterize each sense. Sentences that mention a given word are disambiguated and then ranked by prediction confidence. Top sentences are used as sense usage examples. For more details about the model induction process refer to BIBREF10 . Currently, the following WSD models induced from a text corpus are available: Word senses based on cluster word features. This model uses the cluster words from the induced word sense inventory as sparse features that represent the sense.
Word senses based on context word features. This representation is based on a sum of word vectors of all cluster words in the induced sense inventory weighted by distributional similarity scores.
Super senses based on cluster word features. To build this model, induced word senses are first globally clustered using the Chinese Whispers graph clustering algorithm BIBREF9 . The edges in this sense graph are established by disambiguation of the related words BIBREF11 , BIBREF12 . The resulting clusters represent semantic classes grouping words sharing a common hypernym, e.g. “animal”. This set of semantic classes is used as an automatically learned inventory of super senses: There is only one global sense inventory shared among all words in contrast to the two previous traditional “per word” models. Each semantic class is labeled with hypernyms. This model uses words belonging to the semantic class as features.
Super senses based on context word features. This model relies on the same semantic classes as the previous one but, instead, sense representations are obtained by averaging vectors of words sharing the same class.

Answers:
No
Question: What's the method used here?

Evidence:
raw_evidence: Our model consists of two neural network modules, i.e. an extractor and abstractor. The extractor encodes a source document and chooses sentences from the document, and then the abstractor paraphrases the summary candidates. Formally, a single document consists of $n$ sentences $D=\lbrace s_1,s_2,\cdots ,s_n\rbrace $. We denote $i$-th sentence as $s_i=\lbrace w_{i1},w_{i2},\cdots ,w_{im}\rbrace $ where $w_{ij}$ is the $j$-th word in $s_i$. The extractor learns to pick out a subset of $D$ denoted as $\hat{D}=\lbrace \hat{s}_1,\hat{s}_2,\cdots ,\hat{s}_k|\hat{s}_i\in D\rbrace $ where $k$ sentences are selected. The abstractor rewrites each of the selected sentences to form a summary $S=\lbrace f(\hat{s}_1),f(\hat{s}_2),\cdots ,f(\hat{s}_k)\rbrace $, where $f$ is an abstracting function. And a gold summary consists of $l$ sentences $A=\lbrace a_1,a_2,\cdots ,a_l\rbrace $.
The extractor is based on the encoder-decoder framework. We adapt BERT for the encoder to exploit contextualized representations from pre-trained transformers. BERT as the encoder maps the input sequence $D$ to sentence representation vectors $H=\lbrace h_1,h_2,\cdots ,h_n\rbrace $, where $h_i$ is for the $i$-th sentence in the document. Then, the decoder utilizes $H$ to extract $\hat{D}$ from $D$.
We use LSTM Pointer Network BIBREF22 as the decoder to select the extracted sentences based on the above sentence representations. The decoder extracts sentences recurrently, producing a distribution over all of the remaining sentence representations excluding those already selected. Since we use the sequential model which selects one sentence at a time step, our decoder can consider the previously selected sentences. This property is needed to avoid selecting sentences that have overlapping information with the sentences extracted already.
The abstractor network approximates $f$, which compresses and paraphrases an extracted document sentence to a concise summary sentence. We use the standard attention based sequence-to-sequence (seq2seq) model BIBREF23, BIBREF24 with the copying mechanism BIBREF25 for handling out-of-vocabulary (OOV) words. Our abstractor is practically identical to the one proposed in BIBREF8.

Answers:
Two neural networks: an extractor based on an encoder (BERT) and a decoder (LSTM Pointer Network BIBREF22) and an abstractor identical to the one proposed in BIBREF8.
Question: By how much does their method outperform state-of-the-art OOD detection?

Evidence:
raw_evidence: The goal of this paper is to propose a novel OOD detection method that does not require OOD data by utilizing counterfeit OOD turns in the context of a dialog. Most prior approaches do not consider dialog context and make predictions for each utterance independently. We will show that this independent decision leads to suboptimal performance even when actual OOD utterances are given to optimize the model and that the use of dialog context helps reduce OOD detection errors. To consider dialog context, we need to connect the OOD detection task with the overall dialog task. Thus, for this work, we build upon Hybrid Code Networks (HCN) BIBREF4 since HCNs achieve state-of-the-art performance in a data-efficient way for task-oriented dialogs, and propose AE-HCNs which extend HCNs with an autoencoder (Figure FIGREF8 ). Furthermore, we release new dialog datasets which are three publicly available dialog corpora augmented with OOD turns in a controlled way (exemplified in Table TABREF2 ) to foster further research.
The result is shown in Table TABREF23 . Since there are multiple actions that are appropriate for a given dialog context, we use per-utterance Precision@K as performance metric. We also report f1-score for OOD detection to measure the balance between precision and recall. The performances of HCN on Test-OOD are about 15 points down on average from those on Test, showing the detrimental impact of OOD utterances to such models only trained on in-domain training data. AE-HCN(-CNN) outperforms HCN on Test-OOD by a large margin about 17(20) points on average while keeping the minimum performance trade-off compared to Test. Interestingly, AE-HCN-CNN has even better performance than HCN on Test, indicating that, with the CNN encoder, counterfeit OOD augmentation acts as an effective regularization. In contrast, AE-HCN-Indep failed to robustly detect OOD utterances, resulting in much lower numbers for both metrics on Test-OOD as well as hurting the performance on Test. This result indicates two crucial points: 1) the inherent difficulty of finding an appropriate threshold value without actually seeing OOD data; 2) the limitation of the models which do not consider context. For the first point, Figure FIGREF24 plots histograms of reconstruction scores for IND and OOD utterances of bAbI6 Test-OOD. If OOD utterances had been known a priori, the threshold should have been set to a much higher value than the maximum reconstruction score of IND training data (6.16 in this case).

Answers:
AE-HCN outperforms by 17%, AE-HCN-CNN outperforms by 20% on average
Question: What are dilated convolutions?

Evidence:
raw_evidence: In this work we focus on end-to-end stateless temporal modeling which can take advantage of a large context while limiting computation and avoiding saturation issues. By end-to-end model, we mean a straight-forward model with a binary target that does not require a precise phoneme alignment beforehand. We explore an architecture based on a stack of dilated convolution layers, effectively operating on a broader scale than with standard convolutions while limiting model size. We further improve our solution with gated activations and residual skip-connections, inspired by the WaveNet style architecture explored previously for text-to-speech applications BIBREF10 and voice activity detection BIBREF9 , but never applied to KWS to our knowledge. In BIBREF11 , the authors explore Deep Residual Networks (ResNets) for KWS. ResNets differ from WaveNet models in that they do not leverage skip-connections and gating, and apply convolution kernels in the frequency domain, drastically increasing the computational cost.
Standard convolutional networks cannot capture long temporal patterns with reasonably small models due to the increase in computational cost yielded by larger receptive fields. Dilated convolutions skip some input values so that the convolution kernel is applied over a larger area than its own. The network therefore operates on a larger scale, without the downside of increasing the number of parameters. The receptive field $r$ of a network made of stacked convolutions indeed reads: $r = \sum _i d_i (s_i - 1),$

Answers:
Similar to standard convolutional networks but instead they skip some input values effectively operating on a broader scale.
Question: what was the evaluation metrics studied in this work?

Evidence:
raw_evidence: Performance figures are computed using tail prediction on the test sets: For each test triple $(h,r,t)$ with open-world head $h \notin E$ , we rank all known entities $t^{\prime } \in E$ by their score $\phi (h,r,t^{\prime })$ . We then evaluate the ranks of the target entities $t$ with the commonly used mean rank (MR), mean reciprocal rank (MRR), as well as Hits@1, Hits@3, and Hits@10.

Answers:
mean rank (MR), mean reciprocal rank (MRR), as well as Hits@1, Hits@3, and Hits@10
Question: Do they analyze ELMo?

Evidence:

Answers:
No
Question: What are three possible phases for language formation?

Evidence:
raw_evidence: Three clear domains can be noticed in the behavior of $\langle cc \rangle $ versus $\wp $, at $t_f$, as shown in Fig. FIGREF15 (blue squares). Phase I: $\langle cc \rangle $ increases smoothly for $\wp < 0.4$, indicating that for this domain there is a small correlation between word neighborhoods. Full vocabularies are attained also for $\wp < 0.4$; Phase II: a drastic transition appears at the critical domain $\wp ^* \in (0.4,0.6)$, in which $\langle cc \rangle $ shifts abruptly towards 1. An abrupt change in $V(t_f)$ versus $\wp $ is also found (Fig. FIGREF16) for $\wp ^*$; Phase III: single-word languages dominate for $\wp > 0.6$. The maximum value of $\langle cc \rangle $ indicate that word neighborhoods are completely correlated.

Answers:
Phase I: $\langle cc \rangle $ increases smoothly for $\wp < 0.4$, indicating that for this domain there is a small correlation between word neighborhoods. Full vocabularies are attained also for $\wp < 0.4$, Phase II: a drastic transition appears at the critical domain $\wp ^* \in (0.4,0.6)$, in which $\langle cc \rangle $ shifts abruptly towards 1. An abrupt change in $V(t_f)$ versus $\wp $ is also found (Fig. FIGREF16) for $\wp ^*$, Phase III: single-word languages dominate for $\wp > 0.6$. The maximum value of $\langle cc \rangle $ indicate that word neighborhoods are completely correlated
Question: Do the experiments explore how various architectures and layers contribute towards certain decisions?

Evidence:
raw_evidence: As a document classifier we employ a word-based CNN similar to Kim consisting of the following sequence of layers: $ \texttt {Conv} \xrightarrow{} \texttt {ReLU} \xrightarrow{} \texttt {1-Max-Pool} \xrightarrow{} \texttt {FC} \\ $
Future work would include applying LRP to other neural network architectures (e.g. character-based or recurrent models) on further NLP tasks, as well as exploring how relevance information could be taken into account to improve the classifier's training procedure or prediction performance.

Answers:
No
Question: Do the authors perform experiments using their proposed method?

Evidence:
raw_evidence: We illustrate the concept by discussing some instantiations that we have recently experimented with.
Unlike the Visual Dialogue setting discussed above, this setting ensures informational symmetry between the participants (both have access to the same type of information; but not the same information, as they can't “see” each other). More importantly, however, the constraint that the game only ends if they both agree ensures a “committment symmetry”, where the success of the game must be ensured by both participants. The design also provides for a clear “relevance place” at which an opportunity arises for semantic negotiation, namely, before the final decision is made. An example of this is shown in the example below. (The number in the parentheses indicate the time, relative to the beginning of the interaction, when the utterance was made.)
The MatchIt Game (Ilinykh et al., forthcoming) is a yet further simplified visual game. Here, the goal simply is to decide whether you and your partner are both looking at the same image (of the same genre as in MeetUp). In that sense, it is a reduction of the MeetUP game to the final stage, taking out the navigation aspect. As example SECREF12 shows, this can similarly lead to meta-semantic interaction, where classifications are revised. As SECREF12 shows, even in cases where a decision can be reached quickly, there can be an explicit mutual confirmation step, before the (silent) decision signal is sent.
A third setting that we have explored BIBREF19 brings conceptual negotiation more clearly into the foreground. In that game, the players are presented with images of birds of particular species and are tasked with coming up with a description of common properties. Again, the final answer has to be approved by both participants. As SECREF13 shows, this can lead to an explicit negotiation of conceptual content.

Answers:
Yes
Question: What NLP tasks do the authors evaluate feed-forward networks on?

Evidence:
raw_evidence: We experiment with small feed-forward networks for four diverse NLP tasks: language identification, part-of-speech tagging, word segmentation, and preordering for statistical machine translation.

Answers:
language identification, part-of-speech tagging, word segmentation, and preordering for statistical machine translation
Question: What are three challenging tasks authors evaluated their sequentially aligned representations?

Evidence:
raw_evidence: We consider three tasks representing a broad selection of natural language understanding scenarios: paper acceptance prediction based on the PeerRead data set BIBREF2, Named Entity Recognition (NER) based on the Broad Twitter Corpus BIBREF3, and author stance prediction based on the RumEval-19 data set BIBREF6. These tasks were chosen so as to represent i) different textual domains, across ii) differing time scales, and iii) operating at varying levels of linguistic granularity. As we are dealing with dynamical learning, the vast majority of NLP data sets can unfortunately not be used since they do not include time stamps.

Answers:
paper acceptance prediction, Named Entity Recognition (NER), author stance prediction
Question: What is the difference in findings of Buck et al? It looks like the same conclusion was mentioned in Buck et al..

Evidence:
raw_evidence: Here we perform a qualitative analysis of this communication process to better understand what kind of language the agent has learned. We find that while optimizing its reformulations to adapt to the language of the QA system, AQA diverges from well structured language in favour of less fluent, but more effective, classic information retrieval (IR) query operations. These include term re-weighting (tf-idf), expansion and morphological simplification/stemming. We hypothesize that the explanation of this behaviour is that current machine comprehension tasks primarily require ranking of short textual snippets, thus incentivizing relevance more than deep language understanding.

Answers:
AQA diverges from well structured language in favour of less fluent, but more effective, classic information retrieval (IR) query operations
Question: What is the baseline?

Evidence:
raw_evidence: We compare the results of our model to a baseline multi-task architecture inspired by yang2016multi. In our baseline model there are no explicit connections between tasks - the only shared parameters are in the hidden layer.

Answers:
The baseline is a multi-task architecture inspired by another paper.
Question: What is the unsupervised task in the final layer?

Evidence:
raw_evidence: In our model we represent linguistically motivated hierarchies in a multi-task Bi-Directional Recurrent Neural Network where junior tasks in the hierarchy are supervised at lower layers.This architecture builds upon sogaard2016deep, but is adapted in two ways: first, we add an unsupervised sequence labeling task (Language Modeling), second, we add a low-dimensional embedding layer between tasks in the hierarchy to learn dense representations of label tags. In addition to sogaard2016deep.

Answers:
Language Modeling
Question: Does the latent dialogue state heklp their model?

Evidence:
raw_evidence: Recently, end-to-end approaches have trained recurrent neural networks (RNNs) directly on text transcripts of dialogs. A key benefit is that the RNN infers a latent representation of state, obviating the need for state labels. However, end-to-end methods lack a general mechanism for injecting domain knowledge and constraints. For example, simple operations like sorting a list of database results or updating a dictionary of entities can expressed in a few lines of software, yet may take thousands of dialogs to learn. Moreover, in some practical settings, programmed constraints are essential – for example, a banking dialog system would require that a user is logged in before they can retrieve account information.

Answers:
Yes
Question: Do the authors test on datasets other than bAbl?

Evidence:

Answers:
No