qinchuanhui/UDA-QA huggingface dataset:
Question: what semantically conditioned models did they compare with?

Evidence:
raw_evidence: For goal-oriented generation, many of the models evaluated using the Inform/Request metrics have made use of structured data to semantically condition the generation model in order to generate better responses BIBREF35, BIBREF1, BIBREF0. BIBREF35 proposed to encode each individual dialog act as a unique vector and use it as an extra input feature, in order to influence the generated response. This method has been shown to work well when tested on single domains where the label space is limited to a few dialog acts. However, as the label space grows, using a one-hot encoding representation of the dialog act is not scalable. To deal with this problem, BIBREF1 introduced a semantically conditioned generation model using Hierarchical Disentangled Self-Attention (HDSA) . This model deals with the large label space by representing dialog acts using a multi-layer hierarchical graph that merges cross-branch nodes. For example, the distinct trees for hotel-recommend-area and attraction-recommend-area can be merged at the second and third levels sharing semantic information about actions and slots but maintaining specific information about the domains separate. This information can then be used as part of an attention mechanism when generating a response. This model achieves the state-of-the-art result for generation in both BLEU and Inform/Request metrics.

Answers:
Hierarchical Disentangled Self-Attention
Question: Do they differentiate insights where they are dealing with learned or engineered representations?

Evidence:
raw_evidence: We extract 5 Surface and Lexical features, namely sequence length in number of tokens, average word length, type-token ratio, and lexical to tokens ratio (ratio of adjectives, verbs, nouns, and adverbs to tokens). Bag of INLINEFORM0 -grams features are extracted on the word and POS level. We use frequency cut-offs of INLINEFORM1 for INLINEFORM2 -grams from 1 to 5 respectively for the smaller datasets and ten times higher for the Yahoo! and Amazon datasets. For POS INLINEFORM3 -grams we use cut-offs 10 for unigrams and 20 for bigrams and higher. For the Yahoo! and Amazon datasets we use cut-offs of INLINEFORM4 . The INLINEFORM5 -grams features are then also extracted using the hashing trick with the same cut-offs to reduce the final feature vector size when combined with other features. scikit-learn's BIBREF14 FeatureHasher is used with output vectors sizes of INLINEFORM6 INLINEFORM7 INLINEFORM8 for ngrams from INLINEFORM9 respectively and INLINEFORM10 INLINEFORM11 INLINEFORM12 are used for POS ngrams. We extract lexical and POS level Language model features based on external language models, namely sentence log probabilities, perplexities, and surprisal in units of bits. Building the language model and extracting the features is done by providing the path to the compiled binaries for kenlm BIBREF15 . Finally we extract N-gram Frequency Quantile Distribution features with the same cut-offs as in the bag of ngrams features, with 4 quantiles and an OOV quantile. NLTK BIBREF16 is used for tokenization and POS tagging.
We extracted two features that use a learned representation: Firstly, we get a sentence embedding feature that is built by averaging the word embeddings of an input sentence. Secondly, we extract a fastText representation using the fastText library with the same parameters as reported in Joulin et al. joulin2016bag.

Answers:
Yes
Question: Do they show an example of usage for INFODENS?

Evidence:
raw_evidence: The framework can be used as a standalone toolkit without any modifications given the implemented features and classifiers. For example, it can be used to extract features for usage with other machine learning tools, or to evaluate given features with the existing classifiers or regressors. Extending the framework with new feature extractors or classifiers is as simple as a drag and drop placement of the new code files into the feature_extractor and classifer directories respectively. The framework will then detect the new extensions dynamically at runtime. In this section we explore how each use case is handled.
Since a main use case for the framework is extracting engineered and learned features, it was designed such that developing a new feature extractor would require minimal effort. Figure FIGREF19 demonstrates a simple feature extractor that retrieves the sentence length. More complicated features and learned features are provided in the repository which can be used as a guide for developers. Documentation for adding classifiers and format writers is described in the Wiki of the repository but is left out of this paper due to the limited space.

Answers:
Yes
Question: What models do they compare to?

Evidence:
raw_evidence: To evaluate the summarization performance of AttSum, we implement rich extractive summarization methods. Above all, we introduce two common baselines. The first one just selects the leading sentences to form a summary. It is often used as an official baseline of DUC, and we name it “LEAD”. The other system is called “QUERY_SIM”, which directly ranks sentences according to its TF-IDF cosine similarity to the query. In addition, we implement two popular extractive query-focused summarization methods, called MultiMR BIBREF2 and SVR BIBREF20 . MultiMR is a graph-based manifold ranking method which makes uniform use of the sentence-to-sentence relationships and the sentence-to-query relationships. SVR extracts both query-dependent and query-independent features and applies Support Vector Regression to learn feature weights. Note that MultiMR is unsupervised while SVR is supervised. Since our model is totally data-driven, we introduce a recent summarization system DocEmb BIBREF9 that also just use deep neural network features to rank sentences. It initially works for generic summarization and we supplement the query information to compute the document representation.
To verify the effectiveness of the joint model, we design a baseline called ISOLATION, which performs saliency ranking and relevance ranking in isolation. Specifically, it directly uses the sum pooling over sentence embeddings to represent the document cluster. Therefore, the embedding similarity between a sentence and the document cluster could only measure the sentence saliency. To include the query information, we supplement the common hand-crafted feature TF-IDF cosine similarity to the query. This query-dependent feature, together with the embedding similarity, are used in sentence ranking. ISOLATION removes the attention mechanism, and mixtures hand-crafted and automatically learned features. All these methods adopt the same sentence selection process illustrated in Section "Sentence Selection" for a fair comparison.

Answers:
LEAD, QUERY_SIM, MultiMR, SVR, DocEmb, ISOLATION
Question: Do the authors give any examples of major events which draw the public's attention and the impact they have on stock price?

Evidence:
raw_evidence: Figure FIGREF6 shows the relationship between Tesla stock return and stock sentiment score. According the distribution of the sentiment score, the sentiment on Tesla is slightly skewed towards positive during the testing period. The price has been increased significantly during the testing period, which reflected the positive sentiment. The predicting power of sentiment score is more significant when the sentiment is more extreme and less so when the sentiment is neutral.

Answers:
Yes
Question: Which tweets are used to output the daily sentiment signal?

Evidence:
raw_evidence: There are two options of getting the Tweets. First, Twitter provides an API to download the Tweets. However, rate limit and history limit make it not an option for this paper. Second, scrapping Tweets directly from Twitter website. Using the second option, the daily Tweets for stocks of interest from 2015 January to 2017 June were downloaded.
For this reason, two companies from the same industry, Tesla and Ford are investigated on how Twitter sentiment could impact the stock price. Tesla is an electronic car company that shows consecutive negative operating cash flow and net income but carries very high expectation from the public. Ford, is a traditional auto maker whose stock prices has been stabilized to represent the company fundamentals.
To translate each tweet into a sentiment score, the Stanford coreNLP software was used. Stanford CoreNLP is designed to make linguistic analysis accessible to the general public. It provides named Entity Recognition, co-reference and basic dependencies and many other text understanding applications. An example that illustrate the basic functionality of Stanford coreNLP is shown in Figure. FIGREF5

Answers:
Tesla and Ford are investigated on how Twitter sentiment could impact the stock price
Question: What is the baseline machine learning prediction approach?

Evidence:
raw_evidence: To evaluate if the sentiment feature improves the prediction accuracy, a baseline model is defined. The baseline applies linear logistic regression to a set of stock technical signals to predict the following day’s stock return sign (+/‐). No sentiment features are provided to the baseline model.

Answers:
linear logistic regression to a set of stock technical signals
Question: What are the weaknesses of their proposed interpretability quantification method?

Evidence:
raw_evidence: Our interpretability measurements are based on our proposed dataset SEMCAT, which was designed to be a comprehensive dataset that contains a diverse set of word categories. Yet, it is possible that the precise interpretability scores that are measured here are biased by the dataset used. In general, two main properties of the dataset can affect the results: category selection and within-category word selection. To examine the effects of these properties on interpretability evaluations, we create alternative datasets by varying both category selection and word selection for SEMCAT. Since SEMCAT is comprehensive in terms of the words it contains for the categories, these datasets are created by subsampling the categories and words included in SEMCAT. Since random sampling of words within a category may perturb the capacity of the dataset in reflecting human judgement, we subsample r% of the words that are closest to category centers within each category, where $r \in \lbrace 40,60,80,100\rbrace $ . To examine the importance of number of categories in the dataset we randomly select $m$ categories from SEMCAT where $m \in \lbrace 30,50,70,90,110\rbrace $ . We repeat the selection 10 times independently for each $m$ .
In contrast to the category coverage, the effects of within-category word coverage on interpretability scores can be more complex. Starting with few words within each category, increasing the number of words is expected to more uniformly sample from the word distribution, more accurately reflect the semantic relations within each category and thereby enhance interpretability scores. However, having categories over-abundant in words might inevitably weaken semantic correlations among them, reducing the discriminability of the categories and interpretability of the embedding. Table 3 shows that, interestingly, changing the category coverage has different effects on the interpretability scores of different types of embeddings. As category word coverage increases, interpretability scores for random embedding gradually decrease while they monotonically increase for the GloVe embedding. For semantic spaces $\mathcal {I}$ and $\mathcal {I}^*$ , interpretability scores increase as the category coverage increases up to 80 $\%$ of that of SEMCAT, then the scores decrease. This may be a result of having too comprehensive categories as argued earlier, implying that categories with coverage of around 80 $\%$ of SEMCAT are better suited for measuring interpretability. However, it should be noted that the change in the interpretability scores for different word coverages might be effected by non-ideal subsampling of category words. Although our word sampling method, based on words' distances to category centers, is expected to generate categories that are represented better compared to random sampling of category words, category representations might be suboptimal compared to human designed categories.

Answers:
can be biased by dataset used and may generate categories which are suboptimal compared to human designed categories
Question: What advantages does their proposed method of quantifying interpretability have over the human-in-the-loop evaluation they compare to?

Evidence:
raw_evidence: In the word embedding literature, the problem of interpretability has been approached via several different routes. For learning sparse, interpretable word representations from co-occurrence variant matrices, BIBREF21 suggested algorithms based on non-negative matrix factorization (NMF) and the resulting representations are called non-negative sparse embeddings (NNSE). To address memory and scale issues of the algorithms in BIBREF21 , BIBREF22 proposed an online method of learning interpretable word embeddings. In both studies, interpretability was evaluated using a word intrusion test introduced in BIBREF20 . The word intrusion test is expensive to apply since it requires manual evaluations by human observers separately for each embedding dimension. As an alternative method to incorporate human judgement, BIBREF23 proposed joint non-negative sparse embedding (JNNSE), where the aim is to combine text-based similarity information among words with brain activity based similarity information to improve interpretability. Yet, this approach still requires labor-intensive collection of neuroimaging data from multiple subjects.
In addition to investigating the semantic distribution in the embedding space, a word category dataset can be also used to quantify the interpretability of the word embeddings. In several studies, BIBREF21 , BIBREF22 , BIBREF20 , interpretability is evaluated using the word intrusion test. In the word intrusion test, for each embedding dimension, a word set is generated including the top 5 words in the top ranks and a noisy word (intruder) in the bottom ranks of that dimension. The intruder is selected such that it is in the top ranks of a separate dimension. Then, human editors are asked to determine the intruder word within the generated set. The editors' performances are used to quantify the interpretability of the embedding. Although evaluating interpretability based on human judgements is an effective approach, word intrusion is an expensive method since it requires human effort for each evaluation. Furthermore, the word intrusion test does not quantify the interpretability levels of the embedding dimensions, instead it yields a binary decision as to whether a dimension is interpretable or not. However, using continuous values is more adequate than making binary evaluations since interpretability levels may vary gradually across dimensions.

Answers:
it is less expensive and quantifies interpretability using continuous values rather than binary evaluations
Question: How do they gather data for the query explanation problem?

Evidence:
raw_evidence: WikiTableQuestions BIBREF1 is a question answering dataset over semi-structured tables. It is comprised of question-answer pairs on HTML tables, and was constructed by selecting data tables from Wikipedia that contained at least 8 rows and 5 columns. Amazon Mechanical Turk workers were then tasked with writing trivia questions about each table. In contrast to common NLIDB benchmarks BIBREF2 , BIBREF0 , BIBREF15 , WikiTableQuestions contains 22,033 questions and is an order of magnitude larger than previous state-of-the-art datasets. Its questions were not designed by predefined templates but were hand crafted by users, demonstrating high linguistic variance. Compared to previous datasets on knowledge bases it covers nearly 4,000 unique column headers, containing far more relations than closed domain datasets BIBREF15 , BIBREF2 and datasets for querying knowledge bases BIBREF16 . Its questions cover a wide range of domains, requiring operations such as table lookup, aggregation, superlatives (argmax, argmin), arithmetic operations, joins and unions. The complexity of its questions can be shown in Tables TABREF6 and TABREF66 .

Answers:
hand crafted by users
Question: Which query explanation method was preffered by the users in terms of correctness?

Evidence:
raw_evidence: Results in Table TABREF64 show the correctness rates of these scenarios. User correctness score is superior to that of the baseline parser by 7.5% (from 37.1% to 44.6%), while the hybrid approach outscores both with a correctness of 48.7% improving the baseline by 11.6%. For the user and hybrid correctness we used a INLINEFORM0 test to measure significance. Random queries and tables included in the experiment are presented in Table TABREF66 . We also include a comparison of the top ranked query of the baseline parser compared to that of the user.

Answers:
hybrid approach
Question: Do they conduct a user study where they show an NL interface with and without their explanation?

Evidence:
raw_evidence: We have examined the effect to which our query explanations can help users improve the correctness of a baseline NL interface. Our user study compares the correctness of three scenarios:
Parser correctness - our baseline is the percentage of examples where the top query returned by the semantic parser was correct.
User correctness - the percentage of examples where the user selected a correct query from the top-7 generated by the parser.
Hybrid correctness - correctness of queries returned by a combination of the previous two scenarios. The system returns the query marked by the user as correct; if the user marks all queries as incorrect it will return the parser's top candidate.

Answers:
No
Question: What was the task given to workers?

Evidence:
raw_evidence: We recruited 176 AMT workers to participate in our conceptualization task. Of these workers, 90 were randomly assigned to the Control group and 86 to the AUI group. These workers completed 1001 tasks: 496 tasks in the control and 505 in the AUI. All responses were gathered within a single 24-hour period during April, 2017.

Answers:
conceptualization task
Question: How was lexical diversity measured?

Evidence:
raw_evidence: To study the lexical and semantic diversities of responses, we performed three analyses. First, we aggregated all worker responses to a particular question into a single list corresponding to that question. Across all questions, we found that the number of unique responses was higher for the AUI than for the Control (Fig. FIGREF19 A), implying higher diversity for AUI than for Control.
Second, we compared the diversity of individual responses between Control and AUI for each question. To measure diversity for a question, we computed the number of responses divided by the number of unique responses to that question. We call this the response density. A set of responses has a response density of 1 when every response is unique but when every response is the same, the response density is equal to the number of responses. Across the ten questions, response density was significantly lower for AUI than for Control (Wilcoxon signed rank test paired on questions: INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ) (Fig. FIGREF19 B).

Answers:
By computing number of unique responses and number of responses divided by the number of unique responses to that question for each of the questions
Question: How many responses did they obtain?

Evidence:
raw_evidence: We recruited 176 AMT workers to participate in our conceptualization task. Of these workers, 90 were randomly assigned to the Control group and 86 to the AUI group. These workers completed 1001 tasks: 496 tasks in the control and 505 in the AUI. All responses were gathered within a single 24-hour period during April, 2017.

Answers:
1001
Question: What crowdsourcing platform was used?

Evidence:
raw_evidence: We recruited 176 AMT workers to participate in our conceptualization task. Of these workers, 90 were randomly assigned to the Control group and 86 to the AUI group. These workers completed 1001 tasks: 496 tasks in the control and 505 in the AUI. All responses were gathered within a single 24-hour period during April, 2017.

Answers:
AMT
Question: Are results reported only for English data?

Evidence:
raw_evidence: We evaluate the proposed approach on the Chinese social media text summarization task, based on the sequence-to-sequence model. We also analyze the output text and the output label distribution of the models, showing the power of the proposed approach. Finally, we show the cases where the correspondences learned by the proposed approach are still problematic, which can be explained based on the approach we adopt.
Large-Scale Chinese Short Text Summarization Dataset (LCSTS) is constructed by BIBREF1 . The dataset consists of more than 2.4 million text-summary pairs in total, constructed from a famous Chinese social media microblogging service Weibo. The whole dataset is split into three parts, with 2,400,591 pairs in PART I for training, 10,666 pairs in PART II for validation, and 1,106 pairs in PART III for testing. The authors of the dataset have manually annotated the relevance scores, ranging from 1 to 5, of the text-summary pairs in PART II and PART III. They suggested that only pairs with scores no less than three should be used for evaluation, which leaves 8,685 pairs in PART II, and 725 pairs in PART III. From the statistics of the PART II and PART III, we can see that more than 20% of the pairs are dropped to maintain semantic quality. It indicates that the training set, which has not been manually annotated and checked, contains a huge quantity of unrelated text-summary pairs.

Answers:
No
Question: How is human evaluation performed, what were the criteria?

Evidence:
raw_evidence: Three annotators from a commercial annotation company are recruited to conduct our human evaluation. Responses from different models are shuffled for labeling. 300 test queries are randomly selected out, and annotators are asked to independently score the results of these queries with different points in terms of their quality: (1) Good (3 points): The response is grammatical, semantically relevant to the query, and more importantly informative and interesting; (2) Acceptable (2 points): The response is grammatical, semantically relevant to the query, but too trivial or generic (e.g.,“我不知道(I don't know)", “我也是(Me too)”, “我喜欢(I like it)" etc.); (3) Failed (1 point): The response has grammar mistakes or irrelevant to the query.

Answers:
(1) Good (3 points): The response is grammatical, semantically relevant to the query, and more importantly informative and interesting, (2) Acceptable (2 points): The response is grammatical, semantically relevant to the query, but too trivial or generic, (3) Failed (1 point): The response has grammar mistakes or irrelevant to the query
Question: What automatic metrics are used?

Evidence:
raw_evidence: To evaluate the responses generated by all compared methods, we compute the following automatic metrics on our test set:
BLEU: BLEU-n measures the average n-gram precision on a set of reference responses. We report BLEU-n with n=1,2,3,4.
Distinct-1 & distinct-2 BIBREF5: We count the numbers of distinct uni-grams and bi-grams in the generated responses and divide the numbers by the total number of generated uni-grams and bi-grams in the test set. These metrics can be regarded as an automatic metric to evaluate the diversity of the responses.

Answers:
BLEU, Distinct-1 & distinct-2