qinchuanhui/UDA-QA huggingface dataset:
Question: Do they fine-tune the used word embeddings on their medical texts?

Evidence:
raw_evidence: Three most common classic non-contextual approaches to obtain word embeddings are skip-gram, Continuous Bag of Words (two algorithms from BIBREF19 ) and GloVe (Global Vectors, BIBREF20 , where higher accuracy than in previous algorithms was proved). Some authors use pretrained embeddings (especially when their data set is too small to train their own embeddings) or try to modify these embeddings and adjust to their set. But the biggest drawback of these approaches is that the corpus for training embeddings can be not related to the specific task where embeddings are utilized. A lot of medical concepts are not contained in well-known embeddings bases. Furthermore, the similarity of words may vary in different contexts.
Then, we compute embeddings of concepts (by GloVe) for interview descriptions and for examination descriptions separately. We compute two separate embeddings, because we want to catch the similarity between terms in their specific context, i.e. words similar in the interview may not be similar in the examination description (for example we computed that the nearest words to cough in interview descriptions was runny nose, sore throat, fever, dry cough but in examination description it was rash, sunny, laryngeal, dry cough).

Answers:
No
Question: Which word embeddings do they use to represent medical visits?

Evidence:
raw_evidence: Then, we compute embeddings of concepts (by GloVe) for interview descriptions and for examination descriptions separately. We compute two separate embeddings, because we want to catch the similarity between terms in their specific context, i.e. words similar in the interview may not be similar in the examination description (for example we computed that the nearest words to cough in interview descriptions was runny nose, sore throat, fever, dry cough but in examination description it was rash, sunny, laryngeal, dry cough).
The simplest way to generate text embeddings based on term embeddings is to use some kind of aggregation of term embeddings such as an average. This approach was tested for example by BIBREF21 and BIBREF13 . BIBREF22 computed a weighted mean of term embeddings by the construction of a loss function and training weights by the gradient descent method.
Final embeddings for visits are obtained by concatenation of average embeddings calculated separately for the interview and for the medical examination, see Figure FIGREF4 .

Answers:
GloVe, concatenation of average embeddings calculated separately for the interview and for the medical examination
Question: Do they explore similarity of texts across different doctors?

Evidence:
raw_evidence: Clustering was performed separately for each specialty of doctors. Figure FIGREF11 illustrates two-dimensional projections of visit embeddings coloured by clusters. The projections were created by t-SNE algorithm BIBREF25 . For some domains clusters are very clear and separated (Figure FIGREF11 ). This corresponds with the high stability of clustering measured by Rand index.
We also examined the distribution of doctors' IDs in the obtained clusters. It turned out that some clusters covered almost exactly descriptions written by one doctor. This situation took place in the specialties where clusters are separated with large margins (e.g. psychiatry, pediatrics, cardiology). Figure FIGREF13 (a) shows correspondence analysis between doctors' IDs and clusters for psychiatry clustering.

Answers:
Yes
Question: Which clustering technique do they use on partients' visits texts?

Evidence:
raw_evidence: Based on Euclidean distance between vector representations of visits we applied and compared two clustering algorithms: k-means and hierarchical clustering with Ward's method for merging clusters BIBREF23 . The similarity of these clusterings was measured by the adjusted Rand index BIBREF24 . For the final results we chose the hierarchical clustering algorithm due to greater stability.

Answers:
k-means, hierarchical clustering with Ward's method for merging clusters BIBREF23
Question: What is proof that proposed functional form approximates well generalization error in practice?

Evidence:
raw_evidence: As fig:fit shows, estimated test accuracy is highly correlated with actual test accuracy for various datasets, with worst-case values $\mu <1\%$ and $\sigma <5\%$ . Note that the number of free parameters is small ($||\le 6$) compared to the number of points (42–49 model-data configurations), demonstrating the appropriateness of the proposed function for modeling the complex error landscape.

Answers:
estimated test accuracy is highly correlated with actual test accuracy for various datasets, appropriateness of the proposed function for modeling the complex error landscape
Question: What other non-neural baselines do the authors compare to? 

Evidence:
raw_evidence: is the standard word counting method whereby the feature vector represents the term frequency of the words in a sentence.
is similar to BoW, except that it is derived by the counting of the words in the sentence weighted by individual word's term-frequency and inverse-document-frequency BIBREF31 . This is a very competitive model especially on clean and small dataset.
is derived from clustering of words-embeddings with k-means into 5000 clusters, and follow by BoW representation of the words in 5000 clusters.

Answers:
bag of words, tf-idf, bag-of-means
Question: What baseline function is used in REINFORCE algorithm?

Evidence:
raw_evidence: Following BIBREF21, we sample $K$ captions for each image when applying REINFORCE: ${\hat{c}}_1 \ldots {\hat{c}}_K$, ${\hat{c}}_k \sim p_{\theta }(c|I)$,
The baseline for each sampled caption is defined as the average reward of the rest samples. That is, for caption $\hat{c}_k$, its baseline is

Answers:
baseline for each sampled caption is defined as the average reward of the rest samples
Question: Is Aristo just some modern NLP model (ex. BERT) finetuned od data specific for this task?

Evidence:
raw_evidence: The current configuration of Aristo comprises of eight solvers, described shortly, each of which attempts to answer a multiple choice question. To study particular phenomena and develop solvers, the project has created larger datasets to amplify and study different problems, resulting in 10 new datasets and 5 large knowledge resources for the community.
The solvers can be loosely grouped into:
Statistical and information retrieval methods
Reasoning methods
Large-scale language model methods
Over the life of the project, the relative importance of the methods has shifted towards large-scale language methods.
The field of NLP has advanced substantially with the advent of large-scale language models such as ELMo (BID6), ULMFit (BID37), GPT (BID38), BERT (BID7), and RoBERTa (BID8). These models are trained to perform various language prediction tasks such as predicting a missing word or the next sentence, using large amounts of text (e.g., BERT was trained on Wikipedia + the Google Book Corpus of 10,000 books). They can also be fine-tuned to new language prediction tasks, such as question-answering, and have been remarkably successful in the few months that they have been available.
We apply BERT to multiple choice questions by treating the task as classification: Given a question $q$ with answer options $a_{i}$ and optional background knowledge $K_{i}$, we provide it to BERT as:
[CLS] $K_i$ [SEP] $q$ [SEP] $a_{i}$ [SEP]
The AristoBERT solver uses three methods to apply BERT more effectively. First, we retrieve and supply background knowledge along with the question when using BERT. This provides the potential for BERT to “read” that background knowledge and apply it to the question, although the exact nature of how it uses background knowledge is more complex and less interpretable. Second, we fine-tune BERT using a curriculum of several datasets, including some that are not science related. Finally, we ensemble different variants of BERT together.

Answers:
Yes
Question: Does they focus on any specific product/service domain?

Evidence:
raw_evidence: Our data are collected from the yelp academic dataset, provided by Yelp.com, a popular restaurant review website. The data set contains three types of objects: business, user, and review, where business objects contain basic information about local businesses (i.e. restaurants), review objects contain review texts and star rating, and user objects contain aggregate information about a single user across all of Yelp. Table TABREF31 illustrates the general statistics of the dataset.

Answers:
local businesses (i.e. restaurants)
Question: What are the baselines?

Evidence:
raw_evidence: We show the final results for opinion recommendation, comparing our proposed model with the following state-of-the-art baseline systems:
RS-Average is the widely-adopted baseline (e.g., by Yelp.com), using the averaged review scores as the final score.
RS-Linear estimates the rating score that a user would give by INLINEFORM0 BIBREF49 , where INLINEFORM1 and INLINEFORM2 are the the training deviations of the user INLINEFORM3 and the product INLINEFORM4 , respectively.
RS-Item applies INLINEFORM0 NN to estimate the rating score BIBREF50 . We choose the cosine similarity between INLINEFORM1 to measure the distance between product.
RS-MF is a state-of-the-art recommendation model, which uses matrix factorisation to predict rating score BIBREF8 , BIBREF41 , BIBREF25 .
Sum-Opinosis uses a graph-based framework to generate abstractive summarisation given redundant opinions BIBREF51 .
Sum-LSTM-Att is a state-of-the-art neural abstractive summariser, which uses an attentional neural model to consolidate information from multiple text sources, generating summaries using LSTM decoding BIBREF44 , BIBREF3 .
All the baseline models are single-task models, without considering rating and summarisation prediction jointly. The results are shown in Table TABREF46 . Our model (“ Joint”) significantly outperforms both “RS-Average” and “RS-Linear” ( INLINEFORM0 using INLINEFORM1 -test), which demonstrates the strength of opinion recommendation, which leverages user characteristics for calculating a rating score for the user.

Answers:
RS-Average , RS-Linear, RS-Item, RS-MF, Sum-Opinosis, Sum-LSTM-Att
Question: How is fluency of generated text evaluated?

Evidence:
raw_evidence: We manually reviewed 1,177 pairs of entities and referring expressions generated by the system. We find that 92.2% of the generated referring expressions refer to the correct entity.
From the generated expressions, 325 (27.6%) were pronouns, 192 (16.3%) are repeating a one-token entity as is, and 505 (42.9%) are generating correct shortening of a long entity. In 63 (5.6%) of the cases the system did not find a good substitute and kept the entire entity intact. Finally, 92 (7.82%) are wrong referrals. Overall, 73.3% of the non-first mentions of entities were replaced with suitable shorter and more fluent expressions.

Answers:
manually reviewed
Question: How is faithfulness of the resulting text evaluated?

Evidence:
raw_evidence: The addition of output verification resulted in negligible changes in BLEU, reinforcing that automatic metrics are not sensitive enough to output accuracy. We thus performed manual analysis, following the procedure in BIBREF0. We manually inspect 148 samples from the seen part of the test set, containing 440 relations, counting expressed, omitted, wrong and over-generated (hallucinated) facts. We compare to the StrongNeural and BestPlan systems from BIBREF0. Results in Table indicate that the effectiveness of the verification process in ensuring correct output, reducing the already small number of ommited and overgenerated facts to 0 (with the exhaustive planner) and keeping it small (with the fast neural planner).

Answers:
manually inspect
Question: How are typing hints suggested?

Evidence:
raw_evidence: We incorporate typing information by concatenating to the embedding vector of each input symbol one of three embedding vectors, S, E or R, where S is concatenated to structural elements (opening and closing brackets), E to entity symbols and R to relation symbols.

Answers:
 concatenating to the embedding vector
Question: What is the effectiveness plan generation?

Evidence:
raw_evidence: While the plan generation stage is guaranteed to be faithful to the input, the translation process from plans to text is based on a neural seq2seq model and may suffer from known issues with such models: hallucinating facts that do not exist in the input, repeating facts, or dropping facts. While the clear mapping between plans and text helps to reduce these issues greatly, the system in BIBREF0 still has 2% errors of these kinds.
Recent work in neural text generation and summarization attempt to address these issues by trying to map the textual outputs back to structured predicates, and comparing these predicates to the input data. BIBREF7 uses a neural checklist model to avoid the repetition of facts and improve coverage. BIBREF8 generate $k$-best output candidates with beam search, and then try to map each candidate output back to the input structure using a reverse seq2seq model trained on the same data. They then select the highest scoring output candidate that best translates back to the input. BIBREF9 reconstructs the input in training time, by jointly learning a back-translation model and enforcing the back-translation to reconstruct the input. Both of these approaches are “soft” in the sense that they crucially rely on the internal dynamics or on the output of a neural network module that may or may not be correct.

Answers:
clear mapping between plans and text helps to reduce these issues greatly, the system in BIBREF0 still has 2% errors, work in neural text generation and summarization attempt to address these issues
Question: How is neural planning component trained?

Evidence:
raw_evidence: At training time, we use the plan-to-DFS mapping to perform the correct sequence of traversals, and train a neural classifier to act as a controller, choosing which action to perform at each step. At test time, we use the controller to guide the truncated DFS process. This mechanism is inspired by transition based parsing BIBREF5. The action set at each stage is dynamic. During traversal, it includes the available children at each stage and pop. Before traversals, it includes a choose-i action for each available node $n_i$. We assign a score to each action, normalize with softmax, and train to choose the desired one using cross-entropy loss. At test time, we either greedily choose the best action, or we can sample plans by sampling actions according to their assigned probabilities.

Answers:
plan-to-DFS mapping to perform the correct sequence of traversals, and train a neural classifier to act as a controller
Question: How do they evaluate interpretability in this paper?

Evidence:
raw_evidence: Let all word vectors be normalized and $W$ be the word matrix. Inspired by BIBREF21 , where vector space models are used for evaluating topic coherence, we suggest to estimate the interpretability of $k$ th component as $ \operatorname{interp}_k W = \sum _{i,j=1}^N W_{i,k} W_{j,k} \left(W_i \cdot W_j \right). $

Answers:
we suggest to estimate the interpretability of $k$ th component as $ \operatorname{interp}_k W = \sum _{i,j=1}^N W_{i,k} W_{j,k} \left(W_i \cdot W_j \right). $
Question: How much better performing is the proposed method over the baselines?

Evidence:
raw_evidence: Results are shown in Table TABREF11 . Results show that incorporating self-attention mechanism in the encoder is beneficial for most tasks. However, original models were better in some tasks (CR, MPQA, MRPC), suggesting that self-attention mechanism could sometimes introduce noise in sentence features. Overall, utilizing self-attentive sentence representation further improves performances in 5 out of 8 tasks. Considering that models with self-attention employ smaller LSTM cells (1024) than those without (2048) (Section SECREF6 ), the performance improvements are significant. Results on COCO5K image and caption retrieval tasks (not included in the paper due to limited space) show comparable performances to other more specialized methods BIBREF10 , BIBREF39 .

Answers:
original models were better in some tasks (CR, MPQA, MRPC), utilizing self-attentive sentence representation further improves performances in 5 out of 8 tasks
Question: What baselines are the proposed method compared against?

Evidence:
raw_evidence: Adhering to the experimental settings of BIBREF1 , we concatenate sentence representations produced from our model with those obtained from the state-of-the-art unsupervised learning model (Layer Normalized Skip-Thoughts, ST-LN) BIBREF31 . We evaluate the quality of sentence representations produced from different variants of our encoders on well-known transfer tasks: movie review sentiment (MR) BIBREF32 , customer reviews (CR) BIBREF33 , subjectivity (SUBJ) BIBREF34 , opinion polarity (MPQA) BIBREF35 , paraphrase identification (MSRP) BIBREF36 , binary sentiment classification (SST) BIBREF37 , SICK entailment and SICK relatedness BIBREF38 .
Following the experimental design of BIBREF1 , we conduct experiments on three different learning objectives: Cap2All, Cap2Cap, Cap2Img. Under Cap2All, the model is trained to predict both the target caption and the associated image: INLINEFORM0 . Under Cap2Cap, the model is trained to predict only the target caption ( INLINEFORM1 ) and, under Cap2Img, only the associated image ( INLINEFORM2 ).

Answers:
(Layer Normalized Skip-Thoughts, ST-LN) BIBREF31, Cap2All, Cap2Cap, Cap2Img
Question: What dataset/corpus is this work evaluated over?

Evidence:
raw_evidence: Word embeddings INLINEFORM0 are initialized with GloVe BIBREF25 . The hidden dimension of each encoder and decoder LSTM cell ( INLINEFORM1 ) is 1024. We use Adam optimizer BIBREF26 and clip the gradients to between -5 and 5. Number of layers, dropout, and non-linearity for image feature prediction layers are 4, 0.3 and ReLU BIBREF27 respectively. Dimensionality of hidden attention layers ( INLINEFORM3 ) is 350 and number of attentions ( INLINEFORM4 ) is 30. We employ orthogonal initialization BIBREF28 for recurrent weights and xavier initialization BIBREF29 for all others. For the datasets, we use Karpathy and Fei-Fei's split for MS-COCO dataset BIBREF10 . Image features are prepared by extracting hidden representations at the final layer of ResNet-101 BIBREF30 . We evaluate sentence representation quality using SentEval BIBREF7 , BIBREF1 scripts. Mini-batch size is 128 and negative samples are prepared from remaining data samples in the same mini-batch.

Answers:
Karpathy and Fei-Fei's split for MS-COCO dataset BIBREF10
Question: Does the dataset contain non-English reviews?

Evidence:
raw_evidence: For this aspect and opinion term extraction task, we use tokenized and annotated hotel reviews on Airy Rooms provided by BIBREF1. The dataset consists of 5000 reviews in bahasa Indonesia. The dataset is divided into training and test sets of 4000 and 1000 reviews respectively. The label distribution of the tokens in BIO scheme can be seen in Table TABREF3. In addition, we also see this case as on entity level, i.e. ASPECT, SENTIMENT, and OTHER labels.

Answers:
Yes
Question: Does the paper report the performance of the method when is trained for more than 8 epochs?

Evidence:

Answers:
No
Question: What do the correlation demonstrate? 

Evidence:
raw_evidence: To investigate the extent to which the text obtained from the two platforms of Yahoo! Answers and Twitter reflect the true attributes of neighbourhoods, we first study whether there are significant, strong and meaningful correlations between the terms present in each corpus and the many neighbourhood attributes through the Pearson correlation coefficient $\rho $ . For each term in each corpus, we calculate the correlation between the term and all the selected demographic attributes. To do so, for each term, we define a vector with the dimension of the number of neighbourhoods. The value of each cell in this vector represents the normalised tf-idf value of the term for the corresponding neighbourhood. For each demographic attribute, we also define a vector with the dimension of the number of neighbourhoods. Each cell represents the value for the demographic attribute of the corresponding neighbourhood. We then calculate the Pearson correlation coefficient ( $\rho $ ) between these two vectors to measure the strength of the association between each term and each attribute.

Answers:
the extent to which the text obtained from the two platforms of Yahoo! Answers and Twitter reflect the true attributes of neighbourhoods
Question: On Twitter, do the demographic attributes and answers show more correlations than on Yahoo! Answers?

Evidence:
raw_evidence: As the table shows, terms extracted from Yahoo! Answers tend to be more related, in terms of the number of correlated terms, to attributes related to religion or ethnicity compared to terms from Twitter. However, for two particular attributes (i.e., Price and Buddhist), the number of correlated terms from Twitter is higher than the ones from Yahoo! Answers . These results collectively suggest that there is a wealth of terms, both in Yahoo! Answers and Twitter, which can be used to predict the population demographics.

Answers:
No
Question: How many demographic attributes they try to predict?

Evidence:
raw_evidence: Social media data has been used in many domains to find links to the real-world attributes. Data generated on QA platforms, however, has not been used in the past for predicting such attributes. In this paper, we use discussions on Yahoo! Answers QA platform to make predictions of demographic attribute of city neighbourhoods. Previous work in this domain has mainly focused on predicting the deprivation index of areas BIBREF4 . In this work, we look at a wide range of attributes and report prediction results on 62 demographic attributes. Additionally, work in urban prediction uses geolocation-based platforms such as Twitter. QA data that has been utilised in this paper does not include geolocation information. Utilising such data presents its own challenges.

Answers:
62
Question: What evaluation metrics do they use?

Evidence:
raw_evidence: We conduct our model on the large-scale dataset CNN/Daily Mail BIBREF19, BIBREF1, which is widely used in the task of abstractive document summarization with multi-sentences summaries. We use the scripts provided by BIBREF11 to obtain the non-anonymized version of the dataset without preprocessing to replace named entities. The dataset contains 287,226 training pairs, 13,368 validation pairs and 11,490 test pairs in total. We use the full-length ROUGE F1 and METEOR as our main evaluation metrics.

Answers:
ROUGE F1, METEOR
Question: How do they define local variance?

Evidence:
raw_evidence: As discussed in section SECREF1, the attention model putting most of attention weight on just a few parts of the input tends to achieve good performance. Mathematically, when only a small number of values are large, the shape of the distribution is sharp and the variance of the attention distribution is large. Drawing on the concept of variance in mathematics, local variance loss is defined as the reciprocal of its variance expecting the attention model to be able to focus on more salient parts. The standard variance calculation is based on the mean of the distribution. However, as previous work BIBREF15, BIBREF16 mentioned that the median value is more robust to outliers than the mean value, we use the median value to calculate the variance of the attention distribution. Thus, local variance loss can be calculated as:
where $\hat{\cdot }$ is a median operator and $\epsilon $ is utilized to avoid zero in the denominator.

Answers:
The reciprocal of the variance of the attention distribution