z-uo/qasper-squad huggingface dataset:
Paper: Question Answering by Reasoning Across Documents with Graph Convolutional Networks
In an offline step, we organize the content of each training instance in a graph connecting mentions of candidate answers within and across supporting documents. For a given query $q = \langle s, r, ? \rangle $ , we identify mentions in $S_q$ of the entities in $C_q \cup \lbrace s\rbrace $ and create one node per mention. This process is based on the following heuristic:
Question: How did they get relations between mentions?
Answer: Assign a value to the relation based on whether mentions occur in the same document, if mentions are identical, or if mentions are in the same coreference chain.
Paper: Question Answering by Reasoning Across Documents with Graph Convolutional Networks
Keeping in mind we want an efficient model, we encode words in supporting documents and in the query using only a pre-trained model for contextualized word representations rather than training our own encoder. Specifically, we use ELMo BIBREF20 , a pre-trained bi-directional language model that relies on character-based input representation. ELMo representations, differently from other pre-trained word-based models (e.g., word2vec BIBREF21 or GloVe BIBREF22 ), are contextualized since each token representation depends on the entire text excerpt (i.e., the whole sentence).
Question: How did they detect entity mentions?
Answer: Exact matches to the entity string and predictions from a coreference resolution system
Paper: Question Answering by Reasoning Across Documents with Graph Convolutional Networks
Our model uses a gated version of the original R-GCN propagation rule. At the first layer, all hidden node representation are initialized with the query-aware encodings $\mathbf {h}_i^{(0)} = \mathbf {\hat{x}}_i$ . Then, at each layer $0\le \ell \le L$ , the update message $\mathbf {u}_i^{(\ell )}$ to the $i$ th node is a sum of a transformation $f_s$ of the current node representation $\mathbf {h}^{(\ell )}_i$ and transformations of its neighbours: 
Question: What is the metric used with WIKIHOP?
Paper: Question Answering by Reasoning Across Documents with Graph Convolutional Networks
In this section, we compare our method against recent work as well as preforming an ablation study using the WikiHop dataset BIBREF0 . See Appendix "Implementation and experiments details" in the supplementary material for a description of the hyper-parameters of our model and training details.
Question: What performance does the Entity-GCN get on WIKIHOP?
Answer: During testing: 67.6 for single model without coreference, 66.4 for single model with coreference, 71.2 for ensemble of 5 models
Paper: Effectiveness of Data-Driven Induction of Semantic Spaces and Traditional Classifiers for Sarcasm Detection
Affective computing has raised a great deal of interest in the last years. Picard picard1995affective introduced it as a computing paradigm that relates to, arises from, or influences emotions, letting computers be both more effective in assisting humans and successful in making decisions.
Question: Do they evaluate only on English datasets?
Paper: Effectiveness of Data-Driven Induction of Semantic Spaces and Traditional Classifiers for Sarcasm Detection
The problem of sarcasm detection has been tackled using a wide range of supervised or semi-supervised techniques applied to corpora from different social media sources.
Question: What baseline models are used?
Paper: Effectiveness of Data-Driven Induction of Semantic Spaces and Traditional Classifiers for Sarcasm Detection
We focused our research on the role that fully data-driven models can play in detecting sarcasm. To reach this goal, we exploited the Latent Semantic Analysis paradigm both in its traditional formulation (Landauer et al. 1998) and by using the Truncated Singular Value Decomposition (T-SVD) as a statistical estimator as shown in Pilato et al. pilato2015tsvd. We have chosen to use the LSA paradigm to exploit a well-known and well-founded approach for inducing semantic spaces that have been effectively used in natural language understanding, cognitive modeling, speech recognition, smart indexing, and other statistical natural language processing problems. The sub-symbolic codings of documents obtained by the aforementioned LSA-based approaches are then used as inputs by a set of classifiers to evaluate the differences of performances obtained by using different machine learning approaches and testing them on different sarcasm-detection datasets.
Question: What classical machine learning algorithms are used?
Paper: Effectiveness of Data-Driven Induction of Semantic Spaces and Traditional Classifiers for Sarcasm Detection
The first step of preprocessing for texts is the tokenization using spaces, punctuation and special characters (e.g., $, , @) as separators. Thus one token is a sequence of alphanumeric characters or of punctuation symbols. The set of all the extracted tokens constitutes a “vocabulary” named INLINEFORM0 .
Question: What are the different methods used for different corpora?
Paper: Effectiveness of Data-Driven Induction of Semantic Spaces and Traditional Classifiers for Sarcasm Detection
The matrix INLINEFORM0 is used and further processed to induce proper Semantic Spaces where terms and documents can be mapped. To generate these semantic spaces, we have used both the traditional LSA algorithm (Deerwester et al. 1990, Landauer et al. 1998) and the approach which uses T-SVD as a statistical estimator as proposed in Pilato et al. pilato2015tsvd. For the sake of brevity, we call this last approach Statistical LSA to differentiate it by the Traditional LSA. It is worthwhile to point out that, in the Latent Semantic Analysis paradigm (i.e., both “general” and “statistical”), the corpus used for building the semantic space plays a key role in performances. As a matter of fact, large and heterogeneous corpora may give more noise or too much specific information from a single domain, decreasing the accuracy of the induced models BIBREF36 .
Question: In which domains is sarcasm conveyed in different ways?
Paper: Multi-attention Recurrent Network for Human Communication Comprehension
Humans communicate using a highly complex structure of multimodal signals. We employ three modalities in a coordinated manner to convey our intentions: language modality (words, phrases and sentences), vision modality (gestures and expressions), and acoustic modality (paralinguistics and changes in vocal tones) BIBREF0 . Understanding this multimodal communication is natural for humans; we do it subconsciously in the cerebrum of our brains everyday. However, giving Artificial Intelligence (AI) the capability to understand this form of communication the same way humans do, by incorporating all involved modalities, is a fundamental research challenge. Giving AI the capability to understand human communication narrows the gap in computers' understanding of humans and opens new horizons for the creation of many intelligent entities.
Question: What modalities are being used in different datasets?
Paper: Multi-attention Recurrent Network for Human Communication Comprehension
Modeling multimodal human communication has been studied previously. Past approaches can be categorized as follows:
Question: What is the difference between Long-short Term Hybrid Memory and LSTMs?
Paper: The Effect of Context on Metaphor Paraphrase Aptness Judgments
A metaphor is a way of forcing the normal boundaries of a word's meaning in order to better express an experience, a concept or an idea. To a native speaker's ear some metaphors sound more conventional (like the usage of the words ear and sound in this sentence), others more original. This is not the only dimension along which to judge a metaphor. One of the most important qualities of a metaphor is its appropriateness, its aptness: how good is a metaphor for conveying a given experience or concept. While a metaphor's degree of conventionality can be measured through probabilistic methods, like language models, it is harder to represent its aptness. BIBREF0 define aptness as “the extent to which a comparison captures important features of the topic".
Question: Do they report results only on English data?
Paper: The Effect of Context on Metaphor Paraphrase Aptness Judgments
 BIBREF3 have recently produced a dataset of paraphrases containing metaphors designed to allow both supervised binary classification and gradient ranking. This dataset contains several pairs of sentences, where in each pair the first sentence contains a metaphor, and the second is a literal paraphrase candidate.
Question: What provisional explanation do the authors give for the impact of document context?
Paper: The Effect of Context on Metaphor Paraphrase Aptness Judgments
We found a Pearson correlation of 0.81 between the in-context and out-of-context mean human paraphrase ratings for our two corpora. This correlation is virtually identical to the one that BIBREF5 report for mean acceptability ratings of out-of-context to in-context sentences in their crowd source experiment. It is interesting that a relatively high level of ranking correspondence should occur in mean judgments for sentences presented out of and within document contexts, for two entirely distinct tasks.
Question: What document context was added?
Answer: Preceding and following sentence of each metaphor and paraphrase are added as document context
Paper: The Effect of Context on Metaphor Paraphrase Aptness Judgments
We use the DNN model described in BIBREF3 to predict aptness judgments for in-context paraphrase pairs. It has three main components:
Question: What were the results of the first experiment?
Answer: Best performance achieved is 0.72 F1 score
Paper: Drug-drug Interaction Extraction via Recurrent Neural Network with Multiple Attention Layers
Drug-drug interaction (DDI) is a situation when one drug increases or decreases the effect of another drug BIBREF0 . Adverse drug reactions may cause severe side effect, if two or more medicines were taken and their DDI were not investigated in detail. DDI is a common cause of illness, even a cause of death BIBREF1 . Thus, DDI databases for clinical medication decisions are proposed by some researchers. These databases such as SFINX BIBREF2 , KEGG BIBREF3 , CredibleMeds BIBREF4 help physicians and pharmacists avoid most adverse drug reactions.
Question: How big is the evaluated dataset?
Paper: Drug-drug Interaction Extraction via Recurrent Neural Network with Multiple Attention Layers
In DDI extraction task, NLP methods or machine learning approaches are proposed by most of the work. Chowdhury BIBREF14 and Thomas et al. BIBREF11 proposed methods that use linguistic phenomenons and two-stage SVM to classify DDIs. FBK-irst BIBREF10 is a follow-on work which applies kernel method to the existing model and outperforms it.
Question: By how much does their model outperform existing methods?
Answer: Answer with content missing: (Table II) Proposed model has F1 score of  0.7220 compared to 0.7148 best state-state-of-the-art result.
Paper: Drug-drug Interaction Extraction via Recurrent Neural Network with Multiple Attention Layers
In this section, we present our bidirectional recurrent neural network with multiple attention layer model. The overview of our architecture is shown in figure FIGREF15 . For a given instance, which describes the details about two or more drugs, the model represents each word as a vector in embedding layer. Then the bidirectional RNN layer generates a sentence matrix, each column vector in which is the semantic representation of the corresponding word. The word level attention layer transforms the sentence matrix to vector representation. Then sentence level attention layer generates final representation for the instance by combining several relevant sentences in view of the fact that these sentences have the same drug pair. Followed by a softmax classifier, the model classifies the drug pair in the given instance as specific DDI.
Question: What is the performance of their model?
Answer: Answer with content missing: (Table II) Proposed model has F1 score of  0.7220.
Paper: Drug-drug Interaction Extraction via Recurrent Neural Network with Multiple Attention Layers
The DDI corpus contains thousands of XML files, each of which are constructed by several records. For a sentence containing INLINEFORM0 drugs, there are INLINEFORM1 drug pairs. We replace the interested two drugs with “drug1” and “drug2” while the other drugs are replaced by “durg0”, as in BIBREF9 did. This step is called drug blinding. For example, the sentence in figure FIGREF5 generates 3 instances after drug blinding: “drug1: an increased risk of hepatitis has been reported to result from combined use of drug2 and drug0”, “drug1: an increased risk of hepatitis has been reported to result from combined use of drug0 and drug2”, “drug0: an increased risk of hepatitis has been reported to result from combined use of drug1 and drug2”. The drug blinded sentences are the instances that are fed to our model.
Question: What are the existing methods mentioned in the paper?
Paper: Compositional Neural Machine Translation by Removing the Lexicon from Syntax
Studies of Broca's and Wernicke's aphasia provide evidence that our brains understand an utterance by creating separate representations for word meaning and word arrangement BIBREF0. There is a related thesis about human language, present across many theories of semantics, which is that syntactic categories are partially agnostic to the identity of words BIBREF1. This regularity in how humans derive meaning from an utterance is applicable to the task of natural language translation. This is because, by definition, translation necessitates the creation of a meaning representation for an input. According to the cognitive and neural imperative, we introduce new units to regularize an artificial neural encoder and decoder BIBREF2. These are called the Lexicon and Lexicon-Adversary units (collectively, LLA). Tests are done on a diagnostic task, and naturalistic tasks including semantic parsing, syntactic parsing, and English to Mandarin Chinese translation. We evaluate a Long Short-Term Memory (LSTM) BIBREF3 encoder and decoder, with and without the LLA units, and show that the LLA version achieves superior translation performance. In addition, we examine our model's weights, and its performance when some of its neurons are damaged. We find that the model exhibits the knowledge and the lack thereof expected of a Broca's aphasic BIBREF0 when one module's weights are corrupted. It also exhibits that expected of a Wernicke's aphasic BIBREF0 when another module's weights are corrupted.
Question: Does having constrained neural units imply word meanings are fixed across different context?
Paper: Compositional Neural Machine Translation by Removing the Lexicon from Syntax
BIBREF0 BIBREF0 showed that Broca's aphasics were able to understand “the apple that the boy is eating is red” with significantly higher accuracy than “the cow that the monkey is scaring is yellow,” along with similar pairs. The critical difference between these sentences is that, due to semantic constraints from the words, the first can be understood if it is presented as a set of words. The second cannot. This experiment provides evidence that the rest of the language neurons in the brain (largely Wernicke's area) can yield an understanding of word meanings but not how words are arranged. This also suggests that Broca's area builds a representation of the syntax.
Question: Do they perform a quantitative analysis of their model displaying knowledge distortions?
Paper: Compositional Neural Machine Translation by Removing the Lexicon from Syntax
A tenet of generative grammar theories is that different words can share the same syntactic category BIBREF1. It is possible, for example, to know that the syntax for an utterance is a noun phrase that is composed of a determiner and a noun, followed by a verb phrase that is composed of a verb. One can know this without knowing the words. This also means that there are aspects of a word's meaning that the syntax does not determine; by definition, these aspects are invariant to word arrangement.
Question: How do they damage different neural modules?
Answer: Damage to neural modules is done by randomly initializing their weights, causing the loss of all learned information.
Paper: Compositional Neural Machine Translation by Removing the Lexicon from Syntax
In a natural language translation setting, suppose that an input word corresponds to a set of output tokens independently of its context. Even though this information might be useful to determine the syntax of the input utterance in the first place, the syntax does not determine this knowledge at all (by supposition). So, we can impose the constraint that our model's representation of the input's syntax cannot contain this context-invariant information. This regularization is strictly preferable to allowing all aspects of word meaning to propagate into the input's syntax representation. Without such a constraint, all inputs could, in principle, be given their own syntactic categories. This scenario is refuted by cognitive and neural theories. We incorporate the regularization with neural units that can separate representations of word meaning and arrangement.
Question: Which weights from their model do they analyze?
Paper: Mind Your Language: Abuse and Offense Detection for Code-Switched Languages
With the penetration of internet among masses, the content being posted on social media channels has uptaken. Specifically, in the Indian subcontinent, number of Internet users has crossed 500 mi, and is rising rapidly due to inexpensive data. With this rise, comes the problem of hate speech, offensive and abusive posts on social media. Although there are many previous works which deal with Hindi and English hate speech (the top two languages in India), but very few on the code-switched version (Hinglish) of the two BIBREF0 . This is partially due to the following reasons: (i) Hinglish consists of no-fixed grammar and vocabulary. It derives a part of its semantics from Devnagari and another part from the Roman script. (ii) Hinglish speech and written text consists of a concoction of words spoken in Hindi as well as English, but written in the Roman script. This makes the spellings variable and dependent on the writer of the text. Hence code-switched languages present tough challenges in terms of parsing and getting the meaning out of the text. For instance, the sentence, “Modiji foreign yatra par hai”, is in the Hinglish language. Somewhat correct translation of this would be, “Mr. Modi is on a foriegn tour”. However, even this translation has some flaws due to no direct translation available for the word ji, which is used to show respect. Verbatim translation would lead to “Mr. Modi foreign tour on is”. Moreover, the word yatra here, can have phonetic variations, which would result in multiple spellings of the word as yatra, yaatra, yaatraa, etc. Also, the problem of hate speech has been rising in India, and according to the policies of the government and the various social networks, one is not allowed to misuse his right to speech to abuse some other community or religion. Due to the various difficulties associated with the Hinglish language, it is challenging to automatically detect and ban such kind of speech.
Question: Do all the instances contain code-switching?
Paper: Mind Your Language: Abuse and Offense Detection for Code-Switched Languages
Our methodology primarily consists of these steps: Pre-processing of the dataset, training of word embeddings, training of the classifier model and then using that on HEOT dataset.
Question: What embeddings do they use?
Paper: Mind Your Language: Abuse and Offense Detection for Code-Switched Languages
In this work, we use the datasets released by BIBREF1 and HEOT dataset provided by BIBREF0 . The datasets obtained pass through these steps of processing: (i) Removal of punctuatios, stopwords, URLs, numbers, emoticons, etc. This was then followed by transliteration using the Xlit-Crowd conversion dictionary and translation of each word to English using Hindi to English dictionary. To deal with the spelling variations, we manually added some common variations of popular Hinglish words. Final dictionary comprised of 7200 word pairs. Additionally, to deal with profane words, which are not present in Xlit-Crowd, we had to make a profanity dictionary (with 209 profane words) as well. Table TABREF3 gives some examples from the dictionary.
Question: Do they perform some annotation?
Paper: Mind Your Language: Abuse and Offense Detection for Code-Switched Languages
We tried Glove BIBREF2 and Twitter word2vec BIBREF3 code for training embeddings for the processed tweets. The embeddings were trained on both the datasets provided by BIBREF1 and HEOT. These embeddings help to learn distributed representations of tweets. After experimentation, we kept the size of embeddings fixed to 100.
Question: Do they use dropout?
Paper: Mind Your Language: Abuse and Offense Detection for Code-Switched Languages
Both the HEOT and BIBREF1 datasets contain tweets which are annotated in three categories: offensive, abusive and none (or benign). Some examples from the dataset are shown in Table TABREF4 . We use a LSTM based classifier model for training our model to classify these tweets into these three categories. An overview of the model is given in the Figure FIGREF12 . The model consists of one layer of LSTM followed by three dense layers. The LSTM layer uses a dropout value of 0.2. Categorical crossentropy loss was used for the last layer due to the presence of multiple classes. We use Adam optimizer along with L2 regularisation to prevent overfitting. As indicated by the Figure FIGREF12 , the model was initially trained on the dataset provided by BIBREF1 , and then re-trained on the HEOT dataset so as to benefit from the transfer of learned features in the last stage. The model hyperparameters were experimentally selected by trying out a large number of combinations through grid search.
Question: What definition of hate speech do they use?
Paper: $\rho$-hot Lexicon Embedding-based Two-level LSTM for Sentiment Analysis
Text is important in many artificial intelligence applications. Among various text mining techniques, sentiment analysis is a key component in applications such as public opinion monitoring and comparative analysis. Sentiment analysis can be divided into three problems according to input texts, namely, sentence, paragraph, and document levels. This study focuses on sentence and paragraph levels.
Question: What are the other models they compare to?
Paper: $\rho$-hot Lexicon Embedding-based Two-level LSTM for Sentiment Analysis
Sentiment analysis aims to predict the sentiment polarity of an input text sample. Sentiment polarity can be divided into negative, neutral, and positive in many applications.
Question: What is the agreement value for each dataset?
Answer: 1
Paper: $\rho$-hot Lexicon Embedding-based Two-level LSTM for Sentiment Analysis
Lexicon-based methods are actually in implemented in an unsupervised manner. They infer the sentiment categories of input texts on the basis of polar and privative words. The primary advantage of these methods is that they do not require labeled training data. The key of lexicon-based methods is the lexical resource construction, which maps words into a category (positive, negative, neutral, or privative). Senti-WordNet BIBREF11 is a lexical resource for English text sentiment classification. For Chinese texts, Senti-HowNet is usually used.
Question: How many annotators participated?