z-uo/qasper-squad huggingface dataset:
Paper: BERTQA -- Attention on Steroids
We first focused on directed coattention via context to query and query to context attention as discussed in BIDAF BIBREF9. We then implemented localized feature extraction by 1D convolutions to add local information to coattention based on the QANET architecture BIBREF10. Subsequently, we experimented with different types of skip connections to inject BERT embedding information back into our modified network. We then applied what we learned using the base BERT model to the large BERT model. Finally, we performed hyperparameter tuning by adjusting the number of coattention blocks, the batch size, and the number of epochs trained and ensembled our three best networks. Each part of the project is discussed further in the subsections below.
Question: How much F1 was improved after adding skip connections?
Answer: Simple Skip improves F1 from 74.34 to 74.81
Transformer Skip improes F1 from 74.34 to 74.95
Paper: Non-Parametric Adaptation for Neural Machine Translation
Over the last few years, neural sequence to sequence models BIBREF0 , BIBREF1 , BIBREF2 have revolutionized the field of machine translation by significantly improving translation quality over their phrase based counterparts BIBREF3 , BIBREF4 , BIBREF5 . With more gains arising from continued research on new neural network architectures and accompanying training techniques BIBREF6 , BIBREF7 , BIBREF8 , NMT researchers, both in industry and academia, have doubled down on their ability to train high capacity models on large corpora with gradient based optimization.
Question: Where do they retrieve neighbor n-grams from in their approach?
Paper: Non-Parametric Adaptation for Neural Machine Translation
Standard approaches for Neural Machine Translation rely on seq2seq architectures BIBREF0 , BIBREF1 , where given a source sequence INLINEFORM0 and a target sequence INLINEFORM1 , the goal is to model the probability distribution, INLINEFORM2 .
Question: To which systems do they compare their results against?
Paper: Non-Parametric Adaptation for Neural Machine Translation
Existing approaches have proposed using off the shelf search engines for the retrieval stage. However, our objective differs from traditional information retrieval, since the goal of retrieval in semi-parametric NMT is to find neighbors which might improve translation performance, which might not correlate with maximizing sentence similarity.
Question: Does their combination of a non-parametric retrieval and neural network get trained end-to-end?
Paper: Non-Parametric Adaptation for Neural Machine Translation
To incorporate the retrieved neighbors, INLINEFORM0 , within the NMT model, we first encode them using Transformer layers, as described in subsection UID12 . This encoded memory is then used within the decoder via an attention mechanism, as described in subsection UID15 .
Question: Which similarity measure do they use in their n-gram retrieval approach?
Paper: Multichannel Variable-Size Convolution for Sentence Classification
Different sentence classification tasks are crucial for many Natural Language Processing (NLP) applications. Natural language sentences have complicated structures, both sequential and hierarchical, that are essential for understanding them. In addition, how to decode and compose the features of component units, including single words and variable-size phrases, is central to the sentence classification problem.
Question: Where is MVCNN pertained?
Paper: Multichannel Variable-Size Convolution for Sentence Classification
Much prior work has exploited deep neural networks to model sentences.
Question: How much gain does the model achieve with pretraining MVCNN?
Answer: 0.8 points on Binary; 0.7 points on Fine-Grained; 0.6 points on Senti140; 0.7 points on Subj
Paper: Multichannel Variable-Size Convolution for Sentence Classification
We now describe the architecture of our model MVCNN, illustrated in Figure 1 .
Question: What are the effects of extracting features of multigranular phrases?
Paper: Multichannel Variable-Size Convolution for Sentence Classification
This part introduces two training tricks that enhance the performance of MVCNN in practice.
Question: What are the effects of diverse versions of pertained word embeddings?
Paper: Multichannel Variable-Size Convolution for Sentence Classification
We test the network on four classification tasks. We begin by specifying aspects of the implementation and the training of the network. We then report the results of the experiments.
Question: How is MVCNN compared to CNN?
Paper: Constructing a Natural Language Inference Dataset using Generative Neural Networks
The challenge in Natural Language Inference (NLI), also known as Recognizing Textual Entailment (RTE), is to correctly decide whether a sentence (referred to as a premise) entails or contradicts or is neutral in respect to another sentence (a hypothesis). This classification task requires various natural language comprehension skills. In this paper, we are focused on the following natural language generation task based on NLI. Given the premise the goal is to generate a stream of hypotheses that comply with the label (entailment, contradiction or neutral). In addition to reading capabilities this task also requires language generation capabilities.
Question: What is the highest accuracy score achieved?
Answer: 82.0%
Paper: Constructing a Natural Language Inference Dataset using Generative Neural Networks
NLI has been the focal point of Recognizing Textual Entailment (RTE) Challenges, where the goal is to determine if the premise entails the hypothesis or not. The proposed approaches for RTE include bag-of-words matching approach BIBREF14 , matching predicate argument structure approach BIBREF15 and logical inference approach BIBREF16 , BIBREF17 . Another rule-based inference approach was proposed by BIBREF18 . This approach allows generation of new hypotheses by transforming parse trees of the premise while maintaining entailment. BIBREF19 proposes an approach for constructing training datasets by extracting sentences from news articles that tend to be in an entailment relationship.
Question: What is the size range of the datasets?
Paper: BERT-Based Arabic Social Media Author Profiling
The proliferation of social media has made it possible to collect user data in unprecedented ways. These data can come in the form of usage and behavior (e.g., who likes what on Facebook), network (e.g., who follows a given user on Instagram), and content (e.g., what people post to Twitter). Availability of such data have made it possible to make discoveries about individuals and communities, mobilizing social and psychological research and employing natural language processing methods. In this work, we focus on predicting social media user age, dialect, and gender based on posted language. More specifically, we use the total of 100 tweets from each manually-labeled user to predict each of these attributes. Our dataset comes from the Arabic author profiling and deception detection shared task (APDA) BIBREF0. We focus on building simple models using pre-trained bidirectional encoders from transformers(BERT) BIBREF1 under various data conditions. Our results show (1) the utility of augmenting training data, and (2) the benefit of using majority votes from our simple classifiers.
Question: Does the paper report F1-scores for the age and language variety tasks?
Paper: BERT-Based Arabic Social Media Author Profiling
For the purpose of our experiments, we use data released by the APDA shared task organizers. The dataset is divided into train and test by organizers. The training set is distributed with labels for the three tasks of age, dialect, and gender. Following the standard shared tasks set up, the test set is distributed without labels and participants were expected to submit their predictions on test. The shared task predictions are expected by organizers at the level of users. The distribution has 100 tweets for each user, and so each tweet is distributed with a corresponding user id. As such, in total, the distributed training data has 2,250 users, contributing a total of 225,000 tweets. The official task test set contains 720,00 tweets posted by 720 users. For our experiments, we split the training data released by organizers into 90% TRAIN set (202,500 tweets from 2,025 users) and 10% DEV set (22,500 tweets from 225 users). The age task labels come from the tagset {under-25, between-25 and 34, above-35}. For dialects, the data are labeled with 15 classes, from the set {Algeria, Egypt, Iraq, Kuwait, Lebanon-Syria, Lybia, Morocco, Oman, Palestine-Jordan, Qatar, Saudi Arabia, Sudan, Tunisia, UAE, Yemen}. The gender task involves binary labels from the set {male, female}.
Question: Are the models compared to some baseline models?
Paper: BERT-Based Arabic Social Media Author Profiling
As explained earlier, the shared task is set up at the user level where the age, dialect, and gender of each user are the required predictions. In our experiments, we first model the task at the tweet level and then port these predictions at the user level. For our core modelling, we fine-tune BERT on the shared task data. We also introduce an additional in-house dataset labeled with dialect and gender tags to the task as we will explain below. As a baseline, we use a small gated recurrent units (GRU) model. We now introduce our tweet-level models.
Question: What are the in-house data employed?
Paper: BERT-Based Arabic Social Media Author Profiling
Our baseline is a GRU network for each of the three tasks. We use the same network architecture across the 3 tasks. For each network, the network contains a layer unidirectional GRU, with 500 units and an output linear layer. The network is trained end-to-end. Our input embedding layer is initialized with a standard normal distribution, with $\mu =0$, and $\sigma =1$, i.e., $W \sim N(0,1)$. We use a maximum sequence length of 50 tokens, and choose an arbitrary vocabulary size of 100,000 types, where we use the 100,000 most frequent words in TRAIN. To avoid over-fitting, we use dropout BIBREF2 with a rate of 0.5 on the hidden layer. For the training, we use the Adam BIBREF3 optimizer with a fixed learning rate of $1e-3$. We employ batch training with a batch size of 32 for this model. We train the network for 15 epochs and save the model at the end of each epoch, choosing the model that performs highest accuracy on DEV as our best model. We present our best result on DEV in Table TABREF7. We report all our results using accuracy. Our best model obtains 42.48% for age, 37.50% for dialect, and 57.81% for gender. All models obtains best results with 2 epochs.
Question: What are the three datasets used in the paper?
Answer: Data released for APDA shared task contains 3 datasets.
Paper: Data Innovation for International Development: An overview of natural language processing for qualitative data analysis
Practitioners in the development sector have long recognized the potential of qualitative data to inform programming and gain a better understanding of values, behaviors and attitudes of people and communities affected by their efforts. Some organizations mainly rely on interview or focus group data, some also consider policy documents and reports, and others have started tapping into social media data. Regardless of where the data comes from, analyzing it in a systematic way to inform quick decision-making poses challenges, in terms of high costs, time, or expertise required.
Question: What are the potentials risks of this approach?
Paper: Data Innovation for International Development: An overview of natural language processing for qualitative data analysis
There are two broad approaches to NLP - supervised learning and unsupervised learning BIBREF0 . Supervised learning assumes that an outcome variable is known and an algorithm is used to predict the correct variable. Classifying email as spam based on how the user has classified previous mail is a classic example. In social science, we may want to predict voting behavior of a legislator with the goal of inferring ideological positions from such behavior. In development, interest may center around characteristics that predict successful completion of a training program based on a beneficiary's previous experience or demographic characteristics.
Question: What elements of natural language processing are proposed to analyze qualitative data?
Paper: Empirical Study on Detecting Controversy in Social Media
The financial performance of a corporation is correlated with its social responsibility such as whether their products are environmentally friendly, manufacturing safety procedures protect against accidents, or they use child labors in its third world country factories. Consumers care about these factors when making purchasing decisions in the supermarkets and investors integrate environmental, social and governance factors, known as ESG, in their investment decision-making. It has been shown that corporations financial results have a positive correlation with their sustainability business model and the ESG investment methodology can help reduce portfolio risk and generate competitive returns. However, one barrier for ESG evaluation is the lack of relatively complete and centralized information source. Currently, ESG analysts leverage financial reports to collect the the necessary data for proper evaluation such as greenhouse gas emissions or discrimination lawsuits, but this data is inconsistent and latent. In this study, we consider social media a crowdsourcing data feed to be a new data source for this task.
Question: How does the method measure the impact of the event on market prices?
Paper: Empirical Study on Detecting Controversy in Social Media
There have been a few studies on assessing sustainability of entities. The UN Commission on Sustainable Development (CSD) published a list of about 140 indicators on various dimensions of sustainability BIBREF0 . In BIBREF1 , Singh et al. reviewed various methodologies, indicators, and indices on sustainability assessment, which includes environmental and social domains. All the data, on which the assessments were conducted, mentioned in their works are processed datasets, and some of them are collected from company annual reports and publications, newspaper clips, and management interviews. They stated that the large number of indicators or indices raises the need of data collection. Our work uses the social media data as a new alternative data source to complement the traditional data collection.
Question: How is sentiment polarity measured?
Paper: Humor Detection: A Transformer Gets the Last Laugh
Recent advances in natural language processing and neural network architecture have allowed for widespread application of these methods in Text Summarization BIBREF0, Natural Language Generation BIBREF1, and Text Classification BIBREF2. Such advances have enabled scientists to study common language practices. One such area, humor, has garnered focus in classification BIBREF3, BIBREF4, generation BIBREF5, BIBREF6, and in social media BIBREF7.
Question: Which part of the joke is more important in humor?
Paper: Humor Detection: A Transformer Gets the Last Laugh
In the related work of joke identification, we find a myriad of methods employed over the years: statistical and N-gram analysis BIBREF13, Regression Trees BIBREF14, Word2Vec combined with K-NN Human Centric Features BIBREF15, and Convolutional Neural Networks BIBREF4.
Question: What is improvement in accuracy for short Jokes in relation other types of jokes?
Answer: It had the highest accuracy comparing to all datasets 0.986% and It had the highest improvement comparing to previous methods on the same dataset by 8%
Paper: Humor Detection: A Transformer Gets the Last Laugh
We gathered jokes from a variety of sources, each covering a different type of humor. These datasets include jokes of multiple sentences (the Short Jokes dataset), jokes with only one sentence (the Puns dataset), and more mixed jokes (the Reddit dataset). We have made our code and datasets open source for others to use.
Question: What kind of humor they have evaluated?
Paper: Humor Detection: A Transformer Gets the Last Laugh
Our Reddit data was gathered using Reddit's public API, collecting the most recent jokes. Every time the scraper ran, it also updated the upvote score of the previously gathered jokes. This data collection occurred every hour through the months of March and April 2019. Since the data was already split into body and punchline sections from Reddit, we created separate datasets containing the body of the joke exclusively and the punchline of the joke exclusively. Additionally, we created a dataset that combined the body and punchline together.
Question: How they evaluate if joke is humorous or not?
Paper: Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments
The effectiveness and ubiquity of pretrained sentence embeddings for natural language understanding has grown dramatically in recent years. Recent sentence encoders like OpenAI's Generative Pretrained Transformer BIBREF3 and BERT BIBREF2 achieve the state of the art on the GLUE benchmark BIBREF4 . Among the GLUE tasks, these state-of-the-art systems make their greatest gains on the acceptability task with the Corpus of Linguistic Acceptability BIBREF0 . CoLA contains example sentences from linguistics publications labeled by experts for grammatical acceptability, and written to show subtle grammatical features. Because minimal syntactic differences can separate acceptable sentences from unacceptable ones (What did Bo write a book about? / *What was a book about written by Bo?), and acceptability classifiers are more reliable when trained on GPT and BERT than on recurrent models, it stands to reason that GPT and BERT have better implicit knowledge of syntactic features relevant to acceptability.
Question: Do they report results only on English data?
Paper: Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments
We introduce a grammatically annotated version of the entire CoLA development set to facilitate detailed error analysis of acceptability classifiers. These 1043 sentences are expert-labeled for the presence of 63 minor grammatical features organized into 15 major features. Each minor feature belongs to a single major feature. A sentence belongs to a major feature if it belongs to one or more of the relevant minor features. The Appendix includes descriptions of each feature along with examples and the criteria used for annotation.
Question: Do the authors have a hypothesis as to why morphological agreement is hardly learned by any model?
Paper: Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments
The sentences were annotated manually by one of the authors, who is a PhD student with extensive training in formal linguistics. The features were developed in a trial stage, in which the annotator performed a similar annotation with different annotation schema for several hundred sentences from CoLA not belonging to the development set.
Question: Which models are best for learning long-distance movement?
Paper: Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments
Here we briefly summarize the feature set in order of the major features. Many of these constructions are well-studied in syntax, and further background can be found in textbooks such as adger2003core and sportiche2013introduction.
Question: Where does the data in CoLA come from?
Paper: Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments
We wish to emphasize that these features are overlapping and in many cases are correlated, thus not all results from using this analysis set will be independent. We analyzed the pairwise Matthews Correlation Coefficient BIBREF17 of the 63 minor features (giving 1953 pairs), and of the 15 major features (giving 105 pairs). MCC is a special case of Pearson's INLINEFORM0 for Boolean variables. These results are summarized in Table TABREF25 . Regarding the minor features, 60 pairs had a correlation of 0.2 or greater, 17 had a correlation of 0.4 or greater, and 6 had a correlation of 0.6 or greater. None had an anti-correlation of greater magnitude than -0.17. Turning to the major features, 6 pairs had a correlation of 0.2 or greater, and 2 had an anti-correlation of greater magnitude than -0.2.
Question: How is the CoLA grammatically annotated?
Paper: Question Answering by Reasoning Across Documents with Graph Convolutional Networks
The long-standing goal of natural language understanding is the development of systems which can acquire knowledge from text collections. Fresh interest in reading comprehension tasks was sparked by the availability of large-scale datasets, such as SQuAD BIBREF1 and CNN/Daily Mail BIBREF2 , enabling end-to-end training of neural models BIBREF3 , BIBREF4 , BIBREF5 . These systems, given a text and a question, need to answer the query relying on the given document. Recently, it has been observed that most questions in these datasets do not require reasoning across the document, but they can be answered relying on information contained in a single sentence BIBREF6 . The last generation of large-scale reading comprehension datasets, such as a NarrativeQA BIBREF7 , TriviaQA BIBREF8 , and RACE BIBREF9 , have been created in such a way as to address this shortcoming and to ensure that systems relying only on local information cannot achieve competitive performance.
Question: What baseline did they compare Entity-GCN to?
Answer: Human, FastQA, BiDAF, Coref-GRU, MHPGM, Weaver / Jenga, MHQA-GRN
Paper: Question Answering by Reasoning Across Documents with Graph Convolutional Networks
In this section we explain our method. We first introduce the dataset we focus on, WikiHop by BIBREF0 , as well as the task abstraction. We then present the building blocks that make up our Entity-GCN model, namely, an entity graph used to relate mentions to entities within and across documents, a document encoder used to obtain representations of mentions in context, and a relational graph convolutional network that propagates information through the entity graph.
Question: How many documents at a time can Entity-GCN handle?
Paper: Question Answering by Reasoning Across Documents with Graph Convolutional Networks
The WikiHop dataset comprises of tuples $\langle q, S_q, C_q, a^\star \rangle $ where: $q$ is a query/question, $S_q$ is a set of supporting documents, $C_q$ is a set of candidate answers (all of which are entities mentioned in $S_q$ ), and $a^\star \in C_q$ is the entity that correctly answers the question. WikiHop is assembled assuming that there exists a corpus and a knowledge base (KB) related to each other. The KB contains triples $\langle s, r, o \rangle $ where $s$ is a subject entity, $o$ an object entity, and $r$ a unidirectional relation between them. BIBREF0 used Wikipedia as corpus and Wikidata BIBREF15 as KB. The KB is only used for constructing WikiHop: BIBREF0 retrieved the supporting documents $q$0 from the corpus looking at mentions of subject and object entities in the text. Note that the set $q$1 (not the KB) is provided to the QA system, and not all of the supporting documents are relevant for the query but some of them act as distractors. Queries, on the other hand, are not expressed in natural language, but instead consist of tuples $q$2 where the object entity is unknown and it has to be inferred by reading the support documents. Therefore, answering a query corresponds to finding the entity $q$3 that is the object of a tuple in the KB with subject $q$4 and relation $q$5 among the provided set of candidate answers $q$6 .
Question: Did they use a relation extraction method to construct the edges in the graph?