z-uo/qasper-squad huggingface dataset:
Paper: Improving Visually Grounded Sentence Representations with Self-Attention
Sentence Representations. Since the inception of word embeddings BIBREF3 , extensive work have emerged for larger semantic units, such as sentences and paragraphs. These works range from deep neural models BIBREF4 to log-bilinear models BIBREF5 , BIBREF6 . A recent work proposed using supervised learning of a specific task as a leverage to obtain general sentence representation BIBREF7 .
Question: What baselines are the proposed method compared against?
Paper: Improving Visually Grounded Sentence Representations with Self-Attention
Given a data sample INLINEFORM0 , where INLINEFORM1 is the source caption, INLINEFORM2 is the target caption, and INLINEFORM3 is the hidden representation of the image, our goal is to predict INLINEFORM4 and INLINEFORM5 with INLINEFORM6 , and the hidden representation in the middle serves as the general sentence representation.
Question: What dataset/corpus is this work evaluated over?
Paper: Categorization of Semantic Roles for Dictionary Definitions
This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/ Many natural language understanding tasks such as Text Entailment and Question Answering systems are dependent on the interpretation of the semantic relationships between terms. The challenge on the construction of robust semantic interpretation models is to provide a model which is both comprehensive (capture a large set of semantic relations) and fine-grained. While semantic relations (high-level binary predicates which express relationships between words) can serve as a semantic interpretation model, in many cases, the relationship between words cannot be fully articulated as a single semantic relation, depending on a contextualization that involves one or more target words, their corresponding semantic relationships and associated logical operators (e.g. modality, functional operators).
Question: How many roles are proposed?
Answer: 12
Paper: Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communities
Technology pervades all aspects of society and continues to change the way people access and share information, learn and educate, as well as provide and access services. Language is the main medium through which such transformational technology can be integrated into the socioeconomic processes of a community. Natural Language Processing (NLP) and Speech systems, therefore, break down barriers and enable users and whole communities with easy access to information and services. However, the current trend in building language technology is designed to work on languages with very high resources in terms of data and infrastructure.
Question: What language technologies have been introduced in the past?
Answer: - Font & Keyboard
- Speech-to-Text
- Text-to-Speech
- Text Prediction
- Spell Checker
- Grammar Checker
- Text Search
- Machine Translation
- Voice to Text Search
- Voice to Speech Search
Paper: Aspect and Opinion Term Extraction for Aspect Based Sentiment Analysis of Hotel Reviews Using Transfer Learning
Sentiment analysis BIBREF2 in review text usually consists of multiple aspects. For instance, the following review talks about the location, room, and staff aspects of a hotel, “Excellent location to the Tower of London. We also walked to several other areas of interest; albeit a bit of a trek if you don't mind walking. The room was a typical hotel room in need of a refresh, however clean. The staff couldn't have been more professional, they really helped get us a taxi when our pre arranged pickup ran late.” In this review, some of the sentiment terms are “excellent”, “typical”, “clean”, and “professional”.
Question: Does the dataset contain non-English reviews?
Paper: Aspect and Opinion Term Extraction for Aspect Based Sentiment Analysis of Hotel Reviews Using Transfer Learning
For this aspect and opinion term extraction task, we use tokenized and annotated hotel reviews on Airy Rooms provided by BIBREF1. The dataset consists of 5000 reviews in bahasa Indonesia. The dataset is divided into training and test sets of 4000 and 1000 reviews respectively. The label distribution of the tokens in BIO scheme can be seen in Table TABREF3. In addition, we also see this case as on entity level, i.e. ASPECT, SENTIMENT, and OTHER labels.
Question: Does the paper report the performance of the method when is trained for more than 8 epochs?
Paper: Community Question Answering Platforms vs. Twitter for Predicting Characteristics of Urban Neighbourhoods
Recent years have seen a huge boom in the number of different social media platforms available to users. People are increasingly using these platforms to voice their opinions or let others know about their whereabouts and activities. Each of these platforms has its own characteristics and is used for different purposes. The availability of a huge amount of data from many social media platforms has inspired researchers to study the relation between the data generated through the use of these platforms and real-world attributes.
Question: What do the correlation demonstrate?
Paper: Community Question Answering Platforms vs. Twitter for Predicting Characteristics of Urban Neighbourhoods
The spatial unit of analysis chosen for this work is the neighbourhood. This is identified with a unique name (e.g., Camden) and people normally use this name in QA discussions to refer to specific neighbourhoods. A list of neighbourhoods for London is extracted from the GeoNames gazetteer, a dataset containing names of geographic places including place names. For each neighbourhood, GeoNames provides its name and a set of geographic coordinates (i.e., latitude and longitude) which roughly represents its centre. Note that geographical boundaries are not provided. GeoNames contains 589 neighbourhoods that fall within the boundaries of the Greater London metropolitan area. In the remainder of the paper, we use the terms “neighbourhood” or “area” to refer to our spatial unit of analysis.
Question: On Twitter, do the demographic attributes and answers show more correlations than on Yahoo! Answers?
Paper: Community Question Answering Platforms vs. Twitter for Predicting Characteristics of Urban Neighbourhoods
We collect questions and answers (QAs) from Yahoo! Answers using its public API. For each neighbourhood, the query consists of the name of the neighbourhood together with the keywords London and area. This is to prevent obtaining irrelevant QAs for ambiguous entity names such as Victoria. For each neighbourhood, we then take all the QAs that are returned by the API. Each QA consists of a title and a content which is an elaboration on the title. This is followed by a number of answers. In total, we collect $12,947$ QAs across all London neighbourhoods. These QAs span over the last 5 years. It is common for users to discuss characteristics of several neighbourhoods in the same QA thread. This means that the same QA can be assigned to more than one neighbourhood. Figure 1 shows the histogram of the number of QAs for each neighbourhood. As the figure shows, the majority of areas have less than 100 QAs with some areas having less than 10. Only few areas have over 100 QAs.
Question: How many demographic attributes they try to predict?
Paper: Attention Optimization for Abstractive Document Summarization
Abstractive document summarization BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4 attempts to produce a condensed representation of the most salient information of the document, aspects of which may not appear as parts of the original input text. One popular framework used in abstractive summarization is the sequence-to-sequence model introduced by BIBREF5. The attention mechanism BIBREF6 is proposed to enhance the sequence-to-sequence model by allowing salient features to dynamically come to the forefront as needed to make up for the incapability of memorizing the long input source.
Question: What evaluation metrics do they use?
Paper: Attention Optimization for Abstractive Document Summarization
We adopt the Pointer-Generator Network (PGN) BIBREF11 as our baseline model, which augments the standard attention-based seq2seq model with a hybrid pointer network BIBREF14. An input document is firstly fed into a Bi-LSTM encoder, then an uni-directional LSTM is used as the decoder to generate the summary word by word. At each decoding step, the attention distribution $a_t$ and the context vector $c_t$ are calculated as follows:
Question: How do they define local variance?
Answer: The reciprocal of the variance of the attention distribution
Paper: Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER
Contextual word embeddings BIBREF0 , BIBREF1 , BIBREF2 have been successfully applied to various NLP tasks, including named entity recognition, document classification, and textual entailment. The multilingual version of BERT (which is trained on Wikipedia articles from 100 languages and equipped with a 110,000 shared wordpiece vocabulary) has also demonstrated the ability to perform `zero-resource' cross-lingual classification on the XNLI dataset BIBREF3 . Specifically, when multilingual BERT is finetuned for XNLI with English data alone, the model also gains the ability to handle the same task in other languages. We believe that this zero-resource transfer learning can be extended to other multilingual datasets.
Question: How do they quantify alignment between the embeddings of a document and its translation?
Paper: Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER
Language-adversarial training BIBREF12 was proposed for generating bilingual dictionaries without parallel data. This idea was extended to zero-resource cross-lingual tasks in NER BIBREF5 , BIBREF6 and text classification BIBREF4 , where we would expect that language-adversarial techniques induce features that are language-independent.
Question: Does adversarial learning have stronger performance gains for text classification, or for NER?
Paper: Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER
Self-training, where an initial model is used to generate labels on an unlabeled corpus for the purpose of domain or cross-lingual adaptation, was studied in the context of text classification BIBREF11 and parsing BIBREF13 , BIBREF14 . A similar idea based on expectation-maximization, where the unobserved label is treated as a latent variable, has also been applied to cross-lingual text classification in BIBREF15 .
Question: Do any of the evaluations show that adversarial learning improves performance in at least two different language families?
Paper: Annotating Student Talk in Text-based Classroom Discussions
Current research, theory, and policy surrounding K-12 instruction in the United States highlight the role of student-centered disciplinary discussions (i.e. discussions related to a specific academic discipline or school subject such as physics or English Language Arts) in instructional quality and student learning opportunities BIBREF0 , BIBREF1 . Such student-centered discussions – often called “dialogic" or “inquiry-based” – are widely viewed as the most effective instructional approach for disciplinary understanding, problem-solving, and literacy BIBREF2 , BIBREF3 , BIBREF4 . In English Language Arts (ELA) classrooms, student-centered discussions about literature have a positive impact on the development of students' reasoning, writing, and reading skills BIBREF5 , BIBREF6 . However, most studies have focused on the role of teachers and their talk BIBREF7 , BIBREF2 , BIBREF8 rather than on the aspects of student talk that contribute to discussion quality.
Question: what experiments are conducted?
Paper: Annotating Student Talk in Text-based Classroom Discussions
One discourse feature used to assess the quality of discussions is students' argument moves: their claims about the text, their sharing of textual evidence for claims, and their warranting or reasoning to support the claims BIBREF10 , BIBREF11 . Many researchers view student reasoning as of primary importance, particularly when the reasoning is elaborated and highly inferential BIBREF12 . In Natural Language Processing (NLP), most educationally-oriented argumentation research has focused on corpora of student persuasive essays BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 . We instead focus on multi-party spoken discussion transcripts from classrooms. A second key difference consists in the inclusion of the warrant label in our scheme, as it is important to understand how students explicitly use reasoning to connect evidence to claims. Educational studies suggest that discussion quality is also influenced by the specificity of student talk BIBREF19 , BIBREF20 . Chisholm and Godley found that as specificity increased, the quality of students' claims and reasoning also increased. Previous NLP research has studied specificity in the context of professionally written newspaper articles BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24 . While the annotation instructions used in these studies work well for general purpose corpora, specificity in text-based discussions also needs to capture particular relations between discussions and texts. Furthermore, since the concept of a sentence is not clearly defined in speech, we annotate argumentative discourse units rather than sentences (see Section SECREF3 ).
Question: what opportunities are highlighted?
Paper: Annotating Student Talk in Text-based Classroom Discussions
Our annotation scheme uses argument moves as the unit of analysis. We define an argument move as an utterance, or part of an utterance, that contains an argumentative discourse unit (ADU) BIBREF28 . Like Peldszus and Stede Peldszus:15, in this paper we use transcripts already segmented into argument moves and focus on the steps following segmentation, i.e., labeling argumentation, specificity, and knowledge domain. Table TABREF2 shows a section of a transcribed classroom discussion along with labels assigned by a human annotator following segmentation.
Question: how do they measure discussion quality?
Answer: Measuring three aspects: argumentation, specificity and knowledge domain.
Paper: Annotating Student Talk in Text-based Classroom Discussions
The argumentation scheme is based on BIBREF29 and consists of a simplified set of labels derived from Toulmin's Toulmin:58 model: INLINEFORM0 Claim: an arguable statement that presents a particular interpretation of a text or topic. INLINEFORM1 Evidence: facts, documentation, text reference, or testimony used to support or justify a claim. INLINEFORM2 Warrant: reasons explaining how a specific evidence instance supports a specific claim. Our scheme specifies that warrants must come after claim and evidence, since by definition warrants cannot exist without them.
Question: do they use a crowdsourcing platform?
Paper: Dependency or Span, End-to-End Uniform Semantic Role Labeling
The purpose of semantic role labeling (SRL) is to derive the meaning representation for a sentence, which is beneficial to a wide range of natural language processing (NLP) tasks BIBREF0 , BIBREF1 . SRL can be formed as four subtasks, including predicate detection, predicate disambiguation, argument identification and argument classification. For argument annotation, there are two formulizations. One is based on text spans, namely span-based SRL. The other is dependency-based SRL, which annotates the syntactic head of argument rather than entire argument span. Figure FIGREF1 shows example annotations.
Question: what were the baselines?
Answer: 2008 Punyakanok et al. 
2009 Zhao et al. + ME 
2008 Toutanova et al. 
2010 Bjorkelund et al.  
2015 FitzGerald et al. 
2015 Zhou and Xu 
2016 Roth and Lapata 
2017 He et al. 
2017 Marcheggiani et al.
2017 Marcheggiani and Titov 
2018 Tan et al. 
2018 He et al. 
2018 Strubell et al. 
2018 Cai et al. 
2018 He et al. 
2018 Li et al.
Paper: Learning to Detect Opinion Snippet for Aspect-Based Sentiment Analysis
Aspect-based sentiment analysis BIBREF0, BIBREF1 is a fine-grained sentiment analysis task which has gained much attention from research and industries. It aims at predicting the sentiment polarity of a particular aspect of the text. With the rapid development of deep learning, this task has been widely addressed by attention-based neural networks BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6. To name a few, wang2016attention learn to attend on different parts of the sentence given different aspects, then generates aspect-specific sentence representations for sentiment prediction. tay2018learning learn to attend on correct words based on associative relationships between sentence words and a given aspect. These attention-based methods have brought the ABSA task remarkable performance improvement.
Question: Which soft-selection approaches are evaluated?
Answer: LSTM and BERT
Paper: Learning to Detect Opinion Snippet for Aspect-Based Sentiment Analysis
Traditional machine learning methods for aspect-based sentiment analysis focus on extracting a set of features to train sentiment classifiers BIBREF10, BIBREF11, BIBREF12, which usually are labor intensive. With the development of deep learning technologies, neural attention mechanism BIBREF13 has been widely adopted to address this task BIBREF14, BIBREF2, BIBREF15, BIBREF3, BIBREF16, BIBREF4, BIBREF17, BIBREF6, BIBREF5, BIBREF18, BIBREF19, BIBREF20, BIBREF21. wang2016attention propose attention-based LSTM networks which attend on different parts of the sentence for different aspects. Ma2017Interactive utilize the interactive attention to capture the deep associations between the sentence and the aspect. Hierarchical models BIBREF4, BIBREF17, BIBREF6 are also employed to capture multiple levels of emotional expression for more accurate prediction, as the complexity of sentence structure and semantic diversity. tay2018learning learn to attend based on associative relationships between sentence words and aspect.
Question: Is the model evaluated against the baseline also on single-aspect sentences?
Paper: Learning to Detect Opinion Snippet for Aspect-Based Sentiment Analysis
We first formulate the problem. Given a sentence $S=\lbrace w_1,w_2,...,w_N\rbrace $ and an aspect $A=\lbrace a_1,a_2,...,a_M\rbrace $, the ABSA task is to predict the sentiment of $A$. In our setting, the aspect can be either aspect terms or an aspect category. As aspect terms, $A$ is a snippet of words in $S$, i.e., a sub-sequence of the sentence, while as an aspect category, $A$ represents a semantic category with $M=1$, containing just an abstract token.
Question: Is the accuracy of the opinion snippet detection subtask reported?
Paper: Make Lead Bias in Your Favor: A Simple and Effective Method for News Summarization
The goal of text summarization is to condense a piece of text into a shorter version that contains the salient information. Due to the prevalence of news articles and the need to provide succinct summaries for readers, a majority of existing datasets for summarization come from the news domain BIBREF0, BIBREF1, BIBREF2. However, according to journalistic conventions, the most important information in a news report usually appears near the beginning of the article BIBREF3. While it facilitates faster and easier understanding of the news for readers, this lead bias causes undesirable consequences for summarization models. The output of these models is inevitably affected by the positional information of sentences. Furthermore, the simple baseline of using the top few sentences as summary can achieve a stronger performance than many sophisticated models BIBREF4. It can take a lot of effort for models to overcome the lead bias BIBREF3.
Question: What were the baselines?
Paper: Make Lead Bias in Your Favor: A Simple and Effective Method for News Summarization
End-to-end abstractive text summarization has been intensively studied in recent literature. To generate summary tokens, most architectures take the encoder-decoder approach BIBREF8. BIBREF9 first introduces an attention-based seq2seq model to the abstractive sentence summarization task. However, its output summary degenerates as document length increases, and out-of-vocabulary (OOV) words cannot be efficiently handled. To tackle these challenges, BIBREF4 proposes a pointer-generator network that can both produce words from the vocabulary via a generator and copy words from the source article via a pointer. BIBREF10 utilizes reinforcement learning to improve the result. BIBREF11 uses a content selector to over-determine phrases in source documents that helps constrain the model to likely phrases. BIBREF12 adds Gaussian focal bias and a salience-selection network to the transformer encoder-decoder structure BIBREF13 for abstractive summarization. BIBREF14 randomly reshuffles the sentences in news articles to reduce the effect of lead bias in extractive summarization.
Question: What metric was used in the evaluation step?
Paper: Make Lead Bias in Your Favor: A Simple and Effective Method for News Summarization
In recent years, pretraining language models have proved to be quite helpful in NLP tasks. The state-of-the-art pretrained models include ELMo BIBREF15, GPT BIBREF7, BERT BIBREF6 and UniLM BIBREF16. Built upon large-scale corpora, these pretrained models learn effective representations for various semantic structures and linguistic relationships. As a result, pretrained models have been widely used with considerable success in applications such as question answering BIBREF17, sentiment analysis BIBREF15 and passage reranking BIBREF18. Furthermore, UniLM BIBREF16 leverages its sequence-to-sequence capability for abstractive summarization; the BERT model has been employed as an encoder in BERTSUM BIBREF19 for extractive/abstractive summarization.
Question: What did they pretrain the model on?
Paper: Make Lead Bias in Your Favor: A Simple and Effective Method for News Summarization
News articles usually follow the convention of placing the most important information early in the content, forming an inverted pyramid structure. This lead bias has been discovered in a number of studies BIBREF3, BIBREF14. One of the consequences is that the lead baseline, which simply takes the top few sentences as the summary, can achieve a rather strong performance in news summarization. For instance, in the CNN/Daily Mail dataset BIBREF0, using the top three sentences as summaries can get a higher ROUGE score than many deep learning based models. This positional bias brings lots of difficulty for models to extract salient information from the article and generate high-quality summaries. For instance, BIBREF14 discovers that most models' performances drop significantly when a random sentence is inserted in the leading position, or when the sentences in a news article are shuffled.
Question: What does the data cleaning and filtering process consist of?
Paper: Make Lead Bias in Your Favor: A Simple and Effective Method for News Summarization
In this section, we introduce our abstractive summarization model, which has a transformer-based encoder-decoder structure. We first formulate the supervised summarization problem and then present the network architecture.
Question: What unlabeled corpus did they use?
Paper: A Multi-level Neural Network for Implicit Causality Detection in Web Texts
Automatic text causality mining is a critical but difficult task because causality is thought to play an essential role in human cognition when making decisions BIBREF0. Thus, automatic text causality has been studied extensively in a wide range of areas, such as industry BIBREF1, physics BIBREF2 and healthcare BIBREF3, etc. A tool to automatically scour the plethora of textual content on the web and extract meaningful causal relations could help us construct causal chains to unveil previously unknown relationships between events BIBREF4 and accelerates the discovery of the intrinsic logic of the events BIBREF5.
Question: How efective is MCDN for ambiguous and implicit causality inference compared to state-of-the-art?