z-uo/qasper-squad huggingface dataset:
Paper: Data Interpretation over Plots
Datasets: Over the past few years several large scale datasets for Visual Question Answering have been released. These include datasets such as COCO-QA BIBREF3, DAQUAR BIBREF4, VQA BIBREF5, BIBREF6 which contain questions asked over natural images. On the other hand, datasets such as CLEVR BIBREF7 and NVLR BIBREF8 contain complex reasoning based questions on synthetic images having 2D and 3D geometric objects. There are some datasets BIBREF9, BIBREF10 which contain questions asked over diagrams found in text books but these datasets are smaller and contain multiple-choice questions. FigureSeer BIBREF11 is another dataset which contains images extracted from research papers but this is also a relatively small (60,000 images) dataset. Further, FigureSeer focuses on answering questions based on line plots as opposed to other types of plots such as bar charts, scatter plots, etc. as seen in FigureQA BIBREF0 and DVQA BIBREF1.
Question: How big are FigureQA and DVQA datasets?
Paper: Data Interpretation over Plots
In this section, we describe the PlotQA dataset and the process to build it. Specifically, we discuss the four main stages, viz., (i) curating data such as year-wise rainfall statistics, country-wise mortality rates, etc., (ii) creating different types of plots with a variation in the number of elements, legend positions, fonts, etc., (iii) crowd-sourcing to generate questions, and (iv) extracting templates from the crowd-sourced questions and instantiating these templates using appropriate phrasing suggested by human annotators.
Question: What models other than SAN-VOES are trained on new PlotQA dataset?
Paper: Sentiment Analysis for Twitter : Going Beyond Tweet Text
Analysing sentiment from text is a well-known NLP problem. Several state-of-the-art tools exist that can achieve this with reasonable accuracy. However most of the existing tools perform well on well-formatted text. In case of tweets, the user generated content is short, noisy, and in many cases ( INLINEFORM0 ) doesn't follow proper grammatical structure. Additionally, numerous internet slangs, abbreviations, urls, emoticons, and unconventional style of capitalization are found in the tweets. As a result, the accuracy of the state-of-the art NLP tools decreases sharply. In this project, we develop new features to incorporate the styles salient in short, informal user generated contents like tweets. We achieve an F1-accuracy of INLINEFORM1 for predicting the sentiment of tweets in our data-set. We also propose a method to discover new sentiment terms from the tweets.
Question: Do the authors report only on English language data?
Paper: Sentiment Analysis for Twitter : Going Beyond Tweet Text
Tweets are short messages, restricted to 140 characters in length. Due to the nature of this microblogging service (quick and short messages), people use acronyms, make spelling mistakes, use emoticons and other characters that express special meanings. Following is a brief terminology associated with tweets:
Question: What dataset of tweets is used?
Paper: Sentiment Analysis for Twitter : Going Beyond Tweet Text
Since tweets are informal in nature, some pre-processing is required. Consider the following tweet.
Question: What external sources of information are used?
Paper: Sentiment Analysis for Twitter : Going Beyond Tweet Text
Normalization is done as follows:
Question: What linguistic features are used?
Paper: Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?
Machine learning (ML) has become widely used in many academic fields, as well as across the private and public sector. Supervised machine learning is particularly prevalent, in which training data is collected for a set of entities with known properties (a “ground truth” or “gold standard”), which is used to create a classifier that will make predictions about new entities of the same type. Supervised ML requires high-quality training data to produce high-quality classifiers. “Garbage In, Garbage Out” is a longstanding aphorism in computing about how flawed input data or instructions will produce flawed outputs. BIBREF0, BIBREF1 However, contemporary ML research and education tends to focus less on obtaining and validating such a training dataset, with such considerations often passed over in major textbooks BIBREF2, BIBREF3, BIBREF4. The predominant focus is typically on what is done with the training data to produce a classifier, with heavy emphasis on mathematical foundations and routine use of clean and tidy “toy” datasets. The process of creating a “gold standard” or “ground truth” dataset is routinely black-boxed. Many papers in ML venues are expected to use a standard, public training dataset, with authors comparing various performance metrics on the same dataset. While such a focus on what is done to a training dataset may be appropriate for theoretically-oriented basic research in ML, this is not the case for supervised ML applications.
Question: What are the key issues around whether the gold standard data produced in such an annotation is reliable?
Paper: Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?
All approaches of producing a training dataset involve some form of human judgment, albeit at varying levels of granularity. In this paper, we investigate and discuss a wide range of issues and concerns around the curation of human-labeled or human-annotated data, in which one or more individuals make discrete assessments of items. We report from a study in which a team of six labelers systematically examined a corpus of supervised machine learning application papers in social computing, specifically those that classified tweets from Twitter for various purposes. For each paper, we recorded what the paper does or does not state about the training data used to produce the classifier presented in the paper. The bulk of the papers we examined were a sample of preprints or postprints published on ArXiV.org, plus a smaller set of published papers sampled from Scopus. We determined whether such papers involved an original classification task using supervised ML, whether the training data labels were produced from human annotation, and if so, the source of the human-labeled dataset (e.g. the paper's authors, Mechanical Turk, recruited experts, no information given, etc.). For all papers in which an original human-labeled dataset was produced, we then made a series of further determinations, including if definitions and/or examples were given to labelers, if labelers independently labeled the same items, if inter-rater reliability metrics were presented, if compensation details for crowdworkers were reported, if a public link to the dataset was available, and more.
Question: How were the machine learning papers from ArXiv sampled?
Paper: Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?
In the introduction, we noted training data is frequently black-boxed in machine learning research and applications. We use the term “black-boxed” in a different way than it is typically invoked in and beyond the FAT* community, where often refers to interpretability. In that sense, “black-boxing” means that even for experts who have access to the training data and code which created the classifier, it is difficult to understand why the classifier made each decision. In social science and humanities work on “black-boxing” of ML (and other “algorithmic” systems), there is often much elision between issues of interpretability and intentional concealment, as Burrell BIBREF5 notes. A major focus is on public accountability BIBREF6, where many problematic issues can occur behind closed doors. This is even the case with relatively simple forms of analytics and automation — such as if-then statements, linear regressions, or rule-based expert systems BIBREF7, BIBREF8.
Question: What are the core best practices of structured content analysis?
Paper: Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?
Creating human-labeled training datasets for machine learning often looks like content analysis, a well-established methodology in the humanities and the social sciences (particularly literature, communication studies, and linguistics), which also has versions used in the life, ecological, and medical sciences. Content analysis has taken many forms over the past century, from more positivist methods that formally establish structural ways of evaluating content to more interpretivist methods that embrace ambiguity and multiple interpretations, such as grounded theory BIBREF16. The intersection of ML and interpretivist approaches is outside of the scope of this article, but it is an emerging area of interest BIBREF17.
Question: In what sense is data annotation similar to structured content analysis?
Paper: Large Arabic Twitter Dataset on COVID-19
On December 31, 2019, Chinese public health authorities reported several cases of a respiratory syndrome caused by an unknown disease, which subsequently became known as COVID-19 in the city of Wuhan, China. This highly contagious disease continued to spread worldwide, leading the World Health Organization (WHO) to declare a global health emergency on January 30, 2020. On March 11, 2020 the disease has been identified as pandemic by WHO, and many countries around the world including Saudi Arabia, United States, United Kingdom, Italy, Canada, and Germany have continued reporting more cases of the disease BIBREF0. As the time of writing this paper, this pandemic is affecting more than 208 countries around the globe with more than one million and half confirmed cases BIBREF1.
Question: What additional information is found in the dataset?
Paper: Large Arabic Twitter Dataset on COVID-19
We collected COVID-19 related Arabic tweets from January 1, 2020 until April 15, 2020, using Twitter streaming API and the Tweepy Python library. We have collected more than 3,934,610 million tweets so far. In our dataset, we store the full tweet object including the id of the tweet, username, hashtags, and geolocation of the tweet. We created a list of the most common Arabic keywords associated with COVID-19. Using Twitter’s streaming API, we searched for any tweet containing the keyword(s) in the text of the tweet. Table TABREF1 shows the list of keywords used along with the starting date of tracking each keyword. Furthermore, Table TABREF2 shows the list of hashtags we have been tracking along with the number of tweets collected from each hashtag. Indeed, some tweets were irrelevant, and we kept only those that were relevant to the pandemic.
Question: Is the dataset focused on a region?
Paper: Large Arabic Twitter Dataset on COVID-19
The dataset is accessible on GitHub at this address: https://github.com/SarahAlqurashi/COVID-19-Arabic-Tweets-Dataset
Question: Over what period of time were the tweets collected?
Paper: Large Arabic Twitter Dataset on COVID-19
We are continuously updating the dataset to maintain more aspects of COVID-19 Arabic conversations and discussions happening on Twitter. We also plan to study how different groups respond to the pandemic and analyze information sharing behavior among the users.
Question: Are the tweets location-specific?
Paper: Large Arabic Twitter Dataset on COVID-19
The authors wish to express their thanks to Batool Mohammed Hmawi for her help in data collection.
Question: How big is the dataset?
Paper: Event detection in Twitter: A keyword volume approach
Event detection is important for emergency services to react rapidly and minimize damage. For example, terrorist attacks, protests, or bushfires may require the presence of ambulances, firefighters, and police as soon as possible to save people. This research aims to detect events as soon as they occur and are reported via some Twitter user. The event detection process requires to know the keywords associated with each event and to assess the minimal count of each word to decide confidently that an event has occurred. In this research, we propose a novel method of spike matching to identify keywords, and use probabilistic classification to assess the probability of having an event given the volume of each word.
Question: Do the authors suggest any future extensions to this work?
Paper: Event detection in Twitter: A keyword volume approach
Analyzing social networks for event detection is approached from multiple perspectives depending on the research objective. This can be predicting election results, a contest winner, or predicting peoples' reaction to a government decision through protest. The main perspectives to analyze the social networks are (1) content analysis, where the textual content of each post is analyzed using natural language processing to identify the topic or the sentiment of the authors. (2) Network structure analysis, where the relation between the users are described in a tree structure for the follower-followee patterns, or in a graph structure for friendship and interaction patterns. These patterns can be used to know the political preference of people prior to elections. (3) Behavioural analysis of each user including sentiment, response, likes, retweets, location, to identify responses toward specific events. This might be useful to identify users with terrorist intentions. In this section, we will focus on textual content-based models, where text analysis and understanding can be achieved using keywords, topic modelling or sentiment analysis.
Question: Which of the classifiers showed the best performance?
Answer: Logistic regression
Paper: Event detection in Twitter: A keyword volume approach
Keyword-based approaches focus on sequence analysis of the time series for each keyword. They also consider different forms for each keyword, including n-gram, skip-gram, and word-pairs BIBREF6 . The keyword-based approaches use the concept of the distributional semantics to group semantically-related words as synonyms to be used as a single feature BIBREF7 . In this approach, keywords are usually associated with events by correlation, entropy or distance metrics. Also, Hossny et al. proposed using SVD with K-Means to strengthen keyword signals, by grouping words having similar temporal patterns, then mapping them into one central word that has minimum distance to the other members of the cluster BIBREF8 .
Question: Were any other word similar metrics, besides Jaccard metric, tested?
Paper: Event detection in Twitter: A keyword volume approach
Topic modelling approaches focus on clustering related words according to their meaning, and indexing them using some similarity metric such as cosine similarity or Euclidean distance. The most recognized techniques are (1) Latent Semantic Indexing (LSI), where the observation matrix is decomposed using singular value decomposition and the data are clustered using K-Means BIBREF7 ,(2) Latent Dirichlet Allocation (LDA), where the words are clustered using Gaussian mixture models (GMM) according to the likelihood of term co-occurrence within the same context BIBREF14 , (3) Word2Vec, which uses a very large corpus to compute continuous vector representations, where we can apply standard vector operations to map one vector to another BIBREF15 .
Question: How are the keywords associated with events such as protests selected?
Answer: By using a Bayesian approach  and by using word-pairs, where they extract all the pairs of co-occurring words within each tweet.  They search for the words that achieve the highest number of spikes matching the days of events.
Paper: Analysis of Speeches in Indian Parliamentary Debates
As the world moves towards increasing forms of digitization, the creation of text corpora has become an important activity for NLP and other fields of research. Parliamentary data is a rich corpus of discourse on a wide array of topics. The Lok Sabha website provides access to all kinds of reports, debates, bills related to the proceedings of the house. Similarly, the Rajya Sabha website also contains debates, bills, reports introduced in the house. The Lok Sabha website also contains information about members of the parliament who are elected by the people and debate in the house. Since the data is unstructured , it cannot be computationally analyzed. There is a need to shape the data into a structured format for analysis. This data is important as it can be used to visualize person, party and agenda level semantics in the house.
Question: How many speeches are in the dataset?
Answer: 5575 speeches
Paper: Analysis of Speeches in Indian Parliamentary Debates
Many linguists around the globe are concentrating on creation of parliamentary datasets. BIBREF1 gives an overview of the parliamentary records and corpora from countries with a focus on their availability through Clarin infrastructure. A dataset of Japanese Local Assembly minutes was created and analyzed for statistical data such as number of speakers, characters and words BIBREF2 . BIBREF3 created a highly multilingual parallel corpus of European parliament and demonstrated that it is useful for statistical machine translation. Parliamentary debates are full of arguments. Ruling party members refute the claims made by opposition party members and vice versa. Members provide strong arguments for supporting their claim or refuting other's claim. Analyzing argumentation from a computational linguistics point of view has led very recently to a new field called argumentation mining BIBREF4 . One can perform argument mining on these debates and analyze the results. BIBREF5 worked on detecting perspectives in UK political debates using a Bayesian modelling approach. BIBREF6 worked on claim detection from UK political debates using both linguistic features text and features from speech.
Question: What classification models were used?
Paper: Analysis of Speeches in Indian Parliamentary Debates
Our dataset consists of synopsis of debates in the lower house of the Indian Parliament (Lok Sabha). The dataset consists of :
Question: Do any speeches not fall in these categories?
Paper: BERT Can See Out of the Box: On the Cross-modal Transferability of Text Representations
The BERT language model BIBREF0 is a Deep Bidirectional Transformer BIBREF1 pre-trained on textual corpora (BookCorpus and Wikipedia) using a Masked Language Model (MLM) objective – predicting some words that are randomly masked in the sentence, along with a sentence entailment loss. Recent research efforts BIBREF2 have shown how BERT encodes abstractions that generalize across languages, even when trained on monolingual data only. This contradicts the common belief BIBREF3, BIBREF4 that a shared vocabulary and joint training on multiple languages are essential to achieve cross-lingual generalization capabilities. In this work, we further investigate the generalization potentials of large pre-trained LMs, this time moving to a cross-modal setup: does BERT contain abstractions that generalize beyond text?
Question: What is different in BERT-gen from standard BERT?
Answer: They use a left-to-right attention mask so that the input tokens can only attend to other input tokens, and the target tokens can only attend to the input tokens and already generated target tokens.
Paper: BERT Can See Out of the Box: On the Cross-modal Transferability of Text Representations
Learning unsupervised textual representations that can be applied to downstream tasks is a widely investigated topic in the literature. Text representations have been learned at different granularities: words with Word2vec BIBREF20, sentences with SkipThought BIBREF21, paragraphs with ParagraphVector BIBREF22 and contextualized word vectors with ELMo BIBREF23. Other methods leverage a transfer-learning approach by fine-tuning all parameters of a pre-trained model on a target task, a paradigm which has become mainstream since the introduction of BERT BIBREF0. BERT alleviates the problem of the uni-directionality of most language models (i.e. where the training objective aims at predicting the next word) by proposing a new objective called Masked Language Model (MLM). Under MLM, some words, that are randomly selected, are masked; the training objective aims at predicting them.
Question: How are multimodal representations combined?
Answer: The image feature vectors are mapped into BERT embedding dimensions and treated like a text sequence afterwards.
Paper: Facet-Aware Evaluation for Extractive Text Summarization
In this section, we describe the procedure of annotating CNN/Daily Mail. For each facet (sentence) in the reference summary, we find all its support sentences in the document that can cover its meaning. Note that the support sentences are likely to be more verbose, but we only consider if the sentences cover the semantics of the facet regardless of their length. The reason is that we believe extractive summarization should focus on information coverage and once salient sentences are extracted, one can then compress them in an abstractive way BIBREF0, BIBREF1. Formally, we denote one document-summary pair as $\lbrace d, r\rbrace $, where $d = \lbrace d^j\rbrace _{j=1}^D$, $r = \lbrace r^j\rbrace _{j=1}^R$, and $D$, $R$ denote the number of sentences. We define one support group of facet $\mathcal {F}$ as a minimum set of sentences in the document that express the meaning of $\mathcal {F}$. For each $r^j$, we annotate a FAM $r^j \rightarrow \lbrace \lbrace d^{s_{j, 1}^k}\rbrace _{k=1}^{\textrm {K}_1}, \lbrace d^{s_{j, 2}^k}\rbrace _{k=1}^{\textrm {K}_2}, ..., \lbrace d^{s_{j, N}^k}\rbrace _{k=1}^{\textrm {K}_N}\rbrace $ in which each $\lbrace d^{s_{j, n}^k}\rbrace _{k=1}^{\textrm {K}_n}$ is a support group and $s_{j, n}^k$ is the index of the $k$-th support sentence in group $n$.
Question: What is the problem with existing metrics that they are trying to address?
Answer: Answer with content missing: (whole introduction) However, recent
studies observe the limits of ROUGE and find in
some cases, it fails to reach consensus with human.
judgment (Paulus et al., 2017; Schluter, 2017).
Paper: Facet-Aware Evaluation for Extractive Text Summarization
By utilizing the FAMs, we revisit extractive methods to see how well they perform on facet coverage. Specifically, we compare Lead-3, Refresh BIBREF3, FastRL(E) (E for extractive only) BIBREF0, UnifiedSum(E) BIBREF1, NeuSum BIBREF4, and BanditSum BIBREF5 using both ROUGE and FAMs. As these methods are facet-agnostic (i.e., their outputs are not organized by facets but flat extract sets), we consider one facet is covered as long as one of its support groups is extracted and measure the Facet-Aware Recall ($\textbf {FAR} = \frac{\textrm {\#covered}}{R}$). For a fair comparison, each method extracts three sentences since extracting all would result in a perfect FAR.
Question: How do they evaluate their proposed metric?
Paper: Facet-Aware Evaluation for Extractive Text Summarization
Although the FAMs only need to be annotated once, we investigate whether such human efforts can be further reduced by evaluating approximate approaches that generate extractive labels. Approximate approaches typically transform one abstractive summary to extractive labels heuristically using ROUGE. Previously one could only estimate the quality of these labels by evaluating the extractive models trained using such labels, i.e., comparing the extracted and reference summaries (also approximately via ROUGE). Now that the FAMs serve as ground-truth extractive labels, we can evaluate how well each approach performs accurately. Since the approximate approaches do not have the notion of support group, we flatten all the support sentences in one FAM to a label set.
Question: What is a facet?