z-uo/qasper-squad huggingface dataset:
Paper: Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
 Improving unsupervised learning is of key importance for advancing machine learning methods, as to unlock access to almost unlimited amounts of data to be used as training resources. The majority of recent success stories of deep learning does not fall into this category but instead relied on supervised training (in particular in the vision domain). A very notable exception comes from the text and natural language processing domain, in the form of semantic word embeddings trained unsupervised BIBREF0 , BIBREF1 , BIBREF2 . Within only a few years from their invention, such word representations – which are based on a simple matrix factorization model as we formalize below – are now routinely trained on very large amounts of raw text data, and have become ubiquitous building blocks of a majority of current state-of-the-art NLP applications.
Question: Do they report results only on English data?
Paper: Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
Our model is inspired by simple matrix factor models (bilinear models) such as recently very successfully used in unsupervised learning of word embeddings BIBREF0 , BIBREF1 , BIBREF2 , BIBREF5 as well as supervised of sentence classification BIBREF6 . More precisely, these models can all be formalized as an optimization problem of the form DISPLAYFORM0 
Question: Which other unsupervised models are used for comparison?
Paper: Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
We propose a new unsupervised model, Sent2Vec, for learning universal sentence embeddings. Conceptually, the model can be interpreted as a natural extension of the word-contexts from C-BOW BIBREF0 , BIBREF1 to a larger sentence context, with the sentence words being specifically optimized towards additive combination over the sentence, by means of the unsupervised objective function.
Question: What metric is used to measure performance?
Answer: Accuracy and F1 score for supervised tasks, Pearson's and Spearman's correlation for unsupervised tasks
Paper: Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
In contrast to more complex neural network based models, one of the core advantages of the proposed technique is the low computational cost for both inference and training. Given a sentence INLINEFORM0 and a trained model, computing the sentence representation INLINEFORM1 only requires INLINEFORM2 floating point operations (or INLINEFORM3 to be precise for the n-gram case, see ( EQREF8 )), where INLINEFORM4 is the embedding dimension. The same holds for the cost of training with SGD on the objective ( EQREF10 ), per sentence seen in the training corpus. Due to the simplicity of the model, parallel training is straight-forward using parallelized or distributed SGD.
Question: How do the n-gram features incorporate compositionality?
Paper: Future Word Contexts in Neural Network Language Models
Language models (LMs) are crucial components in many applications, such as speech recognition and machine translation. The aim of language models is to compute the probability of any given sentence INLINEFORM0 , which can be calculated as DISPLAYFORM0 
Question: Which dataset do they use?
Paper: Universal and non-universal text statistics: Clustering coefficient for language identification
Statistical characterization of languages has been a field of study for decadesBIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5. Even simple quantities, like letter frequency, can be used to decode simple substitution cryptogramsBIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10. However, probably the most surprising result in the field is Zipf's law, which states that if one ranks words by their frequency in a large text, the resulting rank frequency distribution is approximately a power law, for all languages BIBREF0, BIBREF11. These kind of universal results have long piqued the interest of physicists and mathematicians, as well as linguistsBIBREF12, BIBREF13, BIBREF14. Indeed, a large amount of effort has been devoted to try to understand the origin of Zipf's law, in some cases arguing that it arises from the fact that texts carry information BIBREF15, all the way to arguing that it is the result of mere chance BIBREF16, BIBREF17. Another interesting characterization of texts is the Heaps-Herdan law, which describes how the vocabulary -that is, the set of different words- grows with the size of a text, the number of which, empirically, has been found to grow as a power of the text size BIBREF18, BIBREF19. It is worth noting that it has been argued that this law is a consequence Zipf's law. BIBREF20, BIBREF21
Question: How do Zipf and Herdan-Heap's laws differ?
Answer: Zipf's law describes change of word frequency rate, while Heaps-Herdan describes different word number in large texts (assumed that Hepas-Herdan is consequence of Zipf's)
Paper: IndoSum: A New Benchmark Dataset for Indonesian Text Summarization
The goal of text summarization task is to produce a summary from a set of documents. The summary should retain important information and be reasonably shorter than the original documents BIBREF0 . When the set of documents contains only a single document, the task is usually referred to as single-document summarization. There are two kinds of summarization characterized by how the summary is produced: extractive and abstractive. Extractive summarization attempts to extract few important sentences verbatim from the original document. In contrast, abstractive summarization tries to produce an abstract which may contain sentences that do not exist in or are paraphrased from the original document.
Question: What was the best performing baseline?
Paper: IndoSum: A New Benchmark Dataset for Indonesian Text Summarization
Fachrurrozi et al. BIBREF3 proposed some scoring methods and used them with TF-IDF to rank and summarize news articles. Another work BIBREF4 used latent Dirichlet allocation coupled with genetic algorithm to produce summaries for online news articles. Simple methods like naive Bayes has also been used for Indonesian news summarization BIBREF2 , although for English, naive Bayes has been used almost two decades earlier BIBREF5 . A more recent work BIBREF6 employed a summarization algorithm called TextTeaser with some predefined features for news articles as well. Slamet et al. BIBREF7 used TF-IDF to convert sentences into vectors, and their similarities are then computed against another vector obtained from some keywords. They used these similarity scores to extract important sentences as the summary. Unfortunately, all these work do not seem to be evaluated using ROUGE, despite being the standard metric for text summarization research.
Question: Which approaches did they use?
Paper: IndoSum: A New Benchmark Dataset for Indonesian Text Summarization
We used a dataset provided by Shortir, an Indonesian news aggregator and summarizer company. The dataset contains roughly 20K news articles. Each article has the title, category, source (e.g., CNN Indonesia, Kumparan), URL to the original article, and an abstractive summary which was created manually by a total of 2 native speakers of Indonesian. There are 6 categories in total: Entertainment, Inspiration, Sport, Showbiz, Headline, and Tech. A sample article-summary pair is shown in Fig. FIGREF4 .
Question: What is the size of the dataset?
Paper: IndoSum: A New Benchmark Dataset for Indonesian Text Summarization
For evaluation, we used ROUGE BIBREF1 , a standard metric for text summarization. We used the implementation provided by pythonrouge. Following BIBREF11 , we report the INLINEFORM0 score of R-1, R-2, and R-L. Intuitively, R-1 and R-2 measure informativeness and R-L measures fluency BIBREF11 . We report the INLINEFORM1 score instead of just the recall score because although we extract a fixed number of sentences as the summary, the number of words are not limited. So, reporting only recall benefits models which extract long sentences.
Question: Did they use a crowdsourcing platform for the summaries?
Paper: BLEURT: Learning Robust Metrics for Text Generation
In the last few years, research in natural text generation (NLG) has made significant progress, driven largely by the neural encoder-decoder paradigm BIBREF0, BIBREF1 which can tackle a wide array of tasks including translation BIBREF2, summarization BIBREF3, BIBREF4, structured-data-to-text generation BIBREF5, BIBREF6, BIBREF7 dialog BIBREF8, BIBREF9 and image captioning BIBREF10. However, progress is increasingly impeded by the shortcomings of existing metrics BIBREF7, BIBREF11, BIBREF12.
Question: How are the synthetic examples generated?
Answer: Random perturbation of Wikipedia sentences using mask-filling with BERT, backtranslation and randomly drop out
Paper: Non-Projective Dependency Parsing with Non-Local Transitions
Greedy transition-based parsers are popular in NLP, as they provide competitive accuracy with high efficiency. They syntactically analyze a sentence by greedily applying transitions, which read it from left to right and produce a dependency tree.
Question: Do they measure the number of created No-Arc long sequences?
Paper: Non-Projective Dependency Parsing with Non-Local Transitions
The original non-projective parser defined by covington01fundamental was modelled under the transition-based parsing framework by Nivre2008. We only sketch this transition system briefly for space reasons, and refer to BIBREF4 for details.
Question: By how much does the new parser outperform the current state-of-the-art?
Answer: Proposed method achieves 94.5 UAS and 92.4 LAS  compared to 94.3 and 92.2 of best state-of-the -art greedy based parser. Best state-of-the art parser overall achieves 95.8 UAS and 94.6 LAS.
Paper: KryptoOracle: A Real-Time Cryptocurrency Price Prediction Platform Using Twitter Sentiments
A cryptocurrency is a digital currency designed to work as a medium of exchange that uses strong cryptography to secure financial transactions, control the creation of additional units, and verify the transfer of assets. They are based on decentralized systems built on block-chain technology, a distributed ledger enforced by a disparate network of computers BIBREF0. The first decentralized cryptocurrency, Bitcoin, was released as open-source software in 2009. After this release, approximately 4000 altcoins (other cryptocurrencies) have been released. As of August 2019, the total market capitalization of cryptocurrencies is $258 billion, where Bitcoin alone has a market capitalization of $179 billion BIBREF1.
Question: Do they evaluate only on English datasets?
Paper: KryptoOracle: A Real-Time Cryptocurrency Price Prediction Platform Using Twitter Sentiments
In this section we present a brief review of the state of the art related to cryptocurrency price prediction. Related works can be divided into three main categories: (i) social media sentiments and financial markets (including cryptocurrency markets); (ii) machine learning for cryptocurrency price prediction; and (iii) big data platforms for financial market prediction.
Question: What experimental evaluation is used?
Answer: root mean square error between the actual and the predicted price of Bitcoin for every minute
Paper: KryptoOracle: A Real-Time Cryptocurrency Price Prediction Platform Using Twitter Sentiments
KryptoOracle is an engine that aims at predicting the trends of any cryptocurrency based on the sentiment of the crowd. It does so by learning the correlation between the sentiments of relevant tweets and the real time price of the cryptocurrency. The engine bootstraps itself by first learning from the history given to it and starts predicting based on the previous correlation. KryptoOracle is also capable of reinforcing itself by the mistakes it makes and tries to improve itself at prediction. In addition, the engine supports trend visualization over time based on records of both incoming data and intermediate results. This engine has been built keeping in mind the increasing data volume, velocity and variety that has been made available and is therefore able to scale and manage high volumes of heterogeneous data.
Question: How is the architecture fault-tolerant?
Answer: By using Apache Spark which stores all executions in a lineage graph and recovers to the previous steady state from any fault
Paper: KryptoOracle: A Real-Time Cryptocurrency Price Prediction Platform Using Twitter Sentiments
The growth of the volume of data inspired us to opt for a big data architecture which can not only handle the prediction algorithms but also the streaming and increasing volume of data in a fault tolerant way.
Question: Which elements of the platform are modular?
Answer: handling large volume incoming data, sentiment analysis on tweets and predictive online learning
Paper: Hate Speech in Pixels: Detection of Offensive Memes towards Automatic Moderation
The spread of misinformation or hate messages through social media is a central societal challenge given the unprecedented broadcast potential of these tools. While there already exist some moderation mechanisms such as crowd-sourced abuse reports and dedicated human teams of moderators, the huge and growing scale of these networks requires some degree of automation for the task.
Question: What is the source of memes?
Paper: Hate Speech in Pixels: Detection of Offensive Memes towards Automatic Moderation
Hate speech is a widely studied topic in the context of social science. This phenomena has been monitored, tracked, measured or quantified in a number of occasions BIBREF4, BIBREF5, BIBREF6. It appears in media such as newspapers or TV news, but one of the main focus of hate speech with very diverse targets has appeared in social networks BIBREF7, BIBREF8, BIBREF9. Most works in hate speech detection has focused in language. The most common approach is to generate an embedding of some kind, using bag of words BIBREF8 or N-gram features BIBREF10 and many times using expert knowledge for keywords. After that, the embedding is fed to a binary classifier to predict hate speech. Up to our knowledge, there is no previous work on detecting hate speech when combining language with visual content as in memes. Our technical solution is inspired by BIBREF11 in which gang violence on social media was predicted from a multimodal approach that fused images and text. Their model extracted features from both modalities using pretrained embeddings for language and vision, and later merged both vectors to feed the multimodal features into a classifier.
Question: Is the dataset multimodal?
Paper: Hate Speech in Pixels: Detection of Offensive Memes towards Automatic Moderation
The overall system expects an Internet meme input, and produces a hate score as an output. Figure FIGREF1 shows a block diagram of the proposed solution.
Question: How is each instance of the dataset annotated?
Paper: A Stable Variational Autoencoder for Text Modelling
Variational Autoencoder (VAE) BIBREF1 is a powerful method for learning representations of high-dimensional data. However, recent attempts of applying VAEs to text modelling are still far less successful compared to its application to image and speech BIBREF2, BIBREF3, BIBREF4. When applying VAEs for text modelling, recurrent neural networks (RNNs) are commonly used as the architecture for both encoder and decoder BIBREF0, BIBREF5, BIBREF6. While such a VAE-RNN based architecture allows encoding and generating sentences (in the decoding phase) with variable-length effectively, it is also vulnerable to an issue known as latent variable collapse (or KL loss vanishing), where the posterior collapses to the prior and the model will ignore the latent codes in generative tasks.
Question: Which dataset do they use for text modelling?
Paper: A Stable Variational Autoencoder for Text Modelling
A variational autoencoder (VAE) is a deep generative model, which combines variational inference with deep learning. The VAE modifies the conventional autoencoder architecture by replacing the deterministic latent representation $\mathbf {z}$ of an input $\mathbf {x}$ with a posterior distribution $P(\mathbf {z}|\mathbf {x})$, and imposing a prior distribution on the posterior, such that the model allows sampling from any point of the latent space and yet able to generate novel and plausible output. The prior is typically chosen to be standard normal distributions, i.e., $P(\mathbf {z}) = \mathcal {N}(\mathbf {0},\mathbf {1})$, such that the KL divergence between posterior and prior can be computed in closed form BIBREF1.
Question: Do they compare against state of the art text generation?
Paper: A Stable Variational Autoencoder for Text Modelling
In this section, we discuss the technical details of the proposed holistic regularisation VAE (HR-VAE) model, a general architecture which can effectively mitigate the KL vanishing phenomenon.
Question: How do they evaluate generated text quality?
Paper: Neural DrugNet
In recent years, there has been a rapid growth in the usage of social media. People post their day-to-day happenings on regular basis. BIBREF0 propose four tasks for detecting drug names, classifying medication intake, classifying adverse drug reaction and detecting vaccination behavior from tweets. We participated in the Task2 and Task4.
Question: Was the system only evaluated over the second shared task?
Paper: Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets
With the growing amount of biomedical information available in textual form, there have been significant advances in the development of pre-training language representations that can be applied to a range of different tasks in the biomedical domain, such as pre-trained word embeddings, sentence embeddings, and contextual representations BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 .
Question: Could you tell me more about the metrics used for performance evaluation?
Answer: BLUE utilizes different metrics for each of the tasks: Pearson correlation coefficient, F-1 scores, micro-averaging, and accuracy
Paper: Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets
There is a long history of using shared language representations to capture text semantics in biomedical text and data mining research. Such research utilizes a technique, termed transfer learning, whereby the language representations are pre-trained on large corpora and fine-tuned in a variety of downstream tasks, such as named entity recognition and relation extraction.
Question: which tasks are used in BLUE benchmark?
Paper: Improving Cross-Lingual Word Embeddings by Meeting in the Middle
Word embeddings are one of the most widely used resources in NLP, as they have proven to be of enormous importance for modeling linguistic phenomena in both supervised and unsupervised settings. In particular, the representation of words in cross-lingual vector spaces (henceforth, cross-lingual word embeddings) is quickly gaining in popularity. One of the main reasons is that they play a crucial role in transferring knowledge from one language to another, specifically in downstream tasks such as information retrieval BIBREF0 , entity linking BIBREF1 and text classification BIBREF2 , while at the same time providing improvements in multilingual NLP problems such as machine translation BIBREF3 .
Question: What are the tasks that this method has shown improvements?
Answer: bilingual dictionary induction, monolingual and cross-lingual word similarity, and cross-lingual hypernym discovery
Paper: Improving Cross-Lingual Word Embeddings by Meeting in the Middle
Bilingual word embeddings have been extensively studied in the literature in recent years. Their nature varies with respect to the supervision signals used for training BIBREF13 , BIBREF14 . Some common signals to learn bilingual embeddings come from parallel BIBREF15 , BIBREF16 , BIBREF17 or comparable corpora BIBREF18 , BIBREF19 , BIBREF20 , or lexical resources such as WordNet, ConceptNet or BabelNet BIBREF21 , BIBREF22 , BIBREF23 . However, these sources of supervision may be scarce, limited to certain domains or may not be directly available for certain language pairs.
Question: Why does the model improve in monolingual spaces as well? 
Answer: because word pair similarity increases if the two words translate to similar parts of the cross-lingual embedding space
Paper: An Interactive Tool for Natural Language Processing on Clinical Text
Electronic Health Records (EHRs) are organized collections of information about individual patients. They are designed such that they can be shared across different settings for providing health care services. The Institute of Medicine committee on improving the patient record has recognized the importance of using EHRs to inform decision support systems and support data-driven quality measures BIBREF0 . One of the biggest challenges in achieving this goal is the difficulty of extracting information from large quantities of EHR data stored as unstructured free text. Clinicians often make use of narratives and first-person stories to document interactions, findings and analyses in patient cases BIBREF1 . As a result, finding information from these volumes of health care records typically requires the use of NLP techniques to automate the extraction process.
Question: What are the categories being extracted?
Paper: Cross-lingual Abstract Meaning Representation Parsing
Abstract Meaning Representation (AMR) parsing is the process of converting natural language sentences into their corresponding AMR representations BIBREF0 . An AMR is a graph with nodes representing the concepts of the sentence and edges representing the semantic relations between them. Most available AMR datasets large enough to train statistical models consist of pairs of English sentences and AMR graphs.
Question: Do the authors test their annotation projection techniques on tasks other than AMR?
Paper: Cross-lingual Abstract Meaning Representation Parsing
AMR is a semantic representation heavily biased towards English, where labels for nodes and edges are either English words or Propbank frames BIBREF5 . The goal of AMR is to abstract away from the syntactic realization of the original sentences while maintaining its underlying meaning. As a consequence, different phrasings of one sentence are expected to provide identical AMR representations. This canonicalization does not always hold across languages: two sentences that express the same meaning in two different languages are not guaranteed to produce identical AMR structures BIBREF6 , BIBREF7 . However, xue2014not show that in many cases the unlabeled AMRs are in fact shared across languages. We are encouraged by this finding and argue that it should be possible to develop algorithms that account for some of these differences when they arise. We therefore introduce a new problem, which we call cross-lingual AMR parsing: given a sentence in any language, the goal is to recover the AMR graph that was originally devised for its English translation. This task is harder than traditional AMR parsing as it requires to recover English labels as well as to deal with structural differences between languages, usually referred as translation divergence. We propose two initial solutions to this problem: by annotation projection and by machine translation.
Question: How is annotation projection done when languages have different word order?
Answer: Word alignments are generated for parallel text, and aligned words are assumed to also share AMR node alignments.