z-uo/qasper-squad huggingface dataset:
Paper: Leveraging Discourse Information Effectively for Authorship Attribution
Authorship attribution (AA) is the task of identifying the author of a text, given a set of author-labeled training texts. This task typically makes use of stylometric cues at the surface lexical and syntactic level BIBREF0 , although BIBREF1 and BIBREF2 go beyond the sentence level, showing that discourse information can help. However, they achieve limited performance gains and lack an in-depth analysis of discourse featurization techniques. More recently, convolutional neural networks (CNNs) have demonstrated considerable success on AA relying only on character-level INLINEFORM0 -grams BIBREF3 , BIBREF4 . The strength of these models is evidenced by findings that traditional stylometric features such as word INLINEFORM1 -grams and POS-tags do not improve, and can sometimes even hurt performance BIBREF3 , BIBREF5 . However, none of these CNN models make use of discourse.
Question: How are discourse embeddings analyzed?
Answer: They perform t-SNE clustering to analyze discourse embeddings
Paper: Leveraging Discourse Information Effectively for Authorship Attribution
Entity-grid model. Typical lexical features for AA are relatively superficial and restricted to within the same sentence. F&H14 hypothesize that discourse features beyond the sentence level also help authorship attribution. In particular, they propose an author has a particular style for representing entities across a discourse. Their work is based on the entity-grid model of BIBREF6 (henceforth B&L).
Question: What was the previous state-of-the-art?
Paper: Leveraging Discourse Information Effectively for Authorship Attribution
Building on shrestha2017's work, we employ their character-bigram CNN (CNN2), and propose two extensions which utilize discourse information: (i) CNN2 enhanced with relation probability vectors (CNN2-PV), and (ii) CNN2 enhanced with discourse embeddings (CNN2-DE). The CNN2-PV allows us to conduct a comparison with F&H14 and F15, which also use relation probability vectors.
Question: How are discourse features incorporated into the model?
Answer: They derive entity grid with grammatical relations and RST discourse relations and concatenate them with pooling vector for the char-bigrams before feeding to the resulting vector to the softmax layer.
Paper: Leveraging Discourse Information Effectively for Authorship Attribution
We begin by introducing the datasets (Section SECREF15 ), followed by detailing the featurization methods (Section SECREF17 ), the experiments (Section SECREF22 ), and finally reporting results (Section SECREF26 ).
Question: What discourse features are used?
Answer: Entity grid with grammatical relations and RST discourse relations.
Paper: Towards Neural Theorem Proving at Scale
Recent advancements in deep learning intensified the long-standing interests in integrating symbolic reasoning with connectionist models BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 . The attraction of said integration stems from the complementing properties of these systems. Symbolic reasoning models offer interpretability, efficient generalisation from a small number of examples, and the ability to leverage knowledge provided by an expert. However, these systems are unable to handle ambiguous and noisy high-dimensional data such as sensory inputs BIBREF5 . On the other hand, representation learning models exhibit robustness to noise and ambiguity, can learn task-specific representations, and achieve state-of-the-art results on a wide variety of tasks BIBREF6 . However, being universal function approximators, these models require vast amounts of training data and are treated as non-interpretable black boxes.
Question: How are proof scores calculated?
Paper: Towards Neural Theorem Proving at Scale
In NTP, the neural network structure is built recursively, and its construction is defined in terms of modules similarly to dynamic neural module networks BIBREF19 . Each module, given a goal, a KB, and a current proof state as inputs, produces a list of new proof states, where the proof states are neural networks representing partial proof success scores.
Question: What are proof paths?
Answer: A sequence of logical statements represented in a computational graph
Paper: Neural Word Segmentation with Rich Pretraining
There has been a recent shift of research attention in the word segmentation literature from statistical methods to deep learning BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . Neural network models have been exploited due to their strength in non-sparse representation learning and non-linear power in feature combination, which have led to advances in many NLP tasks. So far, neural word segmentors have given comparable accuracies to the best statictical models.
Question: What is the size of the model?
Paper: Neural Word Segmentation with Rich Pretraining
Work on statistical word segmentation dates back to the 1990s BIBREF21 . State-of-the-art approaches include character sequence labeling models BIBREF22 using CRFs BIBREF23 , BIBREF24 and max-margin structured models leveraging word features BIBREF25 , BIBREF26 , BIBREF27 . Semi-supervised methods have been applied to both character-based and word-based models, exploring external training data for better segmentation BIBREF11 , BIBREF12 , BIBREF13 , BIBREF28 . Our work belongs to recent neural word segmentation.
Question: What external sources are used?
Answer: Raw data from Gigaword, Automatically segmented text from Gigaword, Heterogenous training data from People's Daily, POS data from People's Daily
Paper: Neural Word Segmentation with Rich Pretraining
Our segmentor works incrementally from left to right, as the example shown in Table TABREF1 . At each step, the state consists of a sequence of words that have been fully recognized, denoted as INLINEFORM0 , a current partially recognized word INLINEFORM1 , and a sequence of next incoming characters, denoted as INLINEFORM2 , as shown in Figure FIGREF4 . Given an input sentence, INLINEFORM3 and INLINEFORM4 are initialized to INLINEFORM5 and INLINEFORM6 , respectively, and INLINEFORM7 contains all the input characters. At each step, a decision is made on INLINEFORM8 , either appending it as a part of INLINEFORM9 , or seperating it as the beginning of a new word. The incremental process repeats until INLINEFORM10 is empty and INLINEFORM11 is null again ( INLINEFORM12 , INLINEFORM13 ). Formally, the process can be regarded as a state-transition process, where a state is a tuple INLINEFORM14 , and the transition actions include Sep (seperate) and App (append), as shown by the deduction system in Figure FIGREF7 .
Question: What submodules does the model consist of?
Paper: Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models
Recent advances in sequence-to-sequence learning architecture BIBREF0 and the transformer model BIBREF1 have raised increasing interest in natural language generation (NLG) tasks, including story generation BIBREF2, open-domain dialogue response generation BIBREF3 and abstractive summarization BIBREF4. Despite the fast advances of models, there remains a huge gap in the evaluation of NLG models and it is hard to measure the progress due to the lack of good evaluation metrics. While perplexity is a good measure of how well a model fits some data, it does not measure performance at the desired task. Word overlap based metrics such as BLEU BIBREF5, METEOR BIBREF6 and ROUGE BIBREF7 capture quality better than the perplexity and are useful in translation and summarization. However, they still correlate poorly with human evaluation BIBREF8 in open domain text generation tasks including story generation and dialogue response generation because two equally good generated texts may have no n-gram overlap. Human evaluation is generally considered to be the gold standard evaluation, however, it does not scale well as it is generally expensive and time-consuming to conduct human evaluation.
Question: How they add human prefference annotation to fine-tuning process?
Paper: Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models
Evaluation of NLG models has been a long-standing open problem. While human evaluation may be ideal, it is generally expensive to conduct and does not scale well. Various automated evaluation approaches are proposed to facilitate the development and evaluation of NLG models. We summarize these evaluation approaches below.
Question: What previous automated evalution approaches authors mention?
Paper: Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models
We present the proposed approach in this section. We begin with the sample-level pairwise comparison model. Afterwards, we introduce how to adopt the skill rating system to perform model-level evaluation of NLG models.
Question: How much better peformance is achieved in human evaluation when model is trained considering proposed metric?
Answer: Pearson correlation to human judgement - proposed vs next best metric
Sample level comparison:
- Story generation: 0.387 vs 0.148
- Dialogue: 0.472 vs 0.341
Model level comparison:
- Story generation:  0.631 vs 0.302
- Dialogue: 0.783 vs 0.553
Paper: Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models
The proposed comparative evaluator is a text pair relation classifier which is trained to compare the task-specific quality of two samples. The motivation of evaluating one sample by comparing it with another sample is drawn from the insight learned when conducting human evaluation for NLG models. We find that when comparing two NLG models, instead of asking human annotator to assign scores separately for samples generated by different models, which resembles the case in the ADEM model BIBREF14, it is much easier for human annotators to directly compare one sample generated by the first model against another sample from the second model pairwisely and compute the win/loss rate. The comparison-based evaluation may also be more accurate, which is demonstrated by a higher inter-annotator agreement score in our preliminary experiments.
Question: Do the authors suggest that proposed metric replace human evaluation on this task?
Paper: Learning to Ask Unanswerable Questions for Machine Reading Comprehension
Extractive reading comprehension BIBREF0 , BIBREF1 obtains great attentions from both research and industry in recent years. End-to-end neural models BIBREF2 , BIBREF3 , BIBREF4 have achieved remarkable performance on the task if answers are assumed to be in the given paragraph. Nonetheless, the current systems are still not good at deciding whether no answer is presented in the context BIBREF5 . For unanswerable questions, the systems are supposed to abstain from answering rather than making unreliable guesses, which is an embodiment of language understanding ability.
Question: What is the training objective of their pair-to-sequence model?
Paper: Learning to Ask Unanswerable Questions for Machine Reading Comprehension
Machine Reading Comprehension (MRC) Various large-scale datasets BIBREF0 , BIBREF1 , BIBREF11 , BIBREF12 , BIBREF5 , BIBREF13 have spurred rapid progress on machine reading comprehension in recent years. SQuAD BIBREF1 is an extractive benchmark whose questions and answers spans are annotated by humans. Neural reading comprehension systems BIBREF14 , BIBREF2 , BIBREF3 , BIBREF15 , BIBREF8 , BIBREF16 , BIBREF4 , BIBREF17 have outperformed humans on this task in terms of automatic metrics. The SQuAD 2.0 dataset BIBREF5 extends SQuAD with more than $50,000$ crowdsourced unanswerable questions. So far, neural reading comprehension models still fall behind humans on SQuAD 2.0. Abstaining from answering when no answer can be inferred from the given document does require more understanding than barely extracting an answer.
Question: How do they ensure the generated questions are unanswerable?
Paper: Learning to Ask Unanswerable Questions for Machine Reading Comprehension
Given an answerable question $q$ and its corresponding paragraph $p$ that contains the answer $a$ , we aim to generate unanswerable questions $\tilde{q}$ that fulfills certain requirements. First, it cannot be answered by paragraph $p$ . Second, it must be relevant to both answerable question $q$ and paragraph $p$ , which refrains from producing irrelevant questions. Third, it should ask for something of the same type as answer $a$ .
Question: Does their approach require a dataset of unanswerable questions mapped to similar answerable questions?
Paper: Evaluating KGR10 Polish word embeddings in the recognition of temporal expressions using BiLSTM-CRF
Recent studies in information extraction domain (but also in other natural language processing fields) show that deep learning models produce state-of-the-art results BIBREF0 . Deep architectures employ multiple layers to learn hierarchical representations of the input data. In the last few years, neural networks based on dense vector representations provided the best results in various NLP tasks, including named entities recognition BIBREF1 , semantic role labelling BIBREF2 , question answering BIBREF3 and multitask learning BIBREF4 . The core element of most deep learning solutions is the dense distributed semantic representation of words, often called word embeddings. Distributional vectors follow the distributional hypothesis that words with a similar meaning tend to appear in similar contexts. Word embeddings capture the similarity between words and are often used as the first layer in deep learning models. Two of the most common and very efficient methods to produce word embeddings are Continuous Bag-of-Words (CBOW) and Skip-gram (SG), which produce distributed representations of words in a vector space, grouping them by similarity BIBREF5 , BIBREF6 . With the progress of machine learning techniques, it is possible to train such models on much larger data sets, and these often outperform the simple ones. It is possible to use a set of text documents containing even billions of words as training data. Both architectures (CBOW and SG) describe how the neural network learns the vector word representations for each word. In CBOW architecture the task is predicting the word given its context and in SG the task in predicting the context given the word.
Question: What conclusions are drawn from these experiments?
Paper: Evaluating KGR10 Polish word embeddings in the recognition of temporal expressions using BiLSTM-CRF
At the time we were testing word embeddings for different applications, there were 2 most popular sources of word vectors. The first one, called IPIPAN, is the result of the project Compositional distributional semantic models for identification, discrimination and disambiguation of senses in Polish texts, the process of creating word embeddings is described in article BIBREF10 and corpora used were National Corpus of Polish (NKJP) BIBREF14 and Wikipedia (Wiki). The second one, called FASTTEXT, is original FastText word embeddings set, created for 157 languages (including Polish). Authors used Wikipedia and Common Crawl as the linguistic data source. Table TABREF6 shows the number of tokens in each corpus and the name of the institution which prepared it. There is also information about the public availability of the resource.
Question: What experiments are presented?
Paper: Evaluating KGR10 Polish word embeddings in the recognition of temporal expressions using BiLSTM-CRF
KGR7 corpus (also called plWordNet Corpus 7.0, PLWNC 7.0) BIBREF15 , BIBREF16 was created at the Wroclaw University of Science and Technology by G4.19 Group. Due to the licences of documents in this corpus, this resource is not publicly available. Table TABREF8 contains KGR7 subcorpora and statistics BIBREF17 . One of the subcorpora in KGR7 is KIPI (the IPI PAN Corpus) BIBREF18 . KGR7 covers texts from a wide range of domains like: blogs, science, stenographic recordings, news, journalism, books and parliamentary transcripts. All texts come from the second half of the 20th century and represent the modern Polish language.
Question: What is specific about the specific embeddings?
Paper: Evaluating KGR10 Polish word embeddings in the recognition of temporal expressions using BiLSTM-CRF
KGR10, also known as plWordNet Corpus 10.0 (PLWNC 10.0), is the result of the work on the toolchain to automatic acquisition and extraction of the website content, called CorpoGrabber BIBREF19 . It is a pipeline of tools to get the most relevant content of the website, including all subsites (up to the user-defined depth). The proposed toolchain can be used to build a big Web corpus of text documents. It requires the list of the root websites as the input. Tools composing CorpoGrabber are adapted to Polish, but most subtasks are language independent. The whole process can be run in parallel on a single machine and includes the following tasks: download of the HTML subpages of each input page URL with HTTrack, extraction of plain text from each subpage by removing boilerplate content (such as navigation links, headers, footers, advertisements from HTML pages) BIBREF20 , deduplication of plain text BIBREF20 , bad quality documents removal utilising Morphological Analysis Converter and Aggregator (MACA) BIBREF21 , documents tagging using Wrocław CRF Tagger (WCRFT) BIBREF22 . Last two steps are available only for Polish.
Question: What embedding algorithm is used to build the embeddings?
Paper: Evaluating KGR10 Polish word embeddings in the recognition of temporal expressions using BiLSTM-CRF
We created a new Polish word embeddings models using the KGR10 corpus. We built 16 models of word embeddings using the implementation of CBOW and Skip-gram methods in the FastText tool BIBREF9 . These models are available under an open license in the CLARIN-PL project DSpace repository. The internal encoding solution based on embeddings of n-grams composing each word makes it possible to obtain FastText vector representations, also for words which were not processed during the creation of the model. A vector representation is associated with character n-gram and each word is represented as the sum of its n-gram vector representations. Previous solutions ignored the morphology of words and were assigning a distinct vector to each word. This is a limitation for languages with large vocabularies and many rare words, like Turkish, Finnish or Polish BIBREF9 . Authors observed that using word representations trained with subword information outperformed the plain Skip-gram model and the improvement was most significant for morphologically rich Slavic languages such as Czech (8% reduction of perplexity over SG) and Russian (13% reduction) BIBREF9 . We expected that word embeddings created that way for Polish should also provide such improvements. There were also previous attempts to build KGR10 word vectors with other methods (including FastText), and the results are presented in the article BIBREF8 . We selected the best models from that article – with embedding ID prefix EP (embeddings, previous) in Table TABREF13 – to compare with new models, marked as embedding ID prefix EC in Table TABREF13 ).
Question: How was the KGR10 corpus created?
Paper: Speech Corpus of Ainu Folklore and End-to-end Speech Recognition for Ainu Language
Automatic speech recognition (ASR) technology has been made a dramatic progress and is currently brought to a pratical levels of performance assisted by large speech corpora and the introduction of deep learning techniques. However, this is not the case for low-resource languages which do not have large corpora like English and Japanese have. There are about 5,000 languages in the world over half of which are faced with the danger of extinction. Therefore, constructing ASR systems for these endangered languages is an important issue.
Question: How big are improvements with multilingual ASR training vs single language training?
Paper: Speech Corpus of Ainu Folklore and End-to-end Speech Recognition for Ainu Language
This section briefly overviews the background of the data collection, the Ainu language, and its writing system. After that, we describe how Ainu recordings are classified and review previous works dealing with the Ainu language.
Question: How much transcribed data is available for for Ainu language?
Answer: Transcribed data is available for duration of 38h 54m 38s for 8 speakers.
Paper: Speech Corpus of Ainu Folklore and End-to-end Speech Recognition for Ainu Language
The Ainu people had total population of about 20,000 in the mid-19th century BIBREF7 and they used to live widely distributed in the area that includes Hokkaido, Sakhalin, and the Kuril Islands. The number of native speakers, however, rapidly decreased through the assimilation policy after late 19th century. At present, there are only less than 10 native speakers, and UNESCO listed their language as critically endangered in 2009 BIBREF8. In response to this situation, Ainu folklore and songs have been actively recorded since the late 20th century in efforts initiated by the Government of Japan. For example, the Ainu Museum started audio recording of Ainu folklore in 1976 with the cooperation of a few Ainu elders which resulted in the collection of speech data with the total duration of roughly 700 hours. This kind of data should be a key to the understanding of Ainu culture, but most of it is not transcribed and fully studied yet.
Question: What is the difference between speaker-open and speaker-closed setting?
Paper: Revealing the Importance of Semantic Retrieval for Machine Reading at Scale
Extracting external textual knowledge for machine comprehensive systems has long been an important yet challenging problem. Success requires not only precise retrieval of the relevant information sparsely restored in a large knowledge source but also a deep understanding of both the selected knowledge and the input query to give the corresponding output. Initiated by chen2017drqa, the task was termed as Machine Reading at Scale (MRS), seeking to provide a challenging situation where machines are required to do both semantic retrieval and comprehension at different levels of granularity for the final downstream task.
Question: What baseline approaches do they compare against?
Answer: HotspotQA: Yang, Ding, Muppet
Fever: Hanselowski, Yoneda, Nie
Paper: Revealing the Importance of Semantic Retrieval for Machine Reading at Scale
Machine Reading at Scale First proposed and formalized in chen2017drqa, MRS has gained popularity with increasing amount of work on both dataset collection BIBREF5, BIBREF6 and MRS model developments BIBREF7, BIBREF8, BIBREF9. In some previous work BIBREF10, paragraph-level retrieval modules were mainly for improving the recall of required information, while in some other works BIBREF4, sentence-level retrieval modules were merely for solving the auxiliary sentence selection task. In our work, we focus on revealing the relationship between semantic retrieval at different granularity levels and the downstream comprehension task. To the best of our knowledge, we are the first to apply and optimize neural semantic retrieval at both paragraph and sentence levels for MRS.
Question: How do they train the retrieval modules?
Paper: Revealing the Importance of Semantic Retrieval for Machine Reading at Scale
In previous works, an MRS system can be complicated with different sub-components processing different retrieval and comprehension sub-tasks at different levels of granularity, and with some sub-components intertwined. For interpretability considerations, we used a unified pipeline setup. The overview of the system is in Fig. FIGREF2.
Question: How do they model the neural retrieval modules?
Paper: Revealing the Importance of Semantic Retrieval for Machine Reading at Scale
Throughout all our experiments, we used BERT-Base BIBREF2 to provide the state-of-the-art contextualized modeling of the input text.
Question: Retrieval at what level performs better, sentence level or paragraph level?