z-uo/qasper-squad huggingface dataset:
Paper: Deep Learning the EEG Manifold for Phonological Categorization from Active Thoughts
As we found the individually-trained parallel networks (CNN and LSTM) to be useful (see Table TABREF12 ), we suspected the combination of these two networks could provide a more powerful discriminative spatial and temporal representation of the data than each independent network. As such, we concatenate the last fully-connected layer from the CNN with its counterpart in the LSTM to compose a single feature vector based on these two penultimate layers. Ultimately, this forms a joint spatio-temporal encoding of the cross-covariance matrix.
Question: How many subjects does the EEG data come from?
Paper: A Human-AI Loop Approach for Joint Keyword Discovery and Expectation Estimation in Micropost Event Detection
Event detection on microblogging platforms such as Twitter aims to detect events preemptively. A main task in event detection is detecting events of predetermined types BIBREF0, such as concerts or controversial events based on microposts matching specific event descriptions. This task has extensive applications ranging from cyber security BIBREF1, BIBREF2 to political elections BIBREF3 or public health BIBREF4, BIBREF5. Due to the high ambiguity and inconsistency of the terms used in microposts, event detection is generally performed though statistical machine learning models, which require a labeled dataset for model training. Data labeling is, however, a long, laborious, and usually costly process. For the case of micropost classification, though positive labels can be collected (e.g., using specific hashtags, or event-related date-time information), there is no straightforward way to generate negative labels useful for model training. To tackle this lack of negative labels and the significant manual efforts in data labeling, BIBREF1 (BIBREF1, BIBREF3) introduced a weak supervision based learning approach, which uses only positively labeled data, accompanied by unlabeled examples by filtering microposts that contain a certain keyword indicative of the event type under consideration (e.g., `hack' for cyber security). Another key technique in this context is expectation regularization BIBREF6, BIBREF7, BIBREF1. Here, the estimated proportion of relevant microposts in an unlabeled dataset containing a keyword is given as a keyword-specific expectation. This expectation is used in the regularization term of the model's objective function to constrain the posterior distribution of the model predictions. By doing so, the model is trained with an expectation on its prediction for microposts that contain the keyword. Such a method, however, suffers from two key problems:
Question: Do they report results only on English data?
Paper: A Human-AI Loop Approach for Joint Keyword Discovery and Expectation Estimation in Micropost Event Detection
Given a set of labeled and unlabeled microposts, our goal is to extract informative keywords and estimate their expectations in order to train a machine learning model. To achieve this goal, our proposed human-AI loop approach comprises two crowdsourcing tasks, i.e., micropost classification followed by keyword discovery, and a unified probabilistic model for expectation inference and model training. Figure FIGREF6 presents an overview of our approach. Next, we describe our approach from a process-centric perspective.
Question: What type of classifiers are used?
Paper: A Human-AI Loop Approach for Joint Keyword Discovery and Expectation Estimation in Micropost Event Detection
This section introduces our probabilistic model that infers keyword expectation and trains the target model simultaneously. We start by formalizing the problem and introducing our model, before describing the model learning method.
Question: Which real-world datasets are used?
Answer: Tweets related to CyberAttack and tweets related to PoliticianDeath
Paper: A Human-AI Loop Approach for Joint Keyword Discovery and Expectation Estimation in Micropost Event Detection
First, we introduce an expectation regularization technique for the weakly supervised learning of the target model $p_{\theta ^{(t)}}(y|x)$. In this setting, the objective function of the target model is composed of two parts, corresponding to the labeled microposts $\mathcal {L}$ and the unlabeled ones $\mathcal {U}$.
Question: How are the interpretability merits of the approach demonstrated?
Answer: By involving humans for post-hoc evaluation of model's interpretability
Paper: A Human-AI Loop Approach for Joint Keyword Discovery and Expectation Estimation in Micropost Event Detection
To learn the keyword-specific expectation $e^{(t)}$ and the crowd worker reliability $\pi ^{(n)}$ ($1\le n\le N$), we model the likelihood of the crowd-contributed labels $\mathbf {A}$ as a function of these parameters. In this context, we view the expectation as the class prior, thus performing expectation inference as the learning of the class prior. By doing so, we connect expectation inference with model training.
Question: How are the accuracy merits of the approach demonstrated?
Answer: By evaluating the performance of the approach using accuracy and AUC
Paper: A Human-AI Loop Approach for Joint Keyword Discovery and Expectation Estimation in Micropost Event Detection
Integrating model training with expectation inference, the overall objective function of our proposed model is given by:
Question: How is the keyword specific expectation elicited from the crowd?
Paper: Macaw: An Extensible Conversational Information Seeking Platform
The rapid growth in speech and small screen interfaces, particularly on mobile devices, has significantly influenced the way users interact with intelligent systems to satisfy their information needs. The growing interest in personal digital assistants, such as Amazon Alexa, Apple Siri, Google Assistant, and Microsoft Cortana, demonstrates the willingness of users to employ conversational interactions BIBREF0. As a result, conversational information seeking (CIS) has been recognized as a major emerging research area in the Third Strategic Workshop on Information Retrieval (SWIRL 2018) BIBREF1.
Question: Does the paper provide any case studies to illustrate how one can use Macaw for CIS research?
Paper: Macaw: An Extensible Conversational Information Seeking Platform
Macaw has a modular design, with the goal of making it easy to configure and add new modules such as a different user interface or different retrieval module. The overall setup also follows a Model-View-Controller (MVC) like architecture. The design decisions have been made to smooth the Macaw's adoptions and extensions. Macaw is implemented in Python, thus machine learning models implemented using PyTorch, Scikit-learn, or TensorFlow can be easily integrated into Macaw. The high-level overview of Macaw is depicted in FIGREF8. The user interacts with the interface and the interface produces a Message object from the current interaction of user. The interaction can be in multi-modal form, such as text, speech, image, and click. Macaw stores all interactions in an “Interaction Database”. For every interaction, Macaw looks for most recent user-system interactions (including the system's responses) to create a list of Messages, called the conversation list. It is then dispatched to multiple information seeking (and related) actions. The actions run in parallel, and each should respond within a pre-defined time interval. The output selection component selects from (or potentially combines) the outputs generated by different actions and creates a Message object as the system's response. This message is logged into the interaction database and is sent to the interface to be presented to the user. Again, the response message can be multi-modal and include text, speech, link, list of options, etc.
Question: What functionality does Macaw provide?
Paper: Macaw: An Extensible Conversational Information Seeking Platform
The overview of retrieval and question answering actions in Macaw is shown in FIGREF17. These actions consist of the following components:
Question: What is a wizard of oz setup?
Answer: a setup where the seeker interacts with a real conversational interface and the wizard, an intermediary, performs actions related to the seeker's message
Paper: Macaw: An Extensible Conversational Information Seeking Platform
We have implemented the following interfaces for Macaw:
Question: What interface does Macaw currently have?
Paper: Macaw: An Extensible Conversational Information Seeking Platform
The current implementation of Macaw lacks the following actions. We intend to incrementally improve Macaw by supporting more actions and even more advanced techniques for the developed actions.
Question: What modalities are supported by Macaw?
Paper: Macaw: An Extensible Conversational Information Seeking Platform
Macaw is distributed under the MIT License. We welcome contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. This project has adopted the Microsoft Open Source Code of Conduct.
Question: What are the different modules in Macaw?
Paper: Automated News Suggestions for Populating Wikipedia Entity Pages
Wikipedia is the largest source of open and collaboratively curated knowledge in the world. Introduced in 2001, it has evolved into a reference work with around 5m pages for the English Wikipedia alone. In addition, entities and event pages are updated quickly via collaborative editing and all edits are encouraged to include source citations, creating a knowledge base which aims at being both timely as well as authoritative. As a result, it has become the preferred source of information consumption about entities and events. Moreso, this knowledge is harvested and utilized in building knowledge bases like YAGO BIBREF0 and DBpedia BIBREF1 , and used in applications like text categorization BIBREF2 , entity disambiguation BIBREF3 , entity ranking BIBREF4 and distant supervision BIBREF5 , BIBREF6 .
Question: Do they report results only on English data?
Paper: Automated News Suggestions for Populating Wikipedia Entity Pages
As we suggest a new problem there is no current work addressing exactly the same task. However, our task has similarities to Wikipedia page generation and knowledge base acceleration. In addition, we take inspiration from Natural Language Processing (NLP) methods for salience detection.
Question: What baseline model is used?
Answer: For Article-Entity placement, they consider two baselines: the first one using only salience-based features, and the second baseline checks if the entity appears in the title of the article. 

For Article-Section Placement, they consider two baselines: the first picks the section with the highest lexical similarity to the article, and the second one picks the most frequent section.
Paper: Automated News Suggestions for Populating Wikipedia Entity Pages
We are interested in named entities mentioned in documents. An entity INLINEFORM0 can be identified by a canonical name, and can be mentioned differently in text via different surface forms. We canonicalize these mentions to entity pages in Wikipedia, a method typically known as entity linking. We denote the set of canonicalized entities extracted and linked from a news article INLINEFORM1 as INLINEFORM2 . For example, in Figure FIGREF7 , entities are canonicalized into Wikipedia entity pages (e.g. Odisha is canonicalized to the corresponding article). For a collection of news articles INLINEFORM3 , we further denote the resulting set of entities by INLINEFORM4 .
Question: What news article sources are used?
Paper: Automated News Suggestions for Populating Wikipedia Entity Pages
We approach the news suggestion problem by decomposing it into two tasks:
Question: How do they determine the exact section to use the input article?
Answer: They use a multi-class classifier to determine the section it should be cited
Paper: Automated News Suggestions for Populating Wikipedia Entity Pages
In this section, we provide an overview of the news suggestion approach to Wikipedia entity pages (see Figure FIGREF7 ). The approach is split into two tasks: (i) article-entity (AEP) and (ii) article-section (ASP) placement. For a Wikipedia snapshot INLINEFORM0 and a news corpus INLINEFORM1 , we first determine which news articles should be suggested to an entity INLINEFORM2 . We will denote our approach for AEP by INLINEFORM3 . Finally, we determine the most appropriate section for the ASP task and we denote our approach with INLINEFORM4 .
Question: What features are used to represent the novelty of news articles to entity pages?
Answer: KL-divergences of language models for the news article and the already added news references
Paper: Automated News Suggestions for Populating Wikipedia Entity Pages
In this step we learn the function INLINEFORM0 to correctly determine whether INLINEFORM1 should be suggested for INLINEFORM2 , basically a binary classification model (0=`non-relevant' and 1=`relevant'). Note that we are mainly interested in finding the relevant pairs in this task. For every news article, the number of disambiguated entities is around 30 (but INLINEFORM3 is suggested for only two of them on average). Therefore, the distribution of `non-relevant' and `relevant' pairs is skewed towards the earlier, and by simply choosing the `non-relevant' label we can achieve a high accuracy for INLINEFORM4 . Finding the relevant pairs is therefore a considerable challenge.
Question: What features are used to represent the salience and relative authority of entities?
Answer: Salience features positional features, occurrence frequency and the internal POS structure of the entity and the sentence it occurs in.
The relative authority of entity features:   comparative relevance of the news article to the different entities occurring in it.
Paper: Named Entities in Medical Case Reports: Corpus and Experiments
The automatic processing of medical texts and documents plays an increasingly important role in the recent development of the digital health area. To enable dedicated Natural Language Processing (NLP) that is highly accurate with respect to medically relevant categories, manually annotated data from this domain is needed. One category of high interest and relevance are medical entities. Only very few annotated corpora in the medical domain exist. Many of them focus on the relation between chemicals and diseases or proteins and diseases, such as the BC5CDR corpus BIBREF0, the Comparative Toxicogenomics Database BIBREF1, the FSU PRotein GEne corpus BIBREF2 or the ADE (adverse drug effect) corpus BIBREF3. The NCBI Disease Corpus BIBREF4 contains condition mention annotations along with annotations of symptoms. Several new corpora of annotated case reports were made available recently. grouin-etal-2019-clinical presented a corpus with medical entity annotations of clinical cases written in French, copdPhenotype presented a corpus focusing on phenotypic information for chronic obstructive pulmonary disease while 10.1093/database/bay143 presented a corpus focusing on identifying main finding sentences in case reports.
Question: Do they experiment with other tasks?
Paper: Named Entities in Medical Case Reports: Corpus and Experiments
Case reports are standardized in the CARE guidelines BIBREF5. They represent a detailed description of the symptoms, signs, diagnosis, treatment, and follow-up of an individual patient. We focus on documents freely available through PubMed Central (PMC). The presentation of the patient's case can usually be found in a dedicated section or the abstract. We perform a manual annotation of all mentions of case entities, conditions, findings, factors and modifiers. The scope of our manual annotation is limited to the presentation of a patient's signs and symptoms. In addition, we annotate the title of the case report.
Question: What baselines do they introduce?
Paper: Named Entities in Medical Case Reports: Corpus and Experiments
We annotate the following entities:
Question: How large is the corpus?
Paper: Named Entities in Medical Case Reports: Corpus and Experiments
We asked medical doctors experienced in extracting knowledge related to medical entities from texts to annotate the entities described above. Initially, we asked four annotators to test our guidelines on two texts. Subsequently, identified issues were discussed and resolved. Following this pilot annotation phase, we asked two different annotators to annotate two case reports according to our guidelines. The same annotators annotated an overall collection of 53 case reports.
Question: How was annotation performed?
Answer: Experienced medical doctors used a linguistic annotation tool to annotate entities.
Paper: Named Entities in Medical Case Reports: Corpus and Experiments
The annotation was performed using WebAnno BIBREF7, a web-based tool for linguistic annotation. The annotators could choose between a pre-annotated version or a blank version of each text. The pre-annotated versions contained suggested entity spans based on string matches from lists of conditions and findings synonym lists. Their quality varied widely throughout the corpus. The blank version was preferred by the annotators. We distribute the corpus in BioC JSON format. BioC was chosen as it allows us to capture the complexities of the annotations in the biomedical domain. It represented each documents properties ranging from full text, individual passages/sentences along with captured annotations and relationships in an organized manner. BioC is based on character offsets of annotations and allows the stacking of different layers.
Question: How many documents are in the new corpus?
Paper: Named Entities in Medical Case Reports: Corpus and Experiments
The corpus consists of 53 documents, which contain an average number of 156.1 sentences per document, each with 19.55 tokens on average. The corpus comprises 8,275 sentences and 167,739 words in total. However, as mentioned above, only case presentation sections, headings and abstracts are annotated. The numbers of annotated entities are summarized in Table TABREF24.
Question: What baseline systems are proposed?
Paper: FacTweet: Profiling Fake News Twitter Accounts
Social media platforms have made the spreading of fake news easier, faster as well as able to reach a wider audience. Social media offer another feature which is the anonymity for the authors, and this opens the door to many suspicious individuals or organizations to utilize these platforms. Recently, there has been an increased number of spreading fake news and rumors over the web and social media BIBREF0. Fake news in social media vary considering the intention to mislead. Some of these news are spread with the intention to be ironic or to deliver the news in an ironic way (satirical news). Others, such as propaganda, hoaxes, and clickbaits, are spread to mislead the audience or to manipulate their opinions. In the case of Twitter, suspicious news annotations should be done on a tweet rather than an account level, since some accounts mix fake with real news. However, these annotations are extremely costly and time consuming – i.e., due to high volume of available tweets Consequently, a first step in this direction, e.g., as a pre-filtering step, can be viewed as the task of detecting fake news at the account level.
Question: How did they obtain the dataset?
Paper: FacTweet: Profiling Fake News Twitter Accounts
Given a news Twitter account, we read its tweets from the account's timeline. Then we sort the tweets by the posting date in ascending way and we split them into $N$ chunks. Each chunk consists of a sorted sequence of tweets labeled by the label of its corresponding account. We extract a set of features from each chunk and we feed them into a recurrent neural network to model the sequential flow of the chunks' tweets. We use an attention layer with dropout to attend over the most important tweets in each chunk. Finally, the representation is fed into a softmax layer to produce a probability distribution over the account types and thus predict the factuality of the accounts. Since we have many chunks for each account, the label for an account is obtained by taking the majority class of the account's chunks.
Question: What activation function do they use in their model?
Answer: Activation function is hyperparameter. Possible values: relu, selu, tanh.
Paper: FacTweet: Profiling Fake News Twitter Accounts
Data. We build a dataset of Twitter accounts based on two lists annotated in previous works. For the non-factual accounts, we rely on a list of 180 Twitter accounts from BIBREF1. This list was created based on public resources where suspicious Twitter accounts were annotated with the main fake news types (clickbait, propaganda, satire, and hoax). We discard the satire labeled accounts since their intention is not to mislead or deceive. On the other hand, for the factual accounts, we use a list with another 32 Twitter accounts from BIBREF19 that are considered trustworthy by independent third parties. We discard some accounts that publish news in languages other than English (e.g., Russian or Arabic). Moreover, to ensure the quality of the data, we remove the duplicate, media-based, and link-only tweets. For each account, we collect the maximum amount of tweets allowed by Twitter API. Table TABREF13 presents statistics on our dataset.
Question: What baselines do they compare to?
Paper: FacTweet: Profiling Fake News Twitter Accounts
In this paper, we proposed a model that utilizes chunked timelines of tweets and a recurrent neural model in order to infer the factuality of a Twitter news account. Our experimental results indicate the importance of analyzing tweet stream into chunks, as well as the benefits of heterogeneous knowledge source (i.e., lexica as well as text) in order to capture factuality. In future work, we would like to extend this line of research with further in-depth analysis to understand the flow change of the used features in the accounts' streams. Moreover, we would like to take our approach one step further incorporating explicit temporal information, e.g., using timestamps. Crucially, we are also interested in developing a multilingual version of our approach, for instance by leveraging the now ubiquitous cross-lingual embeddings BIBREF22, BIBREF23.
Question: How are chunks defined?
Answer: Chunks is group of tweets from single account that  is consecutive in time - idea is that this group can show secret intention of malicious accounts.
sequence of $s$ tweets
Paper: Text Summarization using Abstract Meaning Representation
Summarization of large texts is still an open problem in language processing. People nowadays have lesser time and patience to go through large pieces of text which make automatic summarization important. Automatic summarization has significant applications in summarizing large texts like stories, journal papers, news articles and even larger texts like books.
Question: What is the performance of their method?
Paper: Text Summarization using Abstract Meaning Representation
AMR was introduced by BIBREF1 with the aim to induce work on statistical Natural Language Understanding and Generation. AMR represents meaning using graphs. AMR graphs are rooted, directed, edge and vertex labeled graphs. Figure FIGREF4 shows the graphical representation of the AMR graph of the sentence "I looked carefully all around me" generated by JAMR parser ( BIBREF2 ). The graphical representation was produced using AMRICA BIBREF3 . The nodes in the AMR are labeled with concepts as in Figure FIGREF4 around represents a concept. Edges contains the information regarding the relations between the concepts. In Figure FIGREF4 direction is the relation between the concepts look-01 and around. AMR relies on Propbank for semantic relations (edge labels). Concepts can also be of the form run-01 where the index 01 represents the first sense of the word run. Further details about the AMR can be found in the AMR guidelines BIBREF4 .
Question: Which evaluation methods are used?
Answer: Quantitative evaluation methods using ROUGE, Recall, Precision and F1.