z-uo/qasper-squad huggingface dataset:
Paper: MonaLog: a Lightweight System for Natural Language Inference Based on Monotonicity
The goal of NLI is to determine, given a premise set $P$ and a hypothesis sentence $H$, whether $H$ follows from the meaning of $P$ BIBREF21. In this paper, we look at single-premise problems that involve making a standard 3-way classification decision (i.e., Entailment (H), Contradict (C) and Neutral (N)). Our general monotonicity reasoning system works according to the pipeline in Figure FIGREF1. Given a premise text, we first do Arrow Tagging by assigning polarity annotations (i.e., the arrows $\uparrow ,\downarrow $, which are the basic primitives of our logic) to tokens in text. These surface-level annotations, in turn, are associated with a set of natural logic inference rules that provide instructions for how to generate entailments and contradictions by span replacements over these arrows (which relies on a library of span replacement rules). For example, in the sentence All schoolgirls are on the train, the token schoolgirls is associated with a polarity annotation $\downarrow $, which indicates that in this sentential context, the span schoolgirls can be replaced with a semantically more specific concept (e.g., happy schoolgirls) in order to generate an entailment. A generation and search procedure is then applied to see if the hypothesis text can be generated from the premise using these inference rules. A proof in this model is finally a particular sequence of edits (e.g., see Figure FIGREF13) that derive the hypothesis text from the premise text rules and yield an entailment or contradiction.
Question: How do they combine MonaLog with BERT?
Answer: They use Monalog for data-augmentation to fine-tune BERT on this task
Paper: MonaLog: a Lightweight System for Natural Language Inference Based on Monotonicity
Given an input premise $P$, MonaLog first polarizes each of its tokens and constituents, calling the system described by BIBREF17, which performs polarization on a CCG parse tree. For example, a polarized $P$ could be every$^{\leavevmode {\color {red}\uparrow }}$ linguist$^{\leavevmode {\color {red}\downarrow }}$ swim$^{\leavevmode {\color {red}\uparrow }}$. Note that since we ignore morphology in the system, tokens are represented by lemmas.
Question: How do they select monotonicity facts?
Answer: They derive it from Wordnet
Paper: Question Dependent Recurrent Entity Network for Question Answering
Question Answering is a task that requires capabilities beyond simple NLP since it involves both linguistic techniques and inference abilities. Both the document sources and the questions are expressed in natural language, which is ambiguous and complex to understand. To perform such a task, a model needs in fact to understand the underlying meaning of text. Achieving this ability is quite challenging for a machine since it requires a reasoning phase (chaining facts, basic deductions, etc.) over knowledge extracted from the plain input data. In this article, we focus on two Question Answering tasks: a Reasoning Question Answering (RQA) and a Reading Comprehension (RC). These tasks are tested by submitting questions to be answered directly after reading a piece of text (e.g. a document or a paragraph).
Question: How does the model recognize entities and their relation to answers at inference time when answers are not accessible?
Paper: Smarnet: Teaching Machines to Read and Comprehend Like Human
Teaching machines to learn reading comprehension is one of the core tasks in NLP field. Recently machine comprehension task accumulates much concern among NLP researchers. We have witnessed significant progress since the release of large-scale datasets like SQuAD BIBREF0 , MS-MARCO BIBREF1 , TriviaQA BIBREF2 , CNN/Daily Mail BIBREF3 and Children's Book Test BIBREF4 . The essential problem of machine comprehension is to predict the correct answer referring to a given passage with relevant question. If a machine can obtain a good score from predicting the right answer, we can say the machine is capable of understanding the given context.
Question: What other solutions do they compare to?
Paper: Smarnet: Teaching Machines to Read and Comprehend Like Human
The goal of open-domain MC task is to infer the proper answer from the given text. For notation, given a passage INLINEFORM0 and a question INLINEFORM1 , where INLINEFORM2 and INLINEFORM3 are the length of the passage and the question. Each token is denoted as INLINEFORM4 , where INLINEFORM5 is the word embedding extracts from pre-trained word embedding lookups, INLINEFORM6 is the char-level matrix representing one-hot encoding of characters. The model should read and comprehend the interactions between INLINEFORM7 and INLINEFORM8 , and predict an answer INLINEFORM9 based on a continuous sub-span of INLINEFORM10 .
Question: How does the gatint mechanism combine word and character information?
Paper: Pre-Translation for Neural Machine Translation
In the last years, statistical machine translation (SMT) system generated state-of-the-art performance for most language pairs. Recently, systems using neural machine translation (NMT) were able to outperform SMT systems in several evaluations. These models are able to generate more fluent and accurate translation for most of sentences.
Question: Which dataset do they use?
Paper: Pre-Translation for Neural Machine Translation
The idea of linear combining of machine translation systems using different paradigms has already been used successfully for SMT and rule-based machine translation (RBMT) BIBREF2 , BIBREF3 . They build an SMT system that is post-editing the output of an RBMT system. Using the combination of SMT and RBMT, they could outperform both single systems.
Question: How is the PBMT system trained?
Paper: Pre-Translation for Neural Machine Translation
Starting with the initial work on word-based translation system BIBREF12 , phrase-based machine translation BIBREF13 , BIBREF14 segments the sentence into continuous phrases that are used as basic translation units. This allows for many-to-many alignments.
Question: Which NMT architecture do they use?
Paper: Pre-Translation for Neural Machine Translation
In this work, we want to combine the advantages of PBMT and NMT. Using the combined system we should be able to generate a translation for all words that occur at least once in the training data, while maintaining high quality translations for most sentences from NMT. Motivated by several approaches to simplify the translation process for PBMT using preprocessing, we will translate the source as a preprocessing step using the phrase-base machine translation system.
Question: Do they train the NMT model on PBMT outputs?
Paper: Developing a Fine-Grained Corpus for a Less-resourced Language: the case of Kurdish
Kurdish is an Indo-European language mainly spoken in central and eastern Turkey, northern Iraq and Syria, and western Iran. It is a less-resourced language BIBREF0, in other words, a language for which general-purpose grammars and raw internet-based corpora are the main existing resources. The language is spoken in five main dialects, namely, Kurmanji (aka Northern Kurdish), Sorani (aka Central Kurdish), Southern Kurdish, Zazaki and Gorani BIBREF1.
Question: Is the corpus annotated?
Paper: Developing a Fine-Grained Corpus for a Less-resourced Language: the case of Kurdish
Although the initiative to create a corpus for Kurdish dates back to 1998 BIBREF5, efforts in creating machine-readable corpora for Kurdish are recent. The first machine-readable corpus for Kurdish is the Leipzig Corpora Collection which is constructed using different sources on the Web BIBREF6. Later, Pewan BIBREF2 and Bianet BIBREF7 were developed as general-purpose corpora based on news articles. Kurdish corpora are also constructed for specific tasks such as dialectology BIBREF8, BIBREF3, machine transliteration BIBREF9, and part-of-speech (POS) annotation BIBREF10, BIBREF11. However, to the best of our knowledge, currently, there is no domain-specific corpus for Kurdish dialects.
Question: How is the corpus normalized?
Paper: Developing a Fine-Grained Corpus for a Less-resourced Language: the case of Kurdish
KTC is composed of 31 educational textbooks published from 2011 to 2018 in various topics by the MoE. We received the material from the MoE partly in different versions of Microsoft Word and partly in Adobe InDesign formats. In the first step, we categorized each textbook based on the topics and chapters. As the original texts were not in Unicode, we converted the content to Unicode. This step was followed by a pre-processing stage where the texts were normalized by replacing zero-width-non-joiner (ZWNJ) BIBREF2 and manually verifying the orthography based on the reference orthography of the Kurdistan Region of Iraq. In the normalization process, we did not remove punctuation and special characters so that the corpus can be easily adapted our current task and also to future tasks where the integrity of the text may be required.
Question: What are the 12 categories devised?
Answer: Economics, Genocide, Geography, History, Human Rights, Kurdish, Kurdology, Philosophy, Physics, Theology, Sociology, Social Study
Paper: Developing a Fine-Grained Corpus for a Less-resourced Language: the case of Kurdish
Previously, researchers have addressed the challenges in Kurdish corpora development BIBREF2, BIBREF13, BIBREF3. We highlight two main challenges we faced during the KTC development. First, most of the written Kurdish resources have not been digitized BIBREF14, or they are either not publicly available or are not fully convertible. Second, Kurdish text processing suffers from different orthographic issues BIBREF9 mainly due to the lack of standard orthography and the usage of non-Unicode keyboards. Therefore, we carried out a semi-automatic conversion, which made the process costly in terms of time and human assistance.
Question: Is the corpus annotated with a phonetic transcription?
Paper: Developing a Fine-Grained Corpus for a Less-resourced Language: the case of Kurdish
We presented KTC–the Kurdish Textbook Corpus, as the first domain-specific corpus for Sorani Kurdish. This corpus will pave the way for further developments in Kurdish language processing. We have mad the corpus available at https://github.com/KurdishBLARK/KTC for non-commercial use. We are currently working on a project on the Sorani spelling error detection and correction. As future work, we are aiming to develop a similar corpus for all Kurdish dialects, particularly Kurmanji.
Question: Is the corpus annotated with Part-of-Speech tags?
Paper: Automatic Language Identification in Texts: A Survey
Language identification (“”) is the task of determining the natural language that a document or part thereof is written in. Recognizing text in a specific language comes naturally to a human reader familiar with the language. intro:langid presents excerpts from Wikipedia articles in different languages on the topic of Natural Language Processing (“NLP”), labeled according to the language they are written in. Without referring to the labels, readers of this article will certainly have recognized at least one language in intro:langid, and many are likely to be able to identify all the languages therein.
Question: what evaluation methods are discussed?
Paper: Automatic Language Identification in Texts: A Survey
is in some ways a special case of text categorization, and previous research has examined applying standard text categorization methods to BIBREF7 , BIBREF8 .
Question: what are the off-the-shelf systems discussed in the paper?
Answer: Answer with content missing: (Names of many identifiers missing) TextCat, ChromeCLD, LangDetect, langid.py, whatlang, whatthelang, YALI, LDIG, Polyglot 3000, Lextek Language Identifier and Open Xerox Language Identifier.
Paper: HireNet: a Hierarchical Attention Model for the Automatic Analysis of Asynchronous Video Job Interviews
Among assessment methods, the job interview remains the most common way to evaluate candidates. The interview can be done via phone, live video, face to face, or more recently asynchronous video interview. For the latter, candidates connect to a platform, and record themselves while answering a set of questions chosen by the recruiter. The platform then allows several recruiters to evaluate the candidate, to discuss among themselves and possibly to invite the candidate to a face-to-face interview. Recruiters choose to use these platforms because it gives them access to a larger pool of candidates, and it speeds up the application processing time. In addition, it allows candidates to do the interview whenever and wherever it suits them the most. However, given a large number of these asynchronous interviews it may quickly become unmanageable for recruiters. The highly structured characteristic of asynchronous video interviews (same questions, same amount of time per candidate) enhances their predictive validity, and reduces inter-recruiter variability BIBREF0 . Moreover, recent advances in Social Signal Processing (SSP) BIBREF1 have enabled automated candidate assessment BIBREF2 , and companies have already started deploying solutions serving that purpose. However, previous studies used corpora of simulated interviews with limited sizes. The work proposed in this paper relies on a corpus that has been built in collaboration with a company and that consists of more than 7000 real job interviews for 475 open positions. The size of this corpus enables the exploration of emerging models such as deep learning models, that are known to be difficult to deploy for Social Computing because of the difficulty to obtain large annotations of social behaviors. Based on those facts, we propose HireNet, a new hierarchical attention neural network for the purpose of automatically classifying candidates into two classes: hirable and not hirable. Our model aims to assist recruiters in the selection process. It does not aim to make any automatic decision about candidate selection. First, this model was built to mirror the sequential and hierarchical structure of an interview assessment: recruiters watch a sequence of questions and answers, which are themselves sequences of words or behavioral signals. Second, the HireNet model integrates the context of the open position (questions during the interview and job title) in order both to determine the relative importance between question-answer pairs and to highlight important behavioral cues with regard to a question. Third, HireNet attention mechanisms enhance the interpretability of our model for each modality. In fact, they provide a way for recruiters to validate and trust the model through visualization, and possibly for candidates to locate their strengths or areas of improvement in an interview.
Question: How is "hirability" defined?
Paper: HireNet: a Hierarchical Attention Model for the Automatic Analysis of Asynchronous Video Job Interviews
To the best of our knowledge, only one corpus of interviews with real open positions has been collected and is subject to automatic analysis BIBREF3 . This corpus consists of face-to-face job interviews for a marketing short assignment whose candidates are mainly students. There are video corpora of face-to-face mock interviews that include two corpora built at the Massachusetts Institute of Technology BIBREF4 , BIBREF5 , and a corpus of students in services related to hospitality BIBREF6 . Many corpora of simulated asynchronous video interviews have also been built: a corpus of employees BIBREF7 , a corpus of students from Bangalore University BIBREF8 and a corpus collected through the use of crowdsourcing tools BIBREF2 . Some researchers are also interested in online video resumes and have constituted a corpus of video CVs from YouTube BIBREF9 . A first impressions challenge dataset was also supplemented by hirability annotation BIBREF10 . Some corpora are annotated by experts or students in psychology BIBREF7 , BIBREF2 , BIBREF3 , BIBREF11 . Other corpora have used crowdsourcing platforms or naive observers BIBREF8 for annotation. Table TABREF2 contains a summary of the corpora of job interviews used in previous works.
Question: Have the candidates given their consent to have their videos used for the research?
Paper: HireNet: a Hierarchical Attention Model for the Automatic Analysis of Asynchronous Video Job Interviews
Features Recent advances in SSP have offered toolboxes to extract features from audio BIBREF13 and video streams BIBREF14 . As asynchronous job interviews are videos, features from each modality (verbal content, audio and video) have to be extracted frame by frame in order to build a classification model. Audio cues consist mainly of prosody features (fundamental frequency, intensity, mel-frequency cepstral coefficients, etc) and speaking activity (pauses, silences, short utterances, etc) BIBREF15 , BIBREF12 . Features derived from facial expressions (facial actions units, head rotation and position, gaze direction, etc) constitute the most extracted visual cues BIBREF2 . Finally, advances in automatic speech recognition have enabled researchers to use the verbal content of candidates. In order to describe the verbal content, researchers have used lexical statistics (number of words, number of unique words, etc), dictionaries (Linguistic Inquiry Word Count) BIBREF12 , topic modeling BIBREF5 , bag of words or more recently document embedding BIBREF7 .
Question: Do they analyze if their system has any bias?
Paper: HireNet: a Hierarchical Attention Model for the Automatic Analysis of Asynchronous Video Job Interviews
Neural networks have proven to be successful in numerous Social Computing tasks. Multiple architectures in the field of neural networks have outperformed hand crafted features for emotion detection in videos BIBREF18 , facial landmarks detection BIBREF14 , document classification BIBREF19 These results are explained by the capability of neural networks to automatically perform useful transformations on low level features. Moreover, some architectures such as Recurrent Neural Networks were especially tailored to represent sequences. In addition, attention mechanisms have proven to be successful in highlighting salient information enhancing the performance and interpretability of neural networks. For example, in rapport detection, attention mechanisms allow to focus only on important moments during dyadic conversations BIBREF20 . Finally, numerous models have been proposed to model the interactions between modalities in emotion detection tasks through attention mechanisms BIBREF21 , BIBREF18 .
Question: Is there any ethical consideration in the research?
Paper: Cross-Lingual Adaptation Using Universal Dependencies
Universal Dependencies (UD) BIBREF0, BIBREF1, BIBREF2 is an ongoing project aiming to develop cross-lingually consistent treebanks for different languages. UD provided a framework for consistent annotation of grammar (parts of speech, morphological features, and syntactic dependencies) across different human languages. The annotation schema relies on Universal Stanford Dependencies BIBREF3 and Google Universal POS tags BIBREF4. The general principle is to provide universal annotation ; meanwhile, each language can add language-specific relations to the universal pool when necessary.
Question: What low-resource languages were used in this work?
Paper: Cross-Lingual Adaptation Using Universal Dependencies
The Universal Dependencies project aims to produce consistent dependency treebanks and parsers for many languages BIBREF0, BIBREF1, BIBREF2. The most important achievements of the project are the cross-lingual annotation guidelines and sets of universal POS and the grammatical relation tags. Consequentially many treebanks have been developed for different languages. The general rule of UD project is to provide a universal tag set ; however each language can add language-specific relations to the universal pool or omit some tags.
Question: What classification task was used to evaluate the cross-lingual adaptation method described in this work?
Paper: End-to-End Streaming Keyword Spotting
Keyword detection is like searching for a needle in a haystack: the detector must listen to continuously streaming audio, ignoring nearly all of it, yet still triggering correctly and instantly. In the last few years, with the advent of voice assistants, keyword spotting has become a common way to initiate a conversation with them (e.g. "Ok Google", "Alexa", or "Hey Siri"). As the assistant use cases spread through a variety of devices, from mobile phones to home appliances and further into the internet-of-things (IoT) –many of them battery powered or with restricted computational capacity, it is important for the keyword spotting system to be both high-quality as well as computationally efficient.
Question: How many parameters does the presented model have?
Paper: End-to-End Streaming Keyword Spotting
This paper proposes a new end-to-end keyword spotting system that by subsuming both the encoding and decoding components into a single neural network can be trained to produce directly an estimation (i.e. score) of the presence of a keyword in streaming audio. The following two sections cover the efficient memoized neural network topology being utilized, as well as the method to train the end-to-end neural network to directly produce the keyword spotting score.
Question: How do they measure the quality of detection?
Paper: End-to-End Streaming Keyword Spotting
We make use of a type of neural network layer topology called SVDF (single value decomposition filter), originally introduced in BIBREF14 to approximate a fully connected layer with a low rank approximation. As proposed in BIBREF14 and depicted in equation EQREF2 , the activation INLINEFORM0 for each node INLINEFORM1 in the rank-1 SVDF layer at a given inference step INLINEFORM2 can be interpreted as performing a mix of selectivity in time ( INLINEFORM3 ) with selectivity in the feature space ( INLINEFORM4 ) over a sequence of input vectors INLINEFORM5 of size INLINEFORM6 . DISPLAYFORM0 
Question: What previous approaches are considered?
Paper: Neural Semantic Parsing in Low-Resource Settings with Back-Translation and Meta-Learning
Semantic parsing aims to map natural language questions to the logical forms of their underlying meanings, which can be regarded as programs and executed to yield answers, aka denotations BIBREF0 . In the past few years, neural network based semantic parsers have achieved promising performances BIBREF1 , however, their success is limited to the setting with rich supervision, which is costly to obtain. There have been recent attempts at low-resource semantic parsing, including data augmentation methods which are learned from a small number of annotated examples BIBREF2 , and methods for adapting to unseen domains while only being trained on annotated examples in other domains.
Question: How is the back-translation model trained?
Paper: Neural Semantic Parsing in Low-Resource Settings with Back-Translation and Meta-Learning
We focus on the task of executive semantic parsing. The goal is to map a natural language question/utterance INLINEFORM0 to a logical form/program INLINEFORM1 , which can be executed over a world INLINEFORM2 to obtain the correct answer INLINEFORM3 .
Question: Are the rules dataset specific?
Paper: Neural Semantic Parsing in Low-Resource Settings with Back-Translation and Meta-Learning
We describe our approach for low-resource neural semantic parsing in this section.
Question: How many rules had to be defined?
Answer: WikiSQL - 2 rules (SELECT, WHERE)
SimpleQuestions - 1 rule
SequentialQA - 3 rules (SELECT, WHERE, COPY)