z-uo/qasper-squad huggingface dataset:
Paper: Character-Level Question Answering with Attention
Our work is motivated by three major threads of research in machine learning and natural language processing: semantic-parsing for open-domain question answering, character-level language modeling, and encoder-decoder methods.
Question: What word level and character level model baselines are used?
Answer: None
Word-level Memory Neural Networks (MemNNs) proposed in Bordes et al. (2015)
Paper: Improving the Performance of Neural Machine Translation Involving Morphologically Rich Languages
The use of RNNs in the field of Statistical Machine Translation (SMT) has revolutionised the approaches to automated translation. As opposed to traditional shallow SMT models, which require a lot of memory to run, these neural translation models require only a small fraction of memory used, about 5% BIBREF0 . Also, neural translation models are optimized such that every module is trained to jointly improve translation quality. With that being said, one of the main downsides of neural translation models is the heavy corpus requirement in order to ensure learning of deeper contexts. This is where the application of these encoder decoder architectures in translation to and/or from morphologically rich languages takes a severe hit.
Question: By how much do they improve the efficacy of the attention mechanism?
Paper: Improving the Performance of Neural Machine Translation Involving Morphologically Rich Languages
The corpus selected for this experiment was a combination of different corpora from various domains. The major part of the corpus was made up by the EnTam v2 corpus BIBREF2 . This corpus contained sentences taken from parallel news articles, English and Tamil bible corpus and movie subtitles. It also comprised of a tourism corpus that was obtained from TDIL (Technology Development for Indian Languages) and a corpus created from Tamil novels and short stories from AU-KBC, Anna university. The complete corpus consisted of 197,792 sentences. Fig. FIGREF20 shows the skinny shift and heatmap representations of the relativity between the sentences in terms of their sentence lengths.
Question: How were the human judgements assembled?
Answer: 50 human annotators ranked a random sample of 100 translations by Adequacy, Fluency and overall ranking on a 5-point scale.
Paper: To Swap or Not to Swap? Exploiting Dependency Word Pairs for Reordering in Statistical Machine Translation
Reordering in machine translation (MT) is a crucial process to get the correct translation output word order given an input source sentence, as word order reflects meaning. It remains a major challenge, especially for language pairs with a significant word order difference. Phrase-based MT systems BIBREF0 generally adopt a reordering model that predicts reordering based on the span of a phrase and that of the adjacent phrase BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 .
Question: Did they only experiment with one language pair?
Paper: Fine-Grained Named Entity Recognition using ELMo and Wikidata
Named entity recognition (NER) BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 is the process by which we identify text spans which mention named entities, and to classify them into predefined categories such as person, location, organization etc. NER serves as the basis for a variety of natural language processing (NLP) applications such as relation extraction BIBREF4 , machine translation BIBREF5 , question answering BIBREF6 and knowledge base construction BIBREF7 . Although early NER systems have been successful in producing adequate recognition accuracy, they often require significant human effort in carefully designing rules or features.
Question: Which other approaches do they compare their model with?
Answer: Akbik et al. (2018), Link et al. (2012)
They compare to Akbik et al. (2018) and Link et al. (2012).
Paper: Fine-Grained Named Entity Recognition using ELMo and Wikidata
We evaluate our model on two publicly available datasets. The statistics for both are shown in Table TABREF3 . The details of these datasets are as follows:
Question: What results do they achieve using their proposed approach?
Answer: F-1 score on the OntoNotes is 88%, and it is 53% on Wiki (gold).
Paper: Fine-Grained Named Entity Recognition using ELMo and Wikidata
NER involves identifying both entity boundaries and entity types. With “exact-match evaluation”, a named entity is considered correctly recognized only if both the boundaries and type match the ground truth BIBREF8 , BIBREF17 , BIBREF18 . Precision, Recall, and F-1 scores are computed on the number of true positives (TP), false positives (FP), and false negatives (FN). Their formal definitions are as follows:
Question: How do they combine a deep learning model with a knowledge base?
Answer: Entities from a deep learning model are linked to the related entities from a knowledge base by a lookup.
Paper: A Resource for Computational Experiments on Mapudungun
Recent years have seen unprecedented progress for Natural Language Processing (NLP) on almost every NLP subtask. Even though low-resource settings have also been explored, this progress has overwhelmingly been observed in languages with significant data resources that can be leveraged to train deep neural networks. Low-resource languages still lag behind.
Question: What are the models used for the baseline of the three NLP tasks?
Answer: For speech synthesis, they build a speech clustergen statistical speech synthesizer BIBREF9. For speech recognition, they use Kaldi BIBREF11. For Machine Translation, they use a Transformer architecture from BIBREF15.
Paper: A Resource for Computational Experiments on Mapudungun
Mapudungun (iso 639-3: arn) is an indigenous language of the Americas spoken natively in Chile and Argentina, with an estimated 100 to 200 thousand speakers in Chile and 27 to 60 thousand speakers in Argentina BIBREF0. It is an isolate language and is classified as threatened by Ethnologue, hence the critical importance of all documentary efforts. Although the morphology of nouns is relatively simple, Mapudungun verb morphology is highly agglutinative and complex. Some analyses provide as many as 36 verb suffix slots BIBREF1. A typical complex verb form occurring in our corpus of spoken Mapudungun consists of five or six morphemes.
Question: How is non-standard pronunciation identified?
Answer: Original transcription was labeled with additional labels in [] brackets with nonstandard pronunciation.
Paper: A matter of words: NLP for quality evaluation of Wikipedia medical articles
As observed by a recent article of Nature News BIBREF0 , “Wikipedia is among the most frequently visited websites in the world and one of the most popular places to tap into the world's scientific and medical information". Despite the huge amount of consultations, open issues still threaten a fully confident fruition of the popular online open encyclopedia.
Question: Is it valid to presume a bad medical wikipedia article should not contain much domain-specific jargon?
Paper: Why So Down? The Role of Negative (and Positive) Pointwise Mutual Information in Distributional Semantics
Dense word vectors (or embeddings) are a key component in modern NLP architectures for tasks such as sentiment analysis, parsing, and machine translation. These vectors can be learned by exploiting the distributional hypothesis BIBREF0, paraphrased by BIBREF1 as “a word is characterized by the company that it keeps”, usually by constructing a cooccurrence matrix over a training corpus, re-weighting it using Pointwise Mutual Information ($\mathit {PMI}$) BIBREF2, and performing a low-rank factorization to obtain dense vectors.
Question: What novel PMI variants are introduced?
Answer: clipped PMI; NNEGPMI
Paper: Why So Down? The Role of Negative (and Positive) Pointwise Mutual Information in Distributional Semantics
There is a long history of studying weightings (also known as association measures) of general (not only word-context) cooccurrence matrices; see BIBREF3, BIBREF4 for an overview and BIBREF5 for comparison of different weightings. BIBREF6 show that word vectors derived from $\mathit {PPMI}$ matrices perform better than alternative weightings for word-context cooccurrence. In the field of collocation extraction, BIBREF7 address the negative infinity issue with $\mathit {PMI}$ by introducing the normalized $\mathit {PMI}$ metric. BIBREF8 show theoretically that the popular Skip-gram model BIBREF9 performs implicit factorization of shifted $\mathit {PMI}$.
Question: What semantic and syntactic tasks are used as probes?
Paper: Why So Down? The Role of Negative (and Positive) Pointwise Mutual Information in Distributional Semantics
PMI: A cooccurrence matrix $M$ is constructed by sliding a symmetric window over the subsampled BIBREF9 training corpus and for each center word $w$ and context word $c$ within the window, incrementing $M_{wc}$. $\mathit {PMI}$ is then equal to:
Question: What are the disadvantages to clipping negative PMI?
Answer: It may lead to poor rare word representations and word analogies.
Paper: Why So Down? The Role of Negative (and Positive) Pointwise Mutual Information in Distributional Semantics
In order to identify the role that $\mathit {\texttt {-}PMI}$ and $\mathit {\texttt {+}PMI}$ play in distributional semantics, we train LexVec models that skip SGD steps when target cell values are $>0$ or $\le 0$, respectively. For example, $-\mathit {CPMI}_{\texttt {-}2}$ skips steps when $\mathit {CPMI}_{\texttt {-}2}(w,c) > 0$. Similarly, the $\mathit {\texttt {+}PPMI}$ model skips SGD steps when $\mathit {PPMI}(w,c) \le 0$. We compare these to models that include both negative and positive information to see how the two interact.
Question: Why are statistics from finite corpora unreliable?
Answer: A finite corpora may entirely omit rare word combinations
Paper: Towards speech-to-text translation without speech recognition
Typical speech-to-text translation systems pipeline automatic speech recognition (ASR) and machine translation (MT) BIBREF0 . But high-quality ASR requires hundreds of hours of transcribed audio, while high-quality MT requires millions of words of parallel text—resources available for only a tiny fraction of the world's estimated 7,000 languages BIBREF1 . Nevertheless, there are important low-resource settings in which even limited speech translation would be of immense value: documentation of endangered languages, which often have no writing system BIBREF2 , BIBREF3 ; and crisis response, for which text applications have proven useful BIBREF4 , but only help literate populations. In these settings, target translations may be available. For example, ad hoc translations may be collected in support of relief operations. Can we do anything at all with this data?
Question: what is the domain of the corpus?
Paper: Towards speech-to-text translation without speech recognition
For UTD we use the Zero Resource Toolkit (ZRTools; Jansen and Van Durme, 2011). ZRTools uses dynamic time warping (DTW) to discover pairs of acoustically similar audio segments, and then uses graph clustering on overlapping pairs to form a hard clustering of the discovered segments. Replacing each discovered segment with its unique cluster label, or pseudoterm, gives us a partial, noisy transcription, or pseudotext (Fig. FIGREF4 ).
Question: what challenges are identified?
Paper: Towards speech-to-text translation without speech recognition
Although we did not have access to a low-resource dataset, there is a corpus of noisy multi-speaker speech that simulates many of the conditions we expect to find in our motivating applications: the CALLHOME Spanish–English speech translation dataset (LDC2014T23; Post el al., 2013). We ran UTD over all 104 telephone calls, which pair 11 hours of audio with Spanish transcripts and their crowdsourced English translations. The transcripts contain 168,195 Spanish word tokens (10,674 types), and the translations contain 159,777 English word tokens (6,723 types). Though our system does not require Spanish transcripts, we use them to evaluate UTD and to simulate a perfect UTD system, called the oracle.
Question: what is the size of the speech corpus?
Paper: Understanding language-elicited EEG data by predicting it from a fine-tuned language model
The cognitive processes involved in human language comprehension are complex and only partially identified. According to the dual-stream model of speech comprehension BIBREF1 , sound waves are first converted to phoneme-like features and further processed by a ventral stream that maps those features onto words and semantic structures, and a dorsal stream that (among other things) supports audio-short term memory. The mapping of words onto meaning is thought to be subserved by widely distributed regions of the brain that specialize in particular modalities — for example visual aspects of the word banana reside in the occipital lobe of the brain and are activated when the word banana is heard BIBREF2 — and the different representation modalities are thought to be integrated into a single coherent latent representation in the anterior temporal lobe BIBREF3 . While this part of meaning representation in human language comprehension is somewhat understood, much less is known about how the meanings of words are integrated together to form the meaning of sentences and discourses. One tool researchers use to study the integration of meaning across words is electroencephelography (EEG), which measures the electrical activity of large numbers of neurons acting in concert. EEG has the temporal resolution necessary to study the processes involved in meaning integration, and certain stereotyped electrical responses to word presentations, known as event-related potentials (ERPs), have been identified with some of the processes thought to contribute to comprehension.
Question: Which two pairs of ERPs from the literature benefit from joint training?
Answer: Answer with content missing: (Whole Method and Results sections) Self-paced reading times widely benefit ERP prediction, while eye-tracking data seems to have more limited benefit to just the ELAN, LAN, and PNP ERP components.
Select:
- ELAN, LAN
- PNP ERP
Paper: Understanding language-elicited EEG data by predicting it from a fine-tuned language model
While a full discussion of each ERP component and the features of language thought to trigger each are beyond the scope of this document (for reviews see e.g. BIBREF0 , BIBREF2 , BIBREF4 , BIBREF5 , and BIBREF6 ), we introduce some basic features of ERP components to help in the discussion later. ERP components are electrical potential responses measured with respect to a baseline that are triggered by an event (in our case the presentation of a new word to a participant in an experiment). The name of each ERP component reflects whether the potential is positive or negative relative to the baseline. The N400 is so-named because it is Negative relative to a baseline (the baseline is typically recorded just before a word is presented at an electrode that is not affected by the ERP response) and because it peaks in magnitude at about 400ms after a word is presented to a participant in an experiment. The P600 is Positive relative to a baseline and peaks around 600ms after a word is presented to a participant (though its overall duration is much longer and less specific in time than the N400). The post-N400 positivity is so-named because it is part of a biphasic response; it is a positivity that occurs after the negativity associated with the N400. The early post-N400 positivity (EPNP) is also part of a biphasic response, but the positivity has an eariler onset than the standard PNP. Finally, the LAN and ELAN are the left-anterior negativity and early left-anterior negativity respectively. These are named for their timing, spatial distribution on the scalp, and direction of difference from the baseline. It is important to note that ERP components can potentially cancel and mask each other, and that it is difficult to precisely localize the neural activity that causes the changes in electrical potential at the electrodes where those changes are measured.
Question: What datasets are used?
Answer: Answer with content missing: (Whole Method and Results sections) The primary dataset we use is the ERP data collected and computed by Frank et al. (2015), and we also use behavioral data (eye-tracking data and self-paced reading times) from Frank et al. (2013) which were collected on the same set of 205 sentences.
Select:
- ERP data collected and computed by Frank et al. (2015)
- behavioral data (eye-tracking data and self-paced reading times) from Frank et al. (2013)
Paper: External Lexical Information for Multilingual Part-of-Speech Tagging
Part-of-speech tagging is now a classic task in natural language processing, for which many systems have been developed or adapted for a large variety of languages. Its aim is to associate each “word” with a morphosyntactic tag, whose granularity can range from a simple morphosyntactic category, or part-of-speech (hereafter PoS), to finer categories enriched with morphological features (gender, number, case, tense, mood, etc.).
Question: which datasets did they experiment with?
Answer: Universal Dependencies v1.2 treebanks for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German,
Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish, and Swedish
Paper: External Lexical Information for Multilingual Part-of-Speech Tagging
MElt BIBREF12 is a tagging system based on maximum entropy Markov models (MEMM) BIBREF5 , a class of discriminative models that are suitable for sequence labelling BIBREF5 . The basic set of features used by MElt is given in BIBREF12 . It is a superset of the feature sets used by BIBREF5 and BIBREF24 and includes both local standard features (for example the current word itself and its prefixes and suffixes of length 1 to 4) and contextual standard features (for example the tag just assigned to the preceding word). In particular, with respect to Ratnaparkhi's feature set, MElt's basic feature set lifts the restriction that local standard features used to analyse the internal composition of the current word should only apply to rare words.
Question: which languages are explored?
Paper: Disunited Nations? A Multiplex Network Approach to Detecting Preference Affinity Blocs using Texts and Votes
The polarization of actors' expressed preferences is a fundamental concern for studies of legislatures, court systems, and international politics BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . Because preferences are unobservable, scholars must look for signals in the empirical world. Recent progress has been made in parliamentary and court settings through the employment of textual data BIBREF4 and votes and texts in tandem BIBREF5 , BIBREF6 . Many of these advances rely on spatial, scaling, and item-response type models that are intuitive for settings where a small number of parties or ideological divisions influence outcomes. This is less intuitive for the study of state preferences, because international relations is marked by multiple dimensions that span ideological, economic, and security concerns, among others BIBREF0 .
Question: Do they use number of votes as an indicator of preference?
Paper: Disunited Nations? A Multiplex Network Approach to Detecting Preference Affinity Blocs using Texts and Votes
Polarization in IR is defined as “the degree to which the foreign policies of nations within a single cluster are similar to each other, and the degree to which the foreign policies of nations in different clusters are dissimilar" BIBREF9 . Therefore, operationalizing a concept of preference polarization broadly involves two steps: an approach to estimate preferences from available data on states' observable behavior; and a method of detecting distinct communities of nations, such that nations belonging to the same community share similar preferences, and nations belonging to different communities have dissimilar preferences.
Question: What does a node in the network approach repesent?
Paper: Disunited Nations? A Multiplex Network Approach to Detecting Preference Affinity Blocs using Texts and Votes
The most widely used source for deriving preferences in IR is UN roll call data BIBREF10 . Voting behavior represent a valuable source of revealed preference information, comparable across states and over time. However, UN roll call votes tend to be a weak signal of underlying preferences in cases where states vote for ceremonial purposes, are constrained by agenda-setting power dynamics, or vote as cohorts to maximize their impact within the UN, such as with regional blocs BIBREF11 .
Question: Which dataset do they use?
Paper: The Trumpiest Trump? Identifying a Subject's Most Characteristic Tweets
Social media platforms, particularly microblogging services such as Twitter, have become increasingly popular BIBREF0 as a means to express thoughts and opinions. Twitter users emit tweets about a wide variety of topics, which vary in the extent to which they reflect a user's personality, brand and interests. This observation motivates the question we consider here, of how to quantify the degree to which tweets are characteristic of their author?
Question: What kind of celebrities do they obtain tweets from?
Answer: Amitabh Bachchan, Ariana Grande, Barack Obama, Bill Gates, Donald Trump,
Ellen DeGeneres, J K Rowling, Jimmy Fallon, Justin Bieber, Kevin Durant, Kim Kardashian, Lady Gaga, LeBron James,Narendra Modi, Oprah Winfrey
Celebrities from varioius domains - Acting, Music, Politics, Business, TV, Author, Sports, Modeling.
Paper: Negated LAMA: Birds cannot fly
Pretrained language models like Transformer-XL BIBREF1, ELMo BIBREF2 and BERT BIBREF3 have emerged as universal tools that capture a diverse range of linguistic and factual knowledge.
Question: How did they extend LAMA evaluation framework to focus on negation?
Answer: Create the negated LAMA dataset and  query the pretrained language models with both original LAMA and negated LAMA statements and compare their predictions.
Paper: Mining Supervisor Evaluation and Peer Feedback in Performance Appraisals
Performance appraisal (PA) is an important HR process, particularly for modern organizations that crucially depend on the skills and expertise of their workforce. The PA process enables an organization to periodically measure and evaluate every employee's performance. It also provides a mechanism to link the goals established by the organization to its each employee's day-to-day activities and performance. Design and analysis of PA processes is a lively area of research within the HR community BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 .
Question: What summarization algorithms did the authors experiment with?
Answer: LSA, TextRank, LexRank and ILP-based summary.
LSA, TextRank, LexRank