z-uo/qasper-squad huggingface dataset:
Paper: Collecting Indicators of Compromise from Unstructured Text of Cybersecurity Articles using Neural-Based Sequence Labelling
Fig. FIGREF2 shows the 3 components (layers) of the proposed neural network architecture.
Question: What contextual features are used?
Answer: The words that can indicate the characteristics of the neighbor words as contextual keywords and generate it from the automatically extracted contextual keywords.
Paper: Collecting Indicators of Compromise from Unstructured Text of Cybersecurity Articles using Neural-Based Sequence Labelling
The token embedding layer takes a token as input and outputs its vector representation. As shown in Fig. FIGREF2 , given an input sequence of tokens INLINEFORM0 , the output vector INLINEFORM1 ( INLINEFORM2 ) of each token INLINEFORM3 results from the concatenation of two different types of embeddings: token embedding INLINEFORM4 and the character-based token embeddings INLINEFORM5 , INLINEFORM6 that come from the output of a character-level bi-LSTM encoder.
Question: Where are the cybersecurity articles used in the model sourced from?
Paper: Collecting Indicators of Compromise from Unstructured Text of Cybersecurity Articles using Neural-Based Sequence Labelling
The Sequence Representation Layer takes the sequence of embeddings INLINEFORM0 ( INLINEFORM1 ) as input, and outputs a sequence INLINEFORM2 , where the INLINEFORM3 element of INLINEFORM4 represents the probability that the INLINEFORM5 token has the label INLINEFORM6 .
Question: What type of hand-crafted features are used in state of the art IOC detection systems?
Paper: Boosting Question Answering by Deep Entity Recognition
A Question Answering (QA) system is a computer program capable of understanding questions in a natural language, finding answers to them in a knowledge base and providing answers in the same language. So broadly defined task seems very hard; BIBREF0 describes it as AI-Complete, i.e. equivalent to building a general artificial intelligence. Nonetheless, the field has attracted a lot of attention in Natural Language Processing (NLP) community as it provides a way to employ numerous NLP tools in an exploitable end-user system. It has resulted in valuable contributions within TREC competitions BIBREF1 and, quite recently, in a system called IBM Watson BIBREF2 , successfully competing with humans in the task.
Question: Do they compare DeepER against other approaches?
Paper: Boosting Question Answering by Deep Entity Recognition
As stated in previous chapter, RAFAEL is a computer system solving a task of Polish text-based, open-domain, factoid question answering. It means that provided questions, knowledge base and returned answers are expressed in Polish and may belong to any domain. The system analyses the knowledge base, consisting of a set of plain text documents, and returns answers (as concise as possible, e.g. a person name), supplied with information about supporting sentences and documents.
Question: How is the data in RAFAEL labelled?
Answer: Using a set of annotation tools such as Morfeusz, PANTERA, Spejd, NERF and Liner
Paper: Boosting Question Answering by Deep Entity Recognition
The problem of Question Answering is not new to the Polish NLP community (nor working on other morphologically rich languages), but none of studies presented so far coincides with the notion of plain text-based QA presented above.
Question: How do they handle polysemous words in their entity library?
Paper: Polysemy Detection in Distributed Representation of Word Sense
Distributed representation of word sense provides us with the ability to perform several operations on the word. One of the most important operations on a word is to obtain the set of words whose meaning is similar to the word, or whose usage in text is similar to the word. We call this set the neighbor of the word. When a word has several senses, it is called a polysemic word. When a word has only one sense, it is called a monosemic word. We have observed that the neighbor of a polysemic word consists of words that resemble the primary sense of the polysemic word. We can explain this fact as follows. Even though a word may be a polysemic, it usually corresponds to a single vector in distributed representation. This vector is primarily determined by the major sense, which is most frequently used. The information about a word's minor sense is subtle, and the effect of a minor sense is difficult to distinguish from statistical fluctuation.
Question: How is the fluctuation in the sense of the word and its neighbors measured?
Answer: Our method performs a statistical test to determine whether a given word is used polysemously in the text, according to the following steps:
1) Setting N, the size of the neighbor.
2) Choosing N neighboring words ai in the order whose angle with the vector of the given word w is the smallest.
3) Computing the surrounding uniformity for ai(0 < i ≤ N) and w.
4) Computing the mean m and the sample variance σ for the uniformities of ai .
5) Checking whether the uniformity of w is less than m − 3σ. If the value is less than m − 3σ, we may regard w as a polysemic word.
Paper: Neural Domain Adaptation for Biomedical Question Answering
Question answering (QA) is the task of retrieving answers to a question given one or more contexts. It has been explored both in the open-domain setting BIBREF0 as well as domain-specific settings, such as BioASQ for the biomedical domain BIBREF1 . The BioASQ challenge provides $\approx 900$ factoid and list questions, i.e., questions with one and several answers, respectively. This work focuses on answering these questions, for example: Which drugs are included in the FEC-75 regimen? $\rightarrow $ fluorouracil, epirubicin, and cyclophosphamide.
Question: Among various transfer learning techniques, which technique yields to the best performance?
Paper: Classifying topics in speech when all you have is crummy translations.
Quickly making sense of large amounts of linguistic data is an important application of language technology. For example, after the 2011 Japanese tsunami, natural language processing was used to quickly filter social media streams for messages about the safety of individuals, and to populate a person finder database BIBREF0. Japanese text is high-resource, but there are many cases where it would be useful to make sense of speech in low-resource languages. For example, in Uganda, as in many parts of the world, the primary source of news is local radio stations, which broadcast in many languages. A pilot study from the United Nations Global Pulse Lab identified these radio stations as a potentially useful source of information about a variety of urgent topics related to refugees, small-scale disasters, disease outbreaks, and healthcare BIBREF1. With many radio broadcasts coming in simultaneously, even simple classification of speech for known topics would be helpful to decision-makers working on humanitarian projects.
Question: What is the architecture of the model?
Paper: Classifying topics in speech when all you have is crummy translations.
We use the method of BIBREF5 to train neural sequence-to-sequence Spanish-English ST models. As in that study, before training ST, we pre-train the models using English ASR data from the Switchboard Telephone speech corpus BIBREF7, which consists of around 300 hours of English speech and transcripts. This was reported to substantially improve translation quality when the training set for ST was only tens of hours.
Question: What language do they look at?
Paper: Word, Subword or Character? An Empirical Study of Granularity in Chinese-English NMT
Neural machine translation (NMT) proposed by Kalchbrenner and Blunsom BIBREF0 and Sutskever et al. BIBREF1 has achieved significant progress in recent years. Unlike traditional statistical machine translation(SMT) BIBREF2 , BIBREF3 , BIBREF4 which contains multiple separately tuned components, NMT builds an end-to-end framework to model the entire translation process. For several language pairs, NMT has already achieved better translation performance than SMT BIBREF5 , BIBREF6 .
Question: Where does the vocabulary come from?
Paper: Word, Subword or Character? An Empirical Study of Granularity in Chinese-English NMT
Our models are based on an encoder-decoder architecture with attention mechanism proposed by Luong et al. BIBREF11 , which utilizes stacked LSTM layers for both encoder and decoder as illustrated in Figure FIGREF1 . In this section, we make a review of NMT framework.
Question: What is the worst performing translation granularity?
Paper: Word, Subword or Character? An Empirical Study of Granularity in Chinese-English NMT
We revisit how the source and target sentences ( INLINEFORM0 and INLINEFORM1 ) are represented in NMT. For the source side of any given training corpus, we scan through the whole corpus to build a vocabulary INLINEFORM2 of unique tokens. A source sentence INLINEFORM3 is then built as a sequence of the integer indices. The target sentence is similarly transformed into a target sequence of integer indices.
Question: What dataset did they use?
Paper: A Comparative Evaluation of Visual and Natural Language Question Answering Over Linked Data
The Semantic Web provides a large number of structured datasets in form of Linked Data. One central obstacle is to make this data available and consumable to lay users without knowledge of formal query languages such as SPARQL. In order to satisfy specific information needs of users, a typical approach are natural language interfaces to allow question answering over the Linked Data (QALD) by translating user queries into SPARQL BIBREF0 , BIBREF1 . As an alternative method, BIBREF2 propose a visual method of QA using an iterative diagrammatic approach. The diagrammatic approach relies on the visual means only, it requires more user interaction than natural language QA, but also provides additional benefits like intuitive insights into dataset characteristics, or a broader understanding of the answer and the potential to further explore the answer context, and finally allows for knowledge sharing by storing and sharing resulting diagrams.
Question: How do they measure performance?
Paper: A Comparative Evaluation of Visual and Natural Language Question Answering Over Linked Data
As introduced in BIBREF2 we understand diagrammatic question answering (DQA) as the process of QA relying solely on visual exploration using diagrams as a representation of the underlying knowledge source. The process includes (i) a model for diagrammatic representation of semantic data which supports data interaction using embedded queries, (ii) a simple method for step-by-step construction of diagrams with respect to cognitive boundaries and a layout that boosts understandability of diagrams, (iii) a library for visual data exploration and sharing based on its internal data model, and (iv) an evaluation of DQA as knowledge understanding and knowledge sharing tool. BIBREF3 propose a framework of five perspectives of knowledge visualization, which can be used to describe certain aspects of the DQA use cases, such as its goal to provide an iterative exploration method, which is accessible to any user, the possibility of knowledge sharing (via saved diagrams), or the general purpose of knowledge understanding and abstraction from technical details.
Question: Do they measure the performance of a combined approach?
Paper: A Comparative Evaluation of Visual and Natural Language Question Answering Over Linked Data
The DQA functionality is part of the Ontodia tool. The initial idea of Ontodia was to enable the exploration of semantic graphs for ordinary users. Data exploration is about efficiently extracting knowledge from data even in situations where it is unclear what is being looked for exactly BIBREF11 .
Question: Which four QA systems do they use?
Paper: A Comparative Evaluation of Visual and Natural Language Question Answering Over Linked Data
Here we present the evaluation of DQA in comparison to four QALD systems.
Question: How many iterations of visual search are done on average until an answer is found?
Paper: A Comparative Evaluation of Visual and Natural Language Question Answering Over Linked Data
As evaluation dataset, we reuse questions from the QALD7 benchmark task 4 “QA over Wikidata”. Question selection from QALD7 is based on the principles of question classification in QA BIBREF13 . Firstly, it is necessary to define question types which correspond to different scenarios of data exploration in DQA, as well as the type of expected answers and the question focus. The question focus refers to the main information in the question which help a user find the answer. We follow the model of BIBREF14 who categorize questions by their question word into WHO, WHICH, WHAT, NAME, and HOW questions. Given the question and answer type categories, we created four questionnaires with nine questions each resulting in 36 questions from the QALD dataset. The questions were picked in equal number for five basic question categories.
Question: Do they test performance of their approaches using human judgements?
Paper: Binary and Multitask Classification Model for Dutch Anaphora Resolution: Die/Dat Prediction
Following previous research on automatic detection and correction of dt-mistakes in Dutch BIBREF0, this paper investigates another stumbling block for both native and non-native speakers of Dutch: the correct use of die and dat. The multiplicity of syntactic functions and the dependency on the antecedent's gender and number make this a challenging task for both human and computer. The grammar concerning die and dat is threefold. Firstly, they can be used as dependent or independent demonstrative pronouns (aanwijzend voornaamwoord), with the first replacing the article before the noun it modifies and the latter being a noun phrase that refers to a preceding/following noun phrase or sentence. The choice between the two pronouns depends on the gender and number of the antecedent: dat refers to neuter, singular nouns and sentences, while die refers to masculine, singular nouns and plural nouns independent of their gender. Secondly, die and dat can be used as relative pronouns introducing relative clauses (betrekkelijk voornaamwoord), which provide additional information about the directly preceding antecedent it modifies. Similar rules as for demonstrative pronouns apply: masculine, singular nouns and plural nouns are followed by relative pronoun die, neuter singular nouns by dat. Lastly, dat can be used as a subordinating conjunction (onderschikkend voegwoord) introducing a subordinating clause. An brief overview of the grammar is given in Table TABREF1.
Question: What are the sizes of both datasets?
Paper: 'Warriors of the Word' -- Deciphering Lyrical Topics in Music and Their Connection to Audio Feature Dimensions Based on a Corpus of Over 100,000 Metal Songs
As audio and text features provide complementary layers of information on songs, a combination of both data types has been shown to improve the automatic classification of high-level attributes in music such as genre, mood and emotion BIBREF0, BIBREF1, BIBREF2, BIBREF3. Multi-modal approaches interlinking these features offer insights into possible relations between lyrical and musical information (see BIBREF4, BIBREF5, BIBREF6).
Question: Why are the scores for predicting perceived musical hardness and darkness extracted only for subsample of 503 songs?
Paper: 'Warriors of the Word' -- Deciphering Lyrical Topics in Music and Their Connection to Audio Feature Dimensions Based on a Corpus of Over 100,000 Metal Songs
In our sequential research design, the distribution of textual topics within the corpus was analyzed using latent Dirichlet allocation (LDA). This resulted in a topic model, which was used for a probabilistic assignment of topics to each of the song documents. Additionally, for a subset of these songs, audio features were extracted using models for high-level music dimensions. The use of automatic models for the extraction of both text as well as musical features allows for scalability as it enables a large corpus to be studied without depending on the process of manual annotation for each of the songs. The resulting feature vectors were then subjected to a correlation analysis. Figure FIGREF6 outlines the sequence of the steps taken in processing the data. The individual steps are explained in the following subsections.
Question: How long is the model trained?
Paper: 'Warriors of the Word' -- Deciphering Lyrical Topics in Music and Their Connection to Audio Feature Dimensions Based on a Corpus of Over 100,000 Metal Songs
For gathering the data corpus, a web crawler was programmed using the Python packages Requests and BeautifulSoup. In total, 152,916 metal music lyrics were extracted from www.darklyrics.com.
Question: What are lyrical topics present in the metal genre?
Paper: Abstractive Dialog Summarization with Semantic Scaffolds
Summarization aims to condense a piece of text to a shorter version, retaining the critical information. On dialogs, summarization has various promising applications in the real world. For instance, the automatic doctor-patient interaction summary can save doctors' massive amount of time used for filling medical records. There is also a general demand for summarizing meetings in order to track project progress in the industry. Generally, multi-party conversations with interactive communication are more difficult to summarize than single-speaker documents. Hence, dialog summarization will be a potential field in summarization track.
Question: By how much does SPNet outperforms state-of-the-art abstractive summarization methods on evaluation metrics?
Answer: SPNet vs best baseline:
ROUGE-1: 90.97 vs 90.68
CIC: 70.45 vs 70.25
Paper: Abstractive Dialog Summarization with Semantic Scaffolds
BIBREF7 first applied modern neural models to abstractive summarization. Their approach is based on Seq2Seq framework BIBREF8 and attention mechanism BIBREF9, achieving state-of-the-art results on Gigaword and DUC-2004 dataset. BIBREF10 proposed copy mechanism in summarization, demonstrating its effectiveness by combining the advantages of extractive and abstractive approach. BIBREF5 applied pointing BIBREF11 as copy mechanism and use coverage mechanism BIBREF12 to discourage repetition. Most recently, reinforcement learning (RL) has been employed in abstractive summarization. RL-based approaches directly optimize the objectives of summarization BIBREF13, BIBREF14. However, deep reinforcement learning approaches are difficult to train and more prone to exposure bias BIBREF15.
Question: What automatic and human evaluation metrics are used to compare SPNet to its counterparts?
Paper: Abstractive Dialog Summarization with Semantic Scaffolds
As discussed above, state-of-the-art document summarizers are not applicable in conversation settings. We propose Scaffold Pointer Network (SPNet) based on Pointer-Generator BIBREF5. SPNet incorporates three types of semantic scaffolds to improve abstractive dialog summarization: speaker role, semantic slot and dialog domain.
Question: Is proposed abstractive dialog summarization dataset open source?
Paper: Abstractive Dialog Summarization with Semantic Scaffolds
We first introduce Pointer-Generator BIBREF5. It is a hybrid model of the typical Seq2Seq attention model BIBREF29 and pointer network BIBREF11. Seq2Seq framework encodes source sequence and generates the target sequence with the decoder. The input sequence is fed into the encoder token by token, producing the encoder hidden states $h_i$ in each encoding step. The decoder receives word embedding of the previous word and generates a distribution to decide the target element in this step, retaining decoder hidden states $s_t$. In Pointer-Generator, attention distribution $a^t$ is computed as in BIBREF9:
Question: Is it expected to have speaker role, semantic slot and dialog domain annotations in real world datasets?
Answer: Not at the moment, but summaries can be additionaly extended with this annotations.
Paper: Abstractive Dialog Summarization with Semantic Scaffolds
Our Scaffold Pointer Network (depicted in Figure FIGREF7) is based on Pointer-Generator BIBREF5. The contribution of SPNet is three-fold: separate encoding for different roles, incorporating semantic slot scaffold and dialog domain scaffold.
Question: How does SPNet utilize additional speaker role, semantic slot and dialog domain annotations?
Paper: Abstractive Dialog Summarization with Semantic Scaffolds
Our encoder-decoder framework employs separate encoding for different speakers in the dialog. User utterances $x_t^{usr}$ and system utterances $x_t^{sys}$ are fed into a user encoder and a system encoder separately to obtain encoder hidden states $h_{i}^{usr}$ and $h_{i}^{sys}$ . The attention distributions and context vectors are calculated as described in section SECREF1. In order to merge these two encoders in our framework, the decoder's hidden state $s_0$ is initialized as:
Question: What are previous state-of-the-art document summarization methods used?
Paper: Abstractive Dialog Summarization with Semantic Scaffolds
We integrate semantic slot scaffold by performing delexicalization on original dialogs. Delexicalization is a common pre-processing step in dialog modeling. Specifically, delexicalization replaces the slot values with its semantic slot name(e.g. replace 18:00 with [time]). It is easier for the language modeling to process delexicalized texts, as they have a reduced vocabulary size. But these generated sentences lack the semantic information due to the delexicalization. Some previous dialog system research ignored this issue BIBREF30 or completed single delexicalized utterance BIBREF31 as generated response. We propose to perform delexicalization in dialog summary, since delexicalized utterances can simplify dialog modeling. We fill the generated templates with slots with the copy and pointing mechanism.
Question: How does new evaluation metric considers critical informative entities?
Answer: Answer with content missing: (formula for CIC) it accounts for the most important information within each dialog domain. CIC can be applied to any summarization task with predefined essential entities
Paper: Abstractive Dialog Summarization with Semantic Scaffolds
We integrate dialog domain scaffold through a multi-task framework. Dialog domain indicates different conversation task content, for example, booking hotel, restaurant and taxi in MultiWOZ dataset. Generally, the content in different domains varies so multi-domain task summarization is more difficult than single-domain. We include domain classification as the auxiliary task to incorporate the prior that different domains have different content. Feedback from the domain classification task provides domain specific information for the encoder to learn better representations. For domain classification, we feed the concatenated encoder hidden state through a binary classifier with two linear layers, producing domain probability $d$. The $i^{th}$ element $d_i$ in $d$ represents the probability of the $i^{th}$ domain:
Question: Is new evaluation metric extension of ROGUE?
Paper: CommonGen: A Constrained Text Generation Dataset Towards Generative Commonsense Reasoning
Commonsense reasoning has long been acknowledged as a critical bottleneck of artificial intelligence and especially in natural language processing. It is an ability of combining commonsense facts and logic rules to make new presumptions about ordinary scenes in our daily life. A distinct property of commonsense reasoning problems is that they are generally trivial for human-beings while challenging for machine reasoners.
Question: What measures were used for human evaluation?