pwc-archive/papers-with-abstracts huggingface dataset:
Structural Text Segmentation of Legal Documents
Authors: Dennis Aumiller, Satya Almasian, Sebastian Lackner, Michael Gertz
Date: 2020-12-07 00:00:00
Tasks: Change Detection, Information Retrieval, Passage Retrieval, Retrieval, Segmentation, Semantic Segmentation, Text Segmentation

The growing complexity of legal cases has lead to an increasing interest in legal information retrieval systems that can effectively satisfy user-specific information needs. However, such downstream systems typically require documents to be properly formatted and segmented, which is often done with relatively simple pre-processing steps, disregarding topical coherence of segments. Systems generally rely on representations of individual sentences or paragraphs, which may lack crucial context, or document-level representations, which are too long for meaningful search results. To address this issue, we propose a segmentation system that can predict topical coherence of sequential text segments spanning several paragraphs, effectively segmenting a document and providing a more balanced representation for downstream applications. We build our model on top of popular transformer networks and formulate structural text segmentation as topical change detection, by performing a series of independent classifications that allow for efficient fine-tuning on task-specific data. We crawl a novel dataset consisting of roughly $74,000$ online Terms-of-Service documents, including hierarchical topic annotations, which we use for training. Results show that our proposed system significantly outperforms baselines, and adapts well to structural peculiarities of legal documents. We release both data and trained models to the research community for future work.https://github.com/dennlinger/TopicalChange
Using previous acoustic context to improve Text-to-Speech synthesis
Authors: Pilar Oplustil-Gallegos, Simon King
Date: 2020-12-07 00:00:00
Tasks: Decoder, Speech Synthesis, text-to-speech, Text to Speech, Text-To-Speech Synthesis

Many speech synthesis datasets, especially those derived from audiobooks, naturally comprise sequences of utterances. Nevertheless, such data are commonly treated as individual, unordered utterances both when training a model and at inference time. This discards important prosodic phenomena above the utterance level. In this paper, we leverage the sequential nature of the data using an acoustic context encoder that produces an embedding of the previous utterance audio. This is input to the decoder in a Tacotron 2 model. The embedding is also used for a secondary task, providing additional supervision. We compare two secondary tasks: predicting the ordering of utterance pairs, and predicting the embedding of the current utterance audio. Results show that the relation between consecutive utterances is informative: our proposed model significantly improves naturalness over a Tacotron 2 baseline.
What Meaning-Form Correlation Has to Compose With
Authors: Timothee Mickus, Timothée Bernard, Denis Paperno
Date: 2020-12-07 00:00:00
Tasks: Form

Compositionality is a widely discussed property of natural languages, although its exact definition has been elusive. We focus on the proposal that compositionality can be assessed by measuring meaning-form correlation. We analyze meaning-form correlation on three sets of languages: (i) artificial toy languages tailored to be compositional, (ii) a set of English dictionary definitions, and (iii) a set of English sentences drawn from literature. We find that linguistic phenomena such as synonymy and ungrounded stop-words weigh on MFC measurements, and that straightforward methods to mitigate their effects have widely varying results depending on the dataset they are applied to. Data and code are made publicly available.
Stylometry for Noisy Medieval Data: Evaluating Paul Meyer's Hagiographic Hypothesis
Authors: Jean-Baptiste Camps, Thibault Clérice, Ariane Pinche
Date: 2020-12-07 00:00:00
Tasks: Handwritten Text Recognition

Stylometric analysis of medieval vernacular texts is still a significant challenge: the importance of scribal variation, be it spelling or more substantial, as well as the variants and errors introduced in the tradition, complicate the task of the would-be stylometrist. Basing the analysis on the study of the copy from a single hand of several texts can partially mitigate these issues (Camps and Cafiero, 2013), but the limited availability of complete diplomatic transcriptions might make this difficult. In this paper, we use a workflow combining handwritten text recognition and stylometric analysis, applied to the case of the hagiographic works contained in MS BnF, fr. 412. We seek to evaluate Paul Meyer's hypothesis about the constitution of groups of hagiographic works, as well as to examine potential authorial groupings in a vastly anonymous corpus.
The Lab vs The Crowd: An Investigation into Data Quality for Neural Dialogue Models
Authors: José Lopes, Francisco J. Chiyah Garcia, Helen Hastie
Date: 2020-12-07 00:00:00

Challenges around collecting and processing quality data have hampered progress in data-driven dialogue models. Previous approaches are moving away from costly, resource-intensive lab settings, where collection is slow but where the data is deemed of high quality. The advent of crowd-sourcing platforms, such as Amazon Mechanical Turk, has provided researchers with an alternative cost-effective and rapid way to collect data. However, the collection of fluid, natural spoken or textual interaction can be challenging, particularly between two crowd-sourced workers. In this study, we compare the performance of dialogue models for the same interaction task but collected in two different settings: in the lab vs. crowd-sourced. We find that fewer lab dialogues are needed to reach similar accuracy, less than half the amount of lab data as crowd-sourced data. We discuss the advantages and disadvantages of each data collection method.
Pre-training Protein Language Models with Label-Agnostic Binding Pairs Enhances Performance in Downstream Tasks
Authors: Modestas Filipavicius, Matteo Manica, Joris Cadow, Maria Rodriguez Martinez
Date: 2020-12-05 00:00:00
Tasks: Language Modeling, Language Modelling, Masked Language Modeling, Self-Supervised Learning

Less than 1% of protein sequences are structurally and functionally annotated. Natural Language Processing (NLP) community has recently embraced self-supervised learning as a powerful approach to learn representations from unlabeled text, in large part due to the attention-based context-aware Transformer models. In this work we present a modification to the RoBERTa model by inputting during pre-training a mixture of binding and non-binding protein sequences (from STRING database). However, the sequence pairs have no label to indicate their binding status, as the model relies solely on Masked Language Modeling (MLM) objective during pre-training. After fine-tuning, such approach surpasses models trained on single protein sequences for protein-protein binding prediction, TCR-epitope binding prediction, cellular-localization and remote homology classification tasks. We suggest that the Transformer's attention mechanism contributes to protein binding site discovery. Furthermore, we compress protein sequences by 64% with the Byte Pair Encoding (BPE) vocabulary consisting of 10K subwords, each around 3-4 amino acids long. Finally, to expand the model input space to even larger proteins and multi-protein assemblies, we pre-train Longformer models that support 2,048 tokens. Further work in token-level classification for secondary structure prediction is needed. Code available at: https://github.com/PaccMann/paccmann_proteomics
MLS: A Large-Scale Multilingual Dataset for Speech Research
Authors: Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, Ronan Collobert
Date: 2020-12-07 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition, text-to-speech, Text to Speech

This paper introduces Multilingual LibriSpeech (MLS) dataset, a large multilingual corpus suitable for speech research. The dataset is derived from read audiobooks from LibriVox and consists of 8 languages, including about 44.5K hours of English and a total of about 6K hours for other languages. Additionally, we provide Language Models (LM) and baseline Automatic Speech Recognition (ASR) models and for all the languages in our dataset. We believe such a large transcribed dataset will open new avenues in ASR and Text-To-Speech (TTS) research. The dataset will be made freely available for anyone at http://www.openslr.org.
Foundations for Near-Term Quantum Natural Language Processing
Authors: Bob Coecke, Giovanni De Felice, Konstantinos Meichanetzidis, Alexis Toumi
Date: 2020-12-07 00:00:00

We provide conceptual and mathematical foundations for near-term quantum natural language processing (QNLP), and do so in quantum computer scientist friendly terms. We opted for an expository presentation style, and provide references for supporting empirical evidence and formal statements concerning mathematical generality. We recall how the quantum model for natural language that we employ canonically combines linguistic meanings with rich linguistic structure, most notably grammar. In particular, the fact that it takes a quantum-like model to combine meaning and structure, establishes QNLP as quantum-native, on par with simulation of quantum systems. Moreover, the now leading Noisy Intermediate-Scale Quantum (NISQ) paradigm for encoding classical data on quantum hardware, variational quantum circuits, makes NISQ exceptionally QNLP-friendly: linguistic structure can be encoded as a free lunch, in contrast to the apparently exponentially expensive classical encoding of grammar. Quantum speed-up for QNLP tasks has already been established in previous work with Will Zeng. Here we provide a broader range of tasks which all enjoy the same advantage. Diagrammatic reasoning is at the heart of QNLP. Firstly, the quantum model interprets language as quantum processes via the diagrammatic formalism of categorical quantum mechanics. Secondly, these diagrams are via ZX-calculus translated into quantum circuits. Parameterisations of meanings then become the circuit variables to be learned. Our encoding of linguistic structure within quantum circuits also embodies a novel approach for establishing word-meanings that goes beyond the current standards in mainstream AI, by placing linguistic structure at the heart of Wittgenstein's meaning-is-context.
Grammar-aware sentence classification on quantum computers
Authors: Konstantinos Meichanetzidis, Alexis Toumi, Giovanni De Felice, Bob Coecke
Date: 2020-12-07 00:00:00
Tasks: Binary Classification, Classification, Question Answering, Sentence, Sentence Classification

Natural language processing (NLP) is at the forefront of great advances in contemporary AI, and it is arguably one of the most challenging areas of the field. At the same time, in the area of Quantum Computing (QC), with the steady growth of quantum hardware and notable improvements towards implementations of quantum algorithms, we are approaching an era when quantum computers perform tasks that cannot be done on classical computers with a reasonable amount of resources. This provides a new range of opportunities for AI, and for NLP specifically. In this work, we work with the Categorical Distributional Compositional (DisCoCat) model of natural language meaning, whose underlying mathematical underpinnings make it amenable to quantum instantiations. Earlier work on fault-tolerant quantum algorithms has already demonstrated potential quantum advantage for NLP, notably employing DisCoCat. In this work, we focus on the capabilities of noisy intermediate-scale quantum (NISQ) hardware and perform the first implementation of an NLP task on a NISQ processor, using the DisCoCat framework. Sentences are instantiated as parameterised quantum circuits; word-meanings are embedded in quantum states using parameterised quantum-circuits and the sentence's grammatical structure faithfully manifests as a pattern of entangling operations which compose the word-circuits into a sentence-circuit. The circuits' parameters are trained using a classical optimiser in a supervised NLP task of binary classification. Our novel QNLP model shows concrete promise for scalability as the quality of the quantum hardware improves in the near future and solidifies a novel branch of experimental research at the intersection of QC and AI.
Diverse Melody Generation from Chinese Lyrics via Mutual Information Maximization
Authors: Ruibin Yuan, Ge Zhang, Anqiao Yang, Xinyue Zhang
Date: 2020-12-07 00:00:00
Tasks: Diversity

In this paper, we propose to adapt the method of mutual information maximization into the task of Chinese lyrics conditioned melody generation to improve the generation quality and diversity. We employ scheduled sampling and force decoding techniques to improve the alignment between lyrics and melodies. With our method, which we called Diverse Melody Generation (DMG), a sequence-to-sequence model learns to generate diverse melodies heavily depending on the input style ids, while keeping the tonality and improving the alignment. The experimental results of subjective tests show that DMG can generate more pleasing and coherent tunes than baseline methods.
Aligning geographic entities from historical maps for building knowledge graphs
Authors: Kai Sun, Yingjie Hu, Jia Song, Yunqiang Zhu
Date: 2020-12-05 00:00:00
Tasks: Knowledge Graphs

Historical maps contain rich geographic information about the past of a region. They are sometimes the only source of information before the availability of digital maps. Despite their valuable content, it is often challenging to access and use the information in historical maps, due to their forms of paper-based maps or scanned images. It is even more time-consuming and labor-intensive to conduct an analysis that requires a synthesis of the information from multiple historical maps. To facilitate the use of the geographic information contained in historical maps, one way is to build a geographic knowledge graph (GKG) from them. This paper proposes a general workflow for completing one important step of building such a GKG, namely aligning the same geographic entities from different maps. We present this workflow and the related methods for implementation, and systematically evaluate their performances using two different datasets of historical maps. The evaluation results show that machine learning and deep learning models for matching place names are sensitive to the thresholds learned from the training data, and a combination of measures based on string similarity, spatial distance, and approximate topological relation achieves the best performance with an average F-score of 0.89.
KATRec: Knowledge Aware aTtentive Sequential Recommendations
Authors: Mehrnaz Amjadi, Seyed Danial Mohseni Taheri, Theja Tulabandhula
Date: 2020-12-06 00:00:00
Tasks: Graph Attention, Recommendation Systems, Representation Learning, Sequential Recommendation

Sequential recommendation systems model dynamic preferences of users based on their historical interactions with platforms. Despite recent progress, modeling short-term and long-term behavior of users in such systems is nontrivial and challenging. To address this, we present a solution enhanced by a knowledge graph called KATRec (Knowledge Aware aTtentive sequential Recommendations). KATRec learns the short and long-term interests of users by modeling their sequence of interacted items and leveraging pre-existing side information through a knowledge graph attention network. Our novel knowledge graph-enhanced sequential recommender contains item multi-relations at the entity-level and users' dynamic sequences at the item-level. KATRec improves item representation learning by considering higher-order connections and incorporating them in user preference representation while recommending the next item. Experiments on three public datasets show that KATRec outperforms state-of-the-art recommendation models and demonstrates the importance of modeling both temporal and side information to achieve high-quality recommendations.
Conversational Browsing
Authors: Svitlana Vakulenko, Vadim Savenkov, Maarten de Rijke
Date: 2020-12-07 00:00:00
Tasks: Information Retrieval, Question Answering, Retrieval

How can we better understand the mechanisms behind multi-turn information seeking dialogues? How can we use these insights to design a dialogue system that does not require explicit query formulation upfront as in question answering? To answer these questions, we collected observations of human participants performing a similar task to obtain inspiration for the system design. Then, we studied the structure of conversations that occurred in these settings and used the resulting insights to develop a grounded theory, design and evaluate a first system prototype. Evaluation results show that our approach is effective and can complement query-based information retrieval approaches. We contribute new insights about information-seeking behavior by analyzing and providing automated support for a type of information-seeking strategy that is effective when the clarity of the information need and familiarity with the collection content are low.
Bearings degradation monitoring indicators based on discarded projected space information and piecewise linear representation
Authors: Fei Huang, Alexandre Sava, Kondo H. Adjallah, Wang Zhouhang
Date: 2020-12-07 00:00:00

Condition-based maintenance of rotating mechanics requests efficient bearings degradation monitoring. The accuracy of bearings degradation measure depends largely on degradation indicators. To extract efficient indicators, in this paper we propose a method based on the discarded projected space information and piecewise linear representation (PLR) to build three bearings degradation monitoring indicators which are named SDHT2, VSDHT2 and NVSDHT2. The discarded projected space information is measured by the segmented discarded Hotelling T square we propose in this paper. For illustration, the IEEE PHM 2012 benchmark dataset is used in this paper. The results show that the three new indicators are all sensitive and monotonic during the bearings whole lifecycle. They describe the whole degradation process history and carry the real-time information of bearings degradation. And NVSDHT2 is the generalised version of VSDHT2, which is promising to monitor bearings degradation.
Modeling Updates of Scholarly Webpages Using Archived Data
Authors: Yasith Jayawardana, Alexander C. Nwala, Gavindya Jayawardena, Jian Wu, Sampath Jayarathna, Michael L. Nelson, C. Lee Giles
Date: 2020-12-07 00:00:00

The vastness of the web imposes a prohibitive cost on building large-scale search engines with limited resources. Crawl frontiers thus need to be optimized to improve the coverage and freshness of crawled content. In this paper, we propose an approach for modeling the dynamics of change in the web using archived copies of webpages. To evaluate its utility, we conduct a preliminary study on the scholarly web using 19,977 seed URLs of authors' homepages obtained from their Google Scholar profiles. We first obtain archived copies of these webpages from the Internet Archive (IA), and estimate when their actual updates occurred. Next, we apply maximum likelihood to estimate their mean update frequency ($\lambda$) values. Our evaluation shows that $\lambda$ values derived from a short history of archived data provide a good estimate for the true update frequency in the short-term, and that our method provides better estimations of updates at a fraction of resources compared to the baseline models. Based on this, we demonstrate the utility of archived data to optimize the crawling strategy of web crawlers, and uncover important challenges that inspire future research directions.
Selective Inference for Hierarchical Clustering
Authors: Lucy L. Gao, Jacob Bien, Daniela Witten
Date: 2020-12-05 00:00:00
Tasks: Clustering

Classical tests for a difference in means control the type I error rate when the groups are defined a priori. However, when the groups are instead defined via clustering, then applying a classical test yields an extremely inflated type I error rate. Notably, this problem persists even if two separate and independent data sets are used to define the groups and to test for a difference in their means. To address this problem, in this paper, we propose a selective inference approach to test for a difference in means between two clusters. Our procedure controls the selective type I error rate by accounting for the fact that the choice of null hypothesis was made based on the data. We describe how to efficiently compute exact p-values for clusters obtained using agglomerative hierarchical clustering with many commonly-used linkages. We apply our method to simulated data and to single-cell RNA-sequencing data.
Global Riemannian Acceleration in Hyperbolic and Spherical Spaces
Authors: David Martínez-Rubio
Date: 2020-12-07 00:00:00
Tasks: global-optimization

We further research on the accelerated optimization phenomenon on Riemannian manifolds by introducing accelerated global first-order methods for the optimization of $L$-smooth and geodesically convex (g-convex) or $\mu$-strongly g-convex functions defined on the hyperbolic space or a subset of the sphere. For a manifold other than the Euclidean space, these are the first methods to \emph{globally} achieve the same rates as accelerated gradient descent in the Euclidean space with respect to $L$ and $\epsilon$ (and $\mu$ if it applies), up to log factors. Due to the geometric deformations, our rates have an extra factor, depending on the initial distance $R$ to a minimizer and the curvature $K$, with respect to Euclidean accelerated algorithms As a proxy for our solution, we solve a constrained non-convex Euclidean problem, under a condition between convexity and \emph{quasar-convexity}, of independent interest. Additionally, for any Riemannian manifold of bounded sectional curvature, we provide reductions from optimization methods for smooth and g-convex functions to methods for smooth and strongly g-convex functions and vice versa. We also reduce global optimization to optimization over bounded balls where the effect of the curvature is reduced.
Mixed-SCORE+ for mixed membership community detection
Authors: Huan Qing, Jingli Wang
Date: 2020-12-07 00:00:00
Tasks: Community Detection

Mixed-SCORE is a recent approach for mixed membership community detection proposed by Jin et al. (2017) which is an extension of SCORE (Jin, 2015). In the note Jin et al. (2018), the authors propose SCORE+ as an improvement of SCORE to handle with weak signal networks. In this paper, we propose a method called Mixed-SCORE+ designed based on the Mixed-SCORE and SCORE+, therefore Mixed-SCORE+ inherits nice properties of both Mixed-SCORE and SCORE+. In the proposed method, we consider K+1 eigenvectors when there are K communities to detect weak signal networks. And we also construct vertices hunting and membership reconstruction steps to solve the problem of mixed membership community detection. Compared with several benchmark methods, numerical results show that Mixed-SCORE+ provides a significant improvement on the Polblogs network and two weak signal networks Simmons and Caltech, with error rates 54/1222, 125/1137 and 94/590, respectively. Furthermore, Mixed-SCORE+ enjoys excellent performances on the SNAP ego-networks.