Semi-supervised Ensemble Learning with Weak Supervision for Biomedical Relation Extraction
Abstract
Natural language understanding research has recently shifted towards complex Machine Learning and Deep Learning algorithms.
Such models often outperform their simpler counterparts signiﬁcantly.
However, their performance relies on the availability of large amounts of labeled data, which are rarely available.
To tackle this problem, we propose a methodology for extending training datasets to arbitrarily big sizes and training complex, data-hungry models using weak supervision.
We apply this methodology on biomedical relation extraction, a task where training datasets are excessively time-consuming and expensive to create, yet has a major impact on downstream applications such as drug discovery.
We demonstrate in two small-scale controlled experiments that our method consistently enhances the performance of an LSTM network, with performance improvements comparable to hand-labeled training data.
Finally, we discuss the optimal setting for applying weak supervision using this methodology.
1. Introduction
The amount of scientiﬁc papers in the biomedical ﬁeld is ever increasing.
Published papers contain important information, however, encoded in unstructured text, making it diﬃcult for researchers to locate it.
Extracting this information in a structured format and storing it within a knowledge base can have a remarkable impact on a variety of important tasks, ranging from drug design to detecting adverse drug eﬀects.
During the past decade, there have been eﬀorts towards automation of Information Extraction [Wei et al., 2016, Krallinger et al., 2017a, Wei et al., 2015], due to the fact that manual annotation of documents from domain experts is labor-intensive to perform on a large scale [Krallinger et al., 2017b].
The broader focus of this work is to help automation of semantic triple extraction from biomedical abstracts.
We apply our methodology on two diﬀerent relations: (a) Regulations , indicating that a Chemical increases (up-regulates) or decreases (down-regulates) the production of a Protein (CPR) and (b) Chemically Induced Diseases (CID).
Both relations are particularly important for areas such as drug design, safety, and discovery, as it will enable researchers to ﬁlter out or select chemical substances with speciﬁc properties, faster.
[Table]
Extracting semantic triples is a structured Information Extraction problem.
First, one needs to identify the entities of interest (subjects & objects) in the unstructured text, optionally distinguishing them with a unique identiﬁer, and then build a classiﬁer to recognize whether the text describes the target relationship (Relation Extraction).
Our main focus lies on the subtask of relation extraction.
As this is a complex and challenging task, increasing the capacity of the learning algorithm is justiﬁed, but should coincide with an appropriate increase in the training dataset size [Goodfellow et al., 2016].
However, annotating training datasets for this task is an excessively labor-intensive process.
To tackle this problem, we propose a new methodology based on weak supervision, which combines and incorporates ideas from semi-supervised and ensemble learning.
The outline of this methodology is the following: (a) We train multiple base learners on a small labeled dataset and use them to predict the labels of a bigger unlabeled dataset; (b) we then use a denoiser to derive weak labels for the unlabeled set using the predictions of the base learners; and (c) we ﬁnally train a strong meta-learner using weak supervision on the (previously) unlabeled set.
Our main contributions include: (a) proposing a detailed methodology speciﬁc to relation extraction, which can be adapted and generalized to most supervised learning tasks; (b) demonstrating the eﬀectiveness and usability of this methodology in a small-scale controlled experiment; (c) investigating the eﬀect of various denoising methods on the overall system behavior and performance.
To ensure reproducibility of our results and encourage the research community to further experiment with this methodology, we release the code in our GitHub repository 1 .
2. Related work
In this section we discuss literature related to three topics: information extraction, relation extraction from biomedical text, and semi-supervised and ensemble learning methods.
2.1 Information extraction
Information extraction is typically modeled as a fully-, semi- or un-supervised learning problem.
Unsupervised methods, such as Open Information Extraction [Banko et al., 2007], do not make use of any training data, but they can only be used for unstructured Information
Extraction.
On the contrary, fully-supervised methods rely on labeled examples, which have to be manually annotated.
Semi-supervised or bootstrapping methods are similar to our approach, as they try to leverage both labeled and unlabeled data.
One of the ﬁrst bootstrapping algorithms was DIRPE [Brin, 1998], which starts from some seed (positive) examples, extracts patterns from them and use them for ﬁnding new positive examples.
Other semi-supervised algorithms include Snowball [Agichtein and Gravano, 2000], KnowItAll [Etzioni et al., 2004] and TextRunner [Banko et al., 2007].
Recent approaches also consider contextual data augmentation, which paraphrases the training samples [Kobayashi, 2018].
All of the aforementioned work focuses on bootstrapping new data to avoid excessive manual annotation, however, diﬀerent from this work, it does not use the combination of learning algorithms to do so.
Distant supervision [Mintz et al., 2009] is a method to generate weak labels for unlabeled data on relation extraction problems.
Speciﬁcally, it uses Knowledge Bases (KB) for the creation of weak (positive and negative) labels instead of pre-trained classiﬁers.
Despite the creation of many noisy examples, this approach has proven to be beneﬁcial on large-scale (or web-scale) datasets, as no human annotation is required.
The core idea of weakly-labeled datasets remains the same, but is achieved by diﬀerent means.
Our work is complementary to distant supervision, as such algorithms can constitute weak classiﬁers in our framework.
2.2 Relation extraction from biomedical text
Most of the research on biomedical relation extraction has been motivated by BioCreative competitions and their corresponding annotated documents (datasets).
BioCreative V (CID task) focused on extracting Chemically-induced Diseases on a document-level [Wei et al., 2015].
The best performing team implemented an ensemble of Support Vector Machines [Xu et al., 2015], while more recent research demonstrated that extending the training set using distant supervision improves the performance
[Peng et al., 2016].
A following competition (BioCreative VI - CPR task) focused on identifying relations between Chemicals and Proteins (or Genes) on a sentence level [Krallinger et al., 2017a].
The best-performing team implemented an ensemble of LSTM, CNN and SVMs, using majority voting and stacking [Peng et al., 2018].
The second highest score was achieved using a Support-Vector
Machines algorithm with a rich set of features, while other approaches using solely Deep
Neural Networks demonstrated overﬁtting problems [Mehryary et al., 2018].
Those results highlight the importance of the lack of training data in this particular domain and demonstrate the suitability of ensemble methods for improving generalization, especially when Deep Neural Networks are used.
To that end, our work aims to combine the advantages of those techniques with semi-supervised learning, a subject which to the best of our knowledge has not been investigated for this task.
Further, we provide a framework where all of the aforementioned work can be used as an input.
2.3 Semi-supervised and ensemble learning methods
Both semi-supervised and ensemble learning techniques aim to improve the performance of
Machine Learning models.
Ensemble learning typically reduces high variance by combining multiple learners, while semi-supervised learning tries to take advantage of unlabeled data to improve generalization.
Although ensembles have been studied thoroughly and used in many applications, their combination with semi-supervised learning has not.
Their combination has not been thoroughly studied, despite indications that they can be beneﬁcial to each other [Zhou, 2011].
Ensembles can enhance semi-supervised learning by providing multiple views and therefore allow the system to perform better, earlier (using less data).
On the other hand, expanding the dataset with unlabeled data is likely to increase diversity between the learners.
The ﬁrst system of this kind proposed was co-training [Blum and Mitchell, 1998], a paradigm where two distinct (independent) learning algorithms take advantage of unlabeled data.
Following research indicated that complete independence was too luxurious for a real-world setting and the method can even succeed without it, given an expansion on the underlying data distribution.
[Nigam and Ghani, 2000, Balcan et al., 2005].
Recent work uses expert-deﬁned lexicons and an auxiliary natural language processing system to create noisy annotations, and incorporate co-training to reduce noise and augment signal in distant supervision.
Without using any manually labeled data, their system learned to accurately annotate data samples in functional genomics and outperform state-of-the-art supervised methods trained on tens of thousands of annotated examples.
[Grechkin et al., 2017].
Tri-training [Zhou and Li, 2005] is an extension of co-training to three learners.
In this case, when two learners agree on the label of a new data point, they teach (re-train)
the third learner using this example.
Co-forest
[Li and Zhou, 2007] is an extension to even more learners, where the decision on whether an unlabeled example should be added in the re-training stack is made by an ensemble system, using all learners.
The fundamental diﬀerence between our methodology and those earlier described, is that the learners included in the ensemble system (base learners) are not used for the ﬁnal prediction; they are only used as a mean for generating the weak labels.
For this reason, we do not aim to re-train or improve the base learners.
This allows us to use all of the unlabeled data, whereas in the paradigms above only a few examples annotated with high conﬁdence were added to the stack for re-training.
Last, a diﬀerent line of research has focused on learning language representations and tuning these representations towards speciﬁc machine learning tasks using a small labeled set [Peters et al., 2018, Clark et al., 2018, Devlin et al., 2018].
This work is complementary to our work, and it could comprise our weak classiﬁers.
3. Background on Weak Supervision and Data Programming
In this section, we describe weak supervision and the data programming paradigm, which have both heavily inﬂuenced the development of our methodology.
Weak supervision revolves around the idea of training models using labels of questionable quality [Dehghani et al., 2017a,b, Mintz et al., 2009].
Data programming is a paradigm for programmatic creation of such training sets, focusing on the scenario where no ground-truth labels are available [Ratner et al., 2016, 2017b].
It can be outlined in the following steps:
(1) Provide weak supervision sources: We deﬁne K weak supervision sources and encode them into Labeling Functions (LF) .
For each unlabeled data point, those sources can either provide a label or abstain from voting. Typically, a LF might consist of a textual pattern, a crowd-worker or a distant supervision resource. However, it is possible to incorporate any other kind of weak source which can provide training labels.
We apply the LFs over M unlabeled data points and derive a (possibly incomplete) vote matrix Λ ∈ R KxM . (2) Denoising:
Our objective is to derive M weak labels from Λ , which are as close as possible to the (unknown) true labels.
As a denoiser, data programming uses a probabilistic graphical Generative Model (GM), relying on agreements and disagreements. It incorporates as trainable parameters: the probability (a) that a LF will label a data point and (b) that this label is correct (accuracy).
The structure of the GM is a hyperparameter, representing correlations of the Labeling Functions and can be estimated automatically [Ratner et al., 2016, 2017a, Bach et al., 2018, 2017].
To train the GM without access to the ground truth, data programming maximizes the marginal log-likelihood that the observed votes occur under the Generative Model (for all possible ground-truth labels):
[Equation] Further, we use the predicted label distributions ˆ Y i = P [Equation]
w ( Y i | Λ) as probabilistic weak labels. (3) Train a noise-aware discriminative model: We use the generated labels for training and use this model as the ﬁnal predictor. During training, we minimize a noise-aware variant of the loss function l with respect to the probabilistic weak labels ˆ Y :
[Equation]
4. Methodology
Based on the concepts of weak supervision and data programming, we propose a methodology for semi-supervised learning, with the intent to capitalize the advantages of multiple learners.
In contrast to the scenario where no ground-truth labels are available, we assume that a gold-labeled training set ( D B ) is available, but its size is size insuﬃcient for training a complex (and therefore more data-hungry) model.
We advocate that it would be beneﬁcial to augment additional, lower quality training data to scale the dataset size.
Instead of relying on heuristics or crowd-sourced labels, we use machine learning models of lower complexity as weak supervision sources.
This comes with the major advantage of adapting an already implemented pipeline with little or no additional eﬀort to similar tasks.
4.1 Data collection
In terms of data, we assume the existence of a labeled training set D B of size m , relevant to our task T .
Additionally to that, we require an unlabeled, arbitrarily large dataset D U of size M (cid:29) m , with the requirement of being drawn from the same distribution as D B .
Other requirements are a validation set D V for hyperparameter tuning and a held-out test set D T for evaluation purposes.
4.2 Constructing diverse base learners
We use D B to train K base learners on solving T .
As in a typical ensemble learning scenario, we try to maximize their individual performance while making them capture diﬀerent “views”’ of the data.
To produce multiple learners we rely on varying hyperparameters and design choices throughout the relation extraction pipeline, which results in the creation of a total 162 base learners.
The most important design choices are:
(1) Sentence pruning: It is often the case, that words appearing within a sentence might be irrelevant to the entities of interest.
For this reason, one can keep only the words between the two entities of interest or additionally include words appearing within a certain sized window before/after them.
More complex approaches incorporating syntactic information can also be considered.
One way to do so is to construct the dependency-based parse tree and include only words contained within the Shortest Path connecting the entities of interest.
In this work we investigate (a) Whole sentences, (b) window of 0, (c) window of 5, and (d) Shortest Dependency Path (SDP).
(2) Sequential features: Additionally to the simple bag-of-words approach, it is also possible to include contiguous sets of tokens as features.
In our approach, we use up to tri-grams.
(3) Text representation: To convert our corpus to a numerical representation, we use token occurrences (binary counts) or TF-IDF weights.
(4) Machine learning algorithms: When the feature matrix is ready, we employ diﬀerent machine learning algorithms, including Logistic Regression, Support Vector Machines (using Gaussian and linear kernels), Random Forest Classiﬁers, Long-Short Term Memory
Networks and Convolutional Neural Networks.
It is important to note, that when the last two models (LSTMs & CNNs) were used, some of the aforementioned feature engineering steps, were not applicable.
4.3 Base learner selection
After producing all possible base learners, we select only a subset of them.
This is necessary due to computational cost and the fact that we should avoid including many similar classiﬁers in a disagreement-based method.
Hence, our objective is to maximize both the individual performance of the base learners and their diversity.
Since this is complex and still an open issue [Zhou, 2011], we resort to a simple method, where we discard all classiﬁers having a lower performance than a certain threshold (evaluated on D V ), while maximizing diversity.
Setting a performance threshold is also desirable, due to the fact that the base learners where automatically created with limited hyperparameter tuning.
We set this threshold above the random guess baseline, but low enough to allow less accurate and more diverse classiﬁers to be part of the ensemble.
To select the most diverse classiﬁers, we employ a similarity-based clustering method.
Using the predictions of the K base learners on D V , we construct a KxK similarity matrix.
In this matrix, each row and column refers to a base learner, while each cell consists of the corresponding pairwise inter-annotator agreement rates (Cohen Kappa coeﬃcient).
We perform K-means clustering [MacQueen et al., 1967] on this matrix and pick the base learners closest to the cluster centroids as most representative of their cluster.
To pick an appropriate number of clusters (and therefore base learners), we refer to the silhouette score coeﬃcient [Rousseeuw, 1987].
4.4 Producing weak labels
We predict the labels of D U using the selected base learners and obtain a KxM (binary) prediction matrix, containing the ”knowledge” our base learners have distilled from D B .
Consequently, we use a denoiser to reduce the vote matrix into M weak labels.
We use the probabilistic Generative Model of data programming (described in Section 3) and select the hyperparameters using the validation dataset.
Additionally, we consider two simpler denoisers to unify the label matrix: (a) a Majority Vote denoiser producing binary weak labels and (b) an Average Vote denoiser calculating an unweighted average of all base learner votes, therefore producing marginal weak labels.
4.5 Training a meta-learner
In the last step, we use a discriminative model (as described in Section 3) as a meta-learner.
In practice, when training the meta-learner with weak supervision, we trade label quality for quantity.
This can be proven to be beneﬁcial in cases where the performance of the meta-learner is upper-bounded by the training set size.
By using high-capacity models such as Deep Neural Networks as meta-learners, we allow them to learn their own features and hopefully build a more accurate representation by relying on a much larger, even though noisy, training dataset.
5. Experimental Setup
To perform our experiments we use part of the functionality of Snorkel [Ratner et al., 2017b,a], a framework build for relation extraction with data programming and weak supervision.
5.1 Datasets
In our experiments, we use the oﬃcial BioCreative CHEMPROT and CDR datasets [Krallinger et al., 2017a, Li et al., 2016], which consist of annotated PubMed 2 abstracts (split in a training, development and test set – Table 2).
[Table]
As described in Subsection 4.1, our methodology requires three gold-labeled datasets, along with a held-out test set.
In both cases, we use the original test sets as the held-out test set ( D T ) and report the ﬁnal scores there.
We merge and shuﬄe the original training and development sets to create the remaining three datasets: We use one part to train the base learners ( D B ) and another part for validation and hyperparameter selection ( D V ).
We use the remaining documents as if they were unlabelled, D U .
The restructured dataset statistics are available on Table 3.
Using this setup, we can make sure that two important requirements are satisﬁed: (a) there is no bias during the document selection process, i.e. we ensure that the training, validation, test and unlabeled dataset are drawn from the same distribution, and (b) all documents have gone through the same pre-processing steps., i.e. labeled datasets have manually annotated entities, while a randomly drawn unlabeled dataset would require applying some Named Entity Recognition algorithm.
We do so to better control the eﬀect of such choices on the results of our algorithm.
Another advantage with this controlled approach is that we are able to compare the performance of the meta-learner trained with weak supervision to the optimal performance, which would be achieved if the ground-truth labels were available.
5.2 Text pre-processing and named entity recognition
Most of the steps in our text pre-processing pipeline are performed by SpaCy (v1.0), an open-source library for Natural Language Processing.
More speciﬁcally, SpaCy performs the following tasks in our pipeline: Sentence splitting, Tokenization, and Dependency parsing.
Both datasets contain manually annotated Named Entity Tags, which are required for the
Candidate Extraction step (Subsection 5.3).
5.3 Candidate extraction
Given all entities of interest within the text, we look for relationship candidates.
We do not consider cross-sentence relations and proceed only with candidates found within the same sentence.
We use Snorkel [Ratner et al., 2017b] for candidate extraction and mapping the candidates to their ground-truth labels.
5.4 Entity replacement
A relationship clasiﬁer is important to understand the Natural Language, rather than memorize the pairs which interact with each other.
For this reason, we replace all entities of interest using the tokens ’ENTITY1’ and ’ENTITY2’ for the entities we want to predict.
We also replace additional entities of the same type within the same sentence with the tokens ’CHEMICAL’, ’GENE’ or ’DISEASE’ accordingly.
5.5 Meta-learner selection & training
In our experiments, we use a simple bi-directional Long-Short Term Memory network, which is one of the most commonly used and better performing Deep Neural Networks, on tasks related to Natural Language.
We use randomly initialized word embeddings and perform a random under-sampling to keep an equal class balance.
We also try diﬀerent hyperparameter settings, including diﬀerent dropout values (0 , 0 .
25 and 0 . 5) and 1-30 training epochs.
We make the selection based the validation dataset D V .
6. Research Questions and Experimental Design
In this section, we form the research questions that we aim to answer, accompanied by our motivation to explore them.
We discuss more details regarding RQ1 and RQ2 in Subsections 6.1 and 6.2, respectively.
RQ1 Can we enhance biomedical relation extraction when using Machine Learning classiﬁers as sources of weak supervision?
RQ2 Which is the optimal setting for using weak supervision on this task?
6.1 Machine learning classiﬁers as weak supervision sources
Related literature provides theoretical warranties, that in this setting and given speciﬁc requirements, adding weakly labeled data will improve the performance of the meta-learner
[Ratner et al., 2016].
Additionally, as the amount of weakly labeled data increases, the performance of the meta-learner is expected to improve quasi-linearly (almost as good) compared to the scenario where the ground-truth labels were provided [Ratner et al., 2017a].
Those requirements include that the weak supervision sources should have accuracy better than random guess, overlap and disagree with each other enough (so that their accuracy can be estimated) while capturing diﬀerent ’views’ of the problem (diversity).
In other words, when combined they should be able to model the problem space suﬃciently, so that meaningful weak labels can be produced.
However, to the best of our knowledge, Machine Learning classiﬁers have not been used so far as weak supervision sources in such a setting, nor in this speciﬁc task.
Therefore, it is unclear whether there is a diverse and suﬃciently big set of base learners, that satisﬁes those conditions after being trained on the same dataset.
This is a critical question, which actively aﬀects the usability of the described methodology.
To evaluate whether weak supervision helps, we conduct several experiments under diﬀerent setups and compare the performance of the meta-learner when trained on three diﬀerent modes: (a) full-supervision on D B , (b) weak-supervision on D U , (c) weak-supervision on D U combined with full-supervision on D B .
Additionally, we can evaluate whether weak-supervision can achieve results comparable to full-supervision, after training the meta-learner using all ground-truth labels ( D U + D B ).
6.2 Optimal setting for applying this methodology Number of base learners:
Selecting the optimal number of classiﬁers to be used as base learners is not a straightforward task.
Naturally, we can construct only a few top-performing learners and as we add more of them, we start to sacriﬁce performance in favor of diversity [Zhou, 2011].
To examine this, we gradually increase the number of Base Learners while benchmarking the performance of the weak labels and the meta-learner.
Comparison
of
various denoising
methods:
The denoising component is fundamental to this method, as it dictates the quality of the weak labels the ﬁnal learner will be trained on.
We use the three denosing methods described in Subsection 4.4 and assess the quality of the results.
Meta-learner performance under diﬀerent weak label distributions:
The denoiser can produce either binary or marginal (non-binary) weak labels.
Additionally, marginal weak labels might be following diﬀerent distributions, ranging from extremely U-shaped (almost binary) to more uniformly distributed.
We perform an error analysis to investigate their eﬀect on the training and the ﬁnal performance of the meta-learner.
7. Results and Analysis
In Subsection 7.1 we answer RQ1, that is whether supervised machine learning classiﬁers can be used as weak classiﬁers, while in Subsection 7.2 we answer RQ2, that is what is the optimal setting for applying weak supervision
[Table]
7.1 Weak classiﬁers
To select the base learners
[Table]
we use the strategy described on Subsection 4.3.
We experiment with diﬀerent number of base learners and benchmark results in intervals of 5 and where the silhouette scores are maximized (Table 4).
Speciﬁcally, we report performance of (a) the Base Learners (in detail in Appendix A), (b) the weak labels/marginals produced by the denoisers and (c) the meta-learner when trained on those weak labels
[Table]
[Table] [Table]
[Table]
C I D 49.
We compare the results achieved with weak supervision on D U (Table 4), versus fullsupervision (Table 3).
It is evident that training the meta-learner with weak labels ( D U ) and a 2 − 2 .
5 x increase in the training set always performs better compared to training with the fewer, gold labels ( D B ).
Performance is further improved, when we also include the available ground-truth labels ( D B + D U ).
This proves that we can successfully augment additional training data using weak supervision, as long as they are drawn from the same distribution.
Additionally, weak supervision can often achieve a performance comparable to fullsupervision; in other words almost as good as if we were using the ground-truth labels.
Interestingly enough, there are even cases where weak supervision achieves slightly better results.
However, those diﬀerences are minor and not statistically signiﬁcant, as there is a high variance on the F1 score of the LSTM.
We also recognize that whenever this happened, the under-sampled training set size of the ﬁnal learner in weak supervision was bigger, due to the fact that undersampling was based on the weak labels instead of the real ones.
Another noteworthy observation, is that a simple Majority Vote often outperforms the meta-learner.
However, this is an expected result and does not undermine the importance of our results, as we can verify from Table 3 that this model (LSTM) cannot outperform
Majority Voting with such a small training dataset, even when gold quality labels are used.
At last, we visualize the learning curves of the meta-learner (starting from the groundtruth labels) to ensure that the weak labels are meaningful and do actually improve performance.
Figures 1b, 1c and 1d indicate an upward trend, while the outlined conﬁdence intervals indicate that the results are statistically signiﬁcant.
Moreover, we observe that the
F1 score on the training set is always much higher than the test score.
This is a sign that our meta-learner suﬀers from high variance (overﬁtting), as the model capacity is far from being fulﬁlled.
Therefore, additional training data are expected to improve the performance of the meta-learner.
7.2 Optimal setting
The complexity of the problem and the methodology, along with the small dataset size do not allow us to draw deﬁnite answers in some of the following questions.
However, we will perform an analysis based on our experimental results and discuss our ﬁndings.
Number of base learners:
We can see from Table 4 that the F 1 score of the weak
Majority Vote labels for 5 learners is the lowest in both experiments.
When it comes to the Generative model weak marginals, we cannot observe any signiﬁcant pattern, as the
F1 score always deviates within 1 . 5 & 2 . 5 points respectively.
The performance of the meta-learner when trained with Average Vote marginals deviates to a certain extent when more than 10 base learners are used, but always performs better compared to when onlybase learners are used.
Using Generative model marginals, performance seems to slightly improve as the number of base learners increases, with two exceptions.
Comparison of various denoising methods:
In the vast majority of cases, the metalearner achieves the best greater performance when trained with Average Marginals.
Generative Model marginals also seem to improve its performance compared to Majority Vote weak labels, with one exception.
However, it is worth highlighting that GM marginals depend on hyperparameters, which are chosen based on the F1 score on a validation dataset.
Later on this section, we argue and demonstrate why this particular measure cannot fully
[Figure]
reﬂect the quality of marginal weak labels.
Therefore, it is not certain whether we have achieved the optimal performance in cases where the Generative Model was used as the denoiser.
Performance under diﬀerent weak label distributions:
The denoisers can produce weak labels which are either binary or marginal (non-binary).
We can conclude that marginal weak labels improve the performance of the meta-learner compared to binary labels.
This is a straightforward comparison, as Majority Vote weak labels always perform worse than Average marginals while none of them is subject to hyperparameter tuning.
Moreover, we observe that the Generative Model tends to create marginals following a U-shaped distribution (close to 0 or 1) in contrast to the average marginals, which are spread more uniformly.
This is evident from the error analysis we perform on the validation set, using a classiﬁcation boundary of 0 .
5 (Figures 2a and 2b).
In both cases, the amount of misclassiﬁed weak labels and therefore their F1 score is the same.
However, it is evident that
Average Vote labels are of higher quality, as most of their misclassiﬁed labels are relatively closer to 0 .
This is inevitable, as the vast majority of the GM marginals are very close to 0 and 1.
Furthermore, these ﬁgures demonstrate the unsuitability of the F 1 score for the evaluation of marginal weak labels.
On Figure 3, we can see how the training loss and validation scores change as we train the
LSTM for more epochs.
We can see that when marginal labels are used, the training error
[Figure]
remains relatively high.
This is especially true with the Average weak marginals, which are spread more uniformly.
On the contrary, it only takes a few epochs for the LSTM to start predicting accurately the binary training labels, despite a small delay on the noisy-labeled
MV weak labels.
Ultimately, training a classiﬁer using marginal labels can be thought of as a regression problem.
In practice, we ask the classiﬁer to predict an exact number (the output of the denoiser) and penalize it every time he fails to do so.
Lastly, we can see that the distributions of predicted logits (on unseen examples) become more spread as the training marginals distributions become more uniform (Figures 4a and 4b).
This is something we would also expect to see when doing regression instead of classiﬁcation.
8. Unlabeled dataset expansion
In this section we discuss our eﬀorts to apply our methodology on the CPR task, while expanding the labeled and unlabeled datasets.
We use all of the CHEMPROT documents (excluding validation and test sets) to train the base learners.
To construct D U , we use
[Figure]
the PubMed API 3 and download two diﬀerent collections: (a) the outgoing citations of
CHEMPROT documents (b) the 25 most similar documents of each CHEMPROT document (using the PMRA similarity metric [Lin and Wilbur, 2007]).
In both cases, the performance of the meta-learner decreases as we add weakly labeled data, indicating a problem with the quality of D U or the generated weak labels.
We also observe a predicted class imbalance of 1:14 on the outgoing citations dataset compared to 1:4 on the original, indicating inherently diﬀerent dataset distributions.
To validate this, we use the t-SNE algorithm
[Maaten and
Hinton, 2008] along with features extracted from our best-performing classiﬁer and visualize candidate samples drawn only from the original set (Figure 5a) versus samples drawn from both sets (Figure 5b).
It becomes evident that most candidates of the new dataset lie in speciﬁc regions of the 2 D space, conﬁrming that it is unsuitable for our use case.
To that end, the best practices of constructing appropriate unlabeled datasets, must be further investigated.
[Figure]
9. Conclusions & future work
We have shown that weak supervision is a tool which can be used for enhancing the performance of complex models, such as deep neural networks, while utilizing both unlabeled data and multiple base learners.
Additionally, we have shown that the proposed methodology is practically feasible for the task at hand, as we have succeeded on deﬁning a combination of base learners, which model the problem space suﬃciently and allow us to take advantage of additional, unlabeled data.
This comes under the requirement that the unlabeled data are drawn from the same domain/distribution as our labeled data, so that our base learners can generalize and perform adequately on D U .
In practice, our methodology shifts the human eﬀort from hand-labeling examples to feature engineering and construction of diverse learners.
More importantly, once a satisfactory set of diverse learners is in place, we can use this method to scale the training datasets in arbitrarily high levels while consistently improving the performance over the supervised learning paradigm.
Moreover, the same pipeline can be re-used on similar tasks with the only requirement of providing the appropriate datasets.
On the contrary, in the typical supervised learning paradigm, we would have to repeatedly hand-label large datasets.
Despite demonstrating the usability of our method using a controlled, small-scale dataset, it is crucial to further explore the requirements of constructing a large enough unlabelled dataset and perform the same experiments there.
That would likely improve the metalearner performance further (which is currently upper-bounded by the small dataset size) and allow us to draw stronger conclusions on the research questions of Subsection 6.2.
Additionally it would allow us to inspect how performance improves with the increase of D U in a diﬀerent scale of magnitude and if there seems to be a certain performance threshold, which we cannot surpass using weak supervision.
Our preliminary experiments demonstrate that drawing an appropriate unlabeled dataset given a labeled one is a challenging task itself, along with the deﬁnition of “appropriate”, and semi-supervised algorithms should not take the existence of an appropriate unlabeled dataset for granted.
Further, it would be very important to conclude on a more appropriate metric than the
F1 score for the evaluation of marginal weak labels.
Currently, the absence of an appropriate metric prevents us from drawing conclusions directly from the weak labels, without having to introduce an additional step (train the meta-learner).
This would also allow us to select the optimal hyperparameters of the Generative Model and could have a signiﬁcant impact upon the ﬁnal performance.
Other areas for further investigation include experimenting with the meta-learner (eg. using pre-trained word embeddings or other model architectures) and deﬁning a more appropriate selection method for the Base Learners.
Last, it would be interesting to examine how this system would behave if the Base Learners abstained from voting on the examples they are less certain about.
One could simply delete a percentage of the votes which are closer to the classiﬁcation boundary, or perform a probability calibration on the output of the Base Learners and set a minimum conﬁdence threshold below which they would abstain voting.
This could also provide the Generative Model with a modeling advantage, compared to unweighted methods (such as Majority Voting), as described in an analysis related to the trade-oﬀs of weak supervision [Ratner et al., 2017a].
Appendix A. Base Learner statistics
# pruning representation l e mm a s n - g r a m s s t
o p w o r d s
L S
A [ d i m s ] und e r s a m p li n g model
F1 @
val (cid:55)
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
LogisticRegression 55.73% window=
TfIdf (cid:51) (cid:51) (cid:51)
SVM(RBF)
54.29% window=
BinaryCounts (cid:51) (cid:51) (cid:51)
LogisticRegression 53.57%
SDP
BinaryCounts (cid:51) (cid:51) (cid:51)
RandomForest 53.33% window=
BinaryCounts (cid:51) (cid:55) (cid:51)
RandomForest 53.85% (cid:55)
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
SDP
TfIdf (cid:51) (cid:55) (cid:51)
SVM(linear)
56.79% window=
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
SVM(linear) 60.80% window=
TfIdf (cid:51) (cid:51) (cid:51)
SVM(linear)
52.87% window=
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
RandomForest 53.58% — — — — — — —
LSTM 44.43% window=
TfIdf (cid:51) (cid:55) (cid:51)
SVM(linear) 57.87% (cid:55)
TfIdf (cid:51) (cid:51) (cid:51)
SVM(RBF)
SDP
BinaryCounts (cid:51) (cid:51) (cid:51)
LogisticRegression 51.67% window=
TfIdf (cid:51) (cid:51) (cid:51)
SVM(RBF)
54.29% window=
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
SVM(linear) 60.80% window=
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
52.02% (cid:55)
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
56.84% window=
BinaryCounts (cid:51) (cid:55) (cid:51)
LogisticRegression 57.21%
SDP
TfIdf (cid:51) (cid:55) (cid:51)
LogisticRegression 56.52% window=
TfIdf (cid:51) (cid:51) (cid:51)
SVM(linear)
52.87% window=
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
RandomForest 53.58% — — — — — — —
LSTM 44.43%
SDP
TfIdf (cid:51) (cid:55) (cid:51)
RandomForest 52.35% (cid:55)
TfIdf (cid:51) (cid:51) (cid:51)
SVM(RBF) 51.43%
window=
TfIdf (cid:51) (cid:51) (cid:51)
SVM(RBF)
54.29% window=
BinaryCounts (cid:51) (cid:51) (cid:55) (cid:51)
RandomForest 51.29%
SDP
TfIdf (cid:51) (cid:51) (cid:51)
SVM(RBF)
SDP
TfIdf (cid:51) (cid:51) (cid:51)
SVM(RBF)
51.63% window=
TfIdf (cid:51) (cid:51) (cid:55) (cid:51)
SVM(RBF)
53.12% window=
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
52.02% window=
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
61.30% window=
BinaryCounts (cid:51) (cid:55) (cid:51)
LogisticRegression 57.21% window=
BinaryCounts (cid:51) (cid:51) (cid:51)
SVM(RBF)
SDP
TfIdf (cid:51) (cid:55) (cid:51)
[Table]
[Table]
LSTM 44.43%
SDP
TfIdf (cid:51) (cid:51) (cid:55) (cid:51)
RandomForest 49.78% (cid:55)
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
56.84% window=
TfIdf (cid:51) (cid:51) (cid:51)
SVM(RBF)
54.29% (cid:55)
TfIdf (cid:51) (cid:51) (cid:51)
LogisticRegression 50.95%
SDP
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
RandomForest 49.69% window=
TfIdf (cid:51) (cid:55) (cid:51)
RandomForest 54.76% (cid:55)
BinaryCounts (cid:51) (cid:55) (cid:51)
RandomForest 50.67% (cid:55)
BinaryCounts (cid:51) (cid:55) (cid:51)
SVM(linear) 53.34% window=
TfIdf (cid:51) (cid:51) (cid:51)
SVM(RBF)
SDP
TfIdf (cid:51) (cid:51) (cid:55) (cid:51)
RandomForest 49.78%
SDP
TfIdf (cid:51) (cid:51) (cid:51)
SVM(RBF)
51.63% (cid:55)
TfIdf (cid:51) (cid:51) (cid:51)
LogisticRegression 50.95% (cid:55)
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
56.84% window=
TfIdf (cid:51) (cid:55) (cid:51)
SVM(RBF) 50.34% window=
TfIdf (cid:51) (cid:55) (cid:51)
SVM(RBF)
55.78% window=
BinaryCounts (cid:51) (cid:55) (cid:51)
SVM(RBF)
57.89% window=
TfIdf (cid:51) (cid:51) (cid:51)
RandomForest 53.97%
SDP
TfIdf (cid:51) (cid:55) (cid:51)
LogisticRegression 56.52% — — — — — — —
LSTM 44.43% window=
BinaryCounts (cid:51) (cid:51) (cid:51)
SVM(RBF)
54.18% window=
TfIdf (cid:51) (cid:55) (cid:51)
RandomForest 50.85% window=
BinaryCounts (cid:51) (cid:55) (cid:51)
SVM(linear) 57.45%
SDP
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(linear) 54.43% window=
TfIdf (cid:51) (cid:51) (cid:51)
SVM(RBF)
54.29% window=
BinaryCounts (cid:51) (cid:51) (cid:55) (cid:51)
RandomForest 51.29% window=
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
SDP
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
RandomForest 49.69%
[Table]
representation l e mm a s n - g r a m s s t
o p w o r d s
L S
A [ d i m s ] und e r s a m p li n g model
F1 @
val window=
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
51.88% window=
TfIdf (cid:51) (cid:51) (cid:55) (cid:51)
SVM(linear) 49.94%
SDP
TfIdf (cid:51) (cid:55) (cid:51)
SVM(linear) 56.21%
SDP
BinaryCounts (cid:51) (cid:51) (cid:51)
SVM(RBF) 49.70% window=
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
LogisticRegression 51.51% window=
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF) 51.88%
[Table]
SDP
TfIdf (cid:51) (cid:55) (cid:51)
SVM(linear) 56.21%
[Table]
BinaryCounts (cid:51) (cid:55) (cid:51)
SVM(linear)
SDP
BinaryCounts (cid:51) (cid:51) (cid:51)
SVM(RBF) 49.70% window=
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
LogisticRegression 51.51%
SDP
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
RandomForest 49.84%
SDP
BinaryCounts (cid:51) (cid:51) (cid:55) (cid:51)
SVM(linear) 49.85%
window=
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF) 52.62% window=
BinaryCounts (cid:51) (cid:55) (cid:51)
SVM(RBF) 52.08%
SDP
TfIdf (cid:51) (cid:55) (cid:51)
SVM(linear) 56.21%
SDP
BinaryCounts (cid:51) (cid:51) (cid:55) (cid:51)
RandomForest 50.00% (cid:55)
BinaryCounts (cid:51) (cid:55) (cid:51)
SVM(RBF)
50.28% (cid:55)
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
RandomForest 50.36%
SDP
TfIdf (cid:51) (cid:51) (cid:51)
LogisticRegression 52.47% (cid:55)
TfIdf (cid:51) (cid:55) (cid:51)
SVM(RBF) 49.92% window=
BinaryCounts (cid:51) (cid:51) (cid:55) (cid:51)
LogisticRegression 50.56% window=
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF) 51.88%
SDP
BinaryCounts (cid:51) (cid:51) (cid:55) (cid:51)
SVM(linear) 49.85%
window=
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF) 52.62% (cid:55)
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
RandomForest 50.36%
SDP
TfIdf (cid:51) (cid:55) (cid:51)
SVM(linear) 56.21%
SDP
BinaryCounts (cid:51) (cid:51) (cid:55) (cid:51)
RandomForest 50.00% window=
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
51.20% (cid:55)
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
49.88% (cid:55)
BinaryCounts (cid:51) (cid:55) (cid:51)
SVM(RBF) 50.28%
SDP
TfIdf (cid:51) (cid:51) (cid:51)
LogisticRegression 52.47% window=
TfIdf (cid:51) (cid:55) (cid:51)
RandomForest 50.00% (cid:55)
BinaryCounts (cid:51) (cid:51) (cid:51)
RandomForest 50.38% window=
BinaryCounts (cid:51) (cid:51) (cid:55) (cid:51)
LogisticRegression 50.56% window=
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
51.88% window=
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF) 52.62% (cid:55)
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
RandomForest 50.36% window=
BinaryCounts (cid:51) (cid:51) (cid:51)
SVM(RBF) 52.04%
SDP
TfIdf (cid:51) (cid:55) (cid:51)
SVM(linear)
56.21% window=
BinaryCounts (cid:51) (cid:51) (cid:51)
LogisticRegression 50.50% window=
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
51.20% (cid:55)
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
SDP
BinaryCounts (cid:51) (cid:51) (cid:55) (cid:51)
SVM(linear) 49.85%
SDP
TfIdf (cid:51) (cid:51) (cid:51)
LogisticRegression 52.47% window=
TfIdf (cid:51) (cid:55) (cid:51)
RandomForest 50.00% window=
BinaryCounts (cid:51) (cid:55) (cid:51)
SVM(linear)
50.46% (cid:55)
BinaryCounts (cid:51) (cid:51) (cid:51)
RandomForest 50.38% window=
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
RandomForest 49.85%
SDP
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
RandomForest 49.84% window=
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
51.88% window=
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF) 52.62%
window=
BinaryCounts (cid:51) (cid:51) (cid:51)
SVM(RBF) 52.04% (cid:55)
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
RandomForest 50.36% window=
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
RandomForest 49.85%
SDP
TfIdf (cid:51) (cid:51) (cid:55) (cid:51)
SVM(linear) 49.86% window=
BinaryCounts (cid:51) (cid:51) (cid:51)
LogisticRegression 50.50% (cid:55)
BinaryCounts (cid:51) (cid:51) (cid:51)
RandomForest 50.38% window=
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
51.20% (cid:55)
TfIdf (cid:51) (cid:55) (cid:55) (cid:51)
SVM(RBF)
SDP
TfIdf (cid:51) (cid:51) (cid:51)
LogisticRegression 52.47%
SDP
TfIdf (cid:51) (cid:55) (cid:51)
SVM(RBF)
54.34% window=
BinaryCounts (cid:51) (cid:55) (cid:51)
RandomForest 52.92% window=
TfIdf (cid:51) (cid:55) (cid:51)
RandomForest 50.00%
SDP
BinaryCounts (cid:51) (cid:55) (cid:51)
SVM(RBF)
54.55% window=
BinaryCounts (cid:51) (cid:55) (cid:51)
SVM(linear)
SDP
BinaryCounts (cid:51) (cid:51) (cid:51)
SVM(RBF) 49.70% SDP
TfIdf (cid:51) (cid:55) (cid:55) (cid:51) RandomForest 49.84%
SDP
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM(linear) 56.95% window=
BinaryCounts (cid:51) (cid:55) (cid:55) (cid:51)
SVM
[Table]