Methods for improving the training and prediction quality of weakly supervised machine learning models vary in how much they are tailored to a specific task , or integrated with a specific model architecture. In this work, we propose a software framework Knodle that provides a modularization for separating weak data annotations, powerful deep learning models, and methods for improving weakly supervised training . This modularization gives the training process access to fine-grained information such as data set characteristics, matches of heuristic rules, or elements of the deep learning model ultimately used for prediction. Hence, our framework can encompass a wide range of training methods for improving weak supervision, ranging from methods that only look at the correlations of rules and output classes (independently of the machine learning model trained with the resulting labels), to those methods that harness the interplay of neural networks and weakly labeled data  .
Strategies for improving the training and prediction quality of weakly supervised machine learning models vary in how much they are tailored to a specific task or integrated with a specific model architecture. In this work, we propose a software framework Knodle that treats weak data annotations, deep learning models, and methods for improving weakly supervised training as separate, modular components. The standardized interfaces between these independent parts account for data- and model-agnostic weak supervision method development, but still allow the training process to access fine-grained information such as data set characteristics, matches of heuristic rules, as well as elements of the deep learning model ultimately used for prediction. Hence, our framework can encompass a wide range of training methods for improving weak supervision, ranging from methods that only look at correlations of rules and output classes (independently of the machine learning model trained with the resulting labels), to those that harness the interplay of neural networks and weakly labeled data . We illustrate the benchmarking potential of the framework with a performance comparison of several reference implementations on a selection of datasets that are already available in Knodle .
R
Methods
Strategies
D
,
R
provides a modularization for separating
treats
D
powerful
R
. This modularization gives
as separate, modular components. The standardized interfaces between these independent parts account for data- and model-agnostic weak supervision method development, but still allow
R
access to
to access
R
or
as well as
D
the
D
methods
A
. We illustrate the benchmarking potential of the framework with a performance comparison of several reference implementations on a selection of datasets that are already available in Knodle