Domain-Agnostic Neural Architecture for Class Incremental Continual Learning in Document Processing Platform (2023)

Abstract: Production deployments in complex systems require ML architectures to be highly efficient and usable against multiple tasks. Particularly demanding are classification problems in which data arrives in a streaming fashion and each class is presented separately. Recent methods with stochastic gradient learning have been shown to struggle in such setups or have limitations like memory buffers, and being restricted to specific domains that disable its usage in real-world scenarios. For this reason, we present a fully differentiable architecture based on the Mixture of Experts model, that enables the training of high-performance classifiers when examples from each class are presented separately. We conducted exhaustive experiments that proved its applicability in various domains and ability to learn online in production environments. The proposed technique achieves SOTA results without a memory buffer and clearly outperforms the reference methods.

## introduction solutions based on deep neural networks have already found their applications in almost every domain that can be automated. an essential part of them is nlp, the development of which has gained particular momentum with the beginning of the era of transformers @xcite . complex and powerful models made it possible to solve problems such as text classification with a previously unattainable accuracy. however, exploiting the capabilities of such architectures in real-world systems requires online learning after deployment. this is especially difficult in dynamically changing environments that require the models to be frequently retrained due to domain or class setup shifts. an example of such environment is alphamoon workspace 1 where the presented architecture will be deployed as a model for document 1 https://alphamoon.ai/ classification since we noticed the emerging need for online learning. we observed that the users' data in document classification process is changing frequently and such shifts often decrease the model accuracy. as a result, we have to retrain the models manually ensuing a time-consuming process. our goal was to design an effective approach to incremental learning that will be used in a continual learning module of our system (figure 1 ). recently, neural architectures have become effective and widely used in classification problems @xcite . the parameter optimization process based on gradient descent works well when the data set is sufficiently large and fully available during the training process. otherwise, the catastrophic forgetting (french, 1999) may occur, which makes neural networks unable to be trained incrementally. continual learning aims to develop methods that enable accumulating new knowledge without forgetting previously learnt one. in this paper, we present a domain-agnostic architecture for online class incremental continual learning called de&e @xcite . inspired by the e&e method @xcite , we proposed a method that increases its accuracy, provides full differentiability, and, most importantly, can effectively solve real-world classification problems in production environments. our contribution is as follows: 1) we introduced a differentiable knn layer @xcite into the model architecture, 2) we proposed a novel approach to aggregate classifier predictions in the ensemble, 3) we performed exhaustive experiments showing the ability to learn incrementally and real-world usability, 4) we demonstrate the effectiveness of the proposed architecture by achieving sota results on various data sets without a memory buffer. ## conclusions in this paper, we proposed a domain-agnostic architecture for continual learning with a training procedure specialized in challenging class incremental problems. the presented architecture is based on the mixture of experts technique and handles many practical issues related to the deployment of text classification models in non-trivial real-world systems. as our main contribution, we introduced a fully differentiable soft knn layer and a novel prediction weighting strategy. by conducting exhaustive experiments, we showed improvement in accuracy for all the cases studied and achieved sota results without using a memory buffer. this enables an effective and secure training, especially when working with sensitive textual data. the presented architecture is highly flexible, can effectively solve classification problems in many domains, and can be applied to real-world machine learning systems requiring continuous improvement. such work enables researchers to make further steps toward overrunning many of the current challenges related to language technology applications.