NLP Service API s and Models for Efficient Registration of New Clients (2020)

Abstract: State-of-the-art NLP inference uses enormous neural architectures and models trained for GPU-months, well beyond the reach of most consumers of NLP. This has led to one-size-fits-all public API-based NLP service models by major AI companies, serving millions of clients. They cannot afford traditional fine tuning for individual clients. Many clients cannot even afford significant fine tuning, and own little or no labeled data. Recognizing that word usage and salience diversity across clients leads to reduced accuracy, we initiate a study of practical and lightweight adaptation of centralized NLP services to clients. Each client uses an unsupervised, corpus-based sketch to register to the service. The server modifies its network mildly to accommodate client sketches, and occasionally trains the augmented network over existing clients. When a new client registers with its sketch, it gets immediate accuracy benefits. We demonstrate the proposed architecture using sentiment labeling, NER, and predictive language modeling.

## introduction state-of-the-art nlp uses large neural networks with billions of parameters, enormous training data, and intensive optimization over weeks of gputime, causing more carbon emission than a car over its lifetime @xcite . such training prowess is (mercifully) out of reach for most users of nlp methods. recognizing this, large ai companies have launched nlp cloud services foot_0 and also provided trained models for download and fine tuning. but many clients have too little data or hardware for fine tuning massive networks. neither can the service be expected to fine-tune for each client. distributional mismatch between the giant general-purpose corpus used to train the central service and the corpus from which a client's instances arise leads to lower accuracy. a common source of trouble is mismatch of word salience @xcite between client and server corpora @xcite . in this respect, our setting also presents a new opportunity. clients are numerous and form natural clusters, e.g., healthcare, sports, politics. we want the service to exploit commonalities in existing client clusters, without explicitly supervising this space, and provide some level of generalization to new clients without re-training or fine-tuning. in response to the above challenges and constraints, we initiate an investigation of practical protocols for lightweight client adaptation of nlp services. we propose a system, kyc ("know your client"), in which each client registers with the service using a simple sketch derived from its (unlabeled) corpus. the service network takes the sketch as additional input with each instance later submitted by the client. the service provides accuracy benefits to new clients immediately. what form can a client sketch take? how should the service network incorporate it? while this will depend on the task, we initiate a study of these twin problems focused on predictive language modeling, sentiment labeling, and named entity recognition (ner). we show that a simple late-stage intervention in the server network gives visible accuracy benefits, and provide diagnostic analyses and insights. our code and data can be found here foot_1 . ## proposed service protocol we formalize the constraints on the server and client in the api setting. (1) the server is expected to scale to a large number of clients making it impractical to adapt to individual clients. (2) after registration, the server is expected to provide labeling immediately and response latency per instance must be kept low implying that the server's inference network cannot be too compute-intensive. ## experiments we evaluate kyc on three nlp tasks as services: ner, sentiment classification, and auto-completion based on predictive language modeling. we compare kyc against the baseline model (without the g φ network in figure 1 ) and the mixture of experts (moe) model @xcite (see appendix b). for all three models, the m θ network is identical in structure. in kyc, g φ has two linear layers with relu giving a 128-dim vector g, with slight exceptions (see appendix a). we choose datasets that are partitioned naturally across domains, used to simulate clients. we evaluate in two settings: indistribution (id) on test instances from clients seen during training, and out-of-distribution (ood) on instances from unseen clients. for this, we perform a leave-k-client-out evaluation where given a set d of clients, we remove k clients as ood test and use remaining d -k as the training client set c. named entity recognition (ner) we use ontonotes @xcite which has 18 entity classes from 31 sources which forms our set d of clients. we perform leave-2-out test five times with 29 training clients as c. we train a cased bert-based ner model @xcite and report f-scores. ## conclusion we introduced the problem of lightweight client adaption in nlp service settings. this is a promising area, ripe for further research on more complex tasks like translation. we proposed client sketches and kyc: an early prototype server network for on-the-fly adaptation. three nlp tasks showed considerable benefits from simple, per-label bias correction. alternative architectures and ablations provide additional insights.