Dense Feature Memory Augmented Transformers for COVID -19 Vaccination Search Classification (2022)

Abstract: With the devastating outbreak of COVID-19, vaccines are one of the crucial lines of defense against mass infection in this global pandemic. Given the protection they provide, vaccines are becoming mandatory in certain social and professional settings. This paper presents a classification model for detecting COVID-19 vaccination related search queries, a machine learning model that is used to generate search insights for COVID-19 vaccinations. The proposed method combines and leverages advancements from modern state-of-the-art (SOTA) natural language understanding (NLU) techniques such as pretrained Transformers with traditional dense features. We propose a novel approach of considering dense features as memory tokens that the model can attend to. We show that this new modeling approach enables a significant improvement to the Vaccine Search Insights (VSI) task, improving a strong well-established gradient-boosting baseline by relative +15% improvement in F1 score and +14% in precision.

## introduction though covid-19 continues to be a challenge worldwide, vaccines have provided the much needed hope. countries and governments have significantly ramped up their efforts to improve the reach of the vaccines including booster shots. as such, it is important to understand how users search for vaccine related information such as vaccine efficacy, safety, and regional availability as this information can be very useful to inform policy decision making, create effective public service announcements, implement more efficient distribution of vaccines, etc. to this end, we released a public tool for covid-19 vaccination search insights, an interactive report that provides insights on user searches for covid-19 vaccinations. for an example please see figure 1 . on the backend, this tool performs privacy preserving classification of user search queries and clusters them based on the region they were issued from to present a timeline of how these searches have changed over time. a core task in the vsi tool is the problem of classifying search query intent, and one of them is whether users are seeking information on vaccine access, i.e. queries related to the eligibility, availability, and accessibility of covid-19 vaccines. the task is a challenging one since simply matching for covid related terms is insufficient, as queries such as fully vaccinated travel or proof of covid vaccination are negative classes. hence, this problem is nuanced and may benefit from a coalition of advanced language understanding systems and traditional search-related feature engineering methods. as such, the problem at hand crosses two main modalities, i.e., text and traditional dense features. in this problem, the text features are user search queries that are short and/or lack context. the dense features (discussed in more detail in section 5) are hand crafted features from recent and past activities, previous clicks, and named entities that play a critical role in understanding the user's intent. however, dense features alone fail to capture important contextual language cues, such as those that state-of-the-art natural language understanding systems @xcite have been shown to handle well. we find that both modalities are highly complementary and it is difficult to achieve strong performance using only a single modality. our contributions the overall contributions of this paper can be summarized as follows: • we propose a new model and framework for search query intent classification for our covid-19 vaccination search insight tool. ## related works this section presents related works and the background for this paper. classification with feature-based ml. building machine learned (ml) systems that operate across dense hand-crafted features is a well-established method. it is common to consider this class of ml methods as tabular machine learning whereby tree-based methods @xcite are dominant foot_0 . within the context of text classifiers, feature engineering typically leverages stemming, lemmatization, part-of-speech tags, tf-idf vectors, entities, salient terms, and other features that are relevant to the task at hand. it is also popular to use semantic representations from glove @xcite or bert @xcite as input dense features to a model. nlu with pretrained transformers. transformers (ashish vaswani, 2017), characterized by interleaved self-attention and mlp blocks, have become the dominant sequence model for language processing and understanding (nlu) @xcite @xcite . the key idea behind self-attention is to perform tokento-token alignment where the joint interaction of queries and keys retrieve from a memory store (value). to this end, it is also common for advanced transformer architectures to leverage global memory tokens @xcite @xcite that act as a parameter store for the query to attend to @xcite . a cornerstone of these systems is the pretraining task that learns general purpose language representations, which have been shown to be extremely beneficial @xcite due to the gains from transfer learning @xcite . joint learning of textual and dense features. performing feature extraction to convert text into dense features (e.g. tf-idf @xcite , word2vec @xcite , etc.) for the purpose of learning classifiers jointly with other non-textual (numerical) dense features has been common practice in machine learning for some time @xcite @xcite . previous research in the multi-modal domain adopt a strategy of early fusion or late fusion: joining the two modalities (in this case textual and numerical) either in feature space early in the architecture or in seman-tic/decision space late in the architecture, respectively @xcite . perhaps most related to our approach is a recent work on joint representation of text and tabular data @xcite that pass a flattened representation of a table alongside text during encoding. our work instead provides a method for jointly encoding text with dense features of a more generic, unstructured form. ## problem description given a search query q, the objective is to classify whether the query was issued with the intent of seeking information related to vaccine access. notably, q is short in length and may or may not contain all the information needed to make the correct prediction. additionally, each q is supplemented with numerical features in the form of a dense feature vector x @xmath0, where d f eatures can be any number of features. details about how x f is constructed for our setup is present in section 5, but in short x f represents the topicality scores of phrases related to the query with d f eatures being 60k. however, it is important to note that all methods described in this paper are agnostic to the source of x f and can be applied to any vector of numerical features. ## vsi transformer this section describes the proposed method. ## covid-19 vaccine access dataset due to the novel nature of covid-19, no previous datasets exist to accurately learn a model for the purpose of vaccine access query classification. thus, in this section we outline the process we used to create this dataset. collection. to collect a dataset of queries to be labeled for vaccine access, we sample anonymized queries from real search traffic. since a small minority of our search queries are for covid-19 vaccination topics, we leveraged google's knowledge graph entities to find queries that included high confidence positives, potential positives, and close negatives. for example, for high precision candidates we sample top and random queries associated with the entity "covid-19 vaccination", while for high recall low precision candidates, we sample queries that are only associated with the entity "covid-19" or with "vaccination". labeling. to label this dataset for the specific purpose of vaccine access, we rely on a large pool of search quality raters who have deep experience with how health-related information needs are reflected in search queries. these raters were unknown to and independent of the developers of the classifiers. each query is rated by three independent raters. label expansion. we expand our dataset using label propagation to queries that are very similar to labeled queries. we include examples of positive and negative vaccine access queries in dense feature augmentation. we augment our dataset by supplementing each query using dense features. to generate these dense features we use a combination of (1) entities mentioned in the query via a proprietary library analogous to google cloud entity analysis (goo, 2019) and ( 2 ) related search queries determined by a proprietary algorithm. we pool these two sources and use the 60,000 most common words and phrases to create a dense feature representation with dimension 60,000. at each dimension, we assign a relevance score for the phrase. for mentioned entities, this is analogous to salience in the google cloud entity analysis api. ## experiments below, we dive into experiment setups and results. ## limitations though we have shown that just one memory token is sufficient, assigning tokens to dense features means less number of tokens are available in the sequence for the text input. another limitation is the use of locale specific vocabularies for dense features for each regional dataset as present in ## conclusion this paper presented an important task of classifying search queries with covid-19 vaccination access intent. with an extensive set of experiments and comparing with strong baselines, we presented vsi transformer, a novel and generic approach that consistently and strongly outperforms all existing baselines that operate on either of the two modalities, or late fusion of the modalities. with an ablation study on model size, we show that for online applications where shallower models need to be deployed primarily due to latency constraints, making use of dense features can help bridge the gap in performance compared to deeper models. future work in this direction can help further understand how to choose the optimal number of memory to-kens, and explore more architectures to efficiently combine sequential and dense features.