CERES : Pretraining of Graph-Conditioned Transformer for Semi-Structured Session Data (2022)

Abstract: User sessions empower many search and recommendation tasks on a daily basis. Such session data are semi-structured, which encode heterogeneous relations between queries and products, and each item is described by the unstructured text. Despite recent advances in self-supervised learning for text or graphs, there lack of self-supervised learning models that can effectively capture both intra-item semantics and inter-item interactions for semi-structured sessions. To fill this gap, we propose CERES, a graph-based transformer model for semi-structured session data. CERES learns representations that capture both inter- and intra-item semantics with (1) a graph-conditioned masked language pretraining task that jointly learns from item text and item-item relations; and (2) a graph-conditioned transformer architecture that propagates inter-item contexts to item-level representations. We pretrained CERES using ~468 million Amazon sessions and find that CERES outperforms strong pretraining baselines by up to 9% in three session search and entity linking tasks.

## introduction user sessions are ubiquitous in online e-commerce stores. an e-commerce session contains customer interactions with the platform in a continuous period. within one session, the customer can issue multiple queries and take various actions on the retrieved products for these queries, such as clicking, adding to cart, and purchasing. sessions are important in many e-commerce applications, e.g., product recommendation @xcite , query recommendation @xcite , and query understanding @xcite . this paper considers sessions as semi-structured data, as illustrated in figure 1 . at the higher level, sessions are heterogeneous graphs that contain interactions between items. at the lower level, each graph node has unstructured text descriptions: we can describe queries by search keywords and products by titles, attributes, customer reviews, and other descriptors. our goal is to simultaneously encode both the graph and text aspects of the session data to understand customer preferences and intents in a session context. pretraining on semi-structured session data remains an open problem. first, existing works on learning from session data usually treat a session as a sequence or a graph @xcite @xcite . while they can model inter-item relations, they do not capture the rich intra-item semantics when text descriptions are available. furthermore, these models are usually large neural networks that require massive labeled data to train from scratch. another line of research utilizes large-scale pretrained language models @xcite @xcite ) as text encoders for session items. however, they fail to model the relational graph structure. several works attempt to improve language models with a graph-structured knowledge base, such as in @xcite @xcite . while adjusting the semantics of entities according to the knowledge graph, they fail to encode general graph structures in sessions. we propose ceres (graph conditioned encoder representations for session data), a pretraining model for semi-structured e-commerce session data, which can serve as a generic session encoder that simultaneously captures both intra-item semantics and inter-item relations. beyond training a potent language model for intra-item semantics, our model also conditions the language modeling task on graph-level session information, thus encouraging the pretrained model to learn how to utilize inter-item signals. our model architecture tightly integrates two key components: (1) an item transformer encoder, which captures text semantics of session items; and (2) a graph conditioned transformer, which aggregates and propagates inter-item relations for cross-item prediction. as a result, ceres models the higher-level interactions between items. we have pretrained ceres using 468,199,822 sessions and performed experiments on three session-based tasks: product search, query search, and entity linking. by comparing with publicly available state-of-the-art language models and domain-specific language models trained on alternative representations of session data, we show that ceres outperforms strong baselines on various session-based tasks by large margins. experiments show that ceres can effectively utilize sessionlevel information for downstream tasks, better capture text semantics for session items, and perform well even with very scarce training examples. we summarize our contributions as follows: 1) we propose ceres , a pretrained model for semistructured e-commerce session data. ceres can effectively encode both e-commerce items and sessions and generically support various sessionbased downstream tasks. 2) we propose a new graph-conditioned transformer model for pretraining on general relational structures on text data. 3) we conducted extensive experiments on a largescale e-commerce benchmark for three sessionrelated tasks. the results show the superiority of ceres over strong baselines, including mainstream pretrained language models and state-ofthe-art deep session recommendation models. ## customer sessions a customer session is the search log before a final purchase action. it consists of customer-queryproduct interactions: a customer submits search queries obtains a list of products. the customer may take specific actions, including view and purchase on the retrieved products. hence, a session contains two types of items: queries and products, and various relations between them established by customer actions. we define each session as a relational graph @xmath0, e) that contains all queries and products in a session and their relations. the vertex set @xmath1, p) is partitioned into ordered query set q and unordered product set p. the queries @xmath2, . . . , q n ) are indexed by order of the customer's searches. the edge set e contains two types of edges: {(q i , q j ), @xmath3; and {q i , p j , a ij } are bidirectional edges that connects the ith query and jth product, if the customer took action a ij on product p j retrieved by query q j . the queries and products are represented by textual descriptions. specifically, each query is represented by customer-generated search keywords. each product is represented with a table of textual attributes. each product is guaranteed to have a product title and description. in this paper, we call "product sequence" as the concatenation of title and description. a product may have additional attributes, such as product type, color, brand, and manufacturer, depending on their specific categories. ## our method in this section we present the details of ceres. we first describe our designed session pretraining task in section 3.1, and then describe the model architecture of ceres in section 3.2. ## related work pretrained language models such as bert @xcite , bart @xcite , elec-tra @xcite , roberta @xcite have pushed the frontiers of many nlp tasks by large margins. their effectiveness and efficiency in parallelism have made them popular and generalpurpose language encoders for many text-rich applications. however, they are not designed to model relational and graph data, and hence are not the best fit for e-commerce session data. researchers have also sought to enhance text representations in pretrained models with knowledge graphs @xcite @xcite @xcite . while these models consider a knowledge graph structure on top of text data, they generally use entities or relations in knowledge graphs to enhance text representations, but cannot encode arbitrary graph structures. this is not sufficient in session-related applications as session structures are ignored. many works have been proposed to learn pretrained graph neural networks. initially, methods were proposed for domain-specific graph pretraining @xcite . however, they rely on pre-extracted domain-specific node-level features, and cannot be extended to either session data or text data as nodes. recently, many works have been proposed to pretrain on general graph structure @xcite @xcite . however, they cannot encode the semantics of text data as nodes. contextual information in sessions have been shown beneficial to various related recommendation tasks, such as product recommendation @xcite @xcite and query rewriting @xcite . many existing session-based recommendation methods seek to exploit the transitions between items @xcite @xcite and considering sessions as graphs @xcite @xcite . ## limitations and risks this paper limits the application of ceres to session data with text descriptions. ceres has the potential of being a universal pretraining framework for arbitrary heterogeneous data. for example, sessions can include product images and customer reviews for more informative multimodal graphs. we leave this extension for future work. session data are personalized experience for customers and could cause privacy issues if data are not properly anonymized. in application, the model should be used to avoid exploitation or leakage of customers personal profiles and preferences. ## conclusion we proposed a pretraining framework, ceres, for learning representations for semi-structured ecommerce sessions. we are the first to jointly model intra-item text and inter-item relations in session graphs with an end-to-end pretraining framework. by modeling graph-conditioned masked language modeling, our model is encouraged to learn high-quality representations for both intraitem and inter-item information during its pretraining on massive unlabeled session graphs. furthermore, as a generic session encoder, our model enabled effective leverage of session information in downstream tasks. we conducted extensive experiments and ablation studies on ceres in comparison to state-of-the-art pretrained models and recommendation systems. experiments show that ceres can produce higher quality text representations as well as better leverage of session graph structure, which are important to many ecommerce related tasks, including product search, query search, and query understanding.