Textomics: A Dataset for Genomics Data Summary Generation (2022)

Abstract: Summarizing biomedical discovery from genomics data using natural languages is an essential step in biomedical research but is mostly done manually. Here, we introduce Textomics, a novel dataset of genomics data description, which contains 22,273 pairs of genomics data matrices and their summaries. Each summary is written by the researchers who generated the data and associated with a scientific paper. Based on this dataset, we study two novel tasks: generating textual summary from a genomics data matrix and vice versa. Inspired by the successful applications of k nearest neighbors in modeling genomics data, we propose a k NN-Vec2Text model to address these tasks and observe substantial improvement on our dataset. We further illustrate how Textomics can be used to advance other applications, including evaluating scientific paper embeddings and generating masked templates for scientific paper understanding. Textomics serves as the first benchmark for generating textual summaries for genomics data and we envision it will be broadly applied to other biomedical and natural language processing applications.

## introduction modern genomics research has become increasingly automated through being roughly divided into three sequential steps: next-generation sequencing technology produces a massive amount of genomics data, which are in turn processed by bioinformatics tools to identify key variants and genes, and, ultimately, analyzed by biologists to summarize the discovery @xcite . in contrast to the first two steps that have been automated by new technologies and software, the last step of summarizing discovery is still largely performed manually, substantially slowing down the progress of scientific discovery @xcite . a plausible solution is to automatically summarize the discovery from genomics data using neural text generation, which has been successfully applied to radiology report generation @xcite and clinical notes generation @xcite @xcite . in this paper, we study this novel task of generating sentences to summarize a genomics data matrix. several excisting approaches demonstrate encouraging results in generating short phrases to describe functions of a set of genes @xcite @xcite . however, our task is fundamentally different from these: the input of our task is a matrix that contains tens of thousands of genes, which could be noisier than a set of selected genes; the outputs of our task are sentences instead of short phrases or controlled vocabularies. to study this task, we curate a novel dataset, textomics, by integrating data from pmc, pubmed, and gene expression omnibus (geo) @xcite (figure 1 ). geo is the default database repository for researchers to upload their genomics data matrices, such as gene expression matrices and mutation matrices. each genomics data matrix in geo is a sample by feature matrices, where samples are from often humans or mice that are sequenced together to study a specific biological problem, and features are genes or variants. each matrix is also associated with a few sentences that are written by researchers to summarize this data matrix. after pre-processing, we obtain 22,273 matrix summary pairs, spanning 9 sequencing technology platforms. each matrix has on average 2,475 samples and 22,796 features. each summary has on average 46 words. we further propose a novel approach to automatically generate a summary from a genomics data matrix, which is highly noisy and high-dimensional. k nearest neighbor (knn) approaches have obtained great success in genomics data by capturing the hidden modules within it @xcite . the key idea of our method is to find k nearest summaries according to the genomics data similarity and then exploit the attention mechanism to convert these k nearest summaries to a new summary. our method obtained substantial improvement in comparison to baseline approaches. we further illustrated how we can generate a genomics data matrix from a given summary, offering the possibility to simulate genomics data from textual description. we then introduced how textomics can be used as a novel benchmark for measuring scientific paper similarity and evaluating scientific paper understanding. to the best of our knowledge, textomics and knn-vec2text together build up the first large-scale benchmark for genomics data summary generation, and can be broadly applied to a variety of natural language processing tasks. our paper is written as follows: we first introduce the textomics dataset (section 2) and describe the text2vec and vec2text tasks (section 3). we then propose a baseline model and knn-vec2text model for vec2text task (section 4.1) and the model for text2vec task. we then evaluate our method (section 5) and provide two applications (section 6) based on textomics dataset. we then discussed the related works and the potential direction of future works (section 7 and 8). ## textomics dataset we collected genomics data matrices from gene expression omnibus (geo) @xcite . the feature of each data matrix represents the expression level of a gene or other genomic measurements of a variant (typically real numbers). the sample of each matrix is an experimental subject, such as an experimental animal or a patient. each data matrix is associated with an expert-written summary, describing this data matrix. we obtained in total 164,667 matrix-summary pairs, spanning 12,219 sequencing platforms. samples in different platforms have different features. however, data matrices belonging to the same sequencing platform are from the same species and share the same set of features, thus can be used together for model training. to further alleviate the missing feature problem, we kept the top-20000 features with a lower missing rate and filtered out the rest. we further selected 9 platforms with the average lowest rate of missing value and the largest amount of matrix-summary pairs to guarantee the quality and the scale of the dataset. after all, we imputed the resulted data matrices using averaging imputation across different features. data matrices belonging to the same platform have distinct samples (e.g., patient samples collected from two hospitals). to make them com-parable and provide fixed-size features for machine learning models, we empirically used a fivenumber summary to represent each data matrix. in particular, we calculated the smallest, the first quartile, the median, the third quartile, and the largest value of each feature across samples in a specific data matrix. we then concatenated these values of all features, resulting in a 100k-dimensional feature vector for each data matrix. compared with other statistics such as mean, median, and mode of the features, the five number statistics maintain the patterns hidden in the raw matrices better. this vector will be finally used as the input to the machine learning model. all genomics data summaries we collected were written by the biologists who generate the corresponding genomics data matrices. therefore, these summaries can properly reflect biologists' descriptions of their datasets. since the summary is the first piece of information that one can learn about the dataset, authors often tend to clearly characterize their dataset in the summary. however, directly leveraging raw data of these summaries is questionable. on the syntactic level, the lengths of summary for each sample are different and comments are often used in genomics descriptions. in order to align our data and leverage the advanced transformer model that requires fix-length sentences as well as simplifies the structure of the summary, we empirically removed the text in the brackets and truncated the summaries length to 64 words (the percentage of summaries with a length greater than 64 is 41%). on the semantic level, there could be non-informative summaries such as a simple sentence 'please see our data below' and some outliers that are substantially different from other summaries. in order to increase the quality of these genomics data summaries, we manually inspected and removed the non-informative summary and excluded the outliers based on the pairwise bleu @xcite scores through a progressive automated procedure. specifically, for every summary, we treated it as the query text and calculated the pairwise bleu-1 scores with all other summaries, filtered out those median that is lower than 0.09, and then re-applied the procedure with a higher threshold of 0.13. finally, each of the 9 platforms contains 471 matrix-summary pairs on average, presenting a desirable number of training samples to develop data summary generation models. we summarized the statistics of these 9 platforms in supplementary table s1 . some of the data matrices are associated with a scientific paper, which describes how the authors generated and used the data. therefore, the data matrix and the summary can be used to help embed these papers. we additionally retrieved these papers from pubmed and pmc databases according to the paper titles enclosed in geo. we obtained the full text for those 7,691 freely accessible ones (supplementary table s1 ). we will introduce two applications that jointly use scientific papers and matrix-summary pairs in section 6. ## task description we aim to accelerate genomics discovery by generating a textual summary given the five-number summary-based vector of a genomics data matrix. we refer to the five-number summary-based vector as a gene feature vector for simplicity. ## applications 6.1 evaluate paper embedding via textomics embedding scientific papers is crucial to effectively identify emerging research topics and new knowledge from scientific literature. to this end, many machine learning models have been proposed to embed scientific papers into dense embeddings and then applied these embeddings for a variety of downstream applications @xcite @xcite @xcite . however, there is currently limited golden standard that can measure the similarity between two papers. as a result, existing approaches use surrogate metrics such as citation relationship, keywords, and user activities to evaluate their paper embeddings @xcite ., gpl 198 gpl 13534 gpl 6244 gpl 570 gpl 10558 gpl 6246 gpl 1261 gpl 6887 gpl 96 textomics platform 0.0 0.1 0.2 0.3 0.4 0.5 0.6 spearman correlation abstract introduction method result conclusion gpl 198 gpl 13534 gpl 6244 gpl 570 gpl 10558 gpl 6246 gpl 1261 gpl 6887 gpl 96 textomics platform 0.0 0.1 0.2 0.3 0.4 0.5 0.6 spearman correlation specter sentbert scibert biobert bert a b 2020; @xcite . textomics can be used to measure these paper embedding approaches by examining the consistency between the embedding-based paper similarity and the embedding-based summary similarity since both the paper and the summary are written by the same authors. in particular, for a pair of summaries d i , d @xmath0, let t i , t j be the text (e.g., abstracts) extracted from their corresponding scientific papers. let enc d be the encoder of the paper embedding method we want to evaluate. we first get their embeddings as: we then compute the pairwise euclidean distance between all pairs of summaries and all pairs of paper text as: to evaluate the quality of the encoder enc d , we can calculate the spearman correlation between the pairwise summary similarity and the pairwise text similarity. a larger spearman correlation means the summary / textual contents of two samples in the pair are better aligned with each other, which indicates this enc d is more accurate in embedding scientific papers. as a proof-of-concept, we obtained the full text of 7,691 papers in our dataset from the freely accessible pubmed central. we segmented each paper into five sections, which included abstract, introduction, method, result and conclusion. we first compared different paper embedding methods using the abstract of a paper. the five embedding methods we considered are introduced in section 5.1. since specter takes both the title and paragraph as the input we used the first sentence of the summary as a pseudo-title when encoding the summary. the results are summarized in figure 4a . we found that specter was substantially better than other methods on 8 out of the 9 platforms. specter is specifically developed to embed scientific papers by processing the title and the abstract separately, whereas other pre-trained language models simply concatenated the title and the abstract. the superior performance of specter suggests the importance of separately modeling paper title and abstract when embedding scientific papers. sentbert obtained the best performance among four pre-trained language models, partially due to its prominent performance in sentence-level embedding. we further noticed that the relative performance among different methods is largely consistent with the previous work evaluated on other metrics @xcite , demonstrating the high-quality of textomics. after observing the superior performance of specter, we next investigated which section of the paper can be best used to assess paper similarity. although existing paper embedding approaches often leverage the abstract for embedding, other sections, such as introduction and results might also be informative, especially for papers describing a specific dataset or method. we thus applied specter to embed five different sections of each scientific paper and used textomics to evaluate which section can best reflect paper similarity. we observed a consistent improvement of using the abstract section in comparison to other paper sections (figure 4b ), which is consistent with the intuition that the abstract represents a good summary of the scientific paper, again indicating the reliability of using textomics to evaluate paper embedding methods. ## related work our task is related to existing works that take structured data as the input and then generate the unstructured text. different input data modalities and related datasets have been considered in the literature, including text triplets in rdf graphs @xcite @xcite ), text-data tables @xcite @xcite @xcite , electronic medical records @xcite , radiology reports @xcite @xcite , and other continuous data modalities without explicit textual structures such as image @xcite @xcite , audio @xcite @xcite , and video @xcite @xcite . different from these structures, our dataset takes a high dimensional genomics feature matrix as input, which doesn't exhibit structure and is thus substantially different from other modalities. moreover, our dataset is the first dataset that aims to convert genomics feature vector to textual summary. the substantial noise and high-dimensionality of genomics data matrices further pose unique challenges in text generation. our knn-vec2text model is inspired by the recent success in applying knn-based language models to machine translation @xcite and language models @xcite @xcite . the main difference between our methods and their approaches is that while we try to leverage knn in the genomics vector space to construct reference text, they use knn in the text embedding space during the autoregressive generation process to help adjust the sample distribution. some other methods can be used to generate text from vectors, such as @xcite @xcite @xcite . their inputs are latent vectors that need to be inferred from the data and do not have specific meanings, which are different from our gene feature vectors. ## conclusion and future work in this paper, we have proposed a novel dataset textomics, containing 22,273 pairs of genomics matrices and their corresponding textual summaries. we then introduce a novel task of vec2text based on our dataset. this task aims to generate the textual summary based on the gene feature vector. to address this task, we propose a novel method knn-vec2text, which constructs the reference text using nearest neighbors in the gene feature vector space and then generates a new summary according to this reference text. we further introduce two applications that can be advanced using our dataset. one application aims at evaluating scientific paper similarity according to the similarity of its corresponding data summary, and the other application leverages our dataset to automatically generate masked sentences for scientific paper understanding. to the best of our knowledge, textomics and knn-vec2text serve as the first large-scale genomics data description benchmark, and we envision it will be broadly applied to other natural language processing and biomedical tasks. on the biomedical side, we provide the benchmark to develop new nlp tools that can generate the description for a genomics data. since each public genomics data needs a description, such tools will substantially accelerate this process. also, descriptions generated from textomics could contain new knowledge. while humans write the description almost solely based on that single dataset, de-scription generation models jointly consider thousands of datasets, enabling the transfer of knowledge from other datasets. the generated description can guide biologists to write more informative descriptions, which ultimately leads to better and larger genomics description data. when biologists start to obtain the generated description from nlp tools, they will be able to write more informative descriptions with the assistance from these nlp tools. on the nlp side, the relationship between a summary and a dataset is analogous to the relationship between an abstract and a scientific paper. a high-quality summary ideally contains all perspectives of a study, including problems, methods, and discoveries. moreover, our work will bridge the nlp and the genomics community and motivate people to analyze genomics data using nlp methods based on the multi-modality dataset introduced in this paper. textomics could also be used to help scientific paper analysis tasks, such as paper recommendation @xcite , citation text generation @xcite , and citation prediction @xcite . our method searches for the nearest neighbours by calculating the euclidean distance between fivenumber summary vectors of the genomics feature matrices. however, this might lose useful information hidden in the original matrices. it's challenging and worth exploring end-to-end approaches that can learn embeddings from the genomics feature matrices instead of representing them as fivenumber summary vectors. on the text2vec side, one remaining challenge that could be the future direction of our work is to directly generate the whole genomics feature matrix instead of the five-number summary vector. also, it would be interesting yet challenging to jointly learn the text2vec and the vec2text tasks, and one potential solution is to further decode the generated vector to reconstruct the embedding of summaries in text2vec, and leverage the resulted decoder to predict the embedding of text by using knn method in the text embedding space. also, it is interesting to jointly model data from multiple platforms, which might lead to beneficial results by transferring biological insights learned from different platforms.