Sequence-to-Set Semantic Tagging for Complex Query Reformulation and Automated Text Categorization in Biomedical IR using Self-Attention (2020)

Abstract: Novel contexts, comprising a set of terms referring to one or more concepts, may often arise in complex querying scenarios such as in evidence-based medicine (EBM) involving biomedical literature. These may not explicitly refer to entities or canonical concept forms occurring in a fact-based knowledge source, e.g. the UMLS ontology. Moreover, hidden associations between related concepts meaningful in the current context, may not exist within a single document, but across documents in the collection. Predicting semantic concept tags of documents can therefore serve to associate documents related in unseen contexts, or categorize them, in information filtering or retrieval scenarios. Thus, inspired by the success of sequence-to-sequence neural models, we develop a novel sequence-to-set framework with attention, for learning document representations in a unique unsupervised setting, using no human-annotated document labels or external knowledge resources and only corpus-derived term statistics to drive the training, that can effect term transfer within a corpus for semantically tagging a large collection of documents. Our sequence-to-set modeling approach to predict semantic tags, gives to the best of our knowledge, the state-of-the-art for both, an unsupervised query expansion (QE) task for the TREC CDS 2016 challenge dataset when evaluated on an Okapi BM25–based document retrieval system; and also over the MLTM system baseline baseline (Soleimani and Miller, 2016), for both supervised and semi-supervised multi-label prediction tasks on the del.icio.us and Ohsumed datasets. We make our code and data publicly available.

## introduction recent times have seen an upsurge in efforts towards personalized medicine where clinicians tai-1 https://github.com/mcoqzeug/seq2set-semantic-tagging lor their medical decisions to the individual patient, based on the patient's genetic information, other molecular analysis, and the patient's preference. this often requires them to combine clinical experience with evidence from scientific research, such as that available from biomedical literature, in a process known as evidence-based medicine (ebm). finding the most relevant recent research however, is challenging not only due to the volume and the pace at which new research is being published, but also due to the complex nature of the information need, arising for example, out of a clinical note which may be used as a query. this calls for better automated methods for natural language understanding (nlu), e.g. to derive a set of key terms or related concepts helpful in appropriately transforming a complex query, by reformulation so as to be able to handle and possibly resolve medical jargon, lesser-used acronyms, misspelling, multiple subject areas and often multiple references to the same entity or concept, and retrieve the most related, yet most comprehensive set of useful results. at the same time, tremendous strides have been made by recent neural machine learning models in reasoning with texts on a wide variety of nlp tasks. in particular, sequence-to-sequence (seq2seq) neural models often employing attention mechanisms, have been largely successful in delivering the state-of-the-art for tasks such as machine translation @xcite , @xcite , handwriting synthesis @xcite , image captioning @xcite , speech recognition @xcite and document summarization @xcite . inspired by these successes, we aimed to harness the power of sequential encoder-decoder architectures with attention, to train end-to-end differentiable models that are able to learn the best possible representation of input documents in a collection while being predictive of a set of key terms that best describe the docu-ment. these will be later used to transfer a relevant but diverse set of key terms from the most related documents, for "semantic tagging" of the original input documents so as to aid in downstream query refinement for ir by pseudo-relevance feedback @xcite . to this end and to the best of our knowledge, we are the first to employ a novel, completely unsupervised end-to-end neural attention-based document representation learning approach, using no external labels, in order to achieve the most meaningful term transfer between related documents, i.e. semantic tagging of documents, in a "pseudorelevance feedback"-based @xcite setting for unsupervised query expansion. this may also be seen as a method of document expansion as a means for obtaining query refinement terms for downstream ir. the following sections give an account of our specific architectural considerations in achieving an end-to-end neural framework for semantic tagging of documents using their representations, and a discussion of the results obtained from this approach. ## related work pseudo-relevance feedback (prf), a local context analysis method for automatic query expansion (qe), is extensively studied in information retrieval (ir) research as a means of addressing the word mismatch between queries and documents. it adjusts a query relative to the documents that initially appear to match it, with the main assumption that the top-ranked documents in the first retrieval result contain many useful terms that can help discriminate relevant documents from irrelevant ones @xcite , @xcite . it is motivated by relevance feedback (rf), a well-known ir technique that modifies a query based on the relevance judgments of the retrieved documents @xcite . it typically adds common terms from the relevant documents to a query and re-weights the expanded query based on term frequencies in the relevant documents relative to the non-relevant ones. thus in prf we find an initial set of most relevant documents, then assuming that the top k ranked documents are relevant, rf is done as before, without manual interaction by the user. the added terms are, therefore, common terms from the top-ranked documents. to this end, @xcite employ term classification for retrieval effectiveness, in a "supervised" setting, to select most relevant terms. @xcite employ a deep sentence embedding approach using lstms and show improvement over standard sentence embedding methods, but as a means for directly deriving encodings of queries and documents for use in ir, and not as a method for qe by prf. in another approach, @xcite train autoencoder representations of queries and documents to enrich the feature space for learningto-rank, and show gains in retrieval performance over pre-trained rankers. but this is a fully supervised setup where the queries are seen at train time. @xcite also use an autoencoder-based approach for actual query refinement in pharmacogenomic document retrieval. however here too, their document ranking model uses the encoding of the query and the document for training the ranker, hence the queries are not unseen with respect to the document during training. they mention that their work can be improved upon by the use of seq2seqbased approaches. in this sense, i.e. with respect to qe by prf and learning a sequential document representation for document ranking, our work is most similar to @xcite . however the queries are completely unseen in our case and we use only the documents in the corpus, to train our neural document language models from scratch in a completely unsupervised way. classic sequence-to-sequence models like @xcite demonstrate the strength of recurrent models such as the lstm in capturing short and long range dependencies in learning effective encodings for the end task. works such as @xcite , @xcite , @xcite , further stress the key role that attention, and multi-headed attention @xcite can play in solving the end task. we use these insights in our work. according to the detailed report provided for this dataset and task in @xcite all of the systems described perform direct query reweighting aside from supervised term expansion and are highly tuned to the clinical queris in this dataset. in a related medical ir challenge @xcite the authors specifically mention that with only six partially annotated queries for system development, it is likely that systems were either under-or over-tuned on these queries. since the setup of the seq2set framework is an attempt to model the prf based query expansion method of its closest related work @xcite where the effort is also to train a neural generalized language model for unsupervised semantic tagging, we choose this system as the benchmark to compare against to our end-to-end approach for the same task. ## methodology drawing on sequence-to-sequence modeling approaches for text classification, e.g. textual entailment @xcite and machine translation @xcite , @xcite we adapt from these settings into a sequenceto-set framework, for learning representations of input documents, in order to derive a meaningful set of terms, or semantic tags drawn from a closely related set of documents, that expand the original documents. these document expansion terms are then used downstream for query reformulation via prf, for unseen queries. we employ an endto-end framework for unsupervised representation learning of documents using tfidf-based pseudolabels (figure 1(a) )and a separate cosine similaritybased ranking module for semantic tag inference (figure 1(b) ). we employ various methods such as doc2vec, deep averaging, sequential models such as lstm, gru, bigru, bilstm, bilstm with attention and self-attention, detailed in figure 1(c )-(f), see appendix a, for learning fixed-length input document representations in our framework. we apply methods like dan @xcite , lstm, and bilstm as our baselines and formulate attentional models including a self-attentional transformerbased one @xcite as our proposed augmented document encoders. further, we hypothesize that a sequential, bidirectional or attentional encoder coupled with a decoder, i.e. a sigmoid or softmax prediction layer, that conditions on the encoder output v (similar to an approach by @xcite for learning a neural probabilistic language model), would enable learning of the optimal semantic tags in our unsupervised query expansion setting, while modeling directly for this task in an end-to-end neural framework. in our setup the decoder predicts a meaningful set of concept tags that best describe a document according to the training objective. the following sections describe our setup. ## conclusion we develop a novel sequence-to-set end-toend encoder-decoder-based neural framework for multi-label prediction, by training document representations using no external supervision labels, for pseudo-relevance feedback-based unsupervised semantic tagging of a large collection of documents. we find that in this unsupervised task setting of prf-based semantic tagging for query expansion, a multi-term prediction training objective that jointly optimizes both prediction of the tfidf-based document pseudo-labels and the log likelihood of the labels given the document encoding, surpasses previous methods such as phrase2vecglm @xcite that used neural generalized language models for the same. our initial hypothesis that bidirectional or self-attentional models could learn the most efficient semantic representations of documents when coupled with a loss more effective than cross-entropy at reducing language model perplexity of document encodings, is corroborated in all experimental setups. we demonstrate the effectiveness of our novel framework in every task setting, viz. for unsupervised qe via prf-based semantic tagging for a downstream medical ir challenge task; as well as for both, supervised and semi-supervised task settings, where seq2set statistically significantly outperforms the state-of-art mltm baseline @xcite on the same held out set of documents as mltm, for multi-label prediction on a set of known labels, for automated text categorization; achieving to the best of our knowledge, the current state-of-the-art for multi-label prediction on documents, with or without known labels. we therefore demonstrate the effectiveness of our sequence-to-set framework for multi-label prediction, on any set of documents, applicable especially towards the automated categorization, filtering and semantic tagging for qebased retrieval, of biomedical literature for ebm. future directions would involve experiments replacing tdidf labels with more meaningful terms (using unsupervised term extraction) for query expansion, initialization with pre-trained embeddings for the biomedical domain such as bluebert @xcite and biobert @xcite , and multi-task learning with closely related tasks such as biomedical named entity recognition and relation extraction to learn better document representations and thus more meaningful semantic tags of documents useful for downstream ebm tasks.