A New Neural Search and Insights Platform for Navigating and Organizing AI Research (2020)

Abstract: To provide AI researchers with modern tools for dealing with the explosive growth of the research literature in their field, we introduce a new platform, AI Research Navigator, that combines classical keyword search with neural retrieval to discover and organize relevant literature. The system provides search at multiple levels of textual granularity, from sentences to aggregations across documents, both in natural language and through navigation in a domain specific Knowledge Graph. We give an overview of the overall architecture of the system and of the components for document analysis, question answering, search, analytics, expert search, and recommendations.

## introduction the growth of publications in ai has been explosive in recent years. a big portion of this growth is happening on platforms outside of traditional publishing venues, for instance arxiv e-print archive (see figure 1 ) and blogs. although this encourages broad access to ai expertise and technology, it makes efficient and effective search, monitoring, and discovery in the ai field increasingly difficult. most general-purpose academic search engines lack a specialization on ai content and practical know-how, because they focus on classical bibliographic information across all scientific disciplines. at the same time, academic search engines often do not make use of the latest ai technologies in search, as well as natural language processing (nlp) and insights capabilities. the main reason that limits them is the need to operate at a much larger scale and cover a large amount of knowledge. recent developments in various nlp tasks are showing fast progress towards an almost human level of language understanding @xcite 2019; @xcite . applying these new technologies to the processing of research and engineering literature bears the promise of accelerating scientific discovery. in addition, providing efficient tools to automate some of the drudgery of human scholarly work by machine understanding of scientific knowledge is extremely valuable. similar directions are being explored in recent studies @xcite @xcite . our system, ai research navigator † , aims to help ai researchers with a simple-to-use semantic search for documents ( §4.1), the answering of detailed factual questions ( §4.3), the generation of insights via visual analytics ( §4.4), combined with recommendations to filter the constant flood of new information in their field ( §4.5). these technologies are combined in the platform with both project and task-oriented tools to support a more effective and efficient organization of a researcher's work on multiple projects and topics ( §5). this paper presents a short outline of the system. ## document analysis a key ingredient to an ai insights platform is the content available to users. we ingest, process, and store documents from a variety of sources, aiming to get broad coverage, as well as detailed views on theoretical and applied ai. at the moment our platform contains approximately 140 thousand scientific papers, collected from arxiv.org * and openreview.net, and about 24 thousand posts from data science blogs. our goal for the near future is to expand this set of sources to include different types of content (e.g., source code, news, tweets). we ingest new content from our sources on a daily basis, offering our users the latest insights. newly ingested or updated content is fed to our back-end storage system (section 3) via a distributed processing pipeline that takes the documents through a number of processing and information extraction steps and generates embeddings that capture the intent and meaning of the text. we also extract images from each document to serve as an illustration for the paper in the search engine. we manage the state of each document, as it traverses the pipeline, with a messaging queue platform (apache pulsar). this allows us to scale our processing throughput, while keeping certain processing guarantees. ## storage systems to provide access to the information we obtained from our processing pipeline, we currently store our information in three core systems. ## accessing information processing and storing information is only useful when we can provide meaningful access to it. our platform allows information access in a variety of ways. in this section, we discuss content ( §4.1) and expert ( §4.2) search, qa system ( §4.3), analytics component ( §4.4), and recommendations system ( §4.5). ## productivity tools discovering relevant information in an effective way is key to researchers, knowledge workers, and decision makers. even though an ai-enabled platform like the one described in this paper can be helpful for this purpose, it is only the first step in researching a topic. organizing and accessing this information is a necessary feature. users of our platform are also supported to organize information and knowledge without having to rely on external tools for reading lists and notes. having found a relevant piece of information in our system, users can save this into their own specific project and topic tags. they can also directly write their notes on the papers and projects they are working on in the tool. tags serve to organize lists of documents, as well as the queries used, and notes taken while working on a project (see figure 6 ). the tagging system can also be used to track the status and priority of work. tag-based lists can easily be shared with others within the platform, on social media, and exported into other tools. as described above, these tags are also the starting point that allows users to be alerted about new results relevant to their interests. ## discussion and next steps having introduced a new platform to discover and organize knowledge for ai researchers, we foresee considerable future research to reach real machine understanding of scientific literature, such as extraction of complex entity relations and more advanced use of neural embeddings to reduce the dependency on manual kg curation. we leverage a mix of state-of-the-art ai components to give researchers transparent access to a body of knowledge from a large volume of heterogeneous and non-reviewed content. as a result, it raises the concern of dealing with fairness, factuality, conflicting opinions, and out-of-date information, which requires deeper investigation. finally, we are interested to further explore how the productivity tools in our platform can contribute to better collaboration in teams and improving knowledge sharing and discovery.