T 2 KG : Transforming Multimodal Document to Knowledge Graph (2023)

Abstract: The large amount of information in digital format that exists today makes it unfeasible to use manual means to acquire the knowledge contained in these documents. Therefore, it is necessary to develop tools that allow us to incorporate this knowledge into a structure that is easy to use by both machines and humans. This paper presents a system that can incorporate the relevant information from a document in any format, structured or unstructured, into a semantic network that represents the existing knowledge in the document. The system independently processes from structured documents based on its annotation scheme to unstructured documents, written in natural language, for which it uses a set of sensors that identifies the relevant information and subsequently incorporates it to enrich the semantic network that is created by linking all the information based on the knowledge discovered.

## introduction nowadays, the amount of information available on the web is available in multiple formats. leveraging this data requires the design of software systems that can exploit the information, obtain relevant data, structure it in a specific format, and generate reports that help to evaluate this information. software systems focused on performing all these actions are currently oriented to apply different natural language processing techniques. many early developments were domain-dependent, so domainspecific resources, although costly in terms of time and expertise, were relatively easy to obtain. but recently, general-purpose, domain-independent systems are being developed. however, it would be difficult to imagine a system like chatgpt incorporating a multi-domain ontology in real time. the trend in natural language processing (nlp) today is text-to-text development that does not use manually curated semantic resources such as semantic networks. in other words, text-to-text oriented systems use self-generated resources without the need for external semantic resources. however, systems must take into account that the software created must be maintainable and extensible, using processes and methodologies that make all these aspects possible. this work aims to present a framework capable of extracting knowledge from heterogeneous sources, structured such as comma-separated volumes or relational databases, or unstructured such as plain text from wikipedia. knowledge from different sources is integrated into an ontology. it also allows the user to query the knowledge in natural language while results are analyzed automatically to generate custom graphics or visualizations to ease its interpretation. ## state of the art knowledge representation is the process of modeling information in a way that enables effective reasoning, communication, and decision-making by computers. given the increasing amount of digital data available, it has become more important than ever to conceive ways to represent it in a meaningful way to add knowledge to nlp systems. this knowledge has been used to improve the accuracy of tasks such as sentiment analysis, named entity recognition, and text classification @xcite . two main tasks are involved in this process: knowledge extraction and knowledge integration from different sources. for knowledge extraction, it's necessary the development of sensors to extract pieces of relevant information (e.g. entities and relations) from unstructured documents. knowledge integration needs to deal with linking entities, modeling uncertainty, or solving inconsistencies. one of the challenges of multimodal representation is integrating information from different sources in a meaningful way. this requires the development of novel algorithms and techniques that can effectively capture the relationships between different types of data. several approaches have been proposed for integrating text and image data, including deep neural networks for image and text @xcite . in recent years, there has been significant progress in the field of knowledge representation from unstructured textual data, for example, processing scientific articles. scientific articles contain a wealth of information, including structured data such as references and citations, as well as unstructured data such as text and figures. by representing this information in a structured way, it is possible to create a comprehensive knowledge graph that captures the relationships between different concepts and entities. one common approach is to use natural language processing techniques to extract structured data from the text of scientific articles @xcite . for example, named entity recognition can be used to identify entities such as proteins, genes, and diseases in the text, while relationship extraction can be used to identify the relationships between these entities. another approach is to use machine learning techniques to learn representations of entities and relationships in a knowledge graph directly from the text and image data @xcite . also, it is possible to use image processing techniques to extract information from figures and tables @xcite . for example, optical character recognition (ocr) can be used to extract text from figures, while computer vision techniques can be used to identify patterns and relationships in the data. there are different techniques for knowledge extraction: rule-based approaches can be effective in extracting structured information from scientific articles, but they are limited by the difficulty of designing rules that capture all the relevant knowledge or that apply to other domains @xcite . knowledge graph construction involves the creation of a structured representation. a knowledge graph consists of a set of entities representing objects or concepts and a set of relationships between them. there are different techniques for constructing knowledge graphs: multimodal knowledge extraction and representation have promising applications in healthcare and biomedicine. by representing medical data in a structured way, it is possible to create a more comprehensive understanding of diseases and to develop more effective treatments. for example, knowledge graphs have been used to identify new drug targets for diseases @xcite . in healthcare, knowledge representation techniques are being used to extract valuable insights from electronic health records (ehrs). ehrs contain a wealth of information, including patient demographics, diagnoses, and treatments. by applying knowledge representation techniques to ehrs, researchers can extract valuable insights into disease risk factors, treatment efficacy, and patient outcomes @xcite . in biomedicine, knowledge representation techniques are being used to extract knowledge from large volumes of scientific literature. the aim is to create a comprehensive biomedical knowledge graph that can be used to facilitate drug discovery, disease diagnosis and personalised medicine. knowledge graphs constructed from the biomedical literature can thus capture complex relationships between genes, proteins and diseases, which can be used to identify potential drug targets. (yuan, 2020). ## t2kg framework we present t2gk, a framework for managing (i.e., extracting, storing and retrieving) knowledge from heterogeneous sources. section 3.1 describes how align structured and unstructured data into an unified schema. next, the data mining process is described in the section 3.2. subsequently, in section 3.3 the extracted pieces of information are integrated into the knowledge graph. finally, the section 3.4 presents the retrieval and visualisation of the data, as well as the evaluation of the platform.