A Multi-Modal Knowledge Graph for Classical C hinese Poetry (2022)

Abstract: Classical Chinese poetry has a long history and is a precious cultural heritage of humankind. Displaying the classical Chinese poetry in a visual way, helps to cross cultural barriers in different countries, making it enjoyable for all the people. In this paper, we construct a multi-modal knowledge graph for classical Chinese poetry (PKG), in which the visual information of words in the poetry are incorporated. Then a multi-modal pre-training language model, PKG-Bert, is proposed to obtain the poetry representation with visual information, which bridges the semantic gap between different modalities. PKG-Bert achieves the state-of-the-art performance on the poetry-image retrieval task, showing the effectiveness of incorporating the multi-modal knowledge. The large-scale multi-modal knowledge graph of classical Chinese poetry will be released to promote the researches in classical Chinese culture area.

## introduction classical chinese poetry plays an important role in cultural transmission in ancient china, and helps the cultural understanding between the east and the west. over thousands of years, the language of classical chinese poetry is greatly different from the modern language, such as the semantic meaning of words and the organization of sentences. for example, the word "wulai" (无赖) in modern chinese only means rascally, but in ancient chinese, it also means cute, e.g."i like my cute son most. he is lying in the grass beside the stream, peeling lotus pods" (最喜小儿无赖，溪头卧剥莲蓬) , which increases the difficulty to understand the language in classical chinese poetry. to bridge the culture gap, we propose to display the classical chinese poetry in an intuitively visual way, which makes it easier to enjoy the beauty of classical chinese poetry for all the people. existing studies in classical chinese poetry area @xcite @xcite @xcite mainly focus on the generation and analyzing of poetry. different from them, we make a preliminary attempt to translate the classical chinese poetry into a multi-modal way to help overcome the cultural barriers among different languages, and make it enjoyable for all the people. to bridge the semantic gap between two modalities @xcite , we construct a multimodal knowledge graph for classical chinese poetry (pkg), in which a wealth of information, such as the allusions and visual information of the words in poetry are incorporated. specifically, pkg consists of text nodes and image nodes. each text node is linked to the most similar images by computing the similarity of their representations, which are initialized by pre-training model bert @xcite and brivl @xcite respectively, and then fine-tuned in our proposed multi-modal pre-training language model pkg-bert. with the incorporated image information in pkg, pkg-bert injects the visual knowledge into the semantic learning of poetry words, making it possible to bridge the semantic gap between different modalities. the effectiveness of our method is verified on the poetry-image retrieval task. we invite five experts in classical chinese poetry area to screen out the most matching poetry (or image) corresponding to each image (or poetry) query, and then construct a evaluation dataset for classical chinese poetry-image retrieval task. the contributions of our work are as follows: ## related work in the field of classical chinese poetry, there are some works on the knowledge graph of poetry @xcite @xcite . liu et al. @xcite studies the new word detection in ancient chinese corpus, and use a semi-automated method to construct a classical chinese poetry knowledge graph (ccp-kg) @xcite . they use bert and gat to encode the nodes in the text and knowledge graph respectively to classify the sentiment and theme of classical chinese poetry. knowpoetry @xcite semi-automatically constructs a knowledge graph of tang poetry and poet, and uses inference rules from experts to help the further study of tang poetry e.g., mining social relationship between poets. 2 the retrieval system is available at http://facdbe.com:8080/ there are also some works on the multi-modal knowledge graph in mordern language @xcite . mmkg @xcite takes three existing knowledge graphs as the blueprint of the textual part, and uses the search engines to find images highly related with each text node. richpedia @xcite obtains texts and images entities from the wikipedia and search engines, and uses predefined rules to extract relations between text and image entities. however, due to the lack of data, existing studies are hard to apply in the field of classical chinese poetry. for example, it is difficult for the mainstream search engines to recognize ancient chinese words, and there is no such large-scale knowledge base like wikipedia in the field of classical chinese poetry. ## methods the overall architecture of the multi-modal knowledge graph based pre-training model pkg-bert is shown in fig. 1 . ## conclusions we construct a multi-modal knowledge graph for classical chinese poetry (pkg), and propose a multi-modal pre-training language model pkg-bert. it incorporates the visual information into the semantic learning of classical chinese poetry, which helps to promote the researches in classical chinese language understanding. the poetryimage retrieval application also helps to cross the cultural barriers in different countries, making it enjoyable for all the people.