A Case Study of NLG from Multimedia Data Sources: Generating Architectural Landmark Descriptions (2020)

Abstract: In this paper, we present a pipeline system that generates architectural landmark descriptions using textual, visual and structured data. The pipeline comprises five main components:(i) a textual analysis component, which extracts information from Wikipedia pages; (ii)a visual analysis component, which extracts information from copyright-free images; (iii) a retrieval component, which gathers relevant (property, subject, object) triples from DBpedia; (iv) a fusion component, which stores the contents from the different modalities in a Knowledge Base (KB) and resolves the conflicts that stem from using different sources of information; (v) an NLG component, which verbalises the resulting contents of the KB. We show that thanks to the addition of other modalities, we can make the verbalisation of DBpedia triples more relevant and/or inspirational.

## introduction the bulk of the information reaches the reader nowadays across different media. most of the videos uploaded to youtube come accompanied by written natural language comments and so do most of the audio podcasts; even (online) newspaper articles can be hardly imagined without any visual illustrative material. this means that to generate a comprehensive but, at the same time, not partially repetitive, content summary of the provided information, the input of all media needs to be taken into account and merged. the value of the merge of complementary information from multiple media for the generation of more informative texts has been pointed out already early in the field; cf., e.g., @xcite . however, since then only a few works tackled the problem; see, e.g., @xcite . a few others merge multimedia content in the context of other tasks such as retrieval; cf. @xcite . in most of these works, the integration is done using similarity measures in a multimedia vector space. to the best of our knowledge, none aims for integration (or fusion) at, and subsequent generation from, the level of @xmath0, use inheritance or apply advanced reasoning mechanisms. in this paper, we present a work that addresses this challenge. it fuses triples from dbpedia, textual information from wikipedia, and visual information obtained from images as input to a pipelinebased generator. our work is situated in the context of a larger research initiative, in which the objective is to automatically reconstruct architectural landmarks in 3d, such that they can be used by architects, game designers, journalists, etc. each reconstructed landmark should be accompanied by automatically generated information that describes its main features. the goal is to convey information such as the landmark's architect, some of its facade elements, its date of construction and/or renovation, its architectural style, etc. in terms of a text like: petronas towers, which césar pelli designed, are commercial offices and a tourist attraction in kuala lumpur. the building has 88 floors and 40 elevators and a floor area of 395,000 m 2 . it was the highest building in the world between 1998 and 2004, and was restored in 2001. which is a verbalisation of the triples in in what follows, we describe how this is done, from content extraction to the use of a grammarbased text generator. we thus do not focus exclusively on text generation. rather, we attempt to show how richer input structures can be created using different media for the benefit of more com2 related work as already pointed out above, to the best of our knowledge, only a few works deal with fusion of content from different media as input representation for downstream applications (in our case, text generation) and when they do, they use similarity measures in a multimedia vector space @xcite @xcite rather than mapping multimedia content onto a common ontology. this does not mean though that research related to text generation across different media would be neglected. for instance, generation of image captions @xcite and video descriptions @xcite has recently become a very popular research topic. all of the proposals in this area use sequence-tosequence neural network models. in @xcite , indexes of textual and visual features are integrated via a multi-modality ontology, which is further enriched by dbpedia triples for the purpose of semantics-driven image retrieval. on the other side, text generation from ontological structures is on the rise; cf., e.g., @xcite for overviews and the webnlg challenge @xcite for state-of-the-art works. in general, there are three main approaches to generating texts from ontologies: (i) filling slot values in predefined sentence templates @xcite , (ii) applying grammars that encode different types of linguistic knowledge @xcite @xcite , and (iii) predicting the most appropriate output based on machine learning models @xcite . template-based generators are very robust, but also limited in terms of portability since new templates need to be defined for ev-ery new domain, style, language, etc. machine learning-based generators have the best coverage, but the relevance and the quality of the produced texts cannot be ensured. furthermore, they are fully dependent on the available (still scarce and mostly monolingual) training data. the development of grammar-based generators is time-consuming and they usually have coverage issues. however, they do not require training material, allow for a greater control over the outputs (e.g., for mitigating errors or tuning the output to a desired style), and the linguistic knowledge used for one domain or language can be reused for other domains and languages. a number of systems also combine (i) and (iii), filling the slot values of pre-existing templates using neural network techniques @xcite . in what follows, we opt for a grammar-based generator. we show that information from visual, textual and structured (dbpedia) sources can be successfully fused in order to generate informative descriptions using a pipeline-based text generator. ## system and dataset overview let us first introduce the architecture of our system and then outline the creation of the datasets used for development and testing. ## multimedia content acquisition in this section, we describe the content that we extracted from the three sources (dbpedia, images and texts) used for generating the landmark descriptions, and how we extracted it. ## multimedia information fusion the results of the visual and textual analyses and the retrieved dbpedia properties are mapped using the web annotation data model foot_4 . the model creates a body and a target for each annotation. as main interconnection point between the content from different media, we use the name of the corresponding entity, which is mapped as the target of the annotation. the body of the annotation contains all the other information, which varies according to the nature of each input module. more specifically: (i) for the visual analysis content, the body contains information pertinent to scene, objects, fac ¸ade, structure elements and architectural styles. a mapping example is shown in figure 4; (ii) for the wikipedia analysis outcome, the body contains creator and localisation information related to the entity; (iii) for the retrieved dbpedia triples, the information in the body is pertinent to the landmark, the architecture, the location and more general information about the landmark. a reasoning mechanism applies the following property-based semantic rules at the time of the retrieval of dbpedia triples: (i) extraction of class information about creators and locations: the mechanism detects whether a creator is a person, an organisation or a company, and whether a location is the same rule is applied for literals. (iii) filtering of undesired values: for instance 'buildingtype' cannot contain values such as 'cultural', 'style' cannot contain an affirmation of type "yes", etc. property text dbpedia visual building type localisation hypernyms and buildingtypes scene recognition features creator creator creator -architectural style -style architectural style ## evaluation we evaluate the quality of the generated descriptions from fused representations, first of all, against monomodal descriptions generated solely from dbpedia triples. the goal is to assess to what extent architectural landmark descriptions benefit from additional content from other media. a comparison with wikipedia texts is also carried out. ## conclusions we presented the case of the generation of architectural landmark descriptions from ontological structures that contain fused content from visual, textual and ontological sources. the evaluation showed that when compared to descriptions generated from the dbpedia rdf-triples obtained from textual material only (i.e., wikipedia), descriptions that communicate fused content are considered more interesting and better in terms of textual quality. however, also due to the limited content features that were considered in the experiments, these descriptions cannot compete, in general, with more comprehensive well-written descriptions as encountered in wikipedia. still, it needs to be taken account that by far not all architectural landmarks that are of interest from the professional or cultural viewpoint are covered by wikipedia. fused content descriptions are then a welcomed solution. with the 98 properties from dbpedia. in parentheses, the number of times each property had one or more value(s) for a building. there can be two reasons why there are more than one value for a feature: (i) one property is given more than one value, or/and (ii) multiple properties have one value. alley alcove amphitheater amusement park apartment building outdoor aqueduct arch archaeological excavation atrium public attic auditorium balcony exterior balcony interior ball bar barn barrier-curb barrier-fence barrier-guard-rail barrier-wall bathroom bazaar indoor bazaar outdoor beach house bedroom berth bow window indoor box building facade cafeteria campus car case castle catacomb cemetery chalet chest of drawers children room church indoor church outdoor classroom cloister computerroom concert hall cornice corridor cottage courthouse courtyard crosswalk deco department store dining hall dining room discotheque doorway outdoor downtown driveway eiffel-tower elevator embassy engine room entrance hall escalator indoor excavation fabric store fac ¸ade/wall farm fire escape fire station fireplace flat-bike-lane flat-crosswalk-plain flat-curb-cut flat-parking flat-pedestrian-area flat-rail-track flat-road flat-sidewalk food court formal garden gameroom garage indoor garage outdoor gazebo exterior general store indoor general store outdoor golden-gate-bridge greenhouse indoor greenhouse outdoor gymnasium indoor home office home theater hospital hotel outdoor hotel room house hunting lodge outdoor igloo indoor industrial area inn outdoor kasbah kindergarden classroom kitchen library indoor library outdoor living room lighthouse mansion lobby mausoleum manufactured home market indoor market outdoor meeting room mirror mosque outdoor motel movie theater indoor oast house museum indoor museum outdoor nursery office office building palace pagoda painting pantry park person parking lot patio pavilion pier playground pool/inside pub indoor pyramid restaurant restaurant kitchen restaurant patio river rock arch rope bridge ruin schoolhouse sculpture shed shopfront shopping mall indoor sill ski resort skyscraper smokestack stable stained-glass staircase structure-bridge structure-building structure-tunnel swimming pool indoor swimming pool outdoor synagogue outdoor temple asia throne room tower tower-pisa train station platform tree house village water tower waterfall wind farm windmill window youth hostel zen garden