The Aligned Multimodal Movie Treebank: An audio, video, dependency-parse treebank (2022)

Abstract: Treebanks have traditionally included only text and were derived from written sources such as newspapers or the web. We introduce the Aligned Multimodal Movie Treebank (AMMT), an English language treebank derived from dialog in Hollywood movies which includes transcriptions of the audio-visual streams with word-level alignment, as well as part of speech tags and dependency parses in the Universal Dependencies formalism. AMMT consists of 31,264 sentences and 218,090 words, that will amount to the 3rd largest UD English treebank and the only multimodal treebank in UD. To help with the web-based annotation effort, we also introduce the Efficient Audio Alignment Annotator (EAAA), a companion tool that enables annotators to significantly speed-up their annotation processes.

## introduction treebanks are fundamental resources in natural language processing @xcite . despite their central role, most existing treebanks are derived from single-modality texts such as newspapers, blogs, and other online communities. the vocabulary, syntax, and statistics of spoken and written language can be quite different from one another @xcite . to complement these datasets and aid the advent of multimodal conversational agents, we have created a new dataset, the aligned multimodal movie treebank, ammt, the content of which is derived from language spoken in hollywood movies. ammt is released publicly under an open source license and will be contributed to the universal dependencies (ud) @xcite treebanks. speech based treebanks have proven to be a resource of enormous importance to the nlp research community @xcite . we find treebank-3 of the penn treebank @xcite , which includes the penn treebank switchboard corpus @xcite , to be the closest existing dataset to ammt. this corpus contains nearly one million transcribed words from switchboard annotated with part of speech tags, dysfluencies, and parse trees, and it also in-figure 1 : an overview of ammt, our novel multimodal dataset, consisting of transcriptions and parses for 21 movies aligned at the millisecond level. eaaa is a new transcription and alignment tool introduced below. our contributions are: 1. ammt is the first large-scale treebank to include alignment to both audio and video. 2. ammt includes fine-grained millisecond-level word boundaries. 3. ammt is parsed in the ud framework and is the 3rd largest english ud treebank. 4. a new tool, efficient audio alignment annotator (eaaa), for rapid word boundary annotation in large corpora. ## dataset the ammt dataset is an english language treebank based on 21 hollywood movies that provides transcriptions with word-level alignment to the audio-visual stream, as well as part of speech tags and dependency parses in the ud formalism. annotations for speaker identification will be included at the time of release. due to copyrighted source material, ammt provides multiple 1-second-long audio-visual sample clips from every movie, and a tool chain allowing users to obtain their own copies and verify alignment with the dataset. ammt consists of 31,264 sentences, 218,090 words, 8,541 lemmas and 10,805 unique tokens. the counts of pos tags and dependencies are shown in appendix a. the 21 movies from which the dataset is derived are listed in movies were chosen to be appropriate for many ages, with the highest rating being pg-13. they belong to a variety of movie genres (including action, adventure, animation, comedy, drama, fantasy, fam-ily, and sci-fi, according to imdb's categorization), and their release dates range from 1995 to present. they were selected to have verbose scripts, in the top 50% of randomly sampled movies. movies which included extensive singing such as musicals were omitted. copies of the movies were obtained and extracted in full including opening and closing credits. special features and after-credits scenes were omitted. ## multimodal feature analysis exploring the utility of the corpus as a multimodal resource for grounded language and vision tasks, we quantified the co-occurrence of nouns and their considering noun-object agreements across both object classes and movie types reveals variable distributions. some nouns are highly likely to appear on screen as their corresponding noun is uttered, like person (94.4%), types of vehicles (car: 59.7%, bicycle: 68.3%) and animals (giraffe: 100%, cow: 100%), while others have not co-occurred once despite being uttered multiple times. moreover, unambiguous nouns (e.g. laptop: 50%, tv: 42.8%, toilet: 33.3%) tend to have a significantly higher agreement rate scores than words with multiple pos (e.g. bear: 2.5%, orange: 0%, remote: 0%). some movie categories are also more likely to have high noun-object agreement, such as movies aimed for a younger audience (educational and animation genres), perhaps to enable language learning through multimodality. for example cars-2 and sesame street present 79.2% and 74.3% agreement rate respectively, while the lord of the rings 1 and 2, and avengers infinity war score only 17.6%, 14.2% and 5.9% respectively; see fig. 6 . figure 4 : coco classes noun-object agreement across the corpus (sorted by agreement rate). all nouns corresponding to one of the 80 coco classes (orange) vs their corresponding objects in the video during the noun utterance (blue). objects were manually detected in the middle frame of a word utterance. ## tools to efficiently annotate the alignment between word onsets and offsets and the audio stream, we created a new tool, the efficient audio alignment annotator (eaaa). eaaa enables annotators to start with a rough transcript and approximate alignment between words and the audio track. annotators can simultaneously correct the transcript while annotating new words. an overview of the eaaa interface is shown in fig. 2 . tools such as praat @xcite also allow for annotating audio corpora with word boundaries. unlike praat, eaaa is web-based making it easier for annotators to use. data such as spectrograms and wave files seen by annotators is pre-processed on the server-side, making browsing and accessing movies with eaaa near real-time. since eaaa is a single-purpose tool meant for transcription and fine-grained alignment, it provides custom features which significantly speed up the annotation process like keyboard shortcuts, the ability to handle audio files of any length, and a streamlined interface. eaaa also handles multiple concurrent annotators, sharing and comparing multiple annotations directly. eaaa pre-processes movie files into 4 second segments that overlap by 2 seconds and computes spectrograms for each segment with librosa @xcite . storage is provided by a local redis database which is not exposed to the web. in addition, eaaa includes a telemetry server which collects comprehensive information during the annotation process including every transcript change, keyboard shortcut used, and mouse press. ## conclusion ammt and eaaa are open source and ammt will be contributed to the ud treebanks. in addition to verbatim transcriptions and a treebank, ammt provides a tool chain to enable access and alignment to the source video and audio. most datasets for evaluating and training parsers are focused on written rather than spoken language. with the rise of conversational agents, ammt can serve as a more predictive benchmark in this domain. at present, no end-to-end systems -from videoand-audio to parses -exist, even if humans often use visual information to disambiguate and contextualize auditory information. we hope that ammt and its tooling will support further work on multimodal approaches to conversational agents, end-toend parsing, as well as psychophysics and neuroscience with language in context.