MAST : Multimodal Abstractive Summarization with Trimodal Hierarchical Attention (2020)

Abstract: This paper presents MAST, a new model for Multimodal Abstractive Text Summarization that utilizes information from all three modalities – text, audio and video – in a multimodal video. Prior work on multimodal abstractive text summarization only utilized information from the text and video modalities. We examine the usefulness and challenges of deriving information from the audio modality and present a sequence-to-sequence trimodal hierarchical attention-based model that overcomes these challenges by letting the model pay more attention to the text modality. MAST outperforms the current state of the art model (video-text) by 2.51 points in terms of Content F1 score and 1.00 points in terms of Rouge-L score on the How2 dataset for multimodal language understanding.

## introduction in recent years, there has been a dramatic rise in information access through videos, facilitated by a proportional increase in the number of videosharing platforms. this has led to an enormous amount of information accessible to help with our day-to-day activities. the accompanying transcripts or the automatic speech-to-text transcripts for these videos present the same information in the textual modality. however, all this information is often lengthy and sometimes incomprehensible because of verbosity. these limitations in user experience and information access are improved upon by the recent advancements in the field of multimodal text summarization. multimodal text summarization is the task of condensing this information from the interacting modalities into an output summary. this generated output summary may be unimodal or multimodal @xcite . the textual summary * indicates equal contribution aman khullar is presently at gram vaani may, in turn, be extractive or abstractive. the task of extractive multimodal text summarization involves selection and concatenation of the most important sentences in the input text without altering the sentences or their sequence in any way. @xcite made the selection of these important sentences using visual and acoustic cues from the corresponding visual and auditory modalities. on the other hand, the task of abstractive multimodal text summarization involves identification of the theme of the input data and the generation of words based on the deeper understanding of the material. this is a tougher problem to solve which has been alleviated with the advancements in the abstractive text summarization techniques -rush et al. @xcite , @xcite and @xcite . @xcite introduced the how2 dataset for large-scale multimodal language understanding, and @xcite were able to produce state of the art results for multimodal abstractive text summarization on the dataset. they utilized a sequence-to-sequence hierarchical attention based technique @xcite for combining textual and image features to produce the textual summary from the multimodal input. moreover, they used speech for generating the speech-to-text transcriptions using pre-trained speech recognizers, however it did not supplement the other modalities. though the previous work in abstractive multimodal text summarization has been promising, it has not yet been able to capture the effects of combining the audio features. our work improves upon this shortcoming by examining the benefits and challenges of introducing the audio modality as part of our solution. we hypothesize that the audio modality can impart additional useful information for the text summarization task by letting the model pay more attention to words that are spoken with a certain tone or level of emphasis. through our original text: let's talk now about how to bait a tip up hook with a maggot. typically, you're going to be using this for pan fish. not a real well known or common technique but on a given day it could be the difference between not catching fish and catching fish. all you do, you take your maggot, you can use meal worms, as well, which are much bigger, which are probably more well suited for this because this is a rather large hook. you would just, again, put that hook right through the maggot. with a big hook like this, i would probably put ten of these on it, just line the whole thing. this is going to be more of a technique for pan fish, such as, perch and sunfish, some of your smaller fish but if you had maggots, like this , or a meal worm, or two, on a hook like this, this would be a fantastic setup for trout, as well. text only: ice fishing is used for ice fishing. learn about ice fishing bait with tips from an experienced fisherman artist in this free fishing video. video-text: learn about the ice fishing bait in this ice fishing lesson from an experienced fisherman. mast: maggots are good for catching perch. learn more about ice fishing bait in this ice fishing lesson from an experienced fisherman. in summary, our primary contributions are: ## methodology in this section we describe (1) the dataset used, (2) the modalities, and (3) our mast model's architecture. the code for our model is available online 1 . ## experiments we train trimodal hierarchical attention (mast) and trimodalh2 models on the 300h version of the how2 dataset, using all three modalities. we also train hierarchical attention models considering audio-text and video-text modalities, as well as simple seq2seq models with attention for each modality individually as baselines. as observed by @xcite , the pointer generator model @xcite does not perform as well as seq2seq models on this dataset, hence we do not use that as a baseline in our experiments. we consider another transformer-based baseline for the text modality, bertsumabs @xcite . for all our experiments (except for the bersum-abs baseline), we use the nmtpytorch toolkit @xcite . the source and the target vocabulary consists of 49,329 words on which we train our word embeddings. we use the nll loss and the adam optimizer (kingma and ba, 2014) with learning rate 0.0004 and trained the models for 50 epochs. we generate our summaries using beam search with a beam size of 5, and then evaluate them using the rouge metric @xcite and the content f1 metric @xcite . in our experiments, the text is embedded with an embedding layer of size 256 and then encoded using a bidirectional gru encoder @xcite with a hidden layer of size 128, which gives us a 256-dimensional output encoding corresponding to the text at each timestep. the audio and video frames are encoded using bidirectional lstm encoders @xcite with a hidden layer of size 128, which gives a 256dimensional output encoding corresponding to the audio and video features at each timestep. finally, the gru-based conditional decoder uses a hidden layer of size 128 followed by two linear layers which transform the decoder output to generate the final output vocabulary distribution. to improve generalization of our model, we use two dropout layers within the text encoder and one dropout layer on the output of the conditional decoder, all with a probability of 0.35. we also use implicit regularization by using early stopping mechanism on the validation loss with a patience of 40 epochs. ## conclusion in this work foot_0 , we presented mast, a state of the art sequence to sequence based model that uses information from all three modalities -audio, text and video -to generate abstractive multimodal text summaries. it uses a trimodal hierarchical attention layer to utilize information from all modalities. we explored the role played by adding the audio modality and compared mast with several baseline models, demonstrating the effectiveness of our approach. in the future, we would like to extend this work by looking at alternate audio modality representations including using neural networks for audio feature extraction, and also explore the use of transformers for an end to end attention based learning. we also aim to explore the application of mast to