MEANT : Multimodal Encoder for Antecedent Information (2024)

Abstract: The stock market provides a rich well of information that can be split across modalities, making it an ideal candidate for multimodal evaluation. Multimodal data plays an increasingly important role in the development of machine learning and has shown to positively impact performance. But information can do more than exist across modes— it can exist across time. How should we attend to temporal data that consists of multiple information types? This work introduces (i) the MEANT model, a Multimodal Encoder for Antecedent information and (ii) a new dataset called TempStock, which consists of price, Tweets, and graphical data with over a million Tweets from all of the companies in the S&P 500 Index. We find that MEANT improves performance on existing baselines by over 15%, and that the textual information affects performance far more than the visual information on our time-dependent task from our ablation study. The code and dataset will be made available upon publication.

## introduction recently, multimodal models have garnered serious momentum, with the release of large pretrained architectures such as microsoft's kosmos-1 @xcite ) and openai's @xcite . their general use has exploded in many domains, such as language and image processing @xcite @xcite . particularly interesting to this study is the deployment of multimodal models on timedependent environments, where recent successes have shown that event driven models processing multiple modalities are far more performant on stock market tasks than previously state of the art (sota) algorithms focusing purely on price information @xcite . language data from news and social media sources have shown to greatly increase the performance of * irving.b@northeastern.edu models for price prediction @xcite @xcite @xcite . however, these approaches typically lack attention components specifically designed to process inputs with sequential, time-dependent information @xcite @xcite . this type of data is particularly important when making predictions about stock prices or market movements, as price prediction is a time series task @xcite . in this work, we introduce meant, a multimodal model architecture with a novel, temporally focused self-attention mechanism. we extract image features using the timesformer architecture @xcite to find relationships in longer range information (i.e a graph of stock prices over a month), while extracting language features from social media information to pick up more immediate trends (e.g.: tweets pertaining to stock prices over a five day period). furthermore, we release tempstock, a multimodal stock-market dataset that is designed to be sequentially processed in chunks of varying lag periods. modeling media-aware stock movements is essentially a binary classification problem. many traditional machine learning methods have been deployed to solve it, including svms and bayesian classifiers @xcite @xcite . more recently, researchers have applied deep learning to the problem. @xcite used a convolutional neural network to explore the impact of tweets on the stock market. @xcite and @xcite then employed a recurrent architecture, specifically an lstm, to extract relevant sentiments from twitter data for stock market analysis, making their model multimodal, as it could handel tweets and price information. @xcite built atop this architecture, employing different tensor representations for their lstm input to create more meaningful relationships between the price and tweets data. @xcite introduced stocknet, a large generative architecture built atop generative architectures, particularly the variational auto encoder (vae). stocknet represented the first deep generative model for stock market prediction @xcite . teanet, the most relevant work to our own, similarly used an lstm to process their final output, but used a bert-style transformer to extract relevant features from the tweets @xcite . teanet is a language model equipped to handle lag periods similarly to meant. they concatenate their language features to price data as an input for an lstm and a subsequent softmax temporal encoding. we abandon recurrence altogether, developing a novel temporal mechanism, entirely based upon traditional self-attention methods @xcite . the temporal processing in teanet consists of concatenation methods similar to our own, but they do not employ attention over time. furthermore, their model was built to handle tweets and price inputs alone. meant can handle images as well, employing a dual encoder architecture similar to that of @xcite . financial twitter datasets previous financial datasets have shown the power of twitter data for financial analysis @xcite @xcite . twitter is powerful in its ability to generate real time information about the market before traditional newswires @xcite . @xcite focused on twitter as a resource for examine financial dynamics in the retail sector. @xcite introduced tweetsfinsent, a large corpus specifically for sentiment analysis. @xcite introduced a dataset consisting of tweets and prices, where the tweets information served as a sentiment analysis accompaniment for the price data. @xcite introduced the stocknet-dataset, consisting of tweets and price information for a selection of 88 companies over a two year period from 01/01/2014 to 01/01/2016. @xcite matched tweets with price information from companies in the s&p 500 dataset, which is the most similar to the tempstock dataset that we introduce below. ## tempstock dataset we collected a new dataset containing 1,755,998 tweets and price information from all of the companies in the s&p 500 from 4/10/2022 to 4/10/2023. from the price information, we calculated the moving average convergence-divergence (macd) @xcite for each company over a year. the macd is built on the back of exponential moving average (ema) @xcite . the ema is defined as follows: ## meant meant combines the advantages of image and language processing with temporal attention, in order to extract dependencies from multimodal, sequential information, where 2 displays the full architecture. meant, similarly to most sota multimodal models @xcite @xcite @xcite , is built atop the transformer architecture @xcite . ## results tables 4 and 5 in sections 6.1 and 6.2 show the results for our experiments respectively. ## conclusion and future work we introduced a multimodal encoder with a novel temporal component comprised entirely of selfattention. meant outperforms previous models on the stocknet benchmark by 15%, and proves to be the most performant model on our own temp-stock benchmark. to our knowledge, meant-xl is the largest model to be applied to stocknet, and is the first multimodal model to contain an attention mechanism to deal with data over a lag period of days. meant combines the realms of language, vision, and time to produce sota results. we would like to explore different early fusion methods in order to make meant more robust against other common multi modal benchmarks, and expand upon our query-targeting strategy to emphasize relevant queries automatically, rather manually emphasizing any specific component such as the final lag day. we believe that the meant architecture has the potential to succeed on a wide variety of tasks. furthermore, the image space that we trained meant on was limited. we would like to introduce more variation into our image inputs, to fully utilize the capabilities of that modality in our model.