Exploring Multimodal Language Models for Sustainability Disclosure Extraction: A Comparative Study (2025)

Abstract: Sustainability metrics have increasingly become a crucial non-financial criterion in investment decision-making. Organizations worldwide are recognizing the importance of sustainability and are proactively highlighting their efforts through specialized sustainability reports. Unlike traditional annual reports, these sustainability disclosures are typically text-heavy and are often expressed as infographics, complex tables, and charts. The non-machine-readable nature of these reports presents a significant challenge for efficient information extraction. The rapid advancement of Vision Language Models (VLMs) has raised the question whether these VLMs can address such challenges in domain specific task. In this study, we demonstrate the application of VLMs for extracting sustainability information from dedicated sustainability reports. Our experiments highlight the limitations in the performance of several open-source VLMs in extracting information about sustainability disclosures from different type of pages.

## introduction in recent years, we have witnessed a significant growth in inclusion of non-financial factors particularly sustainability in corporate reporting. as per kpmg's recent sustainability reporting survey foot_0 , reporting on sustainability has become part of business as usual for 96% of the world's largest 250 companies and a majority of the top 100 companies in each country. driven by continued stakeholder demand for transparency and consistency in sustainability data disclosures, several standards have been proposed to harmonize sustainability reporting. frameworks like global reporting initiative (gri), sustainability accounting standards board (sasb), and un sustainable development goals (sdg), have been developed to streamline the reporting around various sustainability indicators @xcite . despite being part of business-critical disclosures, sustainability reporting remains mostly unstructured, unlike the standardized annual financial reports. with no common reporting template, these reports lack consistency, relying mostly on charts, infographics and text, and are generally published in non-machine-readable pdf formats @xcite . extracting relevant information from these unstructured reports takes several person hours of efforts and is prone to mistakes and influence by personal judgment. hence, automation of sustainability information extraction from reports can reduce processing time and let stakeholders focus more on decision making. in this work, we have used the gri framework as reference. the gri framework offers sustainability standards in three categories: economy (gri 200), environment (gri 300), and social (gri 400), for further details refer a.1. organisations use various indicators listed under these three to report their sustainability activities. recent advancements in natural language processing with availability of large language models (llm) and vision language models (vlm) viz llama @xcite , gemini @xcite , @xcite , chatgpt @xcite , llava @xcite , etc have opened a new dimension to multi-modal information extraction. significant number of efforts have been made to utilize llms for extracting information from sustainability or environmental, social, and governance (esg) reports. for instance, climatebert @xcite is a transformer model fine-tuned for climate-related classification tasks, chatreport @xcite is an llm-based tool that evaluates companies' sustainability reports according to the tcfd guidelines. esgreveal @xcite is a tripartite framework leveraging llms and rag to extract and analyze esg data, offering benchmarks for corporate reporting. docqa @xcite is a platform for question answering over sustainability reports using rag framework. bronzini et al. @xcite used llms to construct knowledge graphs for analyzing esg disclosures from sustainability reports. however, to the best of our knowledge, the use of vlms for esg data extraction is yet to be explored. also, llms still face challenges in effectively handling domain-specific tasks in a zero-shot setting @xcite . in this work, we have evaluated the performance of open source vlms on the task of extracting sustainability disclosures from sustainability reports. we highlight the shortcomings of these models on the extraction task. ## data curation & system architecture we collected around 700 english language sustainability reports from the sasb website foot_1 . many organizations include a gri index table in their sustainability reports which lists gri disclosures with their reference, such as page numbers, links, and section headers. the format varies across reports, as can be seen in a.2. we used gri index tables from these reports to generate the benchmark evaluation dataset, focusing on those with internal references only for listed gri disclosures. we applied a keyword driven heuristic-based filtering method to identify reports containing a gri index table, specifically checking if the index mention appeared in the table of contents. this filtering process left us with 380 reports. next, we manually annotated the page ranges in the reports where the gri index tables appeared. these tables were extracted using a combination of table transformer @xcite and llama 3 @xcite with human in the loop @xmath0, page @xmath1. comparison of our approach with vlm based table extractions are shown in appendix a.2. figure 1 shows the process flow followed for the work. after identifying the relevant page image and corresponding gri disclosure for which data needs to be extracted, we provided the page image along ## experiments we experimented with the following vlms for extracting information related to gri disclosures from the report pages: ## results & analysis we conducted experiments to validate the extractions of 74 unique gri disclosures across 10 sus-tainability reports, with a total of 640 pages analyzed. the results were assessed by two independent domain experts, with a cohen's kappa score @xcite of 0.9, indicating strong consistency in the evaluation process. partial extractions were considered incorrect during validation. the requirement for manual validation of the outputs limited the number of reports included in the experiments. these pages included data presented as text, tables, charts, infographics, or a combination of these formats (examples shown in appendix a figure 2 ). around 60% of the pages have data in more than one format. in we expand our analysis by including key metrics that evaluate the effectiveness of vlms in gri disclosure extraction. for validation purposes, we considered three distinct cases: the accuracy of information extraction using vlms is calculated as the percentage of correct, incorrect, or hallucinated instances, out of the total number of instances for a given gri category or page type.