Init: add images via LFS

README.md

@@ -15,30 +15,30 @@ metrics:
 - accuracy
 ---
 
-# TimeAudio: Bridging Temporal Gaps in Large Audio-Language
-
-🚀🚀 The repo of **TimeAudio**!
-
-Recent Large Audio-Language Models (LALMs) exhibit impressive capabilities in understanding audio content for conversational QA tasks. However, these models struggle to accurately understand timestamps for temporal localization (e.g., Temporal Audio Grounding) and are restricted to short audio perception, leading to constrained capabilities on fine-grained tasks. We identify three key aspects that limit their temporal localization and long audio understanding: (i) timestamp representation, (ii) architecture, and (iii) data. To address this, we introduce TimeAudio, a novel method that empowers LALMs to connect their understanding of audio content with precise temporal perception. Specifically, we incorporate unique temporal markers to improve time-sensitive reasoning and apply an absolute time-aware encoding that explicitly grounds the acoustic features with absolute time information. Moreover, to achieve end-to-end long audio understanding, we introduce a segment-level token merging module to substantially reduce audio token redundancy and enhance the efficiency of information extraction. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing audio datasets into a new dataset focused on temporal tasks and establish a series of metrics to evaluate the fine-grained performance. Evaluations show strong performance across a variety of fine-grained tasks, such as dense captioning, temporal grounding, and timeline speech summarization, demonstrating TimeAudio's robust temporal localization and reasoning capabilities.
+## 🚀🚀TimeAudio: Bridging Temporal Gaps in Large Audio-Language
 
 <div style='display:flex; gap: 0.25rem; '>
 <a href='https://arxiv.org/pdf/.pdf'><img src='https://img.shields.io/badge/paper-PDF-green'></a>
 <a href='https://huggingface.co/lysanderism/TimeAudio'><img src='https://img.shields.io/badge/huggingface-checkpoint-yellow'></a> 
 </div>
 
+Recent Large Audio-Language Models (LALMs) exhibit impressive capabilities in understanding audio content for conversational QA tasks. However, these models struggle to accurately understand timestamps for temporal localization (e.g., Temporal Audio Grounding) and are restricted to short audio perception, leading to constrained capabilities on fine-grained tasks. We identify three key aspects that limit their temporal localization and long audio understanding: (i) timestamp representation, (ii) architecture, and (iii) data.  
+
+To address this, we introduce TimeAudio, a novel method that empowers LALMs to connect their understanding of audio content with precise temporal perception. Specifically, we incorporate unique temporal markers to improve time-sensitive reasoning and apply an absolute time-aware encoding that explicitly grounds the acoustic features with absolute time information. Moreover, to achieve end-to-end long audio understanding, we introduce a segment-level token merging module to substantially reduce audio token redundancy and enhance the efficiency of information extraction. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing audio datasets into a new dataset focused on temporal tasks and establish a series of metrics to evaluate the fine-grained performance. Evaluations show strong performance across a variety of fine-grained tasks, such as dense captioning, temporal grounding, and timeline speech summarization, demonstrating TimeAudio's robust temporal localization and reasoning capabilities.
+
 
-## Structure
+## Method
 
 TimeAudio is based on the fundamental architecture of SALMONN. Specifically, TimeAudio is consists of four components: a sliding audio encoder, a window Q-former, a segment-level token merging module, and an LLM to process raw audio. The sliding audio encoder first divides long audio into shorter segments and combines the BEATs and the Whisper encoder to extract features for each segments independently. Then, the window Q-former projects these encoded audio tokens into the language space and applies a segment-level token merging mechanism based on attention scores to filter out unimportant acoustic information.
 Finally, the audio embeddings and the textual token embeddings of user prompts are fed into the LLM to generate response. 
 
-<div align=center><img src="img/overview.png" height="100%" width="75%"/></div>
+<div align=center><img src="img/overview.png" height="100%" width="90%"/></div>
 
-## Demos
+## Compare
 
 Compared with traditional speech and audio processing tasks such as speech recognition and audio caption, Example of failed cases by Qwen2-Audio and Qwen2-Audio-R1 on fine-grained tasks that require both semantics and timestamps as output.
 
-<div align=center><img src="img/case.png" height="100%" width="75%"/></div>
+<div align=center><img src="img/case.png" height="100%" width="90%"/></div>
 
 ## How to inference in CLI