Update README.md

README.md

@@ -1,76 +1,75 @@
----
-license: apache-2.0
-task_categories:
-- audio-classification
-- automatic-speech-recognition
-- question-answering
-tags:
-- Audio
-- Large Audio Language Models
-language:
-- en
-metrics:
-- F1
-- IOU
-- accuracy
----
-
-## 🚀🚀 TimeAudio: Bridging Temporal Gaps in Large Audio-Language
-
-<div style='display:flex; gap: 0.25rem; '>
-<a href='https://arxiv.org/pdf/.pdf'><img src='https://img.shields.io/badge/paper-PDF-green'></a>
-<a href='https://huggingface.co/lysanderism/TimeAudio'><img src='https://img.shields.io/badge/huggingface-checkpoint-yellow'></a> 
-</div>
-
-Recent Large Audio-Language Models (LALMs) exhibit impressive capabilities in understanding audio content for conversational QA tasks. However, these models struggle to accurately understand timestamps for temporal localization (e.g., Temporal Audio Grounding) and are restricted to short audio perception, leading to constrained capabilities on fine-grained tasks. We identify three key aspects that limit their temporal localization and long audio understanding: (i) timestamp representation, (ii) architecture, and (iii) data.  
-
-To address this, we introduce TimeAudio, a novel method that empowers LALMs to connect their understanding of audio content with precise temporal perception. Specifically, we incorporate unique temporal markers to improve time-sensitive reasoning and apply an absolute time-aware encoding that explicitly grounds the acoustic features with absolute time information. Moreover, to achieve end-to-end long audio understanding, we introduce a segment-level token merging module to substantially reduce audio token redundancy and enhance the efficiency of information extraction. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing audio datasets into a new dataset focused on temporal tasks and establish a series of metrics to evaluate the fine-grained performance.
-
-
-## Method
-
-TimeAudio is based on the fundamental architecture of SALMONN. Specifically, TimeAudio is consists of four components: a sliding audio encoder, a window Q-former, a segment-level token merging module, and an LLM to process raw audio. 
-
-<div align=center><img src="img/overview.png" height="100%" width="92%"/></div>
-
-## Compare
-
-Compared with traditional speech and audio processing tasks such as speech recognition and audio caption, Example of failed cases by Qwen2-Audio and Qwen2-Audio-R1 on fine-grained tasks that require both semantics and timestamps as output.
-
-<div align=center><img src="img/case.png" height="100%" width="70%"/></div>
-
-## How to inference in CLI
-
-You need to use the following dependencies:
-
-1. Our environment: The python version is 3.10.16, and other required packages can be installed with the following command: ```pip install -r requirements.txt```.
-2. Download [whisper large v2](https://huggingface.co/openai/whisper-large-v2/tree/main) to ```whisper_path```.
-3. Download [Fine-tuned BEATs_iter3+ (AS2M) (cpt2)](https://valle.blob.core.windows.net/share/BEATs/BEATs_iter3_plus_AS2M_finetuned_on_AS2M_cpt2.pt?sv=2020-08-04&st=2023-03-01T07%3A51%3A05Z&se=2033-03-02T07%3A51%3A00Z&sr=c&sp=rl&sig=QJXmSJG9DbMKf48UDIU1MfzIro8HQOf3sqlNXiflY1I%3D) to `beats_path`.
-4. Download [vicuna 7B v1.5](https://huggingface.co/lmsys/vicuna-7b-v1.5/tree/main) to ```vicuna_path```.
-5. Download [timeaudio](https://huggingface.co/lysanderism/TimeAudio/tree/main) to ```ckpt_path```.
-6. Running with ```python3 cli_inference.py --ckpt_path xxx --whisper_path xxx --beats_path xxx --vicuna_path xxx``` to start cli inference. Please make sure your GPU has more than 40G of memory. If your GPU does not have enough memory (e.g. only 24G), you can quantize the model using the `--low_resource` parameter to reduce the memory usage.
-
-## Launch a QA 
-
-1. Same as **How to inference in CLI: 1-5**.
-2. Running with ```python3 web_demo.py --ckpt_path xxx --whisper_path xxx --beats_path xxx --vicuna_path xxx``` in your machine. You can add `--low_resource` parameter if the GPU memory is not enough, and reduce the LoRA scaling factor to maintain the model's emergent abilities.
-
-
-## Citation
-If you find TimeAudio great and useful, please cite our paper:
-```
-@article{,
-  title={TimeAudio: Bridging Temporal Gaps in Large Audio-Language Models},
-  author={Hualei Wang, Yiming Li, Shuo Ma, Hong Liu, Xiangdong Wang },
-  journal={arXiv preprint arXiv:},
-  year={2025},
-  url={https://arxiv.org/abs/}
-}
-```
-
-## 🙏 Acknowledgments
-
-We gratefully acknowledge the creators of:
-
-- SALMONN(Tang et al.)
+---
+license: apache-2.0
+task_categories:
+- audio-classification
+- automatic-speech-recognition
+- question-answering
+tags:
+- Audio
+- Large Audio Language Models
+language:
+- en
+metrics:
+- F1
+- IOU
+- accuracy
+---
+
+## 🚀🚀 TimeAudio: Bridging Temporal Gaps in Large Audio-Language
+
+<div style='display:flex; gap: 0.25rem; '>
+<a href='https://arxiv.org/pdf/.pdf'><img src='https://img.shields.io/badge/paper-PDF-green'></a>
+<a href='https://huggingface.co/lysanderism/TimeAudio'><img src='https://img.shields.io/badge/huggingface-checkpoint-yellow'></a> 
+</div>
+
+Recent Large Audio-Language Models (LALMs) exhibit impressive capabilities in understanding audio content for conversational QA tasks. However, these models struggle to accurately understand timestamps for temporal localization (e.g., Temporal Audio Grounding) and are restricted to short audio perception, leading to constrained capabilities on fine-grained tasks. We identify three key aspects that limit their temporal localization and long audio understanding: (i) timestamp representation, (ii) architecture, and (iii) data.  
+
+To address this, we introduce TimeAudio, a novel method that empowers LALMs to connect their understanding of audio content with precise temporal perception. Specifically, we incorporate unique temporal markers to improve time-sensitive reasoning and apply an absolute time-aware encoding that explicitly grounds the acoustic features with absolute time information. Moreover, to achieve end-to-end long audio understanding, we introduce a segment-level token merging module to substantially reduce audio token redundancy and enhance the efficiency of information extraction. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing audio datasets into a new dataset focused on temporal tasks and establish a series of metrics to evaluate the fine-grained performance.
+
+
+## Method
+
+TimeAudio is based on the fundamental architecture of SALMONN. Specifically, TimeAudio is consists of four components: a sliding audio encoder, a window Q-former, a segment-level token merging module, and an LLM to process raw audio. 
+
+<div align=center><img src="img/overview.png" height="100%" width="92%"/></div>
+
+## Compare
+
+Compared with traditional speech and audio processing tasks such as speech recognition and audio caption, Example of failed cases by Qwen2-Audio and Qwen2-Audio-R1 on fine-grained tasks that require both semantics and timestamps as output.
+
+<div align=center><img src="img/case.png" height="100%" width="70%"/></div>
+
+## How to inference in CLI
+
+You need to use the following dependencies:
+
+1. Our environment: The python version is 3.10.16, and other required packages can be installed with the following command: ```pip install -r requirements.txt```.
+2. Download [whisper large v2](https://huggingface.co/openai/whisper-large-v2/tree/main) to ```whisper_path```.
+3. Download [Fine-tuned BEATs_iter3+ (AS2M) (cpt2)](https://valle.blob.core.windows.net/share/BEATs/BEATs_iter3_plus_AS2M_finetuned_on_AS2M_cpt2.pt?sv=2020-08-04&st=2023-03-01T07%3A51%3A05Z&se=2033-03-02T07%3A51%3A00Z&sr=c&sp=rl&sig=QJXmSJG9DbMKf48UDIU1MfzIro8HQOf3sqlNXiflY1I%3D) to `beats_path`.
+4. Download [vicuna 7B v1.5](https://huggingface.co/lmsys/vicuna-7b-v1.5/tree/main) to ```vicuna_path```.
+5. Download [timeaudio](https://huggingface.co/lysanderism/TimeAudio/tree/main) to ```ckpt_path```.
+6. Running with ```python3 cli_inference.py --ckpt_path xxx --whisper_path xxx --beats_path xxx --vicuna_path xxx``` to start cli inference. Please make sure your GPU has more than 40G of memory. If your GPU does not have enough memory (e.g. only 24G), you can quantize the model using the `--low_resource` parameter to reduce the memory usage.
+
+## Launch a Demo 
+
+Same as **How to inference in CLI: 1-5**.
+
+
+## Citation
+If you find TimeAudio great and useful, please cite our paper:
+```
+@article{,
+  title={TimeAudio: Bridging Temporal Gaps in Large Audio-Language Models},
+  author={Hualei Wang, Yiming Li, Shuo Ma, Hong Liu, Xiangdong Wang },
+  journal={arXiv preprint arXiv:},
+  year={2025},
+  url={https://arxiv.org/abs/}
+}
+```
+
+## 🙏 Acknowledgments
+
+We gratefully acknowledge the creators of:
+
+- SALMONN(Tang et al.)
 - Qwen2-Audio(Chu et al.)