Evaluating Automatic Speech Recognition Systems for K orean Meteorological Experts (2025)

Abstract: Automatic speech recognition systems often fail on specialized vocabulary in tasks such as weather forecasting. To address this, we introduce an evaluation dataset of Korean weather queries. The dataset was recorded by diverse native speakers following pronunciation guidelines from domain experts and underwent rigorous verification. Benchmarking both open-source models and a commercial API reveals high error rates on meteorological terms. We also explore a lightweight text-to-speech-based data augmentation strategy, yielding substantial error reduction for domain-specific vocabulary and notable improvement in overall recognition accuracy. Our dataset is available at https://huggingface.co/datasets/ddehun/korean-weather-asr.

## introduction meteorologists rely on vast, complex databases for forecasting, yet crafting precise sql queries demands specialized expertise. natural language interfaces address this gap by translating user questions into database queries @xcite @xcite . notably, @xcite developed an integrated search system for korean weather data, enabling users to query extensive meteorological information using natural language. this system simplifies data retrieval and thereby improves operational efficiency. incorporating automatic speech recognition (asr) into these systems can further enhance usability by enabling voice-driven queries, a practical advantage for busy forecasters. however, offthe-shelf asr models-often trained on generaldomain or english-centric corpora-tend to misrecognize korean meteorology terms due to both the language's agglutinative structure and specialized vocabulary @xcite @xcite @xcite @xcite . @xcite . the models compared include zero-shot, fine-tuned on a general domain dataset (i.e., ksponspeech @xcite ), and fine-tuned on a weather domain dataset synthetically generated by a text-to-speech api. wrongly predicted words are manually highlighted in strikethrough by the authors. to quantify these challenges, we created an evaluation set of 5,500 korean weather queries. eleven native speakers recorded queries sourced from @xcite , following expert pronunciation guidelines and a thorough manual validation process to guarantee correctness. we then benchmarked several popular multilingual asr models (e.g., whisper variants) and observed consistent misrecognitions of domain-specific terms-such as weather map, average wind speed, kim (korean integrated model), and units like hpa-even after fine-tuning on a large general-domain korean speech corpus @xcite (fig. 1 ). we also explored lightweight approaches on our dataset, including tts-based data augmentation @xcite and llm-based post-processing @xcite . our contributions are as follows: (1) we construct a domain-specific asr evaluation dataset by recording and validating weather-related queries from native korean speakers; ## dataset construction this study aims to develop a specialized dataset for asr systems within the korean meteorological domain. the construction process includes several key steps: acquiring the natural language questions, obtaining pronunciation guidelines for specialized terms from domain experts, collecting speech data from native speakers, and verifying the correctness of the recorded audio. the detailed data construction pipeline ( §3.1, fig. 2 ) and the analysis of the resulting dataset ( §3.2) are as follows. ## results and analyses in this section, we describe the key observations. zero-shot asr models struggle with domainspecific terminology whisper models show high cer and wer when transcribing meteorological queries, particularly for specialized terms. larger models (e.g., whisper-large-v2) reduce errors relative to smaller variants but still struggle with domain-specific vocabulary, highlighting the need for further adaptation. ## conclusion we introduce a specialized evaluation dataset for assessing asr performance in the korean meteorological domain, addressing the lack of domainspecific benchmarks. our experiments reveal that while fine-tuning on general-domain data improves overall accuracy, specialized terminology remains a major source of error. by releasing this dataset, we aim to support further research in developing robust, domain-adapted systems that better reflect the demands of real-world forecasting scenarios.