Pisets: A Robust Speech Recognition System for Lectures and Interviews (2025)

Abstract: This work presents a speech-to-text system “Pisets” for scientists and journalists which is based on a three-component architecture aimed at improving speech recognition accuracy while minimizing errors and hallucinations associated with the Whisper model. The architecture comprises primary recognition using Wav2Vec2, false positive filtering via the Audio Spectrogram Transformer (AST), and final speech recognition through Whisper. The implementation of curriculum learning methods and the utilization of diverse Russian-language speech corpora significantly enhanced the system’s effectiveness. Additionally, advanced uncertainty modeling techniques were introduced, contributing to further improvements in transcription quality. The proposed approaches ensure robust transcribing of long audio data across various acoustic conditions compared to WhisperX and the usual Whisper model. The source code of “Pisets” system is publicly available at GitHub: https://github.com/bond005/pisets.

## introduction sustainable speech recognition systems are essential for scientists, journalists, and anyone processing audio recordings of interviews and meetings. they not only streamline transcription but also improve the reliability and accuracy of the output, facilitating better decision-making and communication. we present the three-component architecture of the offline speech recognition system designed to enhance speech recognition accuracy while minimizing errors and hallucinations associated with the whisper model. the architecture consists of three key components: primary recognition based on wav2vec2, false positive filtering using the audio spectrogram transformer (ast), and final speech recognition utilizing whisper. we called this system "pisets" (in russian, scribe), because it, like the ancient roman scribe tiro after cicero, shorthand recordings of scientific speeches, interviews and other conversations. ## related works the development of automated transcription systems for lectures and interviews relies critically on speech recognition methodologies. beyond the fundamental task of acoustic-to-text conversion, such systems must address ancillary linguistic processing challenges to ensure output fidelity. these include punctuation restoration, capitalization recovery, numeral normalization, and syntactic disambiguation-operations essential for producing human-interpretable transcripts. historically, these subtasks were addressed through modular subsystems: for instance, kaldi-based frameworks employing classical hidden markov model-gaussian mixture model (hmm-gmm) architectures for speech recognition @xcite , complemented by separate neural modules (e.g., recurrent or transformer-based networks) for punctuation prediction (tilk and alumäe, 2016; @xcite . however, empirical advances in deep learning consistently demonstrate that end-to-end neural architectures outperform component-based pipelines in overall accuracy and generalizability. the introduction of whisper (radford et al., 2023) , a unified neural model combining acoustic feature extraction with autoregressive language modeling, exemplifies this paradigm shift. by jointly optimizing acoustic and linguistic representations, whisper directly generates grammatically coherent, punctuated text from raw audio signals, obviating the need for cascaded subsystems. despite its advancements, whisper exhibits limitations inherent to autoregressive sequence-tosequence models: ## uncertainty modeling an uncertainty in transcription (word-wise or segment-wise) may be beneficial in some use cases: uncertainty modeling is a vast area of research. in a current work we compare only the most straightforward methods that we describe in details later: their comparison may be a future work. ## conclusion this paper presents a novel framework aimed at improving speech recognition systems, addressing challenges such as hallucinations, domain adaptability, and acoustic-linguistic variability. the combination of wav2vec2 for speech segmentation, ast for false positive filtering, and whisper for final transcription significantly reduced errors across various acoustic conditions. the integration of diverse russian speech corpora, along with the use of the birm model for fine-tuning, further enhanced the system's robustness to unfamiliar domains. additionally, the implementation of advanced uncertainty modeling techniques provided practical recommendations for improving transcription quality. these enhancements led to the development of a reliable system capable of delivering high-quality transcription in a variety of scenarios, including automatic dictation and conversational ai systems. future work is planned to expand uncertainty handling capabilities and enhance adaptation to multilingual datasets, allowing for more effective recognition of english speech by non-native speakers, as well as the recognition of bengali, spanish, and other languages.