Investigating large language models for their competence in extracting grammatically sound sentences from transcribed noisy utterances (2024)

Abstract: Selectively processing noisy utterances while effectively disregarding speech-specific elements poses no considerable challenge for humans, as they exhibit remarkable cognitive abilities to separate semantically significant content from speech-specific noise (i.e. filled pauses, disfluencies, and restarts). These abilities may be driven by mechanisms based on acquired grammatical rules that compose abstract syntactic-semantic structures within utterances. Segments without syntactic and semantic significance are consistently disregarded in these structures. The structures, in tandem with lexis, likely underpin language comprehension and thus facilitate effective communication.In our study, grounded in linguistically motivated experiments, we investigate whether large language models (LLMs) can effectively perform analogical speech comprehension tasks. In particular, we examine the ability of LLMs to extract well-structured utterances from transcriptions of noisy dialogues. We conduct two evaluation experiments in the Polish language scenario, using a dataset presumably unfamiliar to LLMs to mitigate the risk of data contamination. Our results show that not all extracted utterances are correctly structured, indicating that either LLMs do not fully acquire syntactic-semantic rules or they acquire them but cannot apply them effectively. We conclude that the ability of LLMs to comprehend noisy utterances is still relatively superficial compared to human proficiency in processing them.

## introduction in the field of natural language understanding (nlu), efforts are directed towards simulating human language comprehension using language modelling techniques. a crucial aspect of this pursuit involves the development of large language models (llms), which play a pivotal role in numerous natural language processing (nlp) tasks @xcite @xcite , tailored for comprehension, generation, and manipulation of natural language. nlu research also aims to identify llms' shortcomings, to reverse-engineer phenomena that llms fail to address. despite impressive capabilities, llms have not achieved the comprehensive and nuanced linguistic competency inner to human beings @xcite and their further study is necessary. llms undergo training on extensive and varied datasets, which include textual data, code-based data, structured datasets, and other data sources. textual data exhibits significant diversity, comprising edited texts, content from social media platforms as well as speech transcriptions, such as parliamentary proceedings or pretended dialogues within narratives or subtitles. despite spoken language's dominance in daily communication and the availability of high-quality transcription tools, it remains unexplored whether processing transcribed utterances is challenging for llms. motivated by this observation, we aim to examine whether llms can effectively address challenges akin to those faced by humans during comprehending utterances. speech understanding is a complex cognitive process that plays a fundamental role in human communication. the nature of speech comprehension is multifaceted, influenced by neurological, cognitive and linguistic factors. this study focuses on the linguistic dimension. when decoding spoken messages, humans struggle with phonological difficulties @xcite , including phonological similarity and ambiguity among words, and individual phonemic variations. this aspect is irrelevant to the current study, as we solely investigate the processing of texts (transcriptions). the comprehension of spoken utterances can be affected by syntactic complexity. processing complex sentences may increase a cognitive cost and result in comprehension difficulties @xcite . the semantic aspects of speech understanding are thoroughly researched. for instance, @xcite investigated the process of word-sense disambiguation and its associated challenges. to comprehend an utterance, separating semantically significant content from speech-specific noise is crucial. the ability to filter out noise and selectively compose only the semantically relevant information is inherent to humans. since it remains unexplored whether llms can perform this task effectively, we address this issue through linguistically motivated evaluation tasks in the polish language scenario. in section 2, we introduce the proposed approach with its primary objective to determine whether llms are capable of identifying well-structured utterances in transcriptions of authentic spontaneous utterances that incorporate noisy speech-specific segments. in sections 3 and 4, we outline the experimental setup and discuss the results of the empirical evaluation. section 5 provides the contextual backdrop for our research, while section 6 concludes our research findings. ## proposed approach processing spoken data is often more challenging when contrasted with processing genuine written texts. firstly, spoken words may be obscured by background sounds, resulting in transcription gaps. secondly, the application of automated transcription and punctuation recovery tools can yield lexical and punctuation errors in transcriptions. thirdly, the written mode tends to be more standardised, whereas the spoken mode often features informal and colloquial language. finally, and most importantly in the context of this study, speech-specific elements such as fillers, self-corrections, and false starts increase the complexity of understanding spoken data compared to written texts. in the era of robust and advanced llms, utilising them for processing transcribed spoken data emerges as a rational choice. nevertheless, uncertainties arise regarding their ability to identify intended content to be comprehended in possibly noisy utterances. this study examines whether llms possess the competence to selectively process noisy utterances while ignoring non-fluency features. we investigate the capabilities of llms in (1) extracting well-formed sentences determined by abstract syntactic-semantic structures (see section 2.1) from noisy utterances; (2) disregarding speech-specific elements (see section 2.2) that do not contribute to utterance understanding. to ascertain the ability of llms to disregard speech-specific elements and to recognise wellformed sentences within noisy utterances, we employ the prompting methodology (see section 2.3). based on predefined prompts, llms are instructed to identify and subsequently output all tokens composing well-structured utterances. llms' performance in extracting refined utterances and filtering non-fluency features is evaluated against a goldstandard dataset (see section 2.4). ## related works probing state-of-the-art lms for their syntactic and semantic knowledge is a widely adopted diagnostic approach. numerous studies have attempted to examine lms using controlled test sets. some studies focus on designing probing tests to directly inspect the model's internal structure and identify its regions correlated with linguistic information @xcite @xcite @xcite . for instance, @xcite demonstrate that bert can effectively execute multiple stages of an nlp pipeline, including pos tagging, parsing, named entity recognition, semantic role labelling, and coreference resolution. they localise bert's regions where linguistic information is embedded and which are responsible for each task. parallel investigations endeavour to probe models to measure their proficiency and limitations in representing language, with a particular focus on syntactic and semantic knowledge @xcite @xcite @xcite . for example, @xcite discover that lms can classify sentences as instances of a particular linguistic construction, but they cannot extract the conveyed meaning and effectively employ it within a given context. our research aligns with the latter line of work, focusing on llm's linguistic competence. since our research partially explores speech understanding, we mention recent studies focusing on probing speech models for syntax. @xcite probe them to discern their ability to encode linguistic information, including the depth of syntax trees. similarly, @xcite conduct probing tests on speech models to identify the loci where syntactic structures are embedded. speech processing typically involves two main stages -automatic speech recognition (asr) and nlu, with an intermediate step often dedicated to detecting and possibly removing disfluencies @xcite . lou and johnson (2020) aim at developing joint models that integrate asr with disfluency removal. this approach results in refined transcripts, which standard nlp and nlu tools can subsequently process. in our evaluation approach, we test the capability of llms to detect and filter out noise. however, our goal is not to employ llms as noise detectors; rather, we seek to determine whether llms can prioritise the meaningful parts of utterances (i.e. wellstructured sentences) while ignoring noise during processing noisy utterances. ## conclusions in this study, we have introduced an approach aimed at evaluating the capabilities of llms within the realm of processing transcribed noisy utterances in polish. our primary focus is to ascertain whether llms possess adequate linguistic competence to detect well-structured sentences in noisy utterances. to conduct this research, we leverage the prompting technique, in which the currently most powerful gpts, two open llms (llama and mistral) and a polish llm (bielik) are tasked with identifying speech-inherent noise and extracting wellstructured utterances. the models' outcomes are rigorously evaluated using the probing dataset derived from the ud-annotated subset of diabiz. recognising speech-specific phenomena, especially false starts, presents a challenge for the tested llms. mistral appears proficient in filtering out false starts and other speech-specific noise. this proficiency, however, does not stem from its language comprehension ability; rather, it arises from its strategy for aggressive filtering, wherein it elim-inates not only noise but also required components of predicate-argument structures, resulting in grammatical errors. gpts generally exclude fewer required arguments and semantically crucial modifiers but erroneously retain many speech-specific segments. numerous studies confirm that transformerbased lms acquire individual syntactic and semantic rules and can perform syntactic-and semanticbased nlp tasks. our experimental results also indicate that llms possess linguistic competence. however, this competence may be superficial or insufficient, as llms struggle to identify complete and coherent sentences in noisy utterances. this superficial competence prevents the full internalisation of ass underlying human language comprehension. deeper syntactic-semantic understanding is necessary for handling restarts and other speech noise to enable seamless conversation of llms (or large multimodal models) with humans. alternatively, llms may be unable to apply all syntactic-semantic rules they have acquired, resulting in limited performance. in this case, psycholinguistic factors, such as shallow heuristics mixed with syntactic algorithms @xcite or rational statistical inference @xcite , could impact the behaviour of llms, as suggested by an anonymous reviewer. the application of psycholinguistic research methods may be highly valuable for the future evaluation of llms. we anticipate that our novel evaluation approach will inspire further research into selective language processing. considering that asr outcomes used in voice assistants and other speech-based systems require additional text processing, and texts are predominantly processed with llms, llms should handle both written texts and spontaneous speech transcriptions. this ability is crucial for enabling human-like dialogue with machines. moreover, by integrating speech and text understanding, our approach lays the groundwork for evaluating llms and potentially large multimodal models.