Human Speech Perception in Noise: Can Large Language Models Paraphrase to Improve It? (2024)

Abstract: Large Language Models (LLMs) can generate text by transferring style attributes like formality resulting in formal or informal text.However, instructing LLMs to generate text that when spoken, is more intelligible in an acoustically difficult environment, is an under-explored topic.We conduct the first study to evaluate LLMs on a novel task of generating acoustically intelligible paraphrases for better human speech perception in noise.Our experiments in English demonstrated that with standard prompting, LLMs struggle to control the non-textual attribute, i.e., acoustic intelligibility, while efficiently capturing the desired textual attributes like semantic equivalence. To remedy this issue, we propose a simple prompting approach, prompt-and-select, which generates paraphrases by decoupling the desired textual and non-textual attributes in the text generation pipeline.Our approach resulted in a 40% relative improvement in human speech perception, by paraphrasing utterances that are highly distorted in a listening condition with babble noise at signal-to-noise ratio (SNR) -5 dB. This study reveals the limitation of LLMs in capturing non-textual attributes, and our proposed method showcases the potential of using LLMs for better human speech perception in noise.

## introduction paraphrase generation is the task of rephrasing a sentence while retaining its meaning @xcite . humans perform paraphrasing in spoken conversations, to enable their listeners to perceive spoken messages as intended @xcite . motivated by human speech production strategies, paraphrasing has also been applied to speech synthesis systems, to enhance the quality, naturalness @xcite , and intelligibility of synthetic speech, especially in challenging acoustic conditions @xcite . recent explorations on why certain sentences are more intelligible than their paraphrases showed that, the observed intelligibility gain in a noisy listening environment is attributed to the rephrasing, which introduces more acoustic cues that survived the masking effect of the noise @xcite . in other words, the enhanced speech perception with paraphrasing is driven by noiserobust acoustic cues. the potential of paraphrasing is however, seldom used to build human-like spoken dialogue systems that are adaptive to human listeners' perception errors in noise, presumably due to the limited investigations to generate paraphrases that are acoustically more intelligible in a noise condition. prior studies relied on human annotations to identify the ideal paraphrase among a set of candidates @xcite @xcite , with little discussion on generating intelligible paraphrases. this raises the question of how to generate text that is semantically equivalent to and acoustically more intelligible than the given input sentence, for a noisy environment. we refer to this task as paraphrase to improve speech perception in noise (pi-spin). this task is particularly interesting in the context of generative llms, which have shown incredible performance in natural language generation (nlg) tasks such as paraphrase generation and dialogue generation @xcite @xcite . moreover, recent studies have demonstrated llms' capability to control text generation for a wide range of style attributes like sentiment, syntax, formality, and politeness @xcite . pi-spin differs from those controllable text generation problems, as it aims to generate text that satisfies the desired textual attributes (e.g., semantic equivalence), in 1 pwr-stoi : 0.86 ❌ llm figure 1 : a schematic representation of the prompt-and-select and standard prompting approach to generate acoustically intelligible paraphrase in a noisy environment. a speech intelligibility metric, short-time objective intelligibility measure (stoi) is employed to select the paraphrase that is more likely to improve speech perception. to explore the potential of llms in pi-spin, we proposed to evaluate llms' inherent capability to generate acoustically intelligible paraphrases, without any model fine-tuning. through standard prompting methods like zero-shot learning (zsl), we found that the model was able to capture textual attributes, while consistently struggling to improve acoustic intelligibility. we also observed that increasing the description of the desired non-textual attribute in the prompt only confuses the model, and it may even lead to a deterioration in textual attributes that were achievable otherwise. to effectively utilize llms for generating acoustically intelligible paraphrases, we propose a simple approach called prompt-and-select, which guides paraphrase generation by introducing the desired non-textual attribute in a post-processing step (see figure 1 ). it is a two-step process beginning with prompting the llm to generate multiple candidates and then selecting the best candidate based on acoustic intelligibility, which is hard to capture in textual mode alone. by conducting a human evaluation with native english listeners, who have no hearing impairments, we verified that the llm-generated paraphrases via prompt-and-select approach are indeed more intelligible than original sentences, in a listening environment with babble noise at snr -5 db. 2 2 see definitions of babble noise and snr in appendix a. our main contributions are as follows: ## pi-spin task description typically, the paraphrase generation task focuses on generating text that is semantically equivalent to the given input text. however, the pi-spin task aims at generating text that is semantically equivalent to, as well as, acoustically more intelligible than the original input text, in an adverse listening condition. for example, consider the following paraphrase triplet (s 1 , s 2 , s 3 ) from the paraphrases-in-noise (pin) dataset foot_1 @xcite : s 1 : "i was raised in a generation we did need all those things." speech intelligibility in noise is better when sentences are simple @xcite , shorter @xcite , and linguistically more predictive @xcite . however, the intelligibility of an utterance in noise is not only driven by its underlying text. the perception is also influenced by the acoustic cues that survived the masking effect of the background noise @xcite . hence, pi-spin is a text generation task, that involves both textual attributes like semantic equivalence and a non-textual attribute that captures the noise-robustness of an utterance. to generate the acoustic realization of a sentence, we used the tacotron2 text-to-speech (tts) system, which demonstrated performance on par with that of a professional voice talent @xcite . more specifically, we used the tacotron2 model foot_2 pre-trained on the ljspeech dataset by speechbrain @xcite . further, to create the noise-distorted signals, the clean audio signals underwent a noise-mixing procedure using an open-sourced tool, audio-snr. 5 the babble noise from the noisex-92 dataset @xcite was mixed with clean audio at snr-5 db. to determine whether the generated text satisfies the desired outcome, we primarily relied on automatic metrics, which are discussed in detail in the following section. ## experimental setup model. for all our experiments, we used chat-gpt foot_4 @xcite , which is one of the most popular llms. it has shown impressive performance on paraphrase generation with textual style attributes, while its ability on acoustically intelligible paraphrasing remains unclear. we adopt default parameters (@xmath0.0, @xmath1.0) for the api calls. dataset. the evaluation dataset consists of 300 short sentences, which are spoken in a conversational scenario. the dataset is created by filtering out sentences with 10 to 12 words from the top 1000 lines of the speech corpus, switchboard @xcite . ## evaluating llms for pi-spin in our experiments, an llm is prompted to generate a paraphrase for each input sentence in the evaluation set with a prompt template: {prompt prefix} + {input text}. in the following section, we described two prompting methods that we employed and evaluated for the task. ## conclusion in this work, we evaluate llms on acoustically intelligible paraphrase generation for better human speech perception in noise. our results demonstrate the limitations of llms in controlling text generation with a non-textual attribute -acoustic intelligibility. to alleviate the struggles of llms in generating text that satisfies both textual and non-textual attributes, we proposed a simple yet effective approach called prompt-and-select. with human evaluation, we found that when the original utterances are highly prone to misperceptions in noise, prompt-and-select can introduce 40% of relative improvement in human perception. we hope the findings of this work inspire further explorations to control llms' text generation with different real-world context cues, thereby building more human-like agents. for future work, we could consider two approaches to improve llms on this task: 1) fine-tuning llms with a large parallel dataset consisting of sentences and their corresponding intelligible paraphrases, and 2) incorporating the acoustic representation of the utterances to control the paraphrase generation.