Speechworthy Instruction-tuned Language Models (2024)

Abstract: Current instruction-tuned language models are exclusively trained with textual preference data and thus may not be aligned to the unique requirements of other modalities, such as speech. To better align language models with the speech domain, we explore i) prompting strategies based on radio-industry best practices and ii) preference learning using a novel speech-based preference data of 20K samples collected by annotators who listen to response pairs. Both human and automatic evaluation show that both prompting and preference learning increase the speech-suitability of popular instruction tuned LLMs. More interestingly, we show that these methods are additive; combining them achieves the best win rates in head-to-head comparison, resulting in responses that are preferred or tied to the base model in 76.2% of comparisons on average. Lastly, we share lexical, syntactical, and qualitative analyses that elicit how our studied methods differ with baselines in generating more speech-suitable responses.

## introduction speech is one of our primary means of communication and a convenient and popular mode for interacting with virtual assistants @xcite . virtual assistants are a prime application of instructiontuned language models (itlm), as both seek to provide helpful responses to user requests @xcite @xcite @xcite . however, current itlms are fine-tuned on textual instructions @xcite @xcite @xcite and preferences obtained from annotators that read textual responses @xcite @xcite . we hypothesize that user preferences for speech and text are different and that the shift from generating written text to spoken language may pose a challenge for itlms @xcite . speech is serial and transient; speech processing is strictly linear @xcite and requires higher cognitive load than reading @xcite . concise and simple sentences, as seen on the right side of figure 1 , are thus often preferred in speech @xcite , yet current itlms optimized with textual preference datasets @xcite produce verbose responses with non-vocalizable content (left side of figure 1 ). we test the hypothesis that current itlms are sub-optimal for speech by surveying how user preferences differ when responses are heard instead of read. our survey results reveal that users negatively perceive responses from current itlms, primarily due to inadequate response length, information volume, and understanding ease. to improve itlm speech-worthiness, we explore methods that improve itlms responses in each of these dimensions. we explore two main methodologies to make these improvements: (i) prompt engineering @xcite @xcite and (ii) preference learning @xcite @xcite . for prompt engineering, we compile rulesof-thumb from the radio industry and findings from literature on listenability @xcite to design prompts for both zeroshot and few-shot in-context learning @xcite @xcite @xcite . for preference learning, we collect the first speech-based preference dataset speechpref, consisting of over 20k response pairs with varying degrees of speechsuitable responses. to obtain responses that vary on the many dimensions of speech-suitability, we use multiple itlms, generation hyperparameters, and system prompts that target different features (e.g. response length, simplicity, listen-ability). in addition, we designed a speech-only preference annotation, in which annotators are only allowed to listen to the response pairs to accurately capture speech-based preferences. we use speech-pref to align models with the resulting speechpreference data using both proximal policy optimization (ppo) @xcite and direct preference optimization (dpo) @xcite . our experiments with falcon 7b instruct @xcite and olmo 7b instruct @xcite show that both prompt-engineering and preference learning yields models that significantly improve over the base model on both human evaluations and automatic evaluations that serve as proxies to speech suitability. we show that these benefits are additive; combining both techniques produces responses that are maximally preferred to other setups, leading to an average of 75.3% win or tie rate. further, we conduct lexical and syntactical analysis to show how our approaches adapt models for speech and share qualitative examples that compare responses from various techniques. lastly, we show that our prompts and in-context examples also benefit black box models like gpt-4, resulting in responses that are preferred or tied 88.3% of the time compared to the unprompted model. foot_0 ## speech-suitability of itlm responses we initiate our study on the speech-suitability of itlm responses by answering these questions: (i) "are itlm responses suitable for speech-based interactions?" and (ii) "if not, what makes them unsuitable?" 2.1 radio industry best practices @xcite and @xcite provide a glimpse to the answers for these questions by illustrating numerous examples of how news reporting and storytelling in text differs from audio. through decades of radio experience, they establish rulesof-thumb to improve information delivery specific to the audio modality. we highlight rules that generalize beyond the news domain below, including: (i) use simple words and sentence structures: allot a sentence to each idea and put the subject at the beginning as much as possible; (ii) do not use atypical syntax, such as "president bush today told members of congress" and "i today went shopping"; (iii) avoid hyphenated adjectives (e.g. mineralrich, tech-heavy); (iv) avoid too many names and numbers; and other minor ones such as (v) avoid tongue twisters and (vi) avoid too much alliteration. similar principles have been echoed in podcasting @xcite , multimedia journalism @xcite , and literature on listenability @xcite @xcite . ## adapting itlms for speech having established the shortcomings of itlm responses, now we explore how to adapt itlms to generate more speech-suitable responses. in this section, we describe our approaches with two main directions: prompt engineering and preference learning. ## conclusion we explore an important yet overlooked challenge of adapting itlms to compose responses that are specifically designed to be verbalized, i.e., speechsuitable. with rules-of-thumb of the radio industry and through our surveys that compare the suitability of a response for both text and audio, we establish that adaptations for speech-suitable responses is necessary for the broader application of itlms to speech settings. to initiate this line of research, we explore prompting and preference learning with a unique speech-based preference we collected. both human and automatic evaluations show that both methods are useful, and combining them yields to additive benefits. further, we showed that speech-suitability is a nuanced factor that is not fully explained by shorter and more easily understandable responses. our work has focused on single-turn interactions and the content of the responses, and we hope to expand this investigation for multi-turn interactions and the acoustic aspect of speech-suitability.