STEER : Semantic Turn Extension-Expansion Recognition for Voice Assistants (2023)

Abstract: In the context of a voice assistant system, steering refers to the phenomenon in which a user issues a follow-up command attempting to direct or clarify a previous turn. We propose STEER, a steering detection model that predicts whether a follow-up turn is a user’s attempt to steer the previous command. Constructing a training dataset for steering use cases poses challenges due to the cold-start problem. To overcome this, we developed heuristic rules to sample opt-in usage data, approximating positive and negative samples without any annotation. Our experimental results show promising performance in identifying steering intent, with over 95% accuracy on our sampled data. Moreover, STEER, in conjunction with our sampling strategy, aligns effectively with real-world steering scenarios, as evidenced by its strong zero-shot performance on a human-graded evaluation set. In addition to relying solely on user transcripts as input, we introduce STEER+, an enhanced version of the model. STEER+ utilizes a semantic parse tree to provide more context on out-of-vocabulary words, such as named entities that often occur at the sentence boundary. This further improves model performance, reducing error rate in domains where entities frequently appear, such as messaging. Lastly, we present a data analysis that highlights the improvement in user experience when voice assistants support steering use cases.

## introduction in the context of voice assistants, steering refers to the phenomenon in which a user issues a follow-up command attempting to direct or clarify a previous turn. however, the current state of voice assistants poorly supports steering, resulting in users having to restate their requests, causing disruptions in the natural flow of conversation and leading to a bad user experience. support for steering use cases in voice assistants enables users to provide * equal contribution unprompted follow-ups, clarifying or refining their previous requests; listing 1 presents several examples of steering use-cases. ## motivation the ability to handle steering for voice assistants is crucial in enhancing the overall user experience. firstly, it allows users to interact with voice as-sistants in a natural and efficient manner, without having to repeat their entire query when they want to refine or clarify a previous command. section 8.1 shows how steer can reduce user friction from this standpoint. secondly, we analyze how support for steering can improve conversational naturalness in section 8.2. in particular, support for steering allows users to pause more often, providing time for them to clarify, refine or adapt their queries through interactions. this is important in achieving more natural conversations, as humans typically have high-level intent before they speak, rather than having a fully formed query in mind. in particular, previous research has studied speech pauses in natural conversation @xcite and in queries to voice assistants @xcite . these studies have shown that pauses before spoken words tend to be longer when the cognitive load on the speaker is higher. in voice assistants, balancing between latency and accuracy is an important factor in determining how long the va system waits for a user to finish their turn @xcite . on one hand, a va may shorten wait time to prioritize responsiveness; on the other, this approach could result in under-specified queries, leading to unsatisfactory responses. steering opens up opportunities to endpoint more aggressively to reduce latency while not worried about ending a request prematurely, as users can just add on to their previous command. this requirement of responsiveness also places several limitations on the overall architecture and permissible model size and latency. thus, although recently popular large language models (such as @xcite ; @xcite ) are able to handle conversation end-to-end without a traditional pipelined approach (although an evaluation on how they perform on steering requests does not, to the best of our knowledge exist), they tend prohibitively expensive in terms of storage, memory, compute and inference time required, particularly if they were to be run completely on a low-power device, a setting in which voice assistants often operate. the examples in listing 1 highlight that queries suitable for steering are also inherently ambiguous. to leverage this observation, we propose using semantic parse trees (spts) of a query as an additional text-based modality for modeling. these spts are obtained as described in cheng et al. (2020) when a steerable request is ambiguous, it may reflect the lack of information to construct a comprehensive spt. this aspect can be captured by the model, facilitating more accurate steering detection. moreover, spts offer a hierarchical representation of tasks, targets, and entity names parsed from a user's query, commonly used in a va's natural language understanding system @xcite . raw text may not reflect the presence of entity names well, since they are likely out-of-vocabulary words for a model. this issue is particularly common in steering, as named entities frequently appear at sentence boundaries, as shown in figure 3 . employing spts as a feature complements raw text by organizing the request and labeling the entities present in it. ## data sampling sampling data for an unsupported task presents a challenge due to the cold-start problem: if a user attempts steering and the assistant fails to respond appropriately, the user is unlikely to attempt steering again. however, we observe that in face of an incomplete query that was incorrectly executed, users tend to reiterate the intended request in full again, resulting in a self-contained valid query. in light of this, we devised a data sampling strategy for positive data, where the follow-up intends to steer the context; and for negative data, where the followup is a separate request, illustrated in fig 1 . both data sampling processes start from an anonymized randomly sampled va dataset, leveraging heuristic rules to sample opt-in usage data to create an unsupervised training set. for positive data sampling, we first start by identifying reiterations from user. this is done by identifying consecutive turns where: 1. the previous turn is an exact prefix of the current turn; 2. the two turns happened within a short time difference. while this approach is simplistic, the resulting data is of surprisingly high quality. next, with pairs of reiterations, we synthetically infer what a user for positive data, during the data sampling phase, we follow the solid lines. given reiterations, synthetic steering follow-ups are generated. during model training, we follow the dotted lines. the model is provided with the context query and the follow-up request, then asked to predict if the follow-up is a steering request. for negative data, we use self-contained follow-ups found in the va logs. for negative data sampling, we capture natural non-steering follow-ups directed to the va. as is previously mentioned, in a va that doesn't support steering, existing steering use cases are extremely rare. therefore, we simply sample consecutive turns from the anonymized va usage logs of users that have opted in. the positive and negative datasets are combined in a 1:1 ratio, resulting in a dataset of four million samples in total. this combined dataset is then randomly split into training (80%), validation (10%), and testing (10%) sets. our model is thus trained on a positive set comprising solely of unlabeled data, obtained without any annotation. while our negative data is reflective of realworld usage, our positive dataset is derived from heuristics. to establish more certainty about this proxy dataset, we further evaluate the performance on a real-world dataset, in which steering followups are manually identified and annotated from opt-in data sampled from the va usage logs. ## model the steering detection task can be formulated as follows: given two turns, determine whether the second turn is a user's attempt to steer the first turn. we experimented with two variations of the transformer encoder architecture: the first model steer, solely utilizes the transcriptions of the turns. the second model, steer+, incorporates an additional feature: the encoding of a linearized semantic parse tree (spt) derived from the first turn. steer, depicted in figure 2 , follows the general architecture of a transformer encoder @xcite . it operates on tokenized queries and incorporates positional encoding and turn encoding, where the turn encoding denotes 0 for the first turn and 1 for the follow-up turn. the three encodings are projected to match the input size of the encoder and are then summed. the model consists of four transformer encoder layers, each comprising 128 hidden dimensions and 8 attention heads. following this, the output head implements mean pooling across the sequence dimension. this pooled output is then passed through a dense classification head for the final prediction. steer+, also illustrated in figure 2 , utilizes the same token, positional, and segment embeddings as the baseline model. however, it differs by incorporating a linearized semantic parse tree (spt) encoding, depicted in figure 2c . each unique tree node, excluding payloads, is assigned a node index. to represent the spt's structure, we introduce two additional indices: a depth index, encoding the node depth, and a sibling index, denoting the node's position among its siblings. as an example, the spt in listing 2 can be encoded as once the linearized spt is encoded into three groups of indices, we map them into three sequences of embeddings. these three spt embeddings are then summed together, and the sum is node node index depth index sibling index create:alarm v 0 0 .name.str("bedtime") w 1 0 .time.time x 1 1 .hour.int(10) y 2 0 .minute.int(30) z 2 1 ## experimental setup both models undergo training for a total of 300 epochs with a batch size of 256. a linear learning rate warmup is applied for the initial 30 epochs, from 1e-7 to 1e-4, followed by a linear learning rate decay throughout the remaining the epochs back to 1e-7, with early stopping. we used adamw as model optimizer and cross-entropy as loss function. we experimented with various training settings, such as a hyperparameter search on the learning rate and learning rate schedule, batch size variations, and other optimizers. additionally, we explored changes in the model architecture, including varying the number of transformer encoder layers from 2 to 6, and experimenting with pooling methods such as average pooling and max pooling. our experimental results are based on the bestperforming configuration of steer. it is important to note that our hyperparameter and architecture search was specifically done on steer, and we maintained an identical configuration for steer+ to ensure a fair comparison. all our experiments were conducted on systems with a single v100 gpu. on average, steer takes about 39 hours to train, and steer+ trains slightly faster, for around 38 hours due to early stopping. both steer and steer+ models are comparable in size, with steer having 4.5 million parameters and steer+ having 4.7 million. ## results task performance is evaluated based on prediction accuracy on a held-out test dataset randomly split from the training data. in comparison, steer+ exhibits improvement over the baseline across all data categories with statistical significance. it achieves a macro accuracy of 96.44% ± 0.03%. furthermore, within each data category, steer+ attains an accuracy of 96.47% ± 0.05% on consecutive reiteration data and 96.40% ± 0.06% on follow-up data. in addition to evaluating the models on the data collected using the heuristic sampling approach, we conducted a human grading task that involved gathering over 800 real-world steering examples. both models were evaluated to assess their capability zero-shot in practical scenarios. our data sampling strategy demonstrated strong alignment with real-world steering use cases, as both models achieved an accuracy of over 90%. moreover, steer+ showcased statistically significantly better performance, achieving an accuracy of 91.20% ± 0.16%, compared to steer with an accuracy of 90.71% ± 0.17%. ## analysis from in addition to model evaluation, we also assessed the benefits of a voice assistant system having steering support brings to end users. this analysis focus on two aspects: we first show that there is a substantial reduction in user friction. then, we demonstrate how support for steering improves conversational naturalness by allowing users to pause and formulate (or refine) their query. we present additional analysis in appendix a.