Beyond Laurel/Yanny: An Autoencoder-Enabled Search for Polyperceivable Audio (2021)

Abstract: The famous “laurel/yanny” phenomenon references an audio clip that elicits dramatically different responses from different listeners. For the original clip, roughly half the population hears the word “laurel,” while the other half hears “yanny.” How common are such “polyperceivable” audio clips? In this paper we apply ML techniques to study the prevalence of polyperceivability in spoken language. We devise a metric that correlates with polyperceivability of audio clips, use it to efficiently find new “laurel/yanny”-type examples, and validate these results with human experiments. Our results suggest that polyperceivable examples are surprisingly prevalent in natural language, existing for >2% of English words.

## introduction how robust is human sensory perception, and to what extent do perceptions differ between individuals? in may 2018, an audio clip of a man speaking the word "laurel" received widespread attention because a significant proportion of listeners confidently reported hearing not the word "laurel," but rather the quite different sound "yanny" @xcite . at first glance, this suggests that the decision boundaries for speech perception vary considerably among individuals. the reality is more surprising: almost everyone has a decision boundary between the sounds "laurel" and "yanny," without a significant "dead zone" separating these classes. the audio clip in question lies close to this decision boundary, so that if the clip is slightly perturbed (e.g. by damping certain frequencies or slowing down the playback rate), individuals switch from confidently perceiving "laurel" to confidently perceiving "yanny," with the exact point of switching varying slightly from person to person. how common is this phenomenon? specifically, what fraction of spoken language is "polyperceivable" in the sense of evoking a multimodal response in a population of listeners? in this work, we provide initial results suggesting a significant density of spoken words that, like the original "laurel/yanny" clip, lie close to unexpected decision boundaries between seemingly unrelated pairs of words or sounds, such that individual listeners can switch between perceptual modes via a slight perturbation. the clips we consider consist of audio signals synthesized by the amazon polly speech synthesis system with a slightly perturbed playback rate (i.e. a slight slowing-down of the clip). though the resulting audio signals are not "natural" stimuli, in the sense that they are very different from the result of asking a human to speak slower (see section 5), we find that they are easy to compute and reliably yield compelling polyperceivable instances. we encourage future work to investigate the power of more sophisticated perturbations, as well as to consider natural, ecologically-plausible perturbations. to find our polyperceivable instances, we (1) devise a metric that correlates with polyperceivability, (2) use this metric to efficiently sample candidate audio clips, and (3) evaluate these candidates on human subjects via amazon mechanical turk. we present several compelling new examples of the "laurel/yanny" effect, and we encourage readers to listen to the examples included in the supplementary materials (also available online at https://theory.stanford.edu/ ˜valiant/polyperceivable/index.html). finally, we estimate that polyperceivable clips can be made @xmath0. ## method to investigate polyperceivability in everyday auditory input, we searched for audio clips of single spoken words that exhibit the desired effect. our method consisted of two phases: (1) sample a large number of audio clips that are likely to be polyperceivable, and (2) collect human perception data on those clips using amazon mechanical turk to identify perceptual modes and confirm polyperceivability. ## results are the words we found polyperceivable? to identify cases where words had multiple perceptual "modes," we looked for clusters in the distribution of responses for each of the 29 candidate words. concretely, we treated responses as "bags of phonemes" and then applied k-means. though this rough heuristic discards information about the order of phonemes within a word, it works sufficiently well for clustering, especially since most of our words have very few syllables (more sophisticated models of phonetic similarity exist, but they would not change our results). we found that the largest cluster typically contained the original word and rhymes, whereas other clusters represented significantly different perceptual modes. some examples of clusters and their relative frequency are available in how prevalent are polyperceivable words? of our initial sample of 200 words, 11 ultimately yielded compelling demonstrations. to compute the prevalence of polyperceivable words in the population of the top 10k words, we have to account for the importance sampling weights we used when sampling in section 2.1. after scaling each word's contribution by the inverse of the probability of including that word in our nonuniform sample of 200, we conclude that polyperceivable clips exist for at least 2% of the population: that is, of the 16 voices under consideration, at least one yields a polyperceivable clip @xmath1. we emphasize that this is a conservative lower bound, because it assumes that there were no other polyperceivable words in the 200 words we sampled, besides the 11 that we selected for the second round. we did not conduct an exhaustive search among those 200 words, instead focusing our mechanical turk resources on only the most promising candidates. is s a good metric? we consider the metric s to be successful because it allowed us to efficiently find several new polyperceivable instances. if the 200 words were sampled uniformly instead of being importance-sampled based on s, we would only have found 4 polyperceivable words in expectation (2% of 200). thus, importance sampling increased our procedure's recall by almost 3×. for a more quantitative understanding, we analyzed the relationship between "autoencoder path length" s and "perceptual path length" t . our measure t of "perceptual path length" for a clip is change in average distance between source word and response as we slow the clip down from 0.75× to 0.6×. as with clustering above, distance perceptual instability in human sensory systems offers insight into ml systems. the question of what fraction of natural inputs lie close to decision boundaries for trained ml systems has received enormous attention. the surprising punchline that has emerged over the past decade is that most natural examples (including points in the training set) actually lie extremely close to unexpected decision boundaries. for most of these points, a tiny but carefully-crafted perturbation can lead the ml system to change the label. such perturbations are analogous to the slight perturbation in playback speed for the polyperceivable clips we consider. in the ml literature, these perturbations, referred to as "adversarial examples" seem pervasive across complex ml systems @xcite @xcite @xcite @xcite . while the initial work on adversarial examples focused on computer vision, more recent work shows the presence of such examples across other settings, including reinforcement learning @xcite , reading comprehension @xcite , and speech recognition @xcite . studying perceptual illusions would provide a much-needed reference when evaluating ml systems in these domains. for vision tasks, for example, human vision provides the only evidence that current ml models are far from optimal in terms of robustness to adversarial examples. however, while humans are certainly not as susceptible to adversarial examples as ml systems, we lack quantified bounds on human robustness. more broadly, understanding which systems (both biological and ml) have decision boundaries that lie surprisingly close to many natural inputs may inform our sense of what settings are amenable to adversarially robust models, and what settings inherently lead to vulnerable classifiers. perceptual instability in ml systems offers insight into human sensory systems. recent research on adversarial robustness of ml models has provided a trove of new tools and perspectives for probing classifiers and exploring the geometry of decision boundaries. these tools cannot directly be applied to study the decision boundaries of biological classifiers (e.g. we cannot reasonably do "gradient descent" on human subjects). however, using standard data-driven deep learning techniques to model human perceptual systems can allow us to apply these techniques by proxy. an example can be found in the study of "transferability." adversarial examples crafted to fool a specific model often also fool other models, even those trained on disjoint training sets @xcite @xcite . this prompts the question of whether adversarial examples crafted for an ml model might also transfer to humans. recent surprising work by @xcite explores this question for vision. humans were shown adversarial examples trained for an image classifier for ≈ 70ms, and asked to choose between the correct label and the classifier's (incorrect) predicted label. humans selected the incorrect label more frequently when shown adversarial examples than when shown unperturbed images. similarly, @xcite trained a lowdimensional representation of "perceptual space," and used the decision boundaries of the model to find images that confused human subjects. ## related work an enormous body of work from cognitive sciences communities explores the quirks of human/animal sensory systems @xcite . these works often have the explicit goal of exploring isolated "illusions" that provide insights into our perceptual systems @xcite . however, there are few efforts to quantify the extent to which "typical" instances are polyperceivable or lie close to decision boundaries. @xcite studies the effect of speaking rate on how listeners perceive phonemes. the perceptual shifts studied therein are between phonetically adjacent perceptions (e.g. "pip" vs. "peep") rather than dramatically different perceptions (e.g. "laurel" vs. "yanny"). the "perturbation" of increasing human speaking rate is much more complex than simply linearly scaling the playback rate of an audio clip. speaking-rate induced shifts also seem to hold more universally across voices, as opposed to the polyperceivable instances we examine. ## future work priming effects it is possible to use additional stimuli to alter perceptions of the "laurel/yanny" audio clip. for example, bosker (2018) demonstrates the ability to control a listener's perception by "priming" them with a carefully crafted recording before the polyperceivable clip is played. similarly, @xcite investigated the "mcgurk effect" @xcite , where what one "sees" affects what one "hears." the work estimated the fraction of spoken words that, when accompanied by a carefully designed video of a human speaker, would be perceived as significantly different words by listeners. such phenomena raise questions about how our autoencoder-based method can be extended to search for "priming-sensitive" polyperceivability. security implications just as adversarial examples for dnns have security implications @xcite @xcite , so too might adversarial examples for sensory systems. for example, if a video clip of a politician happens to be polyperceivable, an adversary could lightly edit it with potentially significant ramifications. a thorough treatment of such security implications is left to future work. ## conclusion in this paper, we leveraged ml techniques to study polyperceivability in humans. by modeling perceptual space as the latent space of an autoencoder, we were able to discover dozens of new polyper-ceivable instances, which were validated with mechanical turk experiments. our results indicate that polyperceivability is surprisingly prevalent in spoken language. more broadly, we suggest that the study of perceptual illusions can offer insight into machine learning systems, and vice-versa.