Word Segmentation as Unsupervised Constituency Parsing (2022)

Abstract: Word identification from continuous input is typically viewed as a segmentation task. Experiments with human adults suggest that familiarity with syntactic structures in their native language also influences word identification in artificial languages; however, the relation between syntactic processing and word identification is yet unclear. This work takes one step forward by exploring a radically different approach of word identification, in which segmentation of a continuous input is viewed as a process isomorphic to unsupervised constituency parsing. Besides formalizing the approach, this study reports simulations of human experiments with DIORA (Drozdov et al., 2020), a neural unsupervised constituency parser. Results show that this model can reproduce human behavior in word identification experiments, suggesting that this is a viable approach to study word identification and its relation to syntactic processing.

## introduction when exposed to speech in an unknown language, humans are faced with the task of finding out what are the basic combinatorial units of the language, such as phonemes, syllables, words and phrases. since speech is continuous, humans need to rely on implicit cues -such as statistical information-to find out the building blocks of the language. one approach that studies which statistical cues can be used by humans in this task is artificial grammar learning (agl). experiments in agl are characterized by the use of artificial languages with carefully controlled statistical properties. to investigate word identification with this paradigm, participants in a typical agl experiment are first exposed to a speech-like sample of the artificial language (usually recorded with synthetic voice). then, they participate in a test that has been designed to show whether participants identified the words in the artificial language. to formalize theories of how humans identify words in agl tasks, a range of computational models have been proposed over the last two decades. these models have explained a wide arrange of phenomena, using a variety of algorithms such as bayesian inference @xcite , normative statistics @xcite , cognitively inspired processes implementing recognition or memorization @xcite , and neural networks @xcite . there is, however, one phenomenon that has not been addressed in the computational literature in agl: the fact that participant's knowledge of their native language influences performance in this type of agl experiments. in particular, results seem to be influenced by co-occurrence statistics of sublexical units @xcite @xcite , and interestingly, also by the presence of left-or right-branching syntactic structures in the native language, which predict the statistics that subjects use to identify words @xcite . one likely reason why this has not been the focus of prior models of word identification in agl is that we are in need a computational framework that can represent this information on the first place. while sensitivity to co-occurrences of sublexical patterns could potentially be accounted for with at least some of the existing models (in particular the neural network approaches, which should show similar output to input with similar representations), the influence of prior syntactic knowledge cannot be readily explained with the existing approaches, as none of these models incorporate syntactic processing. thus, a preliminary step before modelling the influence of prior knowledge is to develop a modelling framework that can relate word identification in agl to syntactic processing in the first place 1 . this work aims to fill this gap by presenting a radically different account of word identification that is isomorphic to syntactic processing: namely, word segmentation as unsupervised constituency parsing. this paper is structured as follows. section 2 reviews the experimental record that this work focuses on. the approach of modelling word segmentation as unsupervised constituency parsing is formalized in section 3. next, section 4 reports an empirical study using diora @xcite , an unsupervised neural inside-outside constituency parser. the results, reported in section 5, show that this approach can be effectively used to model human word identification in agl experiments with human adults. finally, implications of this new perspective on word identification are discussed in section 6, and directions for future studies are proposed in section 7. ## experimental record a long tradition of agl experiments have used artificial languages to discover how humans identify words from a continuous speech-like stream. studies show that humans can segment words based on statistics over syllables, such as frequency of co-occurence @xcite , transitional probabilities @xcite predictive dependencies between non-adjacent syllables @xcite @xcite , or phonotactic patterns @xcite . here, the focus is on the two experiments reported in @xcite (p&d onwards). these experiments showed that humans have the ability to keep track of both forward and backward transitional probabilities (as explained next) and use them for identifying words. it is precisely this ability that is susceptible of being influenced by prior syntactic knowledge @xcite , motivating the choice to focus on these experiments as a starting point. in experiment 1, the authors used an artificial language consisting of 9 bi-syllabic 'words', formed with combinations of 12 different syllables. there were two conditions in the experiment: forward and backward. in the forward condition, the first syllable of each word uniquely predicted the models that allow for some level of hierarchical representations have been proposed @xcite @xcite but have not been evaluated for unsupervised parsing at the syntactic level. the backward condition follows exactly the same design, except that it is the second syllable in the word which uniquely predicts the first: the participants were familiarized with a sample of synthesized speech of this language, consisting of a random concatenation of 115 repetitions of each word. with this design, the co-occurrence frequency of syllables within a word was 3 times larger than for syllables spanning word boundaries. the total duration of the recorded stream was 8 minutes, and there were no pauses or any other acoustic indication that separated the words. thus, the only two cues that participants could use to identify words were the tps between syllables and the co-occurrence frequency of syllables (as it was 3 times higher for syllables within words than for syllables spanning word boundaries). condition words forward ax, bx, cx, dy, ey, fy, gz, hz, iz backward xa, xb, xc, yd, ye, yf, zg, zh, zi after listening to this stream of artificial words, the participants were presented with a 2-alternative forced choice (2afc) test. each trial in the test consisted of a choice between a word of the language, and a 'partword', i.e. a sequence of two syllables that spanned across word boundaries. for instance, in the forward condition, a test trial could involve the word cx and the partword xd (see to disentangle the contribution of each cue, in a second experiment, the authors designed an artificial language in which the frequency of words and partwords in the familiarization stream was controlled. thus, the only way to identify words was to keep track of tps. results of experiment 2 showed that participants were statistically above chance in both conditions, with a slight advantage for the forward condition (although the difference between directions did not reach significance). the authors concluded that human adults can track tps in both directions, and use them to identify words in a continuous stream. ## formalization of the approach the approach presented this paper is to model the task of word identification from a continuous input using the same process for discovering syntactic constituents. a number of adaptations and considerations are required, as described next. ## simulations this section presents simulations with deep inside-outside recursive autoencoder (diora, @xcite , an unsupervised neural constituency parser. diora is an autoencoder network, trained with a fill-in-the-blank objective: it encodes all the words in a sentence except one in a single vector, and then decodes from this vector, predicting all the words (including the removed one). the encoder uses a chart to build a constituency tree, with each cell consisting of a weighted average all the possible subtrees covering the represented span. these subtrees are encoded as independent vectors with their corresponding score, both of which are computed recursively using a composition function. in a recent empirical comparison, diora exhibited some of the best results in unsupervised constituency parsing for english, and outperformed all the competing models in most of the experiments in japanese @xcite . to reproduce the original experiments in p&d, i trained diora with the input data generated according to the procedure described in 3.2 foot_2 . diora can be used with different composition functions: a multilayer feed-forward network (mlp), a version of the mlp that shares the inside and outside parameters (mp shared ), and a treelstm @xcite . the model can be optimized with either max-margin or cross-entropy loss (softmax). simulations are reported with all these variants, with the rest of hyperparameters fixed to the default values, except: batch @xmath0, hidden layer @xmath1, maximum @xmath2. i trained 30 individual models for each configuration and experimental condition. this is roughly the larger number of participants in the experimental conditions in p&d (@xmath3, and the models only differed in their initial state. at the end of training, the models were presented with the stimuli one more time, to produce the final parse trees that would be used for evaluation. the evaluation metric described in 3.3 was computed for each model, and -as in the original paper-the mean performance of the 30 models is submitted to a one-sided student's t-test to find whether the performance is significantly above chance level foot_4 . ## discussion from a computational perspective, word identification from continuous (artificial) input has always been portrayed as a segmentation task, concerned with breaking the continuous stream into combinatorial pieces. this work explores a completely different perspective, in which the identification of words is carried out with a syntactic constituency parser, which groups the syllables hierarchically into tree structures. the results for experiments 1 and 2 show that a model like diora (with treelstm and max-margin loss) can successfully reproduce human behavior in the experiments. from a mechanistic perspective, a tentative conclusion is that, when exposed to speech-like input in an unknown language, human adults group syllables that follow statistically coherent patterns, and this grouping is hierarchical -akin to the hierarchical structures attributed to syntax. how, then, does the process of identifying words relate to finding the syntactic relations between the identified words? given the hierarchical nature of the process, a possibility is that one single process builds a bottom-up hierarchy of units, grouping subword sequences into words and combining those into syntactic constituents. this is consistent with some usage-based theories of language @xcite , which deem all levels of grammatical analyses as homologous. this interpretation would explain the results in onnis and thiessen (2013), which show that humans identify words consistent with tps in the forward or backward direction, depending on grammatical patterns in the native language (in particular, the tendency for head-directionality). although diora reproduced, to a great extent, the pattern of results reported in p&d, there are some differences. to begin with, diora is better than humans in identifying words when those are more frequent than partwords. this is evidenced by the performance in experiment 1, as well as by the distribution of frequency counts reported in section 5.3. on the other hand, only one of the variants of diora identified words in experiment 2, when frequency information was removed. as shown above, there is large variance in the performance of the models, depending on their initial state. this is again consistent with the results observed in @xcite : in the absence of frequency information, humans seem to rely on prior knowledge to guide the discovery of words. nevertheless, to confirm whether the current results speak to the observed behavior in onnis and thiessen (2013), simulations using the same stimuli are required. thus, a prediction from this work is that pre-training the parser with korean or english could set a bias in the model to discover words based on either tp f w or tp bw . the fact that diora was successful in both english and japanese (a language that, like korean, has a tendency for left-branching syntactic structures) bodes well for such experiment @xcite . finally, it must be noted that, to fully understand the role of tps in word identification -specially in the absence of frequency cues-it would be useful to have experimental procedures with stricter tests, as the analyses of subjective frequencies revealed thatsuccess in the 2afc can be achieved with only a slight difference between words and partwords. ## conclusion this paper proposes a novel approach for word identification from continuous speech-like input: word segmentation as unsupervised parsing. using this framework with diora revealed that word identification in agl can be explained from the perspective of unsupervised constituency parsing, sug-gesting this framework can be effectively used to bridge the gap between models of word identification and syntactic syntactic processing. this work paves the way for addressing unanswered questions on the influence of syntactic knowledge in subsequent learning; in particular, an immediate next step for future work is to pre-train diora with head-first and head-last languages to find whether the model can be biased towards tracking forward or backward tps. the implications of this study are not limited to cognitive modelling: the use of techniques from natural language processing to investigate human learning can also be fruitful for this field. in particular, one finding is that, unlike humans, diora discovers constituents best when those are identifiable by the frequency of co-occurrence of the related units -rather than by transitional probabilities-. although this model was not designed to mimic human learning, incorporating the inductive biases of humans (i.e. a tendency for tracking forward or backward dependencies depending on the degree of left-or right-branchness of the language) may be a fruitful avenue to pursue, as humans are, after all, the best-performing syntactic parsers.