Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing (2024)

Abstract: We present the structured average intersection-over-union ratio (STRUCT-IOU), an evaluation metric that compares a constituency parse tree over automatically recognized spoken word boundaries with the ground-truth parse tree over written words. To compute the metric, we (1) project the ground-truth parse tree to the speech domain by forced alignment, (2) align the projected ground-truth constituents with the predicted ones under certain structured constraints, and (3) calculate the average IOU score across all aligned constituent pairs. STRUCT-IOU takes word boundaries into account and overcomes the challenge that the predicted words and ground truth may not have perfect one-to-one correspondence. Extending to the evaluation of text constituency parsing, we demonstrate that STRUCT-IOU shows higher tolerance to syntactically plausible parses than PARSEVAL (Black et al., 1991).

## introduction automatic constituency parsing of written text @xcite and speech transcriptions @xcite , inter alia), as representative tasks of automatic syntactic analysis, have been widely explored in the past few decades. appropriate evaluation metrics have facilitated the comparison and benchmarking of different approaches: the parseval f 1 score @xcite has served as a reliable measure of text parsing across scenarios; for speech transcription parsing, the sparseval @xcite metric extends parseval and accounts for speech recognition errors by allowing word-level editing with a cost. 1 we open-source the code of struct-iou at https: //github.com/explorerfreda/struct-iou . best viewed in color, where nodes with the same color are aligned. numbers in parentheses are the starting and ending times of the corresponding spans (in seconds). recent work @xcite has proposed a new task of textless speech constituency parsing. in contrast to earlier work that parses manually labeled @xcite , inter alia) or automatic @xcite , inter alia) speech transcriptions, these models construct constituency parse trees over automatically recognized spoken word boundaries, where each word is represented with a time range of the spoken utterance, without using any form of text. to evaluate these textless models, we need a metric that compares the predicted tree (over spoken word boundaries) with the manually labeled ground-truth tree (over written words) and faithfully reflects the parsing quality. since the automatically recognized word boundaries may be imperfect, the metric should also reflect the changes in parsing quality due to word boundary errors. to the best of our knowledge, none of the existing metrics meets these requirements, as they are all designed to compare parse trees over discrete word sequences, instead of continuous time ranges. motivated by the need for textless speech parsing evaluation, in this paper, we introduce the structured average intersection-over-union ratio (struct-iou; figure 1 ), a metric that compares two parse trees over time ranges. we relax the definition of segment trees @xcite to represent speech constituency parse trees, where each node is associated with an interval that represents the time range of the corresponding spoken word or constituent. to obtain the "ground-truth" speech parse trees, we use the forced alignment algorithm @xcite , a supervised and highly accurate method that aligns written words to time ranges of the corresponding spoken utterance, to project the ground-truth text parses onto the time domain. struct-iou is calculated by aligning the same-label nodes in the predicted and ground-truth parse trees, following structured constraints that preserve parent-child relations. the calculation of struct-iou can be formulated as an optimization problem ( §4.1) with a polynomial-time solution ( §4.2) in terms of the number of tree nodes. although struct-iou is designed to evaluate speech parsing, it is also applicable to text parsing evaluation. we analyze struct-iou for both purposes: in speech parsing evaluation, struct-iou robustly takes into account both the structure information and word boundaries; in text parsing evaluation, while maintaining a high correlation with the parseval f 1 score, struct-iou shows a higher tolerance to potential syntactic ambiguity. ## related work text constituency parsing and evaluation. in the past decades, there has been much effort in building and improving constituency parsing models @xcite @xcite @xcite @xcite @xcite . parseval @xcite has been the standard evaluation metric for constituency parsing in most scenarios, which takes the ground truth and predicted trees and calculates the harmonic mean of precision and recall of labeled spans. for morphologically rich lan-guages, tedeval @xcite extends parseval to accept multiple morphological analyses over sequences of words. a few alternative approaches have been pursued to address the potential mismatch in words and sentences @xcite . all these metrics are designed to evaluate parses over discrete word sequences, and cannot be easily extended to evaluate speech parses over continuous time ranges. although our metric, struct-iou, is designed to evaluate speech constituency parsing, it can be easily extended for text parsing evaluation, reflecting a different aspect from existing metrics ( §5.2). ## preliminaries we use real-valued open intervals to represent speech spans for simplicity, although most of the following definitions and conclusions can be easily extended to closed intervals and half-open intervals. proof of each corollary and proposition can be found in appendix a. 3.1 open interval operations definition 1. the length of a real-valued open interval @xmath0, b), where @xmath1, is |i| = ba. definition 2. the intersection size of open intervals i 1 and i 2 is i(i 1 , i 2 ) = 0 if i 1 ∩ i 2 = ∅ |i 1 ∩ i 2 | otherwise. definition 3. the union size of open intervals i 1 and i 2 is u(i 1 , i 2 ) = |i 1 | + |i 2 | -i(i 1 , i 2 ). definition 4. the intersection over union (iou) ratio between open intervals i 1 and i 2 is iou(i 1 , i 2 ) = i(i 1 , i 2 ) u(i 1 , i 2 ) . throughout this paper, we will use iou as the similarity metric between two intervals. ## conclusion and discussion in this paper, we present struct-iou, the first metric that computes the similarity between two parse trees over continuous spoken word boundaries. struct-iou enables the evaluation of textless speech parsing @xcite , where no text or speech recognizer is used or available to parse spoken utterances. in the canonical text and speech parsing settings, struct-iou complements the existing evaluation metrics @xcite @xcite . even for the evaluation of english constituency parsing, struct-iou shows a higher tolerance to potential syntactic ambiguity under certain scenarios, providing an alternative interpretation of the parsing quality. faithful evaluation of parsing quality is crucial for developing both speech and text parsing models. in supervised parsing, it has been common sense that higher evaluation metric scores (i.e., parse-val f 1 ) imply better models. however, the misalignment between linguistically annotated ground truths and model predictions, especially unsupervised parsing model predictions, does not necessarily indicate poor parsing quality of the models @xcite -instead, the models may have learned different but equally valid structures. conversely, annotations made by linguistic experts (such as the penn treebank) may exhibit discrepancies when compared to the responses of native speakers who lack formal linguistic training. we suggest that future work investigate what properties of the parses are emphasized by each evaluation metric, and consider multi-dimensional evaluation metrics @xcite .