Breeding F illmore’s Chickens and Hatching the Eggs: Recombining Frames and Roles in Frame-Semantic Parsing (2021)

Abstract: Frame-semantic parsers traditionally predict predicates, frames, and semantic roles in a fixed order. This paper explores the ‘chicken-or-egg’ problem of interdependencies between these components theoretically and practically. We introduce a flexible BERT-based sequence labeling architecture that allows for predicting frames and roles independently from each other or combining them in several ways. Our results show that our setups can approximate more complex traditional models’ performance, while allowing for a clearer view of the interdependencies between the pipeline’s components, and of how frame and role prediction models make different use of BERT’s layers.

## introduction framenet @xcite is a computational framework implementing the theory of frame semantics @xcite . at its core is the notion of linguistic frames, which are used both for classifying word senses and defining semantic roles. for example, in (1), "bought" is said to evoke the com-merce buy frame, and "chuck", "some eggs", and "yesterday" instantiate its associated roles. ## on pipelines, chickens, and eggs according to @xcite , an essential feature of a frame is that it is "any system of concepts related in such a way that to understand any one of them you have to understand the whole structure in which it fits." in particular, linguistic frames are systems of semantic roles, possible predicates, and other semantic information. in this section, we discuss the relationship between these concepts in the context of frame-semantic parsing and highlight interdependencies between the various components. ## dissecting the pipeline we argued that the different components of the frame-semantic parsing task are mutually dependent on each other. here, we take a more practical view and re-define the parsing problem in a way that allows for experimenting with individual parts of the pipeline and different combinations of them. ## re-composing an end-to-end system having defined a basic setup for experimenting with predicting frames and roles alone, we can now design experiments for investigating any interactions between frames and roles. figure 2 provides an overview of the possible ways for combining the framesonly and rolesonly models: simply merging the outputs, predicting the two tasks jointly, or using a sequential pipeline. ## experiments 6.1 setups all experiments were run using a pre-trained bert-base-cased model, fine-tuned with a simple feedforward network decoder. while our main aim remains a deeper understanding of the components of frame-semantic parsing and their interdependencies, we still need to put our scores into perspective and legitimize our sequence labeling approach. thus, we took open-sesame, the only existing, open-source model that we are aware of that is capable of producing end-to-end predictions, as our baseline. 7 we used default settings (i.e., without scaffolding and ensembling) for better comparability with our own models. hence, note that the results reported here for open-sesame are not the state-of-the-art results reported by @xcite . ## conclusions we examined the frame-semantic parsing pipeline theoretically and practically, identifying 'chickenor-egg' issues in the dependencies between subtasks, and studying them empirically within a bert-based sequence-labeling framework. we found that joint frame and role prediction works well, but not always better than using frames as input to roles. by contrast, previous studies @xcite found substantial improvements from joint prediction. however, these systems use gold targets as input, the main advantage of our sequence-labeling setup is the possibility to investigate frame and role prediction independently, as well as their mutual dependency. we found substantial benefits for role prediction from access to frame information through joint prediction or by receiving frames as input. for frame prediction, instead, the picture is less clear: while we found a theoretical benefit of using (gold) roles as input, this benefit disappears when using predicted roles. similarly, when jointly predicting frames and roles, the multitask setup yielded a slight improvement for frame prediction, whereas multilabel deteriorated it. these results can be taken as supporting the traditional pipeline approach, but our results using semeval evaluation, which looks at full frame structures, do not unequivocally confirm this: open-sesame performs best, but amongst our models, reverse-traditional and merged outperform the others, including neo-traditional. this suggests that there might be valid alternatives to the standard pipeline, and exploring these might lead to a deeper understanding of frame semantic parsing task itself. our setup also allows for investigating which bert layers both components use. role prediction strongly prefers high bert layers, while frame prediction is less picky, suggesting that the tasks use different linguistic information. we see several logical extensions of our work. first, qualitative analysis of the overlaps in predictions from different models could shed light on the discrepancies between sequence labeling scores, consistency scores, and semeval scores. a second direction would be to explore how our observations about the relationship between different components of the frame semantic parsing pipeline and bert layers could be used to improve models. finally, one could try more sophisticated architec-tures for sequence-labeling models, in particular by enforcing frame-role consistency within the model itself rather than during post-processing.