Edit-Constrained Decoding for Sentence Simplification (2024)

Abstract: We propose edit operation based lexically constrained decoding for sentence simplification. In sentence simplification, lexical paraphrasing is one of the primary procedures for rewriting complex sentences into simpler correspondences. While previous studies have confirmed the efficacy of lexically constrained decoding on this task, their constraints can be loose and may lead to sub-optimal generation. We address this problem by designing constraints that replicate the edit operations conducted in simplification and defining stricter satisfaction conditions. Our experiments indicate that the proposed method consistently outperforms the previous studies on three English simplification corpora commonly used in this task.

## introduction lexically constrained decoding @xcite @xcite allows to explicitly apply human knowledge in language generation, which has been employed in various tasks. applications include machine translation using a bilingual dictionary of technical terms as constraints @xcite , data augmentation using references as constraints @xcite , style transfer using style-specific vocabulary as constraints @xcite , knowledge-grounded generation @xcite , and commonsense reasoning using common concepts as constraints @xcite . the notable advantage of lexically constrained decoding is its direct applicability to decoders without the need for model (re)training. lexically constrained decoding is particularly appealing for sentence simplification, where the transformation of complex tokens to simpler ones is crucial. @xcite points out that learners of a foreign language need to know 95% of tokens in the input text to comprehend the text message. a recent evaluation metric for sentence simplification @xcite also focuses on word-level edit quality. previous studies employed negative and positive lexical constraints, i.e., to suppress the output of difficult tokens (negative constraints) @xcite and to encourage the output of their corresponding simpler alternatives (positive constraints) @xcite . these studies showed that such simple constraints significantly improve the quality of simplification. however, their naive constraints are not tight enough, which leads to sub-optimal simplification satisfying easier constraints yet failing on difficult ones. to address this problem, we design constraints more directed to simplification associated with stricter satisfaction conditions. sentence simplification conducts three edit operations to rewrite sentences, i.e., insertion, deletion, and substitution of tokens. we expand neurologic decoding @xcite to perform these edit operations via constrained decoding. figure 1 illustrates our method that constrains generation using a substitution operation during beam search. we evaluate our method using standard sentence simplification corpora that are publicly available, namely, turk @xcite , asset @xcite , and automets @xcite . the experiment results using oracle constraints extracted from references confirm that the proposed method largely outperforms previous methods utilising lexically constrained decoding. furthermore, the results when constraints are predicted by a simple model indicate the consistent efficacy of the proposed method over these baselines. the implementation of our edit-constrained decoding, as well as model outputs, are available at https: //github.com/t-zetsu/editconstraineddecoding. ## edit-constrained decoding we expand neurologic decoding @xcite to realise edit-based constraints, for its efficacy and computational efficiency when applied to sentence simplification @xcite . ## experiment settings we conducted two kinds of evaluations. the first evaluation ( §5) aims to compare the performance of lexically constrained decoding with different constraint designs. to make the effects of methodological differences clearer, this evaluation used oracle constraints extracted from references. the second evaluation ( §6) follows to observe the performance under a practical setting with predicted constraints. preceding these evaluations, this section describes the common experiment settings. ## evaluation with oracle constraints we first measure the performance of the proposed method under a setting where highly reliable constraints are given. this setting eliminates possible performance variations depending on the quality of constraints and clarifies how different lexically constrained decoding mechanisms contribute to sentence simplification. ## evaluation with predicted constraints in this section, we evaluate the proposed method under a practical setting, where constraints are predicted by a simple neural model. ## summary and future work in this study, we proposed the edit-constrained decoding for sentence simplification. our constraints directly replicate the edit operations involved in simplification and are associated with strict satisfaction conditions. these features allow us to avoid the sub-optimal generation observed in previous studies. the evaluation with oracle constraints confirmed that our method largely outperforms the previous methods using lexically constrained decoding. furthermore, the evaluation with a simple constraint prediction model confirmed consistent improvement by our method over the previous studies even with imperfect constraints. in future work, we will apply our editconstrained decoding to llms to further enhance the sentence simplification technology. as valentini et al. (2023) revealed, llms still lack the ability to control lexical complexities. the proposed method is promising as complementation on this issue. we will also improve the constraint prediction model, which should boost the quality of simplification by a large margin.