Benchmarking and Improving Compositional Generalization of Multi-aspect Controllable Text Generation (2024)

Abstract: Compositional generalization, representing the model’s ability to generate text with new attribute combinations obtained by recombining single attributes from the training data, is a crucial property for multi-aspect controllable text generation (MCTG) methods. Nonetheless, a comprehensive compositional generalization evaluation benchmark of MCTG is still lacking. We propose CompMCTG, a benchmark encompassing diverse multi-aspect labeled datasets and a crafted three-dimensional evaluation protocol, to holistically evaluate the compositional generalization of MCTG approaches. We observe that existing MCTG works generally confront a noticeable performance drop in compositional testing. To mitigate this issue, we introduce Meta-MCTG, a training framework incorporating meta-learning, where we enable models to learn how to generalize by simulating compositional generalization scenarios in the training phase. We demonstrate the effectiveness of Meta-MCTG through achieving obvious improvement (by at most 3.64%) for compositional testing performance in 94.4%.

## benchmark: compmctg we propose compmctg, a novel benchmark to comprehensively evaluate the compositional generalization capacity of mctg approaches. the superiority and novelty of compmctg are out of its scale of dataset and its three-dimensional evaluation protocol (section 3.1). we select eight representative baseline approaches (section 3.2), evaluate their performance on our compmctg benchmark, and unveil their struggling on compositional testing (section 3.3). moreover, system-atic analysis towards exploring the behaviors of baseline approaches under different evaluation protocols of compmctg is provided in section 3.4, which highlights: 1) its capacity to dig out the potential generalization risk of evaluated approaches and 2) the undervalued compositionality gap in the previous work @xcite as well. ## methodlogy: meta-mctg in section 3.4, we observe that joint-training-based (both parameter-efficient fine-tuning based and allparameter fine-tuning based) baselines generally achieve better overall performance. nonetheless, there still exist non-negligible compositional generalization gaps for all these baselines, which highly calls for our attention. to this end, we propose meta-mctg, a novel meta-learning (finn et al., re-composed novel condition ... base mctg training phase compositional testing phase gpt-2()❄ + the book pos sport present ( ) ❄ ( ) ( ) gpt-2()❄ gpt-2()❄ + + pos sport past neg moviepresent text text control conditions train batch ℬ + + pos movie past neg sport present text text pseudo-comp batch ℬ lebron james delivered a stellar performance in yesterday's game . logit_1 logit_2 logit_3 logit_x text: output: ... logit_1 logit_2 ... logit_9 the movie last night truly uplifted my spirits 2017) based mctg training framework, to further improve compositional generalization capabilities of existing joint-training baselines. the framework is easy to implement and can be directly combined with any joint-training-based methods. we discuss the design of meta-mctg in section 4.1 and demonstrate its effectiveness through experiment results for meta-mctg in combination with three competitive joint-training baselines (ctrl (keskar et al., 2019), contrastiveprefix (qian et al., 2022a) and dcg (zeng et al., 2023)) in section 4.2. ## conclusion we propose compmctg, the first holistic benchmark targeting compositional generalization for multi-aspect controllable text generation (mctg), and conduct extensive experiments on compmctg with eight representative mctg baselines and two llm baselines, unveiling noticeable compositional generalization risk in them and demonstrating the effectiveness of compmctg. in addition, we propose meta-mctg, a framework incorporating meta-learning into the mctg training process to improve its compositional generalization ability, which can be combined with any joint-training-based mctg methods.