Consistent Joint Decision-Making with Heterogeneous Learning Models (2024)

Abstract: This paper introduces a novel decision-making framework that promotes consistency among decisions made by diverse models while utilizing external knowledge. Leveraging the Integer Linear Programming(ILP) framework, we map predictions from various models into globally normalized and comparable values by incorporating information about decisions’ prior probability, confidence (uncertainty), and the models’ expected accuracy. Our empirical study demonstrates the superiority of our approach over conventional baselines on multiple datasets.

## introduction the rapid advance of ai has led to the widespread use of neural networks in tackling complex tasks that involve multiple output decisions, which may be derived from various models @xcite . however, in many cases, these decisions are interrelated and must conform to specific constraints. for example, to comprehend procedural text, multiple neural models collaborate to establish temporal relationships between actions, reveal semantic relations, and discern entity properties like location and temperature @xcite @xcite . each model exhibits distinct decision characteristics, output sizes, uncertainty levels, and varying excepted accuracy levels. resolving inconsistencies and aligning these diverse neural decisions is crucial for a comprehensive understanding of the underlying process. in many instances, raw model outputs lack usability without enforcing consistency. in tasks like hierarchical image classification, with independent models for each hierarchy level, outputs should adhere to the known hierarchical relationships. for example, the combination "plant, chair, armchair" lacks validity and requires post-processing for downstream applications. a similar requirement extends to generative models in text summa-rization @xcite and image captioning @xcite . prior studies have proposed techniques for handling inconsistencies in correlated decisions during both inference @xcite @xcite @xcite and training @xcite @xcite of neural models. this paper focuses on resolving these inconsistencies at inference, where the goal is to ensure that outputs align with task constraints while preserving or enhancing the original model performance without training. in addressing decision inconsistencies, integer linear programming (ilp) @xcite stands out as a robust approach. ilp is a global optimization framework that seeks to find the best assignments to variables while meeting specified constraints. it is known for its efficiency and capability to produce globally optimal solutions, distinguishing it from alternatives like beam search. the ilp formulation is as follows: objective : maximize p ⊤ y subject to c (y) ≤ 0, the mapping of logical constraints to y equations introduced in the domiknows (faghihi et al., 2021) framework. for instance, in order to map the mutual exclusivity constraint in a multi-class classification task c with n possible outputs, where decision variables over a single input are expressed by {(y foot_0 c , p 1 c ), (y 2 c , p 2 c ), ..., (y n c , p n c )}, the constraint is expressed as n @xmath0. ignoring other variables and constraints, the optimization problem becomes, in this simplification, since the problem is to find y i c values in integer space, the best solution sets the y i c value to 1 for the ith element that has the largest p i c . the rest of the values are set to zero. previous use of ilp has proven effective in ensuring decision consistency in certain cases @xcite but did not address model heterogeneity. this problem becomes more dominant in scenarios where output probabilities come from independent models, making them less directly comparable. to address this limitation, we extend the ilp formulation beyond just considering the raw model probabilities. instead, we map these raw scores into globally comparable values, facilitating a more balanced global optimization. we achieve this by incorporating additional information, such as decision confidence, expected model accuracy, and estimated prior probabilities. while previous studies have explored the integration of uncertainty in modeling the training objective @xcite @xcite , our work represents a novel effort in systematically incorporating multiple factors of this nature into the inference process for interrelated decisions to leverage external knowledge effectively. the methods proposed in this paper are now publicly available and have been properly integrated into the ilp inference pipeline of the domiknows framework 1 . ## method figure 1 shows an overview of our general framework and its components. our objective is to devise an improved scoring system, generating new local variable weights (importance) w in the ilp formulation. thus, we modify the original objective function as follows: ## empirical study we assess the impact of integrating proposed factors into the ilp formulation on a series of structured prediction tasks. our approach is particularly suited for hierarchical structures encompassing multiple classes at different granularity levels, such as classical hierarchical classification problems. additionally, we are the first to investigate the influence of enforcing global consistency on the procedural reasoning task, a complex real-world problem. to implement our method, we rely on the domiknows framework @xcite , offering a versatile platform that enables implementing and evaluating techniques to leverage external logical knowledge with minimal effort on structured output prediction tasks. ## conclusion this paper introduced an approach for taking into account the uncertainty and confidence measures, including the decisions' prior probability, entropy, and expected accuracy, alongside raw probabilities when making globally consistent decisions based on diverse models. through experiments on four datasets, we demonstrated the effectiveness of incorporating our idea within the ilp formulation. this contribution presents a high potential in advancing large models by integrating them into a unified decision-making framework for conducting complex tasks requiring interrelated decisions.