E LICITATION I NFERENCE O PTIMIZATION FOR M ULTI -P RINCIPAL -A GENT A LIGNMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
In multi-principal-agent alignment scenarios spanning governance, markets, diplomacy, and AI, it is infeasible to elicit every principal’s view on all perspectives relevant to agent decisions.
Elicitation inference optimization (EIO) aims to minimize the n elicitations needed to approximate N principal’s views across K perspectives.
In this work, we demonstrate an EIO approach where data efficiency ( NK/n ) increases with scale.
We introduce STUMP: an elicitation inference model which integrates an LLM with a latent factor model to enable learning transfer across samples, contexts, and languages.
Then, we characterize STUMP’s performance on a set of elicitation primitives from which scalable elicitation (sampling) protocols can be constructed.
Building from these results, we design and demonstrate two scalable elicitation protocols for STUMP where data efficiency grows boundlessly, scaling like O ( n ) in the number of elicitations n .
This makes it possible to obtain complex, high-dimensional preference signals spanning principal populations at any scale.
I NTRODUCTION
The principal-agent problem involves aligning agent decisions with principal interests.
A challenge is creating situations where agent choices are sufficiently influenced by signals containing principal preferences.
With a single principal, high-complexity preference signals can be elicited directly via open-ended interaction.
Multi-principal-agent scenarios can involve large populations of principals and powerful agents such as: governments & citizens (Giger and Lefkofridi, 2014; Gabriel, 2020), firms & customers(Roberts and Grover, 2012), peacekeepers & conflict parties (United Nations, 2012), and potentially even transformative AI & humanity (Russell et al., 2015; Christiano et al., 2017).
As the number of principals grows, and the domain of agent decisions becomes open-ended, directly eliciting the preference of all principals on all relevant perspectives becomes unfeasible [A.1].
As a result, lower-complexity forms of elicitation like ballot voting (ie. for governments) and price signals (ie. for firms) are used to learn preferences.
While clearly effective—a basis of democracy and the economy—these approaches drastically simplify real preferences.
For example, they do not allow citizens (principals) to express what they would like a government (agent) to do, or why , only if they support predefined options.
Elicitation inference optimization (EIO) aims to decrease the amount of direct elicitation needed to recover a preference signal (enabling the use of more complex, higher-dimension preferences; e.g. in natural language).
Consider an N × K matrix Θ where rows correspond to N principals, columns correspond to K perspectives, and every element captures a principal-perspective relationship.
The goal of elicitation inference optimization is to obtain a sufficient approximation of Θ with a minimal elicitation budget by directly sampling some elements and inferring the rest.
Thus, EIO involves combining a) a sparse elicitation (sampling) protocol with, b) an elicitation inference model.
Closed-ended surveys simplify EIO by constraining the set of relevant perspectives to a predefined set — typically, with K << N .
Matrix sampling techniques elicit responses from each participant on a subset of perspectives selected randomly (Shoemaker, 1973), heuristically (Raghunathan and Grizzle, 1995), or dynamically such that inference accuracy is adaptively optimized (Gonzalez and Eltinge, 2008; Zhang et al., 2020).
Inference models exploit the low-rank-ness of Θ via matrix factorization and other collaborative filtering techniques (Zhang et al., 2020; Sengupta et al., 2021; Oliveira et al., 2021).
Collective dialogue systems (CDS) enable arbitrarily high-complexity signals by allowing the set of relevant perspectives to be open-ended: neither limited in number nor requiring pre-definition.
In a CDS, participants contribute open-ended perspectives in the context of a question, prompt, or conversation and respond to subsets of perspectives contributed by others – typically, yielding
K ∼ O ( N ) .
Inspired by preference models (Thurstone, 1927) and conjoint analysis (Green and
Srinivasan, 1978), one class of approaches elicits randomized pair-choice votes (Konya and Slodov, 2015) and uses hierarchical probit-like models for inference (Salganik and Levy, 2015).
Other approaches elicit agreement votes to learn absolute human-perspective relationships.
Polis samples agreement votes semi-randomly, prioritizes votes that improve clustering, and uses mean imputation to support dimensionality reduction (Small et al., 2021).
Additional work bridges these approaches by randomly sampling both pair-choice and agreement votes; inferring missing votes by modeling both vote types with a single utility matrix learned via regularized matrix completion (Bilich et al., 2019).
Previous approaches to EIO for CDS have been limited in scope to single samples where the set of relevant perspectives all correspond to the same context (ie.
the same question or prompt) (Salganik and Levy, 2015; Konya and Slodov, 2015; Small et al., 2021; Bilich et al., 2019).
This means votes must be elicited from every person for every new context.
As a result, to approximate some Θ spanning an arbitrary number of contexts, the number of direct elicitations needed grows proportional to the number of elements in Θ [A.2] – data efficiency does not increase with scale.
However, previous approaches do not fully leverage all available data: learning from one context is not transferred to support inference in other contexts, and the information contained in the perspective text is not used at all.
Can an EIO approach that better leverages available data, enable increasing data efficiency with scale?
In this work we introduce an approach to EIO for CDS which becomes increasingly data efficient as the amount of data elicited grows.
First, we introduce a novel elicitation inference model – STUMP – which better leverages available data by integrating a pre-trained LLM with a latent factor model.
Second, we characterize STUMP’s performance across a range of elicitation primitives from which arbitrarily scalable elicitation (vote sampling) protocols can be constructed.
Finally, building on these results, we design and demonstrate two arbitrarily scalable elicitation protocols where STUMP can infer with meaningful accuracy while data efficiency increases linearly with scale.
P ROBLEM S ETUP
Elicitation form .
In a collective dialogue system (CDS) participants respond to prompts with open-ended perspectives and vote on perspectives submitted by others.
Let H be the set of N human participants, P be the set of K perspectives, and Θ be the corresponding N × K humanperspective matrix.
Both agreement and pair-choice votes may be elicited on perspectives.
In agreement exercise e aij , participant i is asked if they agree with perspective j .
In pair choice exercise e cijk , participant i is asked if they prefer perspective j or k .
We denote a set of exercise data
E = { e aij , e cijk | i ∈ H ; j, k ∈ P } .
Elicitation inference involves training a model on available data and predicting all missing elements in a target Θ .
We choose binary agreement as the human-perspective relationship for our target Θ .
Inference performance is probed experimentally by training on a subset of exercises including both vote types E t , and computing prediction accuracy on a validation set of only agreement votes E v .
Data efficiency in EIO can be quantified as the ratio of a) the number of votes contained in the target Θ and b) the number of directly elicited votes a model needs to be trained on to meaningfully approximate Θ .
We denote the data efficiency to achieve inference accuracy acc as:
[Equation]
where n ( X ) denotes the size of set X and n t ≡ n ( E t ) .
We refer to the rate data efficiency changes with more elicited votes ( ∂β acc /∂n t ) as data leverage .
EIO becomes more data efficient with scale if data leverage is positive for arbitrarily large n t above a sufficient threshold of accuracy.
To achieve this, the challenge is to combine: a) an arbitrarily scalable elicitation (sampling) protocol where NK grows faster than n t , and b) an inference model that sustains meaningful inference accuracy at an arbitrary scale using the data the protocol generates.
E LICITATION I NFERENCE M ODEL
Elicitation inference (EI) is analogous to collaborative filtering (CF) in recommenders, except instead of items people may prefer, there are perspectives people may agree with.
A standard approach to CF is to learn a low-rank approximation of a user-item preference matrix by optimizing an objective function against observed preferences (Johnson, 2014).
Bilich et al.
(2019) demonstrated a similar approach for EI on CDS data.
They use regularized matrix completion with a low-rank-promoting nuclear norm prior similar to Fazel et al.
to learn a utility matrix that predicts votes.
We refer to this as the nuclear norm model (NN), and adopt it as a baseline (see A.3 for technical details).
Latent factor models (LFMs) learn vectors for each user
z i and item w j which embed them in the same latent space, and describe the preference of user i for item j as a function of u ij =
z i · w j (Johnson, 2014).
Known item features (i.e., director of a movie) can be used to improve performance by projecting features into latent space as a component of item embeddings (Zhang et al., 2021).
Relevant features can also be extracted from item content (i.e., a song’s time sequence) by a link to latent space embeddings (van den Oord et al., 2013).
We consider EI on text-based perspectives.
Collaborative Topic Ranking (CTR) uses information in text by learning a topic model on item texts and equating topic feature vectors to latent space embeddings (Xin et al., 2015).
Pre-trainable large language models (LLMs) (Cer et al., 2018; Devlin et al., 2018; Yang et al., 2019) learn to encode information from text into embeddings suitable for a wide range of downstream tasks (Men; Hao et al., 2020; Schick and Schütze, 2021).
Our approach to using the information in perspective texts is to start with a pre-trained LLM, then similar to Zhang et al.
(2021), learn a transformation of text embeddings into latent space.
A key challenge in CF is maintaining alignment between latent spaces learned from non-overlapping data (Cremonesi and Quadrana, 2014).
Sharing the same embedding model and transformation across perspectives enforces alignment even in the absence of overlapping data (see section 4.3).
Cross-Domain recommendation (CDR) involves aligning latent spaces across items in different domains like movies and songs (Zang et al., 2022).
One class of approaches uses information like tags that can be shared across domains (Sahu et al., 2018).
Others focus on aligning latent space distributional properties (Liu et al., 2021).
In EI the analog to CDR is multi-context inference.
How one votes on a perspective depends on the context it was elicited in.
Take the perspective " I feel happy.
" Consider two scenarios, where it was elicited by asking " What would you say if you saw your parents smile? " or " What wouldn’t you say if you saw your parents smile? ".
One would expect nearly opposite voting behavior.
But, if the same embedding model and transformation are shared in both contexts, then predictions will be the same.
Enabling multi-context inference requires relaxing this condition.
In the spirit of Sahu et al.
our approach is to share the same LLM across contexts, but learn a unique transformation into latent space for each context.
Our model.
We formally introduce our model, the Semantic Transfer Utility Model of Participants:
STUMP.
We denote the event that participant
i reports agreement with perspective j by a ij , and disagreement by d ij .
We denote the event that participant
i reports preference for perspective j over perspective k by c ijk .
We assume a logit utility model of choice similar to Johnson (2014);
Bilich et al.
(2019) where participant i has a latent utility u ij corresponding to perspective j and voting probabilities are given by p ( a ij | u ij )
= σ ( u ij ) , p ( d ij | u ij ) =
σ (1 − u ij ) and p ( c ijk | u ij ) =
σ ( u ij − u ik ) , where σ denotes the logistic function.
The utility u ij for participant i and perspective j is constructed as an LFM:
[Equation]
where z i is a latent embedding for participant i , and w j is a latent embedding for perspective j .
To use information in perspective text, we employ a pre-trained LLM to generate text embeddings.
We denote the embedding of the j th perspective text p j by model F with parameters ϕ , as: [Equation]
Text embeddings are mapped into latent space by context-specific transformations:
[Equation]
where M q and b q are the transformation weight and bias for context q (see Figure 1).
Finally, let
A = { i, j | a ij } , D = { i, j | d ij } , and C
=
i, j, k | c ijk } .
The log loss is defined by:
L
[Equation]
( (cid:88)
[Equation]
where ∥
∥
F denotes the Frobenius norm of a matrix, and λ m is a hyper-parameter controlling regularization strength.
The participant embedding vectors z and the set of transformation parameters for each context { M q , b q } are learned by minimizing the log-loss for observed data.
Optionally, the
LLM parameters ϕ may be fine-tuned through the same minimization.
E LICITATION P RIMITIVES
Our strategy for elicitation protocol design begins with characterizing STUMP’s performance across a range of elicitation primitives from which arbitrarily scalable protocols can be constructed.
To this end, we explore four types of elicitation primitives arising from two orthogonal data set properties: context homogeneity and sample overlap .
Context homogeneity.
Perspectives and votes are elicited under some context — for example, in response to a question.
Thus, a data set (or Θ ) may correspond to one or more contexts.
We denote the set of perspectives corresponding to the q th context as P q ⊆ P .
Sample overlap.
Vote data E may come from one or more samples, where each s th sample involves a subset of humans H s ⊆ H and a subset of perspectives P s ⊆ P .
Two samples are said to have no overlap when they share neither humans nor perspectives, ie H 1 ∩ H 2 = ∅ and P 1 ∩ P 2 = ∅ .
A sample with overlap may correspond to a data set with only a single sample, or with multiple samples where either humans or perspectives are shared, ie H 1 ∩ H 2 ̸ = ∅ or P 1 ∩ P 2 ̸ = ∅ .
Permuting these two properties gives four types of elicitation primitives (Figure 2, Table 1):
4. E LICITATION PRIMITIVE EXPERIMENTS
Baselines.
We use three prediction baselines to contextualize STUMP’s performance: a) a Random baseline where votes are predicted randomly with uniform probability, b) a Most Common (MC)
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
[Figure]
[Figure]
baseline where votes are predicted to be the most common choice observed for each perspective, and c) the nuclear norm (NN) model from Bilich et al (Bilich et al., 2019)
[A.3] which represents a standard approach to CF and is the only prior work we are aware of focused on single-vote-level inference for CDS.
Data and implementation .
Data used for experiments was collected as part of regular load testing for an existing CDS and de-identified prior to use in this study.
Data came from 534 participants solicited through online research panels.
We use data corresponding to seven questions [A.9] asked during the CD, chosen for their nuance and presumed hard-to-predict nature.
Each question is treated as a unique context, meaning only perspectives and votes elicited in response to the same question share the same context transformation weights.
Experiments are constructed by partitioning subsets of data into training and validation sets ( E t and E v ).
Data and training details are described in [A.4].
4. T YPE I: S INGLE CONTEXT , OVERLAPPING SAMPLE
Data elicited from one question during a collective dialogue (CD) represents a type I primitive.
We test type I model performance as a function of vote per participant (VPP) for two kinds of CD; asynchronous and synchronous.
In asynchronous CDs, participation happens on a rolling basis, so participants only vote on perspectives submitted prior to their participation.
In synchronous CDs, all participation happens in the same minute-scale window of time, so any perspective may be voted on by any participant.
We simulate both cases while varying VPP by partitioning the appropriate votes into training and validation sets [A.5.1].
For each experiment, we train the models for each question and compute the corresponding validation accuracy.
STUMP outperforms all baselines at all VPP for both cases with < 10 VPP and reaches peak performance at ∼ 20 VPP (Figure 3).
Type I as a baseline.
As the simplest elicitation primitive, Type I performance across baselines informs what constitutes meaningful accuracy in general.
The Random baseline achieves 50% accuracy (not shown), the Most Common baseline saturates at 64% accuracy, and the NN baseline at
[Figure]
[Figure] [Figure]
70% accuracy.
A reasonable conclusion is that 50 − 64% accuracy is not meaningful, 65 − 69% is arguably meaningful, and 70% or more is meaningful.
4. T YPE II: S INGLE CONTEXT , NON - OVERLAPPING SAMPLE
Type I to Type II transition .
We simulate EI within a single-question context as the overlap between two samples decreases to zero.
For each question, we randomly split the participants and perspectives into two equal sized samples: { H s 1 , P s 1 } and { H s 2 , P s 2 } .
Agreement votes by participants in H son perspectives in P s 2 become the validation set.
The remaining votes become the first training set, where sample overlap is maximum (Type I).
Sample overlap is decreased by removing votes on perspectives in P s 1 by participants in H s 2 until none remain (Type II) [A.5.2].
We define sample overlap p as the fraction of participants in H s 2 with votes in P s 1 , and compute validation accuracy for STUMP and the NN baseline as p varies (Figure 4).
In contrast to the NN baseline, STUMP’s performance remains stable even at p = 0 ; evidence of STUMP’s ability to maintain latent-space alignment across non-overlapping samples.
Type II with Different languages .
We investigate STUMP’s inference capability across two nonoverlapping samples in different languages.
Perspectives in one sample are translated into one of nine different languages using Google Translate API: German (de), Spanish (es), French (fr),
Italian (it), Japanese (ja), Dutch (nl), Portuguese (pt), simplified Chinese (zh) and traditional Chinese (zhtw) [A.5.2].
We compute STUMP’s Type II accuracy for each language pair and compare it to an English-English baseline (Figure 5).
Performance across all language pairs is on par with the baseline.
4. T YPE III: M ULTIPLE CONTEXT , OVERLAPPING SAMPLE
Transfer learning .
To test STUMP’s capability for transfer learning, we pre-train participant embeddings in one context and use them for inference in a new context.
The pre-training set contains votes by all participants on perspectives in context Q 1 and is used to pre-train participant embeddings.
The second training set contains votes for a fraction r of participants on perspectives in a new context Q 2 .
The pre-trained embeddings are frozen while the transformation to latent space for the new context is learned.
Votes on perspectives in Q 2 by participants not in the second training set comprise the type III validation set.
We vary the participant fraction r from 0.05 to 0.9, and compute validation accuracy for STUMP and the NN baseline (Figure 6).
STUMP performance increases with participant fraction when it is small, leveling off at r ∼ 0 .
2 , while the NN baseline only reaches similar performance around r ∼ 0 .
In other words, STUMP can leverage data from a different context via transfer learning to achieve the same performance as the NN baseline with 75% fewer data.
Increased pre-training data .
Next, we investigate how more pre-training data from more contexts affects STUMP’s transfer learning performance.
We increase the number of question contexts whose votes are included in the pre-training set [A.5.3] and compute type III validation accuracy at different participant fractions (Figure 7).
STUMP’s transfer learning performance increases as the number of question contexts included in pre-training increases.
4. T YPE IV: M ULTIPLE CONTEXT , NON - OVERLAPPING SAMPLE
Type III to Type IV transition .
Finally, we explore transfer learning across contexts as sample overlap decreases.
Let sample overlap p be the fraction of participants in the training sample (context
Q 2 ) who are also in the pre-training sample (context Q 1 ).
Moving from Type III to Type IV means decreasing p from 1 to 0.
To simulate this, we start with a Type III setup and iteratively remove participants from the pre-training sample who are also in the training sample [A.5.4].
We use the training procedure and validation set from [4.4] and compute accuracy as p varies (Figure 8).
STUMP performance is arguably meaningful when sample overlap is larger than 50% but drops below that.
I NCREASING D ATA E FFICIENCY WITH S CALE
Based on STUMP’s performance characteristics across elicitation primitives, we design two arbitrarily scalable elicitation protocols where: a) one may expect STUMP to sustain meaningful inference accuracy, and b) data leverage is positive, ie.
∂β∂n t > 0 .
Case I .
STUMP shows meaningful type III inference for low participant fractions (Figure 6).
This means if participants all vote on perspectives from a shared context, only a small fraction need to vote on perspectives from a new context to meaningfully infer the rest.
Thus, the following elicitation protocol is viable:
Elicit an initial sample where m participants each give 1 new perspective and vote on c other perspectives for a single context.
Let each sample after the first be elicited from m new participants where each a) provides 1 new perspective and votes on c other perspectives for a new context, and b) votes on c perspectives for the context shared by all samples (Figure 9).
This protocol results in data efficiency of β = n t / (4 c 2 )+ m/ (2 c )
+ m/ (4 n t ) ∼ O ( n t ) , and data leverage ∂β∂n t ≈ 1 / (4 c 2 ) > 0 .
[A.6.1]
Case II .
STUMP shows arguably meaningful type III/IV inference once sample overlap approaches ∼ 0 .
5 (Figure 8).
In other words, inference for a population of participants across two contexts requires around half of them to vote on perspectives from both contexts.
This implies such an overlap supports latent space alignment between contexts.
To maintain latent space alignment across an arbitrary number of contexts, we chain successive overlaps together with the following elicitation protocol:
Elicit an initial sample where m participants each provide 1 new perspective and vote on c other perspectives for a single context.
Let each sample after the first be elicited from m participants where m/ 2 participants are new, and m/ 2 are from the previous sample.
Let each participant provide 1 new perspective and vote on c other perspectives for a new context (Figure 10).
This protocol results in data efficiency of β = n t / (2 c 2 ) + m/ (2 c ) ∼ O ( n t ) , and ∂β∂n t
= 1 / (2 c 2 ) > 0 .
[A.6.2]
We simulate EI under both protocols for q = 2 , 3 , 4 samples by partitioning the appropriate perspectives, participants, and votes into training and validation sets [A.7].
In both cases STUMP sustains a meaningful accuracy of ∼ 70% as q increases (Figure 11), approximating O ( n 2 t ) votes
[Figure]
n t / 10d a t a e f fi c i e n c y β
Case I
Case II using only O ( n t ) . Thus, the combination of either elicitation protocol with STUMP achieves β 70 ∼ O ( n t ) (Figure 12), giving positive data leverage: ∂β 70 ∂n t = const > 0 . In other words, data efficiency increases linearly with scale.
D ISCUSSION
Improving inference accuracy.
This work prioritized data efficiency with scale over maximizing inference accuracy.
Experiments in (Section 4.4) indicate more, and more diverse, pre-training data can improve accuracy.
STUMP’s use of information in the perspective text is limited by the LLM it employs and we expect a SOTA LLM (Chowdhery et al., 2022;
Zhang et al., 2022; Mitchell et al., 2022) can improve accuracy.
Other directions to improve accuracy may include: using participant side information like demographics, extending STUMP to use information in prompts, using active sampling to elicit votes expected to give the most information (Gonzalez and Eltinge, 2008; Zhang et al., 2020), and fine-tuning LLM weights while training STUMP.
Limitations.
STUMP fails to align latent spaces across non-overlapping samples from different contexts.
This may require an embedding distribution alignment module and Stien Path training (Liu et al., 2021).
Our work does not quantify confidence in aggregate vote statistics, though (Bilich et al., 2019) shows stochastic weight averaging (Maddox et al., 2019) can estimate the posterior variance in this setting.
We only use text perspectives.
Handling perspectives spanning text, images, audio, and video requires swapping STUMP’s LLM for a multi-modal foundation model able to generate aligned embeddings across media types (Jabbar et al., 2022;
Radford et al., 2021; Bommasani et al., 2021; Song et al., 2022;
Reed et al., 2022).
STUMP can only represent changing participant views by treating different times as different contexts, while recommenders often model it explicitly (Chen et al., 2021a).
This work does not show how to make results understandable for human agents, though clustering in latent space is a promising direction (Small et al., 2021).
We don’t show how learned preference signals can directly control other ML models, but jury learning for classifiers (Gordon et al., 2022) and reward learning for RL agents (Ziegler et al., 2019) are encouraging directions.
STUMP is not deferentially private (Dwork, 2008) but approaches developed for LFMs in recommenders may be applicable (Berlioz et al., 2015).
Finally, elicitation inference is just one of the components helpful for achieving the underlying goals of supporting alignment.
Risks.
In CDS, some biases common to recommenders (Chen et al., 2020) are less significant due to the structure of the task.
In particular, if vote exercises are random and can’t be skipped, while users are anonymous and can’t see results, then biases spanning selection, exposure, conformity, position, and popularity are minimized.
However, participants who respond and vote more slowly may still be under-represented in the data, and bad elicitation prompts can still bias participants (Litwak, 1956).
Biases in pre-trained LLMs (Schramowski et al., 2022; Nadeem et al., 2021; Liu et al., 2022) may also propagate through the model to effect inference in unknown ways.
We avoid active sampling (Zhang et al., 2020) due to the risk of compounding small model bias into large inference bias.
Further, non-representative participant populations can arise from biased solicitation, or unequal willingness or ability to participate.
As such systems become used for important decision-making, they risk neglecting views of already marginalized communities due to limited technology access and capacity (Roessler, 2018) unless specifically mitigated for, e g. using existing inclusive voting infrastructure, structurally increasing digital equity, and/or sortition with high touch support.
Conclusion.
We have demonstrated EIO for CDS where data efficiency increases linearly with the size of the elicited data set; our approach approximates O ( n 2 ) votes while requiring only O ( n ) to be elicited directly.
Our experiments suggest this is likely to hold for arbitrarily large data sets spanning contexts, languages and population samples.
This implies data efficiency may increase boundlessly with scale, making it feasible to approximate high-complexity signals for massive principal populations spanning open-ended perspectives across an agent’s decision domain.
These signals may be used to steer complex objective and evaluation functions for multi-principal-agent scenarios spanning governance, markets, diplomacy and AI alignment.
More speculatively, EIO may help enable scenarios where the quality and complexity of human input driving the actions of powerful automated systems increases with the capabilities of those systems [A.8]
A A PPENDIX A. E LICITATION FEASIBILITY
Assume a scenario where there are 10,000 principals and an agent with an open-ended decision space such that each principal may provide a unique perspective related to decisions the agent could make in a given month.
Consider 10,000 perspectives, and assume that directly eliciting a principal’s view on a perspective involves casting a single vote which takes 10 seconds each.
Direct elicitation of all votes equates to 28 hours spent voting per principal per month ( 280,000 hours per month in total).
This is unreasonable but theoretically possible.
However, increasing this to just 100,000 principals equates to 280 hours per principal per month – more than a full-time job –, and increasing it to 1,000,000 principals equates to 2,800 hours per principal per month which is more than the number of hours in a month.
There are more than 300,000,000 principals (US citizens) in the US government agent.
A. S CALING BEHAVIOR OF EIO FOR CDS BASED ON PREVIOUS APPROACHES
Consider a human-perspective matrix Θ corresponding to N humans and K perspectives spanning Q different contexts (ie.
elicited by Q different questions or prompts).
Assume the corresponding CDS generates a finite number of k perspectives for each context
so K
=
Qk .
The number of elements in Θ is NQk ∼ O ( NQ ) .
Bilich et.
al. demonstrate an EIO approach to CDS in (Bilich et al., 2019) where only a finite number of votes per person, c , is needed to recover an approximation of Θ for a single context.
This means the number of votes which need to be sampled for a Θ spanning Q contexts is NQc ∼ O ( NQ ) .
A. N UCLEAR N ORM MODEL DETAILS
The nuclear norm model likelihood is given by:
[Equation]
where A = { i, j | a ij } , D = { i, j | d ij } , and C = { i, j, k | c ijk } .
The NN model posterior is given by
[Equation]
where 1 || u ||∗ <τ denotes a uniform prior over the nuclear norm ball for u of radius τ .
Leaning happens through optimization via SGD of the MAP estimator.
A. D ATA AND TRAINING DETAILS
For each question in the dataset, there are an average of 529 perspectives, 14455 agreement exercises, and 10151 pair-choice exercises.
The full dataset consists of 172,242 votes across 3,703 perspectives made by 534 humans.
In training the STUMP model, we use L 2 regularization on the transformation weight M and apply dropout on the universal embedding.
We use a pre-trained multilingual universal sentence encoder (Apache-2.0 license) to generate perspective embeddings F ( p i | ϕ ) .
Hyper-parameters are tuned by grid search, and the following values are used in our experiments: regularization rate = 0.01, dropout rate = 0.3, latent participant embedding dimensions =
50, and learning rate = 0.001 with the
Adam optimizer.
The models are implemented with PyTorch, and trained on a PC with 4-core Intel i7-1165G7 processor.
A. E LICITATION PRIMITIVE EXPERIMENT DETAILS
The exercises in the training sets consist of a mix of agreement and pair-choice exercises where the agreement exercises account for 60%–75% of the total exercises.
A.5. T YPE I: S INGLE CONTEXT , OVERLAPPING SAMPLE
To simulate the asynchronous CD scenario, participants’ votes pertaining to perspectives submitted after their own become the validation set E v , and votes pertaining to perspectives submitted before their own become the training pool.
We modulate vote density by adjusting the fraction of votes from the training pool which is randomly partitioned into the training set E t .
To simulate the synchronous
CD scenario, we randomly partition votes such that the training set E t and validation set E v are the same size as those in the asynchronous scenarios for each experiment.
A.5. T YPE II: S INGLE CONTEXT , NON - OVERLAPPING SAMPLE
Type I to Type II transition.
For each question, we randomly split the participants into two equal-sized groups H s 1 and H s 2 , and their corresponding perspectives P s 1 and P s 2 .
We select votes on perspectives in P s 2 by participants in H s 1 as the validation set, E v = { e aij , e cijk | i ∈
H s 1 , j, k ∈ P s 2 } .
The train set consists of the following three pieces: votes on perspectives in P sby participants in H s 1 , votes on perspectives in P s 2 by participants in H s 2 , and votes perspectives in P s 1 submitted by a fraction p of participants in H s 2 , where p ∈ [0 , 1] , E t = { e aij , e cijk | i ∈
H s 1 , j, k ∈ P s 1 } ∪ { e aij , e cijk | i ∈ H s 2 , j, k ∈ P s 2 } ∪ { e aij , e cijk | i ∈
H ′ s 2 , j, k ∈ P s 1 ,
H ′ s 2 ⊂
H s 2 , | H ′ s 2 | / | H s 2 | = p } .
When p = 1 , the train data has maximum participant overlap, i.e., it contains max number of participants with votes on perspectives in both P s 1 and P s 2 .
When p = 0 , the train data has zero participant overlap, i.e., it contains with votes on perspectives in both P s 1 and
P s 2 .
Type II with Different languages.
For each question, we randomly split the participants into two groups H s 1 and H s 2 , and their corresponding perspectives P s 1 and P s 2 .
The train set consists of exercises pertaining to perspectives in P s 1 submitted by participants in H s 1 , and exercises pertaining to perspectives in P s 2 submitted by participants in H s 2 , E t
= { e a ij , e c ijk | i ∈ H s 1 , j, k ∈ P s 1 } ∪ { e aij , e cijk | i ∈ H s 2 , j, k ∈ P s 2 } .
The validation set consists of exercises pertaining to perspectives in P s 1 submitted by participants in H s 2 , and exercises pertaining to perspectives in P s 2 submitted by participants in H s 1 , E v = { e aij , e cijk | i ∈ H s 2 , j, k ∈ P s 1 } ∪ { e aij , e cijk | i ∈ H s 1 , j, k ∈ P s 2 } .
The second group perspectives are translated to nine languages respectively: German (de), Spanish (es), French (fr), Italian (it), Japanese (ja), Dutch (nl), Portuguese (pt), simplified Chinese (zh) and traditional Chinese (zhtw).
A.5. T YPE III: M ULTIPLE CONTEXT , OVERLAPPING SAMPLE
Transfer learning.
Given a pair of contexts Q 1 and Q 2 , we denote all perspectives in Q 1 as P 1 and all perspectives in Q 2 as P 2 .
We split the participants who participated in both contexts randomly into two groups H s 1 and H s 2 .
The pre-train set contains exercises pertaining to perspectives in Psubmitted by all participants, E t 1 = { e aij , e cijk | i ∈ H, j, k ∈ P 1 } .
The second train set contains exercises pertaining to perspectives in P 2 submitted by participants in H s 2 , E t 2 = { e aij , e cijk | i ∈
H s 2 , j, k ∈ P 2 } .
The validation set contains exercises pertaining to perspectives in P 2 submitted by participants in H s 1 , E v = { e aij , e cijk | i ∈ H s 1 , j, k ∈ P 2 } .
Participant fraction r is defined as r = n ( H s 2 ) /
[ n ( H s 1 ) + n ( H s 2 )] .
The model is pre-trained using the pre-train set E t 1 , and the learned participant embeddings z are transferred to the next training cycle using the second train set E t 2 .
In the second training cycle, z is fixed while the new context transformation M 2 is learned.
Then the participant embeddings z learned from pre-training and context transformation M 2 learned from the second training are together used to make predictions for exercises in the validation set.
Increased pre-training data.
Given a target context Q t and its perspectives P t , we select a set of N − 1 contexts that do not contain Q t as the pre-train contexts.
We split the participants who participated in all of the selected questions randomly into two equal-sized groups H s 1 and H s 2 .
The pre-train set contains exercises pertaining to perspectives in the pre-train contexts submitted by all participants, E t 1 = ∪ n =1 ...N,n ̸ = t { e aij , e cijk | i ∈ H, j, k ∈ P n } .
The second train set contains exercises pertaining to perspectives in P t submitted by participants in H s 2 , E t 2 = { e aij , e cijk | i ∈
H s 2 , j, k ∈ P t } .
The validation set contains exercises pertaining to perspectives in P t submitted by participants in H s 1 , E v = { e aij , e cijk | i ∈ H s 1 , j, k ∈ P t } .
A.
. T YPE IV: M ULTIPLE CONTEXT , NON - OVERLAPPING SAMPLE
Type III to Type IV transition.
Given a pair of contexts Q 1 and Q 2 and their corresponding perspectives P 1 and P 2 , we split the participants who participated in both contexts randomly into two equal-sized groups H s 1 and H s 2 .
The pre-train set contains exercises pertaining to perspectives in Psubmitted by participants in H s 1 and exercises pertaining to perspectives in P 1 submitted by a fraction p of participants in H s 2 , where p ∈ [0 , 1] , E t 1 = { e aij , e cijk | i ∈ H s 1 , j, k ∈ P 1 } ∪ { e aij , e cijk | i ∈
H ′ s 2 , j, k ∈ P 1 , H ′ s
⊂ H s 2 , | H ′ s 2 | / | H s 2 | = p } .
The second train set contains exercises pertaining to perspectives in P 2 submitted by participants in H s 2 , E t 2 = { e aij , e cijk | i ∈ H s 2 , j, k ∈ P 2 } .
The validation set contains exercises pertaining to perspectives in P 2 submitted by participants in H s 1 ,
E v = { e aij , e cijk | i ∈ H s 1 , j, k ∈ P 2 } .
When p = 1 , E t 1 and E t 2 have maximum participant overlap.
When p = 0, they have zero participant overlap.
A. D ATA EFFICIENCY SCALING FOR ELICITATION PROTOCOLS A.6. C ASE I
The first sample elicits mc votes and initializes Θ with N = m participants and K = m perspectives.
Each additional sample elicits an additional 2 mc votes and adds m new participants and m new perspectives to Θ .
After q samples, n t = mc (2 q − 1) votes have been elicited, and Θ is comprised of K = mq perspectives and N = mq humans resulting in n (Θ)
= m 2 q 2 .
Noting the relation q = ( n t + mc ) /
(2 mc ) , then n (Θ) = n 2 t / (4 c 2 ) + ( n t m ) /
(2 c ) + m/ 4 ∼ O ( n 2 t ) , data leverage is β = n t / (4 c 2 ) + m/ (2 c ) + m/ (4 n t ) ∼ O ( n t ) , and ∂β∂n t ≈ 1 / (4 c 2 ) for n t >> √ m/ 2 .
A.6. C ASE II
The first sample elicits mc votes and initializes Θ with N = m participants and K = m perspectives.
Each additional sample elicits an additional mc votes and adds m/ 2 new participants and m new perspectives to Θ .
After q samples, n t =
mcq votes have been elicited, and Θ is comprised of
K = mq perspectives and N = ( m/ 2)( q + 1) humans resulting in n (Θ)
=
( m 2 q 2 + m 2 q ) / 2 .
Noting the relation q = n t / ( mc ) , then n (Θ) = n 2 t / (2 c 2 ) + ( n t m ) / (2 c ) ∼ O ( n 2 t ) , data leverage is β = n t / (2 c 2 ) + m/ (2 c ) ∼ O ( n t ) , and ∂β ∂n t = 1 / (2 c 2 )
A. P ROTOCOL SIMULATION EXPERIMENTS
Case I Given a set of perspectives corresponding to Q different contexts P 1 ...P Q , and Q equal sized non-overlapping sets of participants H s 1 ...H sQ , we construct the following experiment for each q = 2 ...
Q : let Θ q be the HPM corresponding to ∪ qn =1 { H sn } and ∪ qn =1 { P n } .
Partition the training set in each case to correspond to local samples denoted by the shaded areas in Figure 9 such that E qt = ∪ qn =2 { e aij , e cijk | i ∈ H sn , j, k ∈ P n } ∪ { e aij , e cijk | i ∈ ∪ qn =1 H sn , j, k ∈ P 1 } .
Let the validation set E v contain all other exercises in Θ q , denoted by the white areas in Figure 9.
Case II Given a set of perspectives corresponding to Q contexts P 1 ...P Q , and Q equal sized partially overlapping sets of participants H s 1 ...H sQ with adjacent overlap p defined by | H si ∩ H si +1 | = p ∗| H si | , we construct the following experiment for each q = 2 ...
Q :let Θ q be the HPM corresponding to ∪ qn =1 { H sn } and ∪ qn =1 { P n } , the train set is denoted by the shaded areas in Figure 10, E qt =
∪ qn =1 { e aij , e cijk | i ∈ H sn , j, k ∈ P n } , and the validation set E v contains all other exercises in Θ q , denoted by the white areas in Figure 10.
A. S PECULATION ON ALIGNMENT SCENARIO
As foundation model (FM) capabilities increasingly span LLMs and RL agents (Bommasani et al., 2021;
Reed et al., 2022; Chen et al., 2021b; Zheng et al., 2022; Janner et al., 2021), sharing an FM between an EIO model and an RL agent may enable a scenario where the quality and nuance of normative alignment signals improves as the agent learns.
A.
D ATA SET
Data used for this research comes from periodic load testing done by a private company for internal quality assurance, and was fully de-identified prior to use in the research.
The researchers identified that portions of it could be interesting for this work due to the scale of the data and the predictive challenges associated with the nuance and ambiguity of certain questions that were asked.
Data made available for this research came from the following open-ended questions asked during the collective dialogue load test
: • During the pandemic, what fun experiences have you newly discovered?
These can be any activities, hobbies, or events you consider fun.
• Please tell me more: what is more important to you now?
• And conversely, what is less important to you now?
• What new routines or behaviors have you started, if any, based on what you now think is important?
• How or where did you learn the new routine or behavior?
• Tell me about a product or service you have used MORE than ever in the past year.
Why did you use it more?
• Please tell me more – how is this impacting your decisions, if at all?
A. P ARTICIPANT PAYMENT AND CONSENT
The 534 participants who joined the load testing exercise had been solicited through online panel providers for an incentive of approximately $15 / hour USD, and consented to the policy shown in Figure 13.
While this consent form does mention PII, this was a general consent form for the data provider and no PII was known to be contained in the data set made available for this research.
The researchers were structurally limited from accessing any directly collected PII due to system design, and none of the raw outputs that were inspected directly during the research included any PII.