Adaptive Sampling for Discovery
Abstract
In this paper, we study a sequential decision-making problem, called Adaptive
Sampling for Discovery (ASD).
Starting with a large unlabeled dataset, algorithms for ASD adaptively label the points with the goal to maximize the sum of responses.
This problem has wide applications to real-world discovery problems, for example drug discovery with the help of machine learning models.
ASD algorithms face the well-known exploration-exploitation dilemma.
The algorithm needs to choose points that yield information to improve model estimates but it also needs to exploit the model.
We rigorously formulate the problem and propose a general information-directed sampling (IDS) algorithm.
We provide theoretical guarantees for the performance of IDS in linear, graph and low-rank models.
The beneﬁts of IDS are shown in both simulation experiments and real-data experiments for discovering chemical reaction conditions.
Introduction
Machine Learning (ML) models are becoming increasingly popular for discovery problems in various areas like drug discovery [40, 5, 38] and scientiﬁc discoveries [13, 23, 19].
Despite its success in real-data applications, the theoretical aspects of the problem are not fully understood in the machine learning literature.
In this paper, we consider a particular case of discovery and formalize it as a problem that we call Adaptive Sampling for Discovery (ASD).
In ASD, an algorithm sequentially picks T samples X 1 , . . .
, X T out of a size- n unlabeled dataset S n ( T (cid:28) n ) and obtains their labels
Y 1 , . . .
, Y T with the goal to maximize the sum (cid:80) Tt
=1 Y t .
The labels correspond to the quality of discoveries, which can be continuous or ordered categorical variables depending on the types of discoveries.
As a sequential decision-making problem, we ﬁnd it substantially connected to the existing literature on Multi-arm Bandit (MAB) and other related decision-making problems, which we brieﬂy review in the rest of this section.
Multi-arm bandit.
MAB is a decision-making problem that maximizes the total rewards by sequentially pulling arms and each arm corresponds to a reward distribution [27].
MAB faces the well-known exploration–exploitation trade-off dilemma.
ASD has to deal with the same explorationexploitation dilemma if one treats each arm as an unlabeled point.
The major difference between
ASD and MAB is that each arm after being pulled once can still be pulled for inﬁnite number of times, while we can only label a sample once in ASD.
This is because a discovery can only be made once: one receives no credit making the same discovery twice.
In fact, as we will show later, ASD can be an MAB when each unique value in S n has larger than
T repeated samples and n (cid:28) T 2 .
Moreover, a variant of MAB called Sleeping Expert [22, 24] can be seen as a more general problem setup of ASD.
One can intuitively interpret Sleeping Expert as the bandit problem where the set of available arms varies.
More generally, bandit algorithms that allows adversarial choice of the action set can be used to solve the ASD problem.
However, due to its generality, the regret bounds developed in [24] are loose or vacuous in our setup.
Detailed discussions are deferred to Section [-
-]2.
Due to the similarities between bandits and ASD, we adopt the same performance measurement, regret, from bandit literature.
Regret is the highest sum of labels that could have been achieved minus the actual sum of labels.
There is a long line of work in the bandit literature on solving the exploration-exploitation tradeoff.
Popular methods includes UCB (upper conﬁdence bound), TS (Thompson sampling).
UCB keeps an optimistic view on the uncertain model [4, 26], which encourages the algorithm to select arms with higher uncertainty.
TS [1] adapts a Bayesian framework, which samples a reward vector from its posterior distribution and select arms accordingly.
In this paper, we adopt another family of methods called Information-directed Sampling (IDS).
IDS [33] balances between the expected information gain on the unknown model and instant regret with respect to the posterior distribution under a
Bayesian framework.
IDS has been shown to outperform UCB in problem with rich structural information, e.g. sparse linear bandit [14].
As we show below (Proposition 1), the ASD problem is not interesting in the unstructured case which means we have to use structural assumptions.
That makes IDS a natural choice.
Traditional MAB algorithms designed for ﬁnite-armed bandits are not applicable to ASD since the number of unlabeled samples n can be much larger than horizon T .
There are works considering inﬁnite-armed bandits by assuming structural information across arms.
For instance, linear bandit assumes a linear model for the reward generation [27], [41] assumes Lipschitz-continuity and [42] considers neural network model for reward generation.
However, there are few works studying sleeping expert with structure across arms beyond linear structure.
Other related literature.
With a similar adaptive sampling procedure, Active Learning (AL) aims at achieving a better model accuracy with a limited number of labels.
As the reader may have noticed, the goal of AL aligns with ASD in the early stage when the model has high uncertainty and the predictions are high unreliable, while in the later stage ASD aims at better discovery performance.
Indeed, there are active learning algorithms based on the idea of maximum information gain [3]. [29] applies the idea to the matrix completion problem, which signiﬁcantly improves the prediction accuracy from random selection.
Other works with a similar goal go by the name sequential exploration [6, 7, 17].
Their works lack a theoretical justiﬁcation and the algorithms are not generally applicable.
A more relevant setup in the active learning literature is Active Search (AS).
Active search is an active learning setting with the goal of identifying as many members of a given class as possible under a labeling budget.
AS concerns the Bayesian optimal methods named Efﬁcient Nonmyopic Search (ENS) [21, 20, 32], which is not computationally efﬁcient.
Their approximate algorithms for the
Bayesian optimal method do not provide strong theoretical guarantee.
A more detailed comparison between our proposed approach and ENS is given in Appendix I.
Main contributions.
In this paper, we formulate the adaptive sampling for discovery problem and propose an generic algorithm using information-direct sampling (IDS) strategy.
We theoretically analyze the regret of IDS under generalized linear model, low-rank matrix model and graph model assumptions.
Indeed, the analysis for linear model are directly from IDS for linear bandit.
The regret analysis for low-rank model and graph model are new even in the bandit literature.
The results are complemented by simulation studies.
We apply the algorithms to the real-world problems in reaction condition discovery, where our algorithm can discover about 20% more plausible reaction conditions than random policies and other simple baseline policies adapted from the bandit literature.
Formulation
In this section, we formally formulate ASD and rigorously discuss its connections to MAB.
Notations.
We ﬁrst introduce some notations that are repeatedly used in this paper.
For any ﬁnite set S , we let D ( S ) be the set of all the distributions over the set.
For any positive integer n , we let [ n ] = { 1 , . . . , n } .
We use the standard O ( · ) and Ω( · ) notation to hide universal constant factors.
We also use a (cid:46) b and a (cid:38) b to indicate a = O ( b ) and a = Ω( b ) . ASD is a sequential decision-making problem over discrete time steps.
In general, we let F t be the observations up to decision step t .
We adapt a Bayesian framework and let E t [ · ] = E [ · | F t ] be the conditional expectation given the history up to t .
Let P t be the probability measure of posterior distribution.
Adaptive sampling for discovery.
We formulate ASD problem as follows.
Consider a problem with the covariate space X ⊂ R d and label space Y ⊂ [0 , 1] .
For categorical discoveries, the labels are ordered such that higher labels reﬂects better discoveries.
Given a input x ∈ X , its label is generated from a distribution D θ | x over Y parametrized by unknown parameters θ .
We adopt a Bayesian framework and denote the prior distribution of θ by φ .
Given a sequence of t −observations F t = { ( X i , Y i ) } t − 1 i =1 , the posterior distribution of φ is denoted by φ ( · | F t ) .
We also let f θ ( x ) =
E Y ∼D θ | x [ Y ] be the expected label of input x under model θ .
ASD starts with an unlabeled dataset S n ⊂ X with n samples.
Our goal is to adaptively label
T samples { X 1 , . . .
, X T } ⊂ S n to maximize the sum of labels (cid:80)
Tt =
1 Y t .
The discovered set is removed from available unlabeled set, i.e. each arm can only be pulled once.
At each step t , an algorithm chooses an input X t from S nt : =
S n \ { X 1 , . .
, X t − 1 } .
The environment returns the label
Y t ∼ D θ | X t
[Equation] To measure the performance of any algorithm, we borrow the deﬁnition of Bayesian regret from bandit literature [27].
Note that the regret deﬁnition is also used in Sleeping Expert [22], which is a more general framework than ASD.
With respect to any chosen random sequence ( X 1 , . . .
, X T ) , we deﬁne regret by
[Equation]
Let S nt =
S n \ { X 1 , . . .
, X t − 1 } be the remaining unlabeled set at the start of step t .
An algorithm
A is a sequence of policies ( π 1 , . . .
, π T ) .
Let D ( S nt )
⊂ R n − t − 1 be the set of all the distributions over S nt that are denoted as n − t − 1 vectors.
Each π t is a mapping from history F t to a distribution
D ( S nt ) over S nT .
The Bayesian Regret w.r.t.
an algorithm is
[Equation]
where the expectation is taken over the randomness in the choice of input X t , the labeling Y t and over the prior distribution on θ .
Let X ∗ 1 , . . .
, X ∗ T be the sequence that realizes the maximum in (1).
Note that X ∗ 1 , . . .
, X ∗ T are random variables depending on θ .
In general, the goal the algorithm is to achieve a sublinear regret upper bound.
Connections to bandit problem.
This problem degenerates to a classical bandit problem when X is discrete and S n has inﬁnite number of unlabeled samples for each unique value in X .
In the bandit literature some frequentist results are available.
The problem has the minimax regret rate (cid:112) |X| T .
However, in our problem, we generally consider large unlabeled set ( n (cid:29) T ).
In fact, when |X| is larger than T 2 , Proposition 1 shows that a linear regret is unavoidable without a structural assumption.
Therefore, we discuss different types of structural information that allows information sharing across different inputs.
Proposition 1 (Lower bound for nonstructural case) .
Consider the following set of ASD problems.
Let n = T 2 , S n = [ n ] and D θ | x = N ( θ x , 1) for x ∈ [ n ] and θ ∈ [0 , 1] n .
For any algorithm A , there exists some prior distribution over the set of ASDs described above such that BR ( T, A ) (cid:38) T .
Compared with sleeping expert.
Sleeping expert considers a generalized bandit framework where the available (awake) action set A t are changing.
It is the same as ASD except for that the changes of the unlabeled dataset (or awake action set) follow certain dynamic.
That is whenever an input is labeled, it is taken out of the unlabeled dataset.
However, sleep expert normally assumes a stochastic distribution on A t or adversarial changes [24, 9], which makes their regret bound loose.
To be speciﬁc, [22] derives regret bounds for unstructured bandits, which leads to vacuous regret bound of √ nT .
For structured case, [24] derives the same regret bound in this paper, due to the strong structure assumption for linear model.
There are few literature for sleeping bandits with complicated structures beyond linear model.
The regret bounds in the bandit literature for the other two models, graph and low-rank matrix models, are not applicable both resulting a Ω( T ) regret bound, which assures the necessity of studying ASD problem by itself.
Information-directed Sampling and Generic Regret Bound
In this section, we introduce a strategy called Information-Directed Sampling and develop a generic regret bound.
IDS runs by evaluating the information gain on the indices of the top T inputs with respect to the labeling of a certain input x .
It balances between obtaining low expected regret in the current step and acquiring high information about the inputs that are worth labeling.
We ﬁrst introduce entropy and information gain .
Deﬁnition 1 (Entropy and information gain) .
For any random variable X ∈ X with a density function P , its entropy is deﬁned as H ( X ) =
− (cid:82) x ∈X P ( x ) log( P ( x )) .
Let H ( X | Z ) be the conditional entropy of X given Z .
The mutual information between any two random variables
Y and X can be deﬁned as I ( X ; Y ) =
H ( X ) − H
( X | Y ) .
Conditional mutual information is
I ( X ; Y | Z ) =
H ( X | Z ) − H ( X | Y, Z ) .
Furthermore, we use H t and I t for the entropy and mutual information under the posterior distribution up to step t .
For IDS, we let ∆ t ( x ) =
E t [max x (cid:48) ∈ S nt f θ ( x (cid:48) )
− f θ ( x )] be the expected instant regret.
Let the information gain of selecting x at the step t be g t ( x )
=
I t ( X ∗ t, 1 , . . .
, X ∗ t,T − t +1 ; Y t | F t − 1 , X t = x ) , where ( X ∗ t, 1 , . . .
, X ∗ t,T − t +1 ) =
[Equation]
− t +1 ) ≥ max x (cid:48) (cid:54) = x 1 ,...,x T − t [Equation] f
[Equation]
( x (cid:48) ) (cid:27) , are the random variable for the top T − t + 1 unlabeled points.
Note that { X ∗ t, 1 , . . .
, X ∗ t,T − t +1 } ⊂ { X ∗ 1 , . . .
, X ∗ T } .
Intuitively, g t ( x ) captures the amount of information on the indices of the top
T − t + 1 points gained by labeling x .
IDS minimizes the following ratio
[Equation]
where ∆ t ∈ R n − t +1 and g t ∈ R n − t +1 are the corresponding vectors for the expected single-step regret and information gain.
Constant ratio here weighs between instant regret and information gain.
A higher λ prefers points with lower instant regrets.
An abstract algorithm is given Algorithm 1.
Algorithm 1 IDS ( -Directed Sampling) for Discovery
Input: Unlabeled dataset S n , prior distribution φ , total number of steps T , constant λ .
Initialize history F 0 = {} , S n 1 = S n .
for t = 1 to T do
Calculate ∆ t ( x ) and {+g t ( x ) for each x ∈ S nt using posterior φ ( · | F t − 1 ) . Sample X t ∼ arg max π Ψ t,λ ( π ) and receive label Y t from D θ | X t . Update history F t = F t − 1 ∪ { ( X t , Y t ) } and unlabeled dataset S nt +
+}
= S nt \ { X t } .
end for
We can show that the following generic Bayesian regret upper bound holds.
Lemma 1.
Suppose π IDS =
( π t ) t ∈ [ T ] and Ψ t,λ ( π t ) ≤ Ψ ∗ ,λ .
Then we have IDS using constant λ has the following regret bound
[Equation]
As the case in the bandit problem, the regret bound depends on the worst-case information ratio bound and the entropy of the random variables for the top T unlabeled points.
Now we will discuss how to bound information ratios under various structural information assumptions.
Bounding Information Ratio under Structural Assumptions
In this section, we study three cases with different structural information.
4.1 Generalized linear model.
In this subsection, we discuss a generalized regression model, which includes logistic regression, corresponding to a natural binary discovery problem.
Consider d -dimensional generalized linear regression problem.
The label Y given an input X is generated by
[Equation]
where µ is usually referred as link function.
We assume that Y is uniformly bounded in [0 , 1] .
In addition, we assume an upper bound on the derivatives of µ .
Assumption 1.
The ﬁrst order derivatives of µ are upper-bounded by L µ .
Theorem 1.
Under Assumption 1, we have Ψ ∗ , 2 ≤ L µ d/ 2 , which gives the Bayesian regret bound
[Equation]
A naive bound for H ( X ∗ 1 , . . .
, X ∗ T ) in [33] does not apply here because it introduces a log( n ) dependency.
Instead, we notice that X ∗ 1 , . . .
, X ∗ T is a deterministic function of θ and S n .
We can bound the entropy of θ instead.
Assuming a Gaussian prior on θ , H ( θ ) (cid:46)
d log( d ) .
Indeed, the regret in (2) is analogous to the regret bound in linear bandit [33, 27].
Theoretically, IDS does not show an advantage.
This is partly because carefully designed algorithms for linear model are sufﬁciently utilizing the structural information efﬁciently.
[14] has shown that IDS achieves a much tighter lower bound for sparse linear bandit compared to UCB-type algorithms and Thompson sampling algorithms.
We show in the Appendix G that IDS enjoys a regret bound of ˜ O
( sT 2 / 3 ) in the discovery setting with a sparse linear structure, with s being the sparsity.
4.2 Low-rank matrix
In this subsection, we discuss a novel problem setup: low-rank matrix discovery.
Low-rank matrix completion problem has been extensively studied in the literature [25, 18, 31].
However, most of the algorithms select entries with a uniform random distribution.
In many applications, appropriately identifying a subset of missing entries in an incomplete matrix is of paramount practical importance.
For example, we can encourage users to rate certain movies for movie rating matrix recovery.
While selecting the users that reveal more information is important, it is also important to push the movies aligning with users tastes.
Active matrix completion has long been studied in the literature [8, 29, 15].
As far as we know, no work has considered a discovery setup, where the goal of adaptive sampling is not only to get a better recovery accuracy but also to maximize the sum of observed entries.
Formulation.
Consider an unknown matrix Y ∈
R m 1 × m 2 , whose entries ( i, j ) ∈ [ m 1 ] × [ m 2 ] are sampled in the following manner:
[Equation]
noise
[Equation]
where M ∈ R m 1 × m 2 is the unknown rank- r matrix and r (cid:28) min { m 1 , m 2 } .
The matrix M admits
SVD, i.e. M = UDV T , where U ∈ R m 1 × r and V ∈ R r × m 2 are orthogonal matrix and D ∈ R r × r .
For simplicity, we let m
=
m
= m .
Our analysis can be easily extended to the case m 1 (cid:54) = m 2 .
Note that this setup is closely connected to the low-rank bandit problem except for that the actions are standard basis and can only be selected once.
The best results in the literature [28] achieve a regret bound of ˜ O
( m 3 / 2 √ rT ) .
Since in our case, T ≤ m 2 , the regret bound becomes vacuous.
This ﬁnding implies that we need some further assumptions on the structure of the matrix.
A common assumption in the matrix completion is incoherence , an important concept that measures how the subspace aligns with the standard basis.
Deﬁnition 2 (Coherence) .
The coherence of subspace U with respect to the standard i -th basis vector e i is deﬁned as µ i ( U ) =
(cid:107) P
[Equation]
Assumption 2 (Incoherence) .
There exists some constant γ > 0 , such that the unknown matrix M is incoherent, i.e. max i ∈ [ m ] µ i ( U ) ≤ (cid:112) γr/m and max i [Equation] m [Equation]
≤ [Equation] γr/m. To introduce our results, we further deﬁne some constants.
Remark 1.
Let ¯ d be the maximum singular value of the unknown matrix M .
We have
[Equation]
Theorem 2.
Under the low-rank matrix model with Assumption 2, the worst-case information ratio can be bounded by Ψ ∗ , 3 ≤ 4 B ( B 2 + 1) r 3 γ 2 , which gives us a regret bound of
[Equation]
The entropy term H ( M ) depends on the distribution of M .
To give an example, let M = UV T , where U, V (cid:62) ∈
R m × r and each element in U and V are sampled independently from standard
Gaussian.
Then H ( M ) ≤ H (( U, V ))
=
H ( U ) + H ( V ) (cid:46)
mr log( mr ) .
In general, the regret bound is still sublinear when mr 4 (cid:28) T (cid:28) m 2 .
4.3 Graph
We consider the discovery problem with graph feedback.
Speciﬁcally, we are given a graph G = ( S n , E ) , where the unlabeled set S n is the node set.
By labeling a node x ∈ S n , we receive noisy outcomes for all the nodes connecting to x .
Let the side information at step t be O t ( X t ) =
{ ˜ Y x (cid:48) } x (cid:48) :(
x (cid:48) ,X t ) ∈ E , where ˜ Y x (cid:48)
=
x (cid:48)
+ (cid:15) x (cid:48) for (cid:15) x (cid:48) being the zero-mean noise given x being selected.
In addition to the side information, we also receive the label for X t , which is sampled in the same manner.
This setup is also studied in the sleeping expert literature [9].
[9] gives a regret bound depending on E [ (cid:80) x
T x /Q x ] , where T x is the total number of visits on action x and Q x is the total number of observations for action x .
The ratio can be low when the algorithms tend to select nodes with high degrees and their algorithms are not designed for that purpose.
Therefore, the structure are not sufﬁciently exploited and the regret bound can be loose.
To measure the complexity of a graph, we introduce the following two deﬁnitions.
Deﬁnition 3 (Maximal independent set) .
An independent set is a set of vertices in a graph such that no two of which are adjacent.
A maximal independent set (MIS) is an independent set that is not a subset of any other independent set.
We denote the cardinality of the smallest maximum independent set of a graph G by C ( G ) .
Deﬁnition 4 (Clique cover number) .
A Clique of a graph G =
( K , E ) is a subset S ⊆ K such that the sub-graph formed by S and E is a complete graph.
A Clique cover of a graph G =
( K , E ) is a partition of K , denoted by Q , such that Q is a clique for each Q ∈ Q .
The cardinality of the smallest clique cover is called the clique cover number, which is denoted by χ ( G ) .
Remark 2.
Clique cover number is guaranteed to be larger than the cardinality of the smallest maximal independent set, i.e. C ( G )
≤ X ( G ) .
Since the nodes in the graph are diminishing, we further deﬁne a stable version of MIS.
Deﬁnition 5.
Let G t ( G ) be the set of all subgraphs of G with N − t nodes and deﬁne C t ( G ) = max G t ∈G t ( G ) C ( G t ) .
Assumption 3.
We assume that x + (cid:15) x ≤ B almost surely for all x ∈ S n and some constant B > 0 .
Theorem 3.
Under Assumption 3, the Bayesian regret of the IDS with λ = 2 can be upper bounded by
[Equation]
Let us discuss certain graphs where the regret bound can be low.
First, if the graph can be decomposed into K (cid:28) T complete subgraphs, then X ( G ) ≤ K and we have a regret bound of √ KT .
An example, in which C ( G ) is low while X ( G ) is high, is a star-shaped graph, with a set of nodes in the center of the graph connecting to all the other nodes.
See Figure 2 in the Appendix for an illustration.
4.4 A generic results for models with structural information
As shown in sections 4.2 and 4.3, a T 2 / 3 regret upper bound with a small coefﬁcient can be derived when the model has certain structural information.
We give the following generic result, stating that as long as there exists a policy µ , whose information gain can be lower bounded by the instant regret of Thompson sampling policy an upper bound for Ψ (cid:63), 3 can be derived.
Proposition 2.
Assume that there exists a policy µ such that g (cid:62) t µ ≥ φ (∆ (cid:62) t π tst ) 2 , where π tst is the
Thompson sampling policy at the step t for some constant φ and assume that the instant regrets are uniformly bounded by B .
Then Ψ (cid:63), 3 ≤ 2 B/φ, which gives a regret bound of O (( BTH ( θ ) /φ ) 2 / 3 ) .
Experiments
We start with simulation studies on the three problems: (generalized) linear model, low-rank matrix and graph model.
Since information gain can be hard to calculate in some problems, we introduce sampling-based approximate algorithms, which generate random samples from the posterior distribution and evaluate a variance term instead of the original information gain.
We will also introduce general Thompson Sampling algorithms for all the three problem setups.
5.1 Approximate algorithms
It may not be efﬁcient to evaluate the information gain in certain problem setups.
For example, one way to generate random low-rank matrix is to generate its row and column spaces from Gaussian distribution.
We do not have a closed form for the posterior distribution of the resulting matrix.
Neither can we evaluate the information gain.
To that end, we follow an approximate algorithmic design in [33], which replaces the information gain with a conditional variance v t ( x ) =
Var t ( E [ Y t,x |
X ∗ 1 , X t = x ]) , which will be evaluated under random samples from posterior distribution using
MCMC.
The conditional variance is a lower bound of the information gain.
An approximate algorithm is given by Algorithm 2 in Appendix H.
Proposition 3.
The following lower bound for information gain holds, g t ( x ) ≥ v t ( x ) .
Compared algorithms.
For the rest of section, we compare IDS (the approximate algorithm), TS,
UCB (if the bandit version is available) and random policy.
As a method that is closely connected to
IDS, Thompson sampling (also referred as posterior sampling) is also compared in our experiments.
Thompson sampling [36] at each round samples a θ t from its posterior distribution and then selects the input with lowest instant regret w.r.t to θ t .
TS does not account for the amount of information gain, thus can be insufﬁcient in utilizing the structural information.
UCB is an important algorithm design in bandit literature.
Though it is not speciﬁcally designed for ASD, we apply UCB to (generalized) linear ASD by manually removing arms that have been selected.
5.2 Simulation studies
Linear and logistic regression model.
The data generating distribution are speciﬁed below.
Unlabeled dataset is sampled from N (0 , σ 2 x ) and the prior distribution of θ ∗ is N (0 , σ 2 θ ) .
The noise (cid:15) is sampled from N (0 , σ 2 (cid:15) ) .
We study the effects of d = 20 , 50 , 100 .
The same setup is used for logistic regression model.
The posterior distribution for θ in simple linear regression is also Gaussian.
For logistic regression, we used a Laplace approximation for the posterior distribution [16].
We use the
GLM-UCB [12] for logistic regression.
Graph model.
We consider different types of graphs.
For each graph, the node values are sampled from standard Gaussian distribution and the noise are sampled from Gaussian with zero mean and σ 2 (cid:15) variance.
We tested σ 2 (cid:15) = 0 .
We experiment on graphs with N = 900 .
For a better demonstration of the early stage performance, we only show the ﬁrst 50 steps.
We experiment the following types of graphs. 1)
Random connections: any node has a probability of p (= 0 . 01) to be connected; 2) Complete graph: every pair of nodes has an edge; 3) Star graph: 2 / 3 of nodes are randomly connected with a probability of p (= 0 . 01) .
And the rest of 1 / 3 nodes are connected to all the nodes.
Note that an example of star graph is given in Figure 2.
Since, no node in complete graph provides more information than others, we expect TS and IDS perform the same.
We expect IDS outperforms TS more signiﬁcantly in star graphs than it does in random graphs as there are more nodes gaining global information.
Low-rank matrix We sample each entry of U ∈ R d × r and V ∈ R d × r from N (0 , σ 20 ) and let the unknown matrix Y = UV T + (cid:15), with each entry of (cid:15) from N (0 , σ 2 (cid:15) ) .
Since we do not have a closed form for the posterior distribution, posterior samples are generated from a MCMC algorithm.
In the experiment, we let m = 30 , which gives us 900 steps at maximum.
For a better demonstration of the early stage, we show the cumulative regrets in the ﬁrst 100 steps.
Results.
Figure 1 shows the simulations results for the four models above.
IDS has lower cumulative regrets over all the steps in linear model, logistic model, graph random and graph star model.
TS and IDS always outperform random policy and UCB (if available), showing a beneﬁt of developing algorithms for ASD.
In the graph experiments, IDS and TS performs almost the same for complete graph because any input gains information of the full graph.
IDS has slightly lower regrets than TS in random graph because of the weak structural information.
It has signiﬁcantly lower regrets than TS and random because of the stronger structural information.
As shown in (b, d), the cumulative regrets at steo 100 grows linearly with d .
A higher noise also leads to a higher regret as shown in (f, h, j, l).
IDS performs better in different levels of d for linear models and different levels of σ for graph random, graph star, matrix.
L i n ea r m od e l
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
r a ph (r a ndo m )
[Figure]
a ph ( s t a r)
[Figure]
240 260 100 - s t e p R e g r e t s
IDSTSRandom (l) Regret at T =
5.3 Reaction condition discovery
To further test the usefulness of IDS and TS for ASD in real-world problems we consider a reaction condition discovery problem in organic chemistry.
Background and dataset.
In the chemical literature, a single reaction condition that gives the highest yield for one set of reacting molecules is often demonstrated on a couple dozen analogous molecules.
However, for molecules with structures that signiﬁcantly differ from this baseline, one needs to rediscover working reaction conditions.
This provides a signiﬁcant challenge as the number of plausible reaction conditions can grow rapidly, due to factors such as the many combinations of individual reagents, varying concentrations of the reagents, and temperature.
Therefore, strategies that can identify as many high-yielding reaction conditions as early as possible would greatly facilitate the preparation of functional molecules.
Similar reaction condition discovery problem is studied under a transfer learning framework [30, 37].
The IDS algorithms may therefore be highly useful in examining experimental data compared to UCB,
TS and random policies.
Two experimental datasets on reactions that form carbon-nitrogen bonds, which are of high importance for synthesizing drug molecules, were therefore selected as speciﬁc test cases.
The ﬁrst dataset, named Photoredox Nickel Dual-Catalysis (PNDC)
[11], comes from a reaction that involves two types of catalysts (a photocatalyst and a nickel catalyst) that work together to form carbon-nitrogen bonds.
It has n = 80 reaction conditions for discovery.
The variables in the dataset are the identity of photocatalysts, relative catalyst concentration, and relative substrate concentration, totaling 80 reaction conditions.
The second C-N Cross-Coupling with Isoxazoles (CNCCI) dataset [2] comes from a collection of a cross-coupling reactions in the presence of an additive.
This dataset was designed to test the reaction’s tolerance to the additive (isoxazole) and includes 330 candidate reactions.
Both datasets have continuous response variables (reaction yield).
Results.
Table 1 demonstrated the results of linear IDS, TS, UCB for PNDC and CNCCI.
We tuned hyperparameters for all three algorithms.
Details are given in Appendix H. Over 17 steps,
IDS showed a dominant performance in all 11 molecules tested in PNDC with an average of 1. more discoveries than random policy.
Either IDS or TS performs the best in the 9 combinations of catalyst and base tested in CNCCI dataset.
Over 66 steps, IDS discovers 14.1 more plausible reaction conditions on average than random.
[Table]
Discussion
In this paper, we posed and comprehensively studied the Adaptive Sampling for Discovery problem and proposed the Information-directed sampling algorithm along with its regret analysis.
The results are complemented with both simulation and real-data experiments.
There are a few open directions following the results in the paper.
First, although our paper shows bandit-like regret bounds, ASD problem by nature is different from bandit in the way that the unlabeled set is diminishing, which may further reduce the complexity of the problem.
For example, in the linear model, all the unlabeled points may fall on a lower dimensional hyperplane, which would reduce the dependence on d .
Second, frequentist regret may be studied instead of adopting a Bayesian framework.
More practical algorithms e.g. random forests and neural network may be considered in the future work.
Sampling from posterior can be slow for certain models.
A fast and generalizable algorithm using IDS for more complicated and larger scale real-data applications may be considered.
Acknowledgement
We acknowledge the support of National Science Foundation via grant IIS-2007055 and CHE1551994.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Supplementary material.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] The limitations are discussed in the Section(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] Depending on the area of application, the impact can be different.
The potential impact of reaction discovery is in 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] The proofs are given in Appendix 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The codes and dataset are in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] They are provided along with code (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Figure 1 and Table 1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] They are given in Appendix H 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We include our code in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] The data are realised with their publications.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes] The data does not have personally identiﬁable information or offensive content.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]