Learning to Extrapolate: A Transductive Approach
Abstract
Machine learning systems, especially overparameterized deep neural networks, can generalize to novel testing instances drawn from the same distribution as the training data.
However, they fare poorly when evaluated on out-of-support testing points.
In this work, we tackle the problem of developing machine learning systems that retain the power of overparametrized function approximators, while enabling extrapolation to out-of-support testing points when possible.
This is accomplished by noting that under certain conditions, a “transductive” reparameterization can convert an out-of-support extrapolation problem into a problem of within-support combinatorial generalization.
We propose a simple strategy based on bilinear embeddings to enable this type of combinatorial generalization, thereby addressing the out-of-support extrapolation problem.
We instantiate a simple, practical algorithm applicable to various supervised learning problems and imitation learning tasks.
Introduction
Generalization is a central problem in machine learning.
Typically, one expects generalization when the test data is sampled from the same distribution as the training set, i e out-of-sample generalization.
However, in many scenarios of interest, test data is sampled from a different distribution from the training set, i.e out-of-distribution ( OOD ).
In some OOD scenarios, the test-distribution is assumed to be known during training – a common assumption made by meta-learning methods.
Several works have tackled a more general scenario of “reweighted” distribution shift [13, 15] where the test distribution shares support with the training distribution, but has a different and unknown probability density; this setting can be tackled via distributional robustness approaches [20, 16].
We explore the scenario where test data is drawn from a distribution which has support outside that of the train distribution.
Formally, assume the problem of learning function h :
ˆ y =
h θ ( x ) using data { ( x i , y i ) }
Ni =1 ∼ D train , where x i ∈ X train , the train domain.
We are interested in making accurate predictions h ( x ) for x / ∈ X train .
Consider an example task of predicting actions to reach a desired goal (Fig 1).
During train, goals are provided from the blue cuboid ( x ∈ X train ) , but test time goals are from the orange cuboid ( x / ∈ X train ).
If f is modelled using a deep neural network, its predictions on test goals in the blue area are likely to be accurate, but in the orange area the performance can be arbitrarily poor.
This challenge manifests itself in a variety of real world problems, ranging from object classification [6] to sequential decision making with reinforcement learning [12] and imitation learning [8].
Reliably deploying learning algorithms in unconstrained environments requires one to account for this type of “out-of-support” distribution shift.
If one can identify some structure in the training data that constrains the behavior of optimal predictors on novel data, then extrapolation may become possible.
Several methods can extrapolate if the nature of distribution shift is known apriori: convolution neural networks are appropriate if a test-time training pattern appears at an out-of-distribution translation.
Similarly, accurate predictions can be made for object point-clouds in out-of-support orientations by building in SE(3) equivariance [9, 18].
Another way to extrapolate is if the model class is known apriori: fitting a linear function to a linear problem will extrapolate.
Similarly, methods like NeRF [26] use physics of image formation to learn a 3D model of a scene which can synthesize images from novel viewpoints.
In this work, we propose an alternative structural condition under which out-of-support extrapolation is feasible.
Typical machine learning approaches are inductive: decision making rules are inferred from train data and employed for test predictions.
An alternative to induction is transduction [10] or analogy-making where a test example is compared with training examples to make predictions.
Our main insight is that in the transductive view of machine learning, out-of-support extrapolation can be reparameterized as a combinatorial generalization problem, which, under certain low-rank and coverage conditions [17, 2, 4, 3], admits a solution.
In this work we show how we can (i) re-parameterize out-of-support inputs h ( x test )
→ h (∆ x, x ′ ) , where x ′ ∈ X train , when provided a representation of measure of difference ∆ x between x test and x ′ .
(ii) Provide conditions under which h (∆ x, x ′ ) makes accurate predictions for unseen combinations (∆ x, x ′ )
(iii) based on a theoretically justified bilinear modeling approach: h (∆ x, x ′ )
→ f (∆ x ) T g ( x ′ ) , where f and g map their inputs into same dimension vector spaces.
(iv)
Show empirical results demonstrating generality of extrapolation of our algorithm on: (a) regression for analytical functions and high-dimensional data; (b) sequential decision making tasks.
Setup
Notation.
Given a space of inputs X and targets Y , we aim to learn a predictor h θ : X → P ( Y )which best fits a ground truth function h
⋆ : X → Y .
Given some non-negative loss function ℓ : Y × Y → R ≥ 0 on the outputs (e.g., square loss), and a distribution D over X , risk is defined as
[Equation]
Various choices of train ( D train ) and test ( D test ) distributions yield different generalization settings:
In-Distribution Generalization.
This setting assumes D test =
D train .
The challenge is to ensure that with N samples from D train , the expected risk R ( h θ ; D test )
=
R ( h θ ; D train ) is small.
This is a common paradigm in both empirical supervised learning (e.g. [19]) and in standard statistical learning theory (e.g. [23]).
Out-of-Distribution ( OOD ).
This is more challenging and requires accurate predictions when
D train ̸ = D test .
When the ratio between the density function of D test to that of D train is bounded, rigorous OOD extrapolation guarantees exist and are detailed in Appendix A.3.
Such a situation arises when D test shares support with D train but is differently distributed as depicted in Fig 2a.
Out-of-Support ( OOS ).
There are innumerable forms of distribution shift in which density ratios are not bounded.
The most extreme case is when the support of D test is not contained in that of D train .
I.e., when there exists some X ′ ⊂
X such that P x ∼D test [ x ∈ X ′ ] > 0 , but P x ∼D train [ x ∈ X ′ ]
=(see Fig 2b).
We term the problem of achieving low risk on such a D test as OOS extrapolation.
Out-of-Combination ( OOC ).
This is a special case of OOS .
Let X =
X 1 ×
X 2 be the product of two spaces.
Let D train , X 1 , D train , X 2 denote the marginal distributions of x 1 ∈ X 1 , x 2 ∈ X 2 under D train , and D test , X 1 , D test , X 2 under D test .
In OOC learning, D test , X 1 , D test , X 2 are in the support of D train , X 1 , D train , X 2 , but the joint distributions D test need not be in the support of D train .
2 Throughout, we let P ( Y ) denote the set of distributions supported on Y .
[Figure]
Bilinear Transduction
To convert OOS to OOC , we require that X have a subtraction operator such that x − x ′ is well-defined for x, x ′ ∈ X .
Let ∆ X := { x − x ′ : x, x ′ ∈ X} .
We propose a transductive re-parameterization h θ : X → P ( Y ) with a deterministic function ¯ h θ : ∆ X × X
→ Y as h θ ( x ) :
=
¯ h θ ( x − x ′ , x ′ ) , where x ′ is referred to as an anchor point for a query point x .
Our basic proposal for OOS extrapolation is unweighted transduction , depicted in Algorithm 1: at train time, a predictor ¯ h θ is trained to make predictions for train points x i drawn from the training set
D train based on their similarity with other points x j also drawn from D train :
¯ h θ ( x i − x j , x j ) .
The train pairs x i , x j are sampled uniformly from D train .
At test time, for an OOS point x test , we first select an anchor point x i from the train set which has similarity with the test point x test − x i that is within some radius ρ of the train similarities distribution ∆ X train .
We then predict the value for x test based on the anchor point x i and the similarity of the test and anchor points: ¯
h θ ( x test − x i , x i ) .
1: Input: distance parameter ρ , training set ( x 1 , y 1 ) , . .
, ( x n , y n ) .
2: Train:
Train θ on loss L ( θ ) =
(cid:80) ni
=1 (cid:80)
j ̸ = i ℓ (¯ h θ ( x i − x j , x j ) , y i ) 3: Test:
For the supervised regression setting, we compute differences directly between inputs x i , x j ∈ X .
For goal-conditioned imitation learning, we compute difference between states x i , x j ∈ X sampled uniformly over demonstration trajectories.
At test time, we select an anchor trajectory based on the goal, and transduce each anchor state in the anchor trajectory to predict a sequence of actions for a test goal.
In practice, we select ρ to be some percentile of differences ∥ x test − x i − ∆ x ∥ . We provide formal theoretical analysis of transductive bilinear predictors and conditions under which
OOS extrapolation can be achieved in Appendix A.
Experiments
What types of problems satisfy the assumptions for extrapolation?
We consider functions with different structure: a periodic function with mixed periods (Fig 3a), a sawtooth function (Fig 3b) and a polynomial function (Fig 3c).
Standard deep networks (yellow) fit the training points well (blue), but fail to extrapolate to OOS inputs (orange).
In comparison, our approach (pink) accurately extrapolates on periodic functions but is much less effective on polynomials.
This is because the periodic functions have symmetries which induce low rank structure under the proposed re-parameterization.
Going Beyond Known Inductive Biases In Fig 4, we show that bilinear transduction is able to extrapolate even in cases that the ground truth function is not simply translation invariant, but is translation equivariant .
Fig 5 demonstrates bilinear transduction extrapolation for a function neither invariant nor equivariant to translation, compared with an equivariant baseline (green).
How does the relationship between the training distribution and testing distribution affect extrapolation behavior?
We show in Fig 6 that for a particular “width” of the training distribution
[Figure]
(size of the training set), OOS extrapolation only extends for one “width” beyond the training range since the conditions for ∆ X being in-support are no longer valid beyond this point.
4.1 Analyzing OOS extrapolation on larger scale decision making problems
Baselines: Linear Model : linear function approximator to check whether linear models can solve the problem.
Neural Networks : typical training of overparameterized neural network function approximators.
Alternative Techniques with Neural Networks ( DeepSets [25]): an alternative architecture for combining multiple inputs, that are meant to be permutation invariant and encourage a degree of generalization between different pairings of states and goals.
Transductive Method without a mechanism for Structured Extrapolation ( Transduction ): transduction with no special structure, to check the impact of bilinear embeddings and low rank structure.
This baseline uses reparameterization, and h θ is a standard neural network.
OOS Extrapolation in Sequential Decision Making :
• Extrapolation to OOS Goals: We consider two tasks from the Meta-World benchmark [24] where a simulated robotic agent needs to reach or push a target object to a goal location (Fig 7).
Given a set of expert demonstrations reaching/pushing to goals in the blue box, we tested generalization to OOS goals in the orange box, using a simple extension of our method to perform transduction over trajectories
rather than individual states.
We quantify performance by measuring the distance between the conditioned and reached goal.
Results in Table 1 show that on the easy task of reaching, training a typical linear or a neural network based predictor extrapolate as well as our method.
However, for the more challenging task of pushing an object, our extrapolation is better by an order of magnitude than other baselines, showing the ability to generalize to goals in a completely different direction.
• Extrapolation with large state and action space: Next we tested our method on grasping and placing an object to OOS goal-locations in R 3 with an anthropomorphic “Adroit" hand that has a much larger action ( R 30 ) and state ( R 39 ) space.
Bilinear transduction is able to scale up to high dimensional state-action spaces as well and is naturally able to grasp the ball and move it to new target locations (with train and test distributions indicated in Fig 7).
• Extrapolation to OOS Dynamics: Lastly, we consider a slider task where the goal is to move a slider on a table to strike a ball such that it rolls to a fixed target position.
The mass of the ball varies across episodes and is provided as input to the policy.
We train and test on a range of masses (Fig 7).
Bilinear transduction is able to successfully extrapolate to new masses and adjust behavior accordingly, showing the ability to extrapolate not just to goals, but also to varying dynamics.
OOS Extrapolation in Higher Dimensional Regression Problems .
To scale up the dimension of the input space, we consider the problem of predicting valid grasping points (in R 3 ) from point clouds of various objects (bottles, mugs and teapots) with different orientations, positions and scales (Fig 7).
In this domain, we represent entire point clouds by a low dimensional representation of the point cloud obtained via PCA.
We consider situations where the objects are not individually identified but instead a single grasp point predictor is trained on the entire set of bottles, mugs and teapots.
We assume access to category labels at training time, but do not require this at test time.
[Table]
Discussion
In this work, we consider the problem of out-of-support extrapolation in regression and sequential decision making problems.
We show that under some assumptions, extrapolation problems can be reparameterized using transduction to be viewed as combinatorial generalization problems.
This allows us to leverage techniques from low-rank matrix completion in order to solve the combinatorial generalization problem.
While our work serves as an initial study of the circumstances under which problem structure can be both discovered and exploited for understanding extrapolation, there are a number of natural questions for further research.
First, can we classify which set of real-world domains fit our assumptions, beyond the domains we have demonstrated?
Second, can we learn a latent space in which differences ∆ x are meaningful for high dimensional domains?
And lastly, are there more effective schemes for selecting anchor points?
A Appendix A.1 Transductive Predictors: Converting OOS to OOC
To convert OOS to OOC , we require the input space X to have group structure, i.e. there are addition and subtraction operators such that x + x ′ , x − x ′ are well-defined for x, x ′ ∈ X .
Let ∆ X := { x − x ′ : x, x ′ ∈ X} .
We propose a transductive re-parameterization h θ : X → P ( Y ) with a deterministic function ¯ h θ : ∆ X × X → Y as
[Equation]
where x ′ is referred to as an anchor point for a query point x .
Under this re-parameterization, the training distribution can be rewritten as a joint distribution of train ∆ x = x − x ′ and x ′ as follows
[Equation]
This is just representing the prediction for every point from the training distribution in terms of it’s relationship to other points in the training distribution.
At test time, we are presented with point x ∼ D test that may be from an OOS distribution.
To make a prediction on this OOS x , we make the observation that with a careful selection of an anchor point x ′ , our reparameterization may be able to convert this OOS problem into a more manageable OOC one, since representing the test point x in terms of it’s difference from training points can still be an “in-support” problem.
To do so, we select an anchor point x ′ from D train as follows.
For a radius parameter ρ > 0 , define the distribution of chosen anchor points D trns ( x ) (referred to as a transducing distribution) as
[Equation]
where X train denotes the set of x in our training set, and denote ∆ X train :
= { x 1 − x 2 : x 1 , x 2 ∈
X train } .
Intuitively, our choice of D trns ( x ) selects anchor points x ′ to transduce from the training distribution, subject to the resulting differences ( x − x ′ ) being close to a “seen” ∆ x ∈ ∆ X train .
In doing so, both the anchor point x ′ and the difference ∆ x have been seen individually at training time, albeit not in combination.
This allows us to express the prediction for a OOS query point in terms of an in-support anchor point x ′ and an in-support difference ∆ x (but not jointly in support).
This choice of anchor points induces a joint test distribution of ∆ x = x − x ′ and x ′ :
[Equation]
As seen from Fig 8, the marginals of ∆ x and x ′ under ¯ D test , are individually in the support of those under ¯ D train .
Still, as Fig 8 reveals, since x test is out-of-support, the joint distribution of ¯ D test is not covered by that of ¯ D train (i.e. the combination of x 1 and x test have not been seen together before); precisely the OOC regime.
Moreover, if one tried to transduce all x ′ ∼ D train to x ∼ D test at test time (e.g. transduce point x 3 to x test in the figure) then we would lose coverage of the ∆ x -marginal.
By doing transduction to keep both the marginal x ′ and ∆ x in support, we are ensuring that we can convert difficult OOS problems into (potentially) more manageable OOC ones.
A.2 Bilinear representations for OOC learning
Without additional assumptions, OOC extrapolation may be just as challenging as OOS .
However, with certain low-rank structure it can be feasible [17, 2, 5].
This is best illustrated in the case of matrix completion (see Fig 9a): let us consider a finite set of x, ∆ x , such that the OOC problem can be viewed as one of matrix completion (with rows and columns as ∆ x , x respectively).
Consider a rank- p matrix M ∈ R n × m , and let n 1 ≤ n and m 1 ≤ m be such that the top-left n 1 × m 1 block of
M , denoted M 11 , has rank p .
Then, one can complete entries of M , given only access to all entries ( i, j ) for which either i ≤ n 1 or j ≤ m 1 .
Such completion can be performed using SVD, where
M = U Σ V T , where U ∈ R n × p , V ∈ R m × p , Σ ∈ R p × p .
M can we written as a bilinear function:
M = U ′ V T , where U ′ =
U Σ .
[Figure]
[Equation]
Following [1], we recognize that this low-rank property can be leveraged implicitly for our reparameterized OOC problem even in the continuous case (where x, ∆ x do not explicitly form a finite dimensional matrix) using a bilinear representation of the transductive predictor in Eq.
(A.1), ¯ h θ (∆ x, x ′ )
= ⟨ f θ f (∆ x ) , g θ g ( x ′ ) ⟩ .
Here f θ f , g θ g map their respective inputs into a vector space of the same dimension (say p ).
If the output space is K dimensional, then we independently model the prediction for each dimension using a set of K bilinear embeddings:
[Equation]
While ¯ h θ ,k are bilinear in embeddings f θ ,k , g θ ,k , the embeddings themselves may be parameterized by general function approximators.
The effective “rank" of the transductive predictor is controlled by the dimension of the continuous embeddings f θ ,k (∆ x ) , g θ ,k ( x ′ ) .
To illustrate the connection to matrix completion, we can imagine our predictors in Eq.
(A.5) as matrices defining large look-up tables for each (∆ x, x ′ ) pair.
See Fig 9b and a more detailed exposition in Section A.4.
Leveraging the analysis of matrix completion in [17], we next provide formal theoretical analysis of transductive bilinear predictors and conditions under which OOS extrapolation can be achieved.
A.3 Generalization under bounded density ratio
The following gives a robust, quantitative notion of when one distribution is in the support of another.
For generality, we state this condition in terms of general positive measures µ 1 , µ 2 , which need not be normalized and sum to one.
Definition A.1 ( κ -bounded density ratio) .
Let µ 1 , µ 2 be two measures over a space Ω .
We say µhas κ -bounded density with respect to µ 2 , which we denote µ 1 ≪ κ µ 2 , if for all measurable 3 A ⊂ Ω , µ 1 [ A ] ≤ κµ 2 [ A ] .
Stating Definition A.1 for general probability affords us the flexibility to write example, P 1 ≪ κ P 2 + P 3 , as P 2 + P 3 is a nonnegative measure with total mass 1 + 1 = 2 .
The following lemma motivates the use of Definition A.1.
Its proof is standard but included for completeness.
Lemma A.1.
Let µ 1 , µ 2 be measures on the same measurable space Ω , and suppose that µ 2 ≪ κ µ 1 .
Then, for any nonnegative function ϕ , µ 2 [ ϕ ] ≤ κµ 1 [ ϕ ] .
In particular, if D test ≪ κ D train , then as long as our loss function ℓ ( · , · ) is nonnegative,
[Equation]
Thus, up to a κ -factor, R ( h θ ; D test ) inherits any in-distribution generalization guarantees for
R ( h θ ; D train ) .
Proof.
As in standard measure theory (c.f.
[7, Chpater 1]), we can approximate any ϕ ≥ 0 by a sequence of simple functions ϕ n ↑ ϕ , where ϕ n ( ω ) =
(cid:80) k n
i =1 c n,i I { ω ∈
A n,i } , with A n,i ⊂ Ω and c n,i ≥ 0 .
For each ϕ n , we have
[Equation]
The result now follows from the monotone convergence theorem.
To derive the special case for D test and D train , apply the general result with nonnegative function ϕ ( x ) =
E y ∼ h θ ( x ) ℓ ( y,
h ⋆ ( x ))
(recall ℓ ( · , · ) ≥ 0 by assumption), µ 1 = D train and µ 2 = D test .
A.4 Extrapolation for Matrix Completion
In what follows, we derive a simple extrapolation guarantee for matrix completion.
The following is in the spirit of the Nyström column approximation (see e.g. [11]), and our proof follows the analysis due to [17].
Throughout, consider
[Equation]
where we decompose ˆ M , M ⋆ into blocks ( i, j ) ∈ { 1 , 2 } 2 for dimension n i × m j .
Lemma A.2.
Suppose that ˆ M is rank at most p , M ⋆ is rank p , and
[Equation]
and ∥ M ⋆i,j ∥ F ≤ M, where ϵ ≤ σ p ( M ⋆ 11 ) / 2 .
Then,
[Equation]
Proof.
The proof mirrors that of [17, Proposition 13].
We shall show below that ˆ M is of rank exactly p .
Hence, [17, Lemma 12] gives the following exact expression for the bottom-right blocks,
[Equation]
where above ( · ) † denotes the Moore-Penrose pseudoinverse.
Since ∥ ˆ M 11 − M ⋆ 11 ∥ op ≤ ∥ ˆ M 11 −
M ⋆ 11 ∥ F ≤ ϵ ≤ σ p ( M ⋆ 11 ) / 2 , Weyls inequality implies that ˆ M 11 is rank p (as promised), and ∥ ˆ M † 11 ∥ op ≤ 2 σ p ( M ⋆ 11 ) − 1 .
Similarly, as ∥ ˆ M 12 − M ⋆ 12 ∥ op ≤ σ p ( M ⋆ 11 ) / 2 ≤ M/ 2 , so ∥ ˆ M 12 ∥ op ≤ 32 M .
Thus,
[Equation]
Next, using a perturbation bound on the pseudoinverse 5 due to [14, Theorem 2.1],
[Equation]
Therefore, we conclude
[Equation]
A.5 General Analysis for Combinatioral Extrapolation
We now provide our general analysis for combinatorial extrapolation.
To avoid excessive subscripts, we write X =
W × V rather than X =
X 1 × X 2 as in the main body.
We consider extrapolation under the following definition of combinatorial support.
Definition A.2 (Bounded combinatorial density ratio, generic definition) .
We say D test has κbounded combinatorial density ratio with respect to D train , written as D test comb ≪ κ D train , if there exist distributions D W ,i and D V ,j , i, j ∈ { 1 , 2 } , over W and V , respectively,
such that D i ⊗ j :=
D W ,i ⊗ D V ,j satisfy
[Equation]
where we recall the definition of κ -bounded density ratio notation ≪ κ in Definition A.1.
Remark A.1 (Connection to matrix completion) .
This definition of combinatorial support is the distributional equivalent of the four-block matrix completion setting depicted in Fig 9, where D train covers the top-left, bottom-left, and top-right blocks (corresponding to D i,j for ( i, j ) ̸
=
(2 , 2) , but
D test may also include samples from the bottom-right block as well (corresponding to D 2 × 2 ).
It is this structure that allows us to leverage the matrix-completion guarantee Lemma A.2 from the previous section to establish a combinatorial extrapolation guarantee below.
For simplicity, we consider scalar predictors, as the general result for vector valued estimators can be obtained by stacking the components.
Specifically, we consider a ground-truth predictor h ⋆ and estimator ˆ h of the form
[Equation]
Lastly, we choose the (scalar) square-loss, yielding the following risk
[Equation]
Throughout, we assume that all expectations that arise are finite.
Our main guarantee is as follows.
Theorem 1.
Define the effective singular value
[Equation]
and suppose that max 1 ≤ i,j ≤ 2 E D i ⊗ j | h ⋆ ( w, v ) | 2 ≤ M 2 ⋆ .
Then, if R (ˆ h ; D train ) ≤ σ 2 ⋆ 4 κ ,
[Equation]
A.5.1 Proof of Theorem
First, let us assume the following two conditions hold; we shall derive these conditions from the conditions of Theorem 1 at the end of the proof:
[Equation]
(A.8)
Our strategy is first to prove a version of Theorem 1 for when W and V have finite cardinality by reduction to the analysis of matrix completion in Lemma A.2, and then extend to arbitrary domains via a limiting argument.
Lemma A.3.
Suppose that Eq.
(A.8) hold, and in addition, that W and V have finite cardinality.
Then,
[Equation]
Proof Lemma A.3.
By adding additional null elements to either W or V , we may assume without loss of generality that |W| = |V| = d , and enumerate their elements { w 1 , . . .
, w d } and { v 1 , . .
, v d } .
Let p i,a =
Pr w ∼D W ,i [ w = w a ] and q j,b
= Pr v ∼D V ,j [ v = v b ] .
Consider matrices ˆ M , M ⋆ ∈ R 2 d × 2 d , with d × d blocks
[Equation]
We then verify that
[Equation]
and thus ∥ ˆ M ij − M ⋆ij ∥ 2F ≤ ϵ 2 for ( i, j ) ̸ = (2 , 2) .
Furthermore, define the matrices ˆ A
i , ˆ B j via
[Equation]
and define A ⋆i , B ⋆j similarly.
Then,
[Equation]
, showing that rank( ˆ M 1 ) , rank( ˆ M 2 ) ≤ p .
Finally, by Eq.
(A.7),
[Equation]
Lastly, we have
[Equation]
Thus
[Equation]
Eq.
(A.9) and Lemma A.2 imply that
[Equation]
Lemma A.4.
Suppose that Eq.
(A.8) hold, but unlike Lemma A.4, W and V need not be finite spaces.
Then, still, it holds that
[Equation]
Proof of Lemma A.4.
For n ∈ N , define h ⋆,n = ⟨
f ⋆,n , g ⋆,n ⟩ and ˆ h n = ⟨ ˆ f n , ˆ g n ⟩ , where f ⋆,n , ˆ f n , ˆ g n , g ⋆,n are simple functions (i.e. finite range, see the proof of Lemma A.1) converging to f ⋆ , ˆ f, g ⋆ , ˆ g .
Define
[Equation]
By the dominated convergence theorem 7 ,
[Equation]
In particular, as ϵ 2 ≤ σ 2 / 4 , then applying Lemma A.3 for n sufficiently large,
[Equation]
Indeed, for any fixed n , all of ˆ f n , ˆ g n , f ⋆,n , g ⋆,n are simple functions, so we can partition W and V into sets on which these embeddings are constant, and thus treat W and V as finite domains; this enables the application of Lemma A.3 applies.
Finally, using the dominated covergence theorem one last time,
[Equation]
We can now conclude the proof of our proposition.
Proof of Theorem 1.
As D test ≪ κ (cid:80) i,j D i ⊗ j and (cid:80) i,j ̸ =(2 ,
[Equation]
D i ⊗ j ≪ κ D train , Lemma A.1 and additivity of the integral implies
[Equation]
Moreover, setting ϵ 2 :=
κ R (ˆ h ; D train ) , we have
[Equation]
Thus, for R (ˆ h ; D train ) <
σ 2 ⋆ 4 κ , Eq.
(A.8) holds and thus Lemma A.4 entails
[Equation]
Thus, combining with Eq.
(A.10),
[Equation]
D test ) ≤
[Equation]
completing the proof.
A.6 Extrapolation for Transduction
Leveraging Theorem 1, this section provides a formal theoretical justification for predictors of the form Eq.
(A.5).
We begin by stipulating the requisite conditions.
First, we require well-specification : that h ⋆ ( · ) can also be expressed in the form Eqs.
(A.1) and (A.5); to ensure ¯ h ⋆ ( · , · ) is well-defined as a deterministic predictor (whereas h θ ( · ) need not be), we need the following, rather strong condition on h ⋆ ( · ) .
Assumption A.1.
We assume that h ⋆ is bilinearly transducible ; that is, there exists f ⋆,k : ∆ X → R p and g ⋆,k : X → R p such that for all x ∈ X , the following holds with probability 1 over x ′ ∼
D trns ( x ) :
[Equation]
where ∆ x = x − x ′ .
Assumption A.1 means that for any feature x , any feature x ′ in the support of D trns ( x ) be transduced to x via bilinear embeddings.
Next, our theory requires that the distributions ¯ D train , ¯ D test defined in Eqs.
(A.2) and (A.4) satisfy the notion of combinatorial support given in the previous section.
Definition A.3 (Bounded combinatorial density ratio, specialized to transduction) .
We say that ¯ D test has κ -bounded combinatorial density ratio with respect to ¯ D train , written as ¯ D test comb ≪ κ ¯ D train , if it abides by Definition A.2.
That is, there exists distributions D ∆ X ,i and D X ,j , i, j ∈ { 1 , 2 } , over ∆ X and X , respectively, such that D i ⊗ j :
=
D ∆ X ,i ⊗ D X ,j satisfy (cid:88)
[Equation]
and ¯ D
[Equation]
≪
[Equation]
(cid:88)
[Equation]
where we recall the definition of κ -bounded density ratio notation ≪ κ in Definition A.1.
Let us recall the discussion of Remark A.1.
Building upon Definition A.1, Definition A.3 introduces a notion of bounded density ratio between ¯ D train and ¯ D test in the OOC setting.
Take the discrete case of matrix completion as an example, as illustrated in Fig 9, the training distribution of (∆ x, x ′ ) covers the support of the (1 , 1) , (1 , 2) , (2 , 1) blocks of the matrix, while the testing distribution of (∆ x, x ′ ) might be covered by any product of the marginals of the 2 × 2 blocks.
With this connection in mind, it is possible to establish the OOC guarantees on ¯ D test as in matrix completion tasks, if the bilinear embedding admits some low-rank structure.
Theorem 2.
Suppose that h ⋆ is bilinearly transducible (Assumption A.1), h θ takes the form of
Eqs.
(A.1) and (A.5) , and for each k ∈
[ K ] , the embeddings f ⋆,k , f θ ,k , g ⋆,k , g θ ,k are all of dimension p .
Further, suppose there exist κ ≥ 1 and M ≥ σ > 0 such that ¯ D test comb ≪ κ ¯ D train , and for all k ∈ [ K ] ,
[Equation]
Finally, suppose the loss ℓ ( · , · ) is the square loss.
Then, if R ( h θ ; D train ) ≤ σ 2 4 κ , we have
[Equation]
D
[Equation]
The additional conditions of Theorem 2 beyond those stated in Assumption A.1 and Definition A. are discussed at the end of the section.
Proof of Theorem 2.
We argue by reducing to Theorem 1.
The parameterization of the stochastic predictor h θ in Eq.
(A.1), followed by Assumption A.1 allows us to write
[Equation]
In the above display, the joint distribution of ( x − x ′ , x ′ ) is precisely given by ¯ D train (see Eq.
(A.2)).
Hence,
[Equation]
Further, as ℓ ( y, y ′ ) =
∥
y − y ′
∥ 2 is the square loss and decomposes across coordinates,
[Equation]
D
[Equation]
By the same token,
[Equation]
To conclude the proof, we remain to show that for all k ∈ [ K ] , we have
[Equation]
where C prob = κ 2 (cid:18) 1 + 64 M 4 σ 4 (cid:19) .
[Equation]
Indeed, for each k ∈ [ K ] , we have
[Equation]
Hence Eq.
(A.13) holds by invoking Theorem 1 with the correspondences W ← ∆ X , V ← X , σ ⋆ ← σ , M ⋆ ← M and κ ← κ .
This concludes the proof.
Remarks on additional conditions.
We comment on the three additional conditions of Theorem 2.
• The singular value condition, σ p ( E
[Equation]
mirrors non-degeneracy conditions given in past work in matrix completion (c.f.
• The support condition sup ∆ x,x ′ | ¯ h ⋆,k (∆ x, x ′ )
| ≤ M is a mild boundedness condition, which (in light of Theorem 1) can be weakened further to
[Equation]
where D i ⊗ j are the contituent distributions witnessing ¯ D test comb ≪ κ ¯ D train .
• The final condition, R ( h θ ; D train ) ≤ σ 2 4 κ , is mostly for convenience.
Indeed, as M ≥ σ and κ ≥ 1 , then as soon as R ( h θ ; D train ) >
σ 2 4 κ , our upper-bound on R ( h θ ; D test ) is no better than
[Equation]
which is essentially vacuous.
Indeed, if we also inflate M and stipulate that sup ∆ x,x
[Equation]
| ¯ h θ
[Equation]
∆ x, x ′ ) | ≤ √ 6 M , we can remove the condition R ( h θ ; D train ) ≤ σ 2 4 κ altogether.