Original: Finding Maximally Informative Patches in Images
Revised: Finding Maximally Informative Patches in Images

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: We consider the problem of distilling an image into an ordered set of maximallyinformative patches, given prior data from the same domain.
Revised: We consider the problem of distilling an image into an ordered set of maximally informative patches, given prior data from the same domain.

Original: We cast this problemas one of maximizing a pointwise mutual information (PMI) objective between asubset of an image’s patches and the perceptual content of the entire image.
Revised: We cast this problem as one of maximizing a pointwise mutual information (PMI) objective between a subset of an image’s patches and the perceptual content of the entire image.

Original: Wetake an image synthesis-based approach, reasoning that the patches that are mostinformative would also be most useful for predicting other pixel values.
Revised: We take an image synthesis-based approach, reasoning that the patches that are most informative would also be most useful for predicting other pixel values.

Original: We capturethis idea with an image completion CNN trained to model the PMI between animage’s perceptual content and any of its subregions.
Revised: We capture this idea with an image completion CNN trained to model the PMI between an image’s perceptual content and any of its subregions.

Original: Because our PMI objective isa submodular, monotonic function, we can greedily construct patch sets using the
Revised: Because our PMI objective is a submodular, monotonic function, we can greedily construct patch sets using the

Original: CNN to obtain a provably close approximation to the intractable optimal solution.
Revised: CNN to obtain a provably close approximation to the intractable optimal solution.

Original: We evaluate our approach on datasets of faces, common objects, and line drawings.
Revised: We evaluate our approach on datasets of faces, common objects, and line drawings.

Original: For all datasets, we ﬁnd that a surprisingly few number of patches are neededto reconstruct most images, demonstrating a particular type of redundancy ofinformation in images, and new potentials in their sparse representations.
Revised: For all datasets, we ﬁnd that a surprisingly few number of patches are needed to reconstruct most images, demonstrating a particular type of redundancy of information in images, and new potentials in their sparse representations.

Original: We alsoshow that these minimal patch sets may be used effectively for downstream taskssuch as image classiﬁcation.
Revised: We also show that these minimal patch sets may be used effectively for downstream tasks such as image classiﬁcation.

Original:  Introduction
Revised:  Introduction

Original: Natural image datasets often lie on surprisingly low-dimensional manifolds due to signiﬁcant correlations between image pixels and across images [40].
Revised: Natural image datasets often lie on surprisingly low-dimensional manifolds due to signiﬁcant correlations between image pixels and across images [40].

Original: As a result, the pixel domain tends to be ahighly redundant representation for many vision tasks, where only a subset of a scene needs to beobserved to ensure success.
Revised: As a result, the pixel domain tends to be a highly redundant representation for many vision tasks, where only a subset of a scene needs to be observed to ensure success.

Original: This is illustrated by “attention”-based classiﬁers that focus on imagesubregions [45, 49] and cognitive science studies on “minimal” images for recognition [3, 44].
Revised: This is illustrated by “attention”-based classiﬁers that focus on image subregions [45, 49], and cognitive science studies on “minimal” images for recognition [3, 44].

Original: Putanother way, by just observing a subset of an image’s pixels, we can often infer more about the imagethan just those pixels .
Revised: Put another way, by just observing a subset of an image’s pixels, we can often infer more about the image than just those pixels .

Original: Studying what in an image is predictable given a very limited view of it revealsthe regularities within and across images and can inﬂuence how we choose to store or process them.
Revised: Studying what in an image is predictable given a very limited view of it reveals the regularities within and across images, and can inﬂuence how we choose to store or process them.

Original: This paper reveals the strength of those image regularities through the following task: given an image
Revised: This paper reveals the strength of those image regularities through the following task: given an image

Original: I , a ﬁxed patch size, and training images from the same domain, sequentially determine which patchin I provides the most new information about I .
Revised: I , a ﬁxed patch size, and training images from the same domain, sequentially determine which patch in I provides the most new information about I .

Original: Consider the example in Fig.
Revised: Consider the example in Fig.

Original: Suppose we knowthat the image is in the CelebaHQ dataset, consisting of faces of various celebrities.
Revised: Suppose we know that the image is in the CelebaHQ dataset, consisting of faces of various celebrities.

Original: Our approachwill sequentially select patches coinciding with informative regions — areas both within and outsidethe face — that help to infer the full image.
Revised: Our approach will sequentially select patches coinciding with informative regions — areas both within and outside the face — that help to infer the full image.

Original: In addition to revealing fundamental properties ofimage collections, the answer to this question has potential impact in the design of efﬁcient visionalgorithms, and may relate to human cognition studies of attention and foveation.
Revised: In addition to revealing fundamental properties of image collections, the answer to this question has potential impact in the design of efﬁcient vision algorithms, and may relate to human cognition studies of attention and foveation.

Original: We propose a method for this task that leverages a combination of probabilistic modeling, imagesynthesis, and submodular function optimization.

Original: We ﬁrst propose a measure of pointwise mutualinformation (PMI) between an image and any subset of its patches with a probabilistic notion ofperceptual similarity.

Original: We then design a convolutional neural network that yields an approximation ofthis PMI measure for a data domain of interest.

Original: Given a subset of observed image patches as input,

Original: Minimal Patch Sets
Revised: Minimal Patch Sets

Original: Inferred Images
Revised: Inferred Images

Original: New Image 

Revised: New Image

Original: Patches 1 Patch 2 Patches 3 Patches 5 Patches 10 Patches 20 Patches
Revised: Patches 1 Patch 2 Patches 3 Patches 5 Patches 10 Patches 20 Patches

Original: OriginalImage 

Revised: OriginalImage

Original: Patches 1 Patch 2 Patches 3 Patches 5 Patches 10 Patches 20 Patches
Revised: Patches 1 Patch 2 Patches 3 Patches 5 Patches 10 Patches 20 Patches

Original: OriginalImage 

Revised: OriginalImage

Original: Patches 1 Patch 2 Patches 3 Patches 5 Patches 10 Patches 20 Patches
Revised: Patches 1 Patch 2 Patches 3 Patches 5 Patches 10 Patches 20 Patches

Original: OriginalImage
Revised: OriginalImage

Revised: Prior Images

Revised: We propose a method for this task that leverages a combination of probabilistic modeling, image synthesis, and submodular function optimization.

Revised: We ﬁrst propose a measure of pointwise mutual information (PMI) between an image and any subset of its patches with a probabilistic notion of perceptual similarity.

Revised: We then design a convolutional neural network that yields an approximation of this PMI measure for a data domain of interest.

Original: Prior Images this network synthesizes a completed image, along with uncertainty estimates in perceptual space.
Revised: Given a subset of observed image patches as input, this network synthesizes a completed image, along with uncertainty estimates in perceptual space.

Original: We then use this network to sequentially ﬁnd maximally-informative patches for a test image in agreedy manner, which is an efﬁcient and provably good approximation to the optimal solution.
Revised: We then use this network to sequentially ﬁnd maximally-informative patches for a test image in a greedy manner, which is an efﬁcient and provably good approximation to the optimal solution.

Original: We evaluate our work on several real datasets with different characteristics: face images (CelebaHQ [23, 29]) , common objects (STL-10 [9]) and line drawings (QuickDraw [16]).
Revised: We evaluate our work on several real datasets with different characteristics: face images (CelebaHQ [23, 29]) , common objects (STL-10 [9]) and line drawings (QuickDraw [16]).

Original: We show that,given training data from a particular domain, surprisingly few image patches need to be observed toreconstruct the full image.
Revised: We show that, given training data from a particular domain, surprisingly few image patches need to be observed to reconstruct the full image.

Original: We also explore how this ﬁnding may be useful for performing downstreamtasks like classiﬁcation.
Revised: We also explore how this ﬁnding may be useful for performing downstream tasks like classiﬁcation.

Original:  Related Work
Revised:  Related Work

Original: Our computational work has ties to several areas in cognition, the closest being visual attention.
Revised: Our computational work has ties to several areas in cognition, the closest being visual attention.

Original: Humans selectively attend to parts of the visual space to acquire information, and combine informationfrom these views to form an internal representation of the scene [33].
Revised: Humans selectively attend to parts of the visual space to acquire information, and combine information from these views to form an internal representation of the scene [33].

Original: Attention has been studied inrelation to human reasoning [4, 10], saccades and eye movements [12, 13, 18], motor control [1, 5],and peripheral vision [35].
Revised: Attention has been studied in relation to human reasoning [4, 10], saccades and eye movements [12, 13, 18], motor control [1, 5], and peripheral vision [35].

Original: Our work also has ties to ﬁndings on “minimal images,” the smallestregions of an image that remain recognizable for humans [3, 44].
Revised: Our work also has ties to ﬁndings on “minimal images,” the smallest regions of an image that remain recognizable for humans [3, 44].

Original: Finally, games like Bubbles and
Revised: Finally, games like Bubbles and

Original: Peekaboom were developed to use human attention to annotate salient regions of image data inan interactive way [11, 47].
Revised: Peekaboom were developed to use human attention to annotate salient regions of image data in an interactive way [11, 47].

Original: We draw inspiration from these works, though our method is purelydata-driven and not strongly based on the cognitive system.
Revised: We draw inspiration from these works, though our method is purely data-driven and not strongly based on the cognitive system.

Original: Inspired by the human cognitive system, computer vision researchers have incorporated attentioninto various learning models.
Revised: Inspired by the human cognitive system, computer vision researchers have incorporated attention into various learning models.

Original: Several studies use attention to localize features for ﬁne-grained imageclassiﬁcation [14, 48, 49, 51] and for image captioning and visual question answering tasks [2, 8, 31].
Revised: Several studies use attention to localize features for ﬁne-grained image classiﬁcation [14, 48, 49, 51] and for image captioning and visual question answering tasks [2, 8, 31].

Original: Attention can also provide computational savings for image processing, by isolating interestingregions to explore further [30].
Revised: Attention can also provide computational savings for image processing, by isolating interesting regions to explore further [30].

Original: In contrast to these studies, we focus on understanding how little ofan image must be attended to to understand its contents.
Revised: In contrast to these studies, we focus on understanding how little of an image must be attended to to understand its contents.

Original: Mutual information and submodular function optimization are topics with wide applications in AIand other ﬁelds.
Revised: Mutual information and submodular function optimization are topics with wide applications in AI and other ﬁelds.

Original: They have had a few uses in computer vision too.
Revised: They have had a few uses in computer vision too.

Original: Mutual information has beenused for registration between pairs of images [15, 36, 37, 46], and pointwise mutual informationwas proposed for grouping pixels within an image [19].
Revised: Mutual information has been used for registration between pairs of images [15, 36, 37, 46], and pointwise mutual information was proposed for grouping pixels within an image [19].

Original: Submodular function optimization has beenused to measure image region saliency and to segment image regions [20, 21, 25].
Revised: Submodular function optimization has been used to measure image region saliency and to segment image regions [20, 21, 25].

Original: These worksdonot aim to maximize the information of the entire image from a few patches, and donot optimizea synthesis-based objective. Submodular functions have also been used for tracking and detectingobjects [38, 52], and image collection summarization [43]. While this approach utilizes synthesis-based objective similar to image inpainting works, whatdifferentiates this approach is that the primary focus is to retrieve the maximally informative patches [Figure]
Revised: These works do

Original: [Figure]

Original: Mask ( " # )

Original: [Figure]
Revised: [Figure]

Original: VGG

Original: [Figure]
Revised: [Figure]

Original: Synthesis
Revised: While this approach utilizes synthesis-based objective similar to image inpainting works, what differentiates this approach is that the primary focus is to retrieve the maximally informative patches instead of maximizing reconstruction quality.

Original: [Figure] of maximizing patch method  use isnetwork-agnostic and generalizes to different architectures.
Revised: Therefore, the patch-selection method we use is network-agnostic and generalizes to different architectures.

Original: -the- architectures may providecrisper image reconstructions [32, 28, 7], but the UNet-style architecture we choose sufﬁces inreconstruction quality and achieves the paper’s main purpose of calculating mutual information.
Revised: State-of-the-art architectures may provide crisper image reconstructions [32, 28, 7], but the UNet-style architecture we choose sufﬁces in reconstruction quality and achieves the paper’s main purpose of calculating mutual information.

Original:  Methods
Revised:  Methods

Original: We assume a training dataset D of images from a domain of interest and a ﬁxed patch size s as input.
Revised: We assume a training dataset D of images from a domain of interest and a ﬁxed patch size s as input.

Original: Given an image I ∈ R d from this domain at test time, we aim to construct an ordered set of k patches
Revised: Given an image I ∈ R d from this domain at test time, we aim to construct an ordered set of k patches

Original: R ∗ k ∈ I carrying maximal information about I ’s visual content.
Revised: R ∗ k ∈ I carrying maximal information about I ’s visual content.

Original: Formally, we deﬁne R ∗ k as the set ofk patches with maximal pointwise mutual information ( ρ ) with I :
Revised: Formally, we deﬁne R ∗ k as the set of k patches with maximal pointwise mutual information ( ρ ) with I :

Original: [Equation]
Revised: [Equation]

Original: We estimate p ( I | R k ) using a synthesis-based approach (see Figs.
Revised: We estimate p ( I | R k ) using a synthesis-based approach (see Figs.

Original: 1 and 2).
Revised: 1 and 2).

Original: We ﬁrst specify a form forthis distribution that captures both per-pixel and higher-order perceptual features.
Revised: We ﬁrst specify a form for this distribution that captures both per-pixel and higher-order perceptual features.

Original: This formulationleads to a loss function with trainable parameters θ , which we implement with a CNN that completesimages from a subset of visible patches.
Revised: This formulation leads to a loss function with trainable parameters θ , which we implement with a CNN that completes images from a subset of visible patches.

Original: Once trained, this network serves as a functional approximator to the pointwise mutual information,ρ ( · , · ) .
Revised: Once trained, this network serves as a functional approximator to the pointwise mutual information, ρ ( · , · ) .

Original: Unfortunately, ﬁnding the optimal set R ∗ k for a given k is a combinatorial, intractable problem.
Revised: Unfortunately, ﬁnding the optimal set R ∗ k for a given k is a combinatorial, intractable problem.

Original: We instead leverage the fact that ρ ( · , · ) is a monotone submodular function, and use a greedypatch selection strategy that is a provably good approximation to the optimal solution.
Revised: We instead leverage the fact that ρ ( · , · ) is a monotone submodular function, and use a greedy patch selection strategy that is a provably good approximation to the optimal solution.

Original: Using asynthesis-driven objective to measure information content is attractive for several reasons.
Revised: Using a synthesis-driven objective to measure information content is attractive for several reasons.

Original: First, it isunsupervised, and therefore applicable to any image dataset.
Revised: First, it is unsupervised, and therefore applicable to any image dataset.

Original: Second, since it is not tied to a particulartask like classiﬁcation or segmentation, our ﬁndings are applicable to any downstream task operatingon the images.
Revised: Second, since it is not tied to a particular task like classiﬁcation or segmentation, our ﬁndings are applicable to any downstream task operating on the images.

Original: Finally, a synthesis network can tractably model long-range pixel dependencies, unlikeapproaches that only compare pixels over local neighborhoods [6, 19, 27].
Revised: Finally, a synthesis network can tractably model long-range pixel dependencies, unlike approaches that only compare pixels over local neighborhoods [6, 19, 27].

Original: 3.1 Probabilistic Formulation
Revised: 3.1 Probabilistic Formulation

Original: We build on existing works that combine per-pixel and “perceptual” losses for realistic imagesynthesis [22, 50].
Revised: We build on existing works that combine per-pixel and “perceptual” losses for realistic image synthesis [22, 50].

Original: A per-pixel loss treats each pixel as an independent observation, and the perceptualloss measures image differences in a more complex feature space φ ( · ) : R d → R | φ | deﬁned bythe activations of an L -layer image classiﬁer, e.g., VGG16 [39] ( L = 5 in our experiments).
Revised: A per-pixel loss treats each pixel as an independent observation, and the perceptual loss measures image differences in a more complex feature space φ ( · ) : R d → R | φ | deﬁned by the activations of an L -layer image classiﬁer, e.g., VGG16 [39] ( L = 5 in our experiments).

Original: Weapproximate p ( I | R ) with these ideas using a factored form:
Revised: We approximate p ( I | R ) with these ideas using a factored form:

Original: [Equation]
Revised: [Equation]

Original: where θ are function parameters to be trained over data.
Revised: where θ are function parameters to be trained over data.

Original: We deﬁne the per-pixel distribution as:
Revised: We deﬁne the per-pixel distribution as:

Original: [Equation]
Revised: [Equation]

Original: where µ I ( · ; θ ) ∈ R
Revised: where µ I ( · ; θ ) ∈ R

Original: d is a mean image, and Σ I ( · ; θ
Revised: d is a mean image, and Σ I ( · ; θ

Original: ) ∈ R
Revised: ) ∈ R

Original: d × d is a diagonal covariance matrix (i.e.,effectively also in R d ).
Revised: d × d is a diagonal covariance matrix (i.e., effectively also in R d ).

Original: Estimating a different variance for each pixel accounts for the heteroscedasticuncertainty of our task [24] due to certain pixels being easier to model than others given R .
Revised: Estimating a different variance for each pixel accounts for the heteroscedastic uncertainty of our task [24] due to certain pixels being easier to model than others given R .

Original: We deﬁne the perceptual distribution as:
Revised: We deﬁne the perceptual distribution as:

Original: [Equation]
Revised: [Equation]

Original: where ˆ I is a sample from p pixel ( ·| R ; θ ) , φ l ( · ) are features from layer l of the perceptual network, andσ 2 φ,l ( · ; θ ) ∈ R is a variance estimate for layer l .
Revised: where ˆ I is a sample from p pixel ( ·| R ; θ ) , φ l ( · ) are features from layer l of the perceptual network, and σ 2 φ,l ( · ; θ ) ∈ R is a variance estimate for layer l .

Original: 3.2 Synthesis CNN and Loss Function
Revised: 3.2 Synthesis CNN and Loss Function

Original: We use a UNet-style architecture [34] to implement the three functions µ I ( · ; θ ) , Σ I ( · ; θ ) and σ 2 φ ( · ; θ )(see Fig. 2).
Revised: We use a UNet-style architecture [34] to implement the three functions µ I ( · ; θ ) , Σ I ( · ; θ ) and σ 2 φ ( · ; θ ) (see Fig. 2).

Original: We output µ I and Σ I from the ﬁnal convolutional layer of the UNet, at full imageresolution.
Revised: We output µ I and Σ I from the ﬁnal convolutional layer of the UNet, at full image resolution.

Original: We estimate the vector σ 2 φ with a fully-connected layer applied to the activations of themiddle (smallest) convolutional block of the UNet.
Revised: We estimate the vector σ 2 φ with a fully-connected layer applied to the activations of the middle (smallest) convolutional block of the UNet.

Original: We represent R with a masked version of image I , obtained by multiplying a binary mask M Relement-wise with I .
Revised: We represent R with a masked version of image I , obtained by multiplying a binary mask M R element-wise with I .

Original: During training, we populate M R with a random number of patches at randomlocations, to teach the model to complete arbitrarily-occluded images.
Revised: During training, we populate M R with a random number of patches at random locations, to teach the model to complete arbitrarily-occluded images.

Original: We optimize the network parameters by minimizing the negative log-likelihood of Eq.
Revised: We optimize the network parameters by minimizing the negative log-likelihood of Eq.

Original: (2) over thetraining dataset D :
Revised: (2) over the training dataset D :

Original: [Equation]
Revised: [Equation]

Original: 3.3 Patch Selection
Revised: 3.3 Patch Selection

Original: At test time we use the trained network to calculate PMI between any set R and test image I .
Revised: At test time we use the trained network to calculate PMI between any set R and test image I .

Original: Constructing the optimal set R ∗ k for value k requires a combinatorial search over many candidateregions with complexity O ( k d ) , and is therefore intractable.
Revised: Constructing the optimal set R ∗ k for value k requires a combinatorial search over many candidate regions with complexity O ( k d ) , and is therefore intractable.

Original: Fortunately, the PMI between a set of variables and any of its subsets is  submodular in expectationand monotone (see supplementary material for proof).
Revised: Fortunately, the PMI between a set of variables and any of its subsets is monotone and submodular in expectation (see supplementary material for proof).

Original: This means that the incremental informationadded by a patch about I decreases as the number of visible patches increases, and that if R ∈ S ∈ I ,
Revised: This means that the incremental information added by a patch about I decreases as the number of visible patches increases, and that if R ∈ S ∈ I ,

Original: PMI ( R, I ) ≤ PMI ( S, I ) .
Revised: PMI ( R, I ) ≤ PMI ( S, I ) .

Original: For such functions, a greedy selection of candidates is a provably goodapproximation, to within a factor of 1 − 1 e of the optimal solution [26].
Revised: For such functions, a greedy selection of candidates is a provably good approximation, to within a factor of 1 − 1 e of the optimal solution [26].

Original: We perform the greedysearch in a brute-force manner, for each patch over a ﬁxed grid of image locations.
Revised: We perform the greedy search in a brute-force manner, for each patch over a ﬁxed grid of image locations.

Original: Evaluating everypixel location is computationally intensive, so we set grid points to be a half-patch-size apart.
Revised: Evaluating every pixel location is computationally intensive, so we set grid points to be a half-patch-size apart.

Original:  Experiments
Revised:  Experiments

Original: We evaluate our work on three popular image datasets: CelebaHQ [23, 29] (high-res faces), STL10 [9] (common animals/objects), and QuickDraw [16] (line drawings).
Revised: We evaluate our work on three popular image datasets: CelebaHQ [23, 29] (high-res faces), STL10 [9] (common animals/objects), and QuickDraw [16] (line drawings).

Original: We randomly split all of the
Revised: We randomly split all of the

Original: CelebaHQ images to construct 28000/1000/1000 train/validation/test sets.
Revised: CelebaHQ images to construct 28000/1000/1000 train/validation/test sets.

Original: We used a subset of the
Revised: We used a subset of the

Original: QuickDraw object classes, by taking the union of the 10 classes in the original study’s experiments,and the object classes in STL-10, resulting in 17 classes 1 .
Revised: QuickDraw object classes, by taking the union of the 10 classes in the original study’s experiments, [Figure] NearestNeighbor and the object classes in STL-10, resulting in 17 classes 1 .

Original: We randomly sampled 5400/300/300train/validation/test images per class.
Revised: We randomly sampled 5400/300/ train/validation/test images per class.

Original: We found that STL-10 contains duplicate images betweenits unlabeled and labeled datasets, so we removed them from the unlabeled set, which we used fortraining
Revised: We found that STL-10 contains duplicate images between its unlabeled and labeled datasets, so we removed them from the unlabeled set, which we used for training.

Original: We used an 87,
Revised: We used an 87,700/5000/1000 split of data.

Original: [Figure]
Revised: We used constant patch areas of 1 / 64 th of the full image area of each dataset.

Original: NearestNeighbor
Revised: For CelebAHQ and QuickDraw this resulted in 32 × 32 patches, and for STL-10 this resulted in 12 × 12 patches. We varied the number of patches from 1-25, covering roughly 1 − 40% of the full image area. We trained separate models for each dataset, and our code is written in PyTorch.

Original: 4.1 Reconstruction Results
Revised: 4.1 Reconstruction Results

Original: Figs. 3, 4 and 5 present sample reconstruction results for several test images from each dataset.
Revised: Figs. 3, 4 and 5 present sample reconstruction results for several test images from each dataset.

Original: Weshow both the reconstructed image, as well as the greedily selected patches from our method fordifferent patch set sizes.
Revised: We show both the reconstructed image, as well as the greedily selected patches from our method for different patch set sizes.

Original: We also display the nearest neighbor to the ground truth image in the trainingset, evaluated using the LPIPS perceptual error [50], for further context.
Revised: We also display the nearest neighbor to the ground truth image in the training set, evaluated using the LPIPS perceptual error [50], for further context.

Original: For QuickDraw, roughly 5 patches give close approximations of the ground truth for most cases, afterwhich smaller details are added with subsequent patches.
Revised: For QuickDraw, roughly 5 patches give close approximations of the ground truth for most cases, after which smaller details are added with subsequent patches.

Original: Selected patches tend to straddle areaswith multiple semantic parts or corners.
Revised: Selected patches tend to straddle areas with multiple semantic parts or corners.

Original: For example, in Fig.
Revised: For example, in Fig.

Original: 3, the ﬁrst patch selected for the face(example 1) straddles the chin and mouth, and the ﬁrst patch for the truck (3rd example) is placed atits top-left corner.
Revised: 3, the ﬁrst patch selected for the face [Figure] [Figure] NearestNeighbor

Original: CelebaHQ contains variabilities in facial attributes, poses, apparel, and background (see Fig. 4).
Revised: CelebaHQ contains variabilities in facial attributes, poses, apparel, and background (see Fig. 4).

Original: Our synthesis method is usually able to establish gender with one greedily-selected patch, such asin examples 1,3, and 4.
Revised: Our synthesis method is usually able to establish gender with one greedily-selected patch, such as in examples 1,3, and 4.

Original: The ﬁrst patch on the lip for example 1, for instance, helps establish bothgender and pose.
Revised: The ﬁrst patch on the lip for example 1, for instance, helps establish both gender and pose.

Original: The fourth example is a case where patches are placed to help ﬁll in both complexbackground details along with the face.
Revised: The fourth example is a case where patches are placed to help ﬁll in both complex background details along with the face.

Original: The second example is particularly challenging due to theatypical hair, pose, and apparel combination.[Figure]
Revised: The second example is particularly challenging due to the atypical hair, pose, and apparel combination.

Original: NearestNeighbor STL-10 is the most challenging of the three datasets, consisting of more unconstrained images, withcomplex backgrounds.
Revised: STL-10 is the most challenging of the three datasets, consisting of more unconstrained images, with complex backgrounds.

Original: But even for this dataset, our method performs well, and is able to recover asurprising number of details within 10 patches.
Revised: But even for this dataset, our method performs well, and is able to recover a surprising number of details within 10 patches.

Original: Some interesting landmarks captured by the patchesinclude the ear of one dog (1st example, 2 patches) as well as the nose of a second dog in the samescene (1st example, 5 patches), part of the leg and beak of the bird (2nd example), and the centerridge of the boat (3rd example, 5 patches).
Revised: Some interesting landmarks captured by the patches include the ear of one dog (1st example, 2 patches) as well as the nose of a second dog in the same scene (1st example, 5 patches), part of the leg and beak of the bird (2nd example), and the center ridge of the boat (3rd example, 5 patches).

Original: We numerically evaluated the reconstruction results against a baseline, random patch selectionapproach.
Revised: We numerically evaluated the reconstruction results against a baseline, random patch selection approach.

Original: The random selector places patches with equal probability at the same grid locations weconsider for the greedy search (see Sec. 3.3).
Revised: The random selector places patches with equal probability at the same grid locations we [Figure] OriginalImage

Revised: NearestNeighbor consider for the greedy search (see Sec. 3.3).

Original: For QuickDraw, we added a constraint that the patchmust contain at least one black (foreground) pixel.
Revised: For QuickDraw, we added a constraint that the patch must contain at least one black (foreground) pixel.

Original: Fig.
Revised: Fig.

Original: 6 presents our results.
Revised: 6 presents our results.

Original: Greedy selectionoutperforms random, with the difference more accentuated as the number of patches approaches 25 .
Revised: Greedy selection outperforms random, with the difference more accentuated as the number of patches approaches 25 .

Original: These results demonstrate that not all patches in an image are equal in regards to information content,and that careful selection is critical.
Revised: These results demonstrate that not all patches in an image are equal in regards to information content, and that careful selection is critical.

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: l a ss i ﬁ c a t i o n A cc u r a c y

Original: OriginalImage
Revised: QuickDraw [Figure] CelebaHQ [Figure]

Original: NearestNeighbor
Revised: STL [Figure] [Figure] 4.2 Patches for Classiﬁcation

Original: We explore the viability of using our patch sets for image classiﬁcation.
Revised: We explore the viability of using our patch sets for image classiﬁcation.

Original: Instead of designing a newclassiﬁer that handles arbitrary patches as input, we evaluate a standard image classiﬁer’s accuracyon synthesized images produced from different sets of input patches.
Revised: Instead of designing a new classiﬁer that handles arbitrary patches as input, we evaluate a standard image classiﬁer’s accuracy on synthesized images produced from different sets of input patches.

Original: Each of the three datasetsprovide annotations: QuickDraw and STL-10 for object classes, and CelebaHQ for 40 facial features.
Revised: Each of the three datasets provide annotations: QuickDraw and STL-10 for object classes, and CelebaHQ for 40 facial features.

Original: We selected seven attributes for CelebaHQ capturing a broad span of face characteristics.
Revised: We selected seven attributes for CelebaHQ capturing a broad span of face characteristics.

Original: We use
Revised: We use

Original: [Figure]

Original: [Figure]

Original: l a ss i ﬁ c a t i o n A cc u r a c y

Original: QuickDraw

Original: [Figure] [Figure] ResNet-51 [17] architectures for each classiﬁer.
Revised: ResNet-51 [17] architectures for each classiﬁer.

Original: We use a multiclass softmax loss for QuickDraw(17-way) and STL-10 (10-way), and a binary cross-entropy loss for each CelebaHQ attribute.
Revised: We use a multiclass softmax loss for QuickDraw (17-way) and STL-10 (10-way), and a binary cross-entropy loss for each CelebaHQ attribute.

Original: Fig.
Revised: Fig.

Original: 7 presents our method’s accuracy versus patch set size.
Revised: 7 presents our method’s accuracy versus patch set size.

Original: We show random patch selection as abaseline for QuickDraw and STL-10, and omit it for CelebaHQ to avoid overcrowding the ﬁgure (fullresults are in supplementary material).
Revised: We show random patch selection as a baseline for QuickDraw and STL-10, and omit it for CelebaHQ to avoid overcrowding the ﬁgure (full results are in supplementary material).

Original: Greedy selection outperforms random for QuickDraw and
Revised: Greedy selection outperforms random for QuickDraw and

Original: STL-10 by a considerable margin, in line with our results for image reconstruction in Fig.
Revised: STL-10 by a considerable margin, in line with our results for image reconstruction in Fig.

Original: Greedyselection achieves within 5% of the true classiﬁer accuracy at roughly 10-15 patches for QuickDraw,and 5-10 patches for STL-10.
Revised: Greedy selection achieves within 5% of the true classiﬁer accuracy at roughly 10-15 patches for QuickDraw, and 5-10 patches for STL-10.

Original: Greedy selection performs well on CelebaHQ as well, though we do observe variability acrossattributes.
Revised: Greedy selection performs well on CelebaHQ as well, though we do observe variability across attributes.

Original: For example, it struggles with few patches on high cheekbones and smiling , likely becauseslight variations to these features (e.g., a neutral vs. smiling mouth) are not particularly informativefor reconstructing the full image.
Revised: For example, it struggles with few patches on high cheekbones and smiling , likely because slight variations to these features (e.g., a neutral vs. smiling mouth) are not particularly informative for reconstructing the full image.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: There are several avenues of further exploration given our results.
Revised: There are several avenues of further exploration given our results.

Original: It would be interesting to disentangle the information of a patch into its location and content components.
Revised: It would be interesting to disentangle the information of a patch into its location and content components.

Original: For instance, the ﬁrst
Revised: For instance, the ﬁrst

Original: QuickDraw example in Fig.
Revised: QuickDraw example in Fig.

Original: 3 of a face shows how one carefully placed patch straddling the mouthand chin immediately constructs most of the face.
Revised: 3 of a face shows how one carefully placed patch straddling the mouth and chin immediately constructs most of the face.

Original: Given that the image is from QuickDraw, clearlyboth the visual feature and its coordinate in image space say something about the object class.
Revised: Given that the image is from QuickDraw, clearly both the visual feature and its coordinate in image space say something about the object class.

Original: Itis likely that our method uses such biases, which exist widely in vision datasets [41, 42], to itsadvantage.
Revised: It is likely that our method uses such biases, which exist widely in vision datasets [41, 42], to its advantage.

Original: Our experiments demonstrate that our minimal patch sets also contain appropriate information forclassiﬁcation tasks.
Revised: Our experiments demonstrate that our minimal patch sets also contain appropriate information for classiﬁcation tasks.

Original: Admittedly, more analysis may be required to apply those ideas in practice.
Revised: Admittedly, more analysis may be required to apply those ideas in practice.

Original: First, it would be useful to show that the patches can be informative when given sequentially toan attention-based classiﬁer to minimize computation.
Revised: First, it would be useful to show that the patches can be informative when given sequentially to an attention-based classiﬁer to minimize computation.

Original: Second, our process for returning patchesinvolves synthesizing a full image for each considered patch location, for all k iterations.
Revised: Second, our process for returning patches involves synthesizing a full image for each considered patch location, for all k iterations.

Original: This may beimproved upon by using the per-pixel uncertainty map σ I returned by our network to prune locationsworth exploring.
Revised: This may be improved upon by using the per-pixel uncertainty map σ I returned by our network to prune locations worth exploring.

Original: This would be more similar in spirit to ideas in sequential attention [12, 30].
Revised: This would be more similar in spirit to ideas in sequential attention [12, 30].

Original: Our results show that images from several popular datasets can be accurately inferred by justobserving a few patches in sequential order, given prior data from those domains.
Revised: Our results show that images from several popular datasets can be accurately inferred by just observing a few patches in sequential order, given prior data from those domains.

Original: This form ofdata redundancy has not been explored in depth by past work, and has natural ties to sequentialattention processes of humans and recent vision algorithms, as well as image compression.
Revised: This form of data redundancy has not been explored in depth by past work, and has natural ties to sequential attention processes of humans and recent vision algorithms, as well as image compression.

Original: At thecore of our approach is a measure of pointwise mutual information captured by a synthesis modelthat does not rely on any supervised information.
Revised: At the core of our approach is a measure of pointwise mutual information captured by a synthesis model that does not rely on any supervised information.

Original: Quantifying information about a scene from asubset of patches speciﬁes where an attention-based learner should look next in a task-agnosticway.
Revised: Quantifying information about a scene from a subset of patches speciﬁes where an attention-based learner should look next in a task-agnostic way.

Original: In addition to image reconstruction, our experiments demonstrate these minimal patch setscontain appropriate information for potential downstream tasks like classiﬁcation.
Revised: In addition to image reconstruction, our experiments demonstrate these minimal patch sets contain appropriate information for potential downstream tasks like classiﬁcation.

Original: In the case of low-bandwidth computing, this approach could process high-resolution images piecemeal and feedpatches into the classiﬁer.
Revised: In the case of low-bandwidth computing, this approach could process high-resolution images piecemeal and feed patches into the classiﬁer.

Original: Broader Impact

Original: This paper addresses a fundamental question about images within a collection: which patches ofthe image, and how many of them, need to be seen in order to reconstruct the image?

Original: Our workprobes the regularities and biases that exist over collections of images.

Original: A better understanding ofthose regularities may lead to improved storage or processing algorithms for computer vision andphotography.

Original: Piecemeal classiﬁcation of high-resolution images is another possible application inface of low-bandwidth communication.

Original: The pointwise mutual information approach is both task-agnostic and network-agnostic.
Revised: The pointwise mutual information approach is both task-agnostic and network-agnostic.

Original: Otherpotential applications of this method include quantifying how much information a network can learnfrom seeing a face occluded by a face-mask and potentially detect which patches a face recognitionsystem must focus on to authenticate a user.
Revised: Other potential applications of this method include quantifying how much information a network can learn from seeing a face occluded by a face-mask and potentially detect which patches a face recognition system must focus on to authenticate a user.

Original: Furthermore, neural networks with attention-mechanisms,such as transformers, identify which parts of the image the model focuses on.
Revised: Furthermore, neural networks with attention-mechanisms, such as transformers, identify which parts of the image the model focuses on.

Original: Alternatively, agreedy search with among patches can also help us dissect where the network gains information tounderstand a scene.
Revised: Alternatively, a greedy search with among patches can also help us dissect where the network gains information to understand a scene.

Original: While this paper is not geared toward one single application, the method wepropose demonstrates which information neural networks ﬁnd most valuable, which can then beleveraged for various downstream tasks.
Revised: While this paper is not geared toward one single application, the method we propose demonstrates which information neural networks ﬁnd most valuable, which can then be leveraged for various downstream tasks.

Original: C Additional visual examples
Revised: B Additional visual examples

Original: See Figs. 8, 9, and 10 below for additional reconstruction results for each dataset.
Revised: See Figs. 8, 9, and 10 below for additional reconstruction results for each dataset.

Original: D Celeba-HQ classiﬁcation plots
Revised: C Celeba-HQ classiﬁcation plots

Original: See Fig.
Revised: See Fig.

Original: 11 for a breakdown of accuracies of our approach vs. random patch selection for eachattribute.
Revised: 11 for a breakdown of accuracies of our approach vs. random patch selection for each attribute.

Original: E Visual examples ordered by difﬁculty
Revised: D Visual examples ordered by difﬁculty

Original: Figs.
Revised: Figs.

Original: 12, 13 and 14 show test images ordered by the number of patches needed by our method toreconstruct them to a given accuracy threshold.
Revised: 12, 13 and 14 show test images ordered by the number of patches needed by our method to reconstruct them to a given accuracy threshold.

Original: This metric correlates well with the visual complexityof the images.
Revised: This metric correlates well with the visual complexity of the images.

Original: [Figure]
Revised: [Figure]

Original: OriginalImage
Revised: OriginalImage

Original: NearestNeighbor
Revised: NearestNeighbor

Original: [Figure]
Revised: [Figure]

Original: OriginalImage
Revised: OriginalImage

Original: NearestNeighbor
Revised: NearestNeighbor

Original: [Figure]
Revised: [Figure]

Original: OriginalImage
Revised: OriginalImage

Original: NearestNeighbor
Revised: NearestNeighbor

Original: Male true accuracy greedy random     25 0.
Revised: Male true accuracy greedy random     25 0.

Original: Bangs true accuracy greedy random     25 0.
Revised: Bangs true accuracy greedy random     25 0.

Original: Heavy Makeup
Revised: Heavy Makeup

Original: true accuracy greedy random     25 0.
Revised: true accuracy greedy random     25 0.

Original: Smiling true accuracy greedy random     25 0.
Revised: Smiling true accuracy greedy random     25 0.

Original: Wearing Earrings true accuracy greedy random     25 0.
Revised: Wearing Earrings true accuracy greedy random     25 0.

Original: Young true accuracy greedy random     25 0.
Revised: Young true accuracy greedy random     25 0.

Original: High Cheekbones true accuracy greedy random
Revised: High Cheekbones true accuracy greedy random

Original: [Figure]
Revised: [Figure]

Original: r                      25 > 25 >
Revised: r                      25 > 25 >