Hand-Object Interaction Image Generation
Anonymous Author(s)
AffiliationAddress email
Abstract
In this work, we are dedicated to a new task, i e., hand-object interaction imagegeneration, which aims to conditionally generate the hand-object image under thegiven hand, object and their interaction status.
This task is challenging and researchworthy in many potential application scenarios, such as VR / AR games and onlineshopping, etc.
To address this problem, we propose a novel HOGAN framework,which utilizes the expressive model-aware hand-object representation and leveragesits inherent topology to build the unified surface space.
In this space, we explicitlyconsider the complex self- and mutual occlusion during interaction.
During finalimage synthesis, we consider different characteristics of hand and object andgenerate the target image in a split-and-combine manner.
For evaluation, we build acomprehensive protocol to access both the fidelity and structure preservation of thegenerated image.
Extensive experiments on two large-scale datasets, i e., HO3Dv3and DexYCB, demonstrate the effectiveness and superiority of our framework bothquantitatively and qualitatively.
Introduction
As a crucial step for analyzing human actions, hand-object interaction understanding is researchworthy in a broad range of applications related to virtual or augmented reality.
Current workslargely focus on hand-object pose estimation (HOPE) [16, 19, 21], which aims to capture thepose configuration of the given hand-object image.
In contrast, its inverse counterpart is seldomconsidered.
In this work, we aim to explore this novel task and term it Hand-Object Interaction image
Generation (HOIG).
Its objective is to generate the interacting hand-object image under the guidanceof the target posture, while preserving the appearance of the source image, as illustrated in Figure 1.
This HOIG task is of both application and research value to the community.
On one hand, HOIGcan be potentially applied to many scenarios, such as VR / AR games, online shopping, etc.
On theother hand, HOIG explicitly involves simultaneous generation of two instances (hand and object)with high interaction relationship.
These characteristics thus bring new challenges.
1) It requiresto process the complex self- and mutual occlusion between the interacting hand and object.
2) Itinvolves image translation of co-occurring instances, where different appearance characteristics ofthe hand and object need to be considered.
The effectiveness of Generative Adversarial Networks (GANs) has been validated in realistic imagesynthesis, such as human, face and hand [8, 27, 29, 38].
These GAN-based synthesis methods can beconditioned on different input information, such as simple-to-draw sketches, 2D sparse keypoints anddense semantic masks, etc.
Among them, GestureGAN [38] resolves the isolated hand generation,which is mostly related to our task.
It utilizes 2D sparse hand keypoints as the condition and attemptsto generate the target hand image according to the optical flow learned from the source and targetconditions.
However, these methods do not consider the generation of two interacting instances,which cannot meet the requirement of the HOIG task.
[Figure]
Info.
In this work, we propose the HOGAN framework to deal with the challenges of this novel task.
HOGAN utilizes the expressive model-aware representation as the condition and leverages its inherentstructure topology to build the unified hand-object surface space.
In this space, complex self- andmutual occlusion among hand and object are explicitly modeled.
Specifically, the visible parts of handand object are mapped to the target image plane, along with their corresponding fine-grained topologymap.
Meanwhile, the transformation flow is computed between the source and target.
These middleresults provide abundant information for final image synthesis.
During synthesis, our frameworkconsiders appearance difference between hand and object and generates the target hand-object imagein a split-and-combine manner.
To systematically explore this task, we build the comparison baselines via adopting representativemethods from the most related single hand generation task with a few modifications.
The imagegeneration quality is carefully evaluated from multiple perspectives, including the fidelity (FID and
LPIPS), structure preservation (AUC, PA-MPJPE and ADD-0.1D) and subjective user study.
Our contributions are summarized as follows,
• To our best knowledge, we are the first to explore the novel task named the hand-object interactionimage generation, i.e., conditionally generating the hand-object image under the given hand, object
and their interaction status.
This task is of board research and application value to the community.
• To deal with the challenges of this task, we propose the HOGAN framework, which considers
the
hand-object occlusion and generates the target image in a split-and-combine manner.
• To systematically explore this task, we present comparison baselines from related single-hand
generation.
Besides, the comprehensive metrics are chosen to evaluate the both fidelity and
structure preservation of the generated image.
Extensive experiments on two datasets demonstrate
the effectiveness and superiority of our method over baselines.
Related Work
In this section, we will briefly review the related topics, including pose-guided image synthesis andhand-object interaction.
Pose-guided image synthesis.
Pose-guided image synthesis is a conditional generation task, whichaims to generate an image under the condition of the target pose while preserving the identity ofthe source image.
This problem involves instances with rigid parts such as bodies [1, 5, 11, 12, 13,20, 29, 35, 33, 37, 42], faces [8, 18, 24, 28, 34], and hands [17, 22, 38, 39], and can be utilized invarious scenarios, such as image animation, face reenactment and sign language production, etc.
Inrecent work, Ren et al.
[29] generate person images under target posture with a differential globalflow local-attention framework in a multi-scale manner.
Deng et al.
[8] utilize 3DMM [2] whichparameterizes pose and shape to disentangle posture representation for face generation.
Hu et al.
[17]incorporate hand prior for pose-guided hand image synthesis instead of 2D joint representation.
However, previous methods mainly focus on single-object pose translation problems.
Different fromthem, the novel problem we formulate, i.e. , HOIG, involves generation of the co-occurring subjects,which brings new characteristics worth exploring.
Hand-object interaction.
Most current works on hand-object interaction focus on simultaneouslyestimating hand-object pose aligning the given image [4, 7, 9, 15, 16, 19, 21, 23, 25, 32].
To betterdepict hand-object interaction, they resort to dense triangle meshes with the pre-defined topologyas the representations, which are produced by the MANO hand model [30] and the known objectmodel [3, 40].
Hasson et al.
[16] leverage physical constraints to better estimate hand and objectmeshes.
Cao et al.
[4] propose a optimization-based method, which leverages 2D image cues and3D contact priors for hand-object interaction estimation.
Liu et al.
[21] further boost the estimationperformance by semi-supervised learning with the assistance of in-the-wild hand-object videos.
Toour best knowledge, there exists no work on the inverse task of hand-object interaction estimation.
Inthis work, we aim to explore this novel task and term it hand-object interaction image generation.
Methodology
In this section, we first present the detailed problem formulation.
Then we elaborate the architectureof our proposed HOGAN framework.
Finally, we introduce the loss function during optimization.
3.1 Problem Formulation
Hand-object interaction image generation is a conditional generation task.
Its objective is to generatethe interacting hand-object image under the target pose condition, while preserving the sourceappearance.
Specifically, the object is well-modeled with texture known.
Resolving this task ischallenging, since it is non-trivial to understand the complex interacting relationship between handand object during image generation.
We summarize the main challenges as follows.
Firstly , it requires modeling the occlusion betweenco-occurring instances.
In the interacting hand-object scenario, complex self- and mutual occlusionusually occur.
Since the occlusion leads to more transformation complexity between the sourceand target, the occluded regions should be located and identified, which eases the final synthesis.
Secondly , it needs to take into account different characteristics of two instances during generation.
Specifically, hand is articulated and encounters self-occlusion among joints, while object is usuallyrigid with fine-grained texture.
The framework should generate realistic appearance of co-occurringinstances, jointly with reasonable manipulation between them.
3.2 HOGAN
Overview.
As illustrated in Figure 2, our framework first utilizes the expressive model-awarerepresentation as the pose condition.
Then we leverage the pre-defined structure in the model tobuild the unified space to perform occlusion-aware topology modeling.
This modeling identifiesthe occluded regions, and provides the coarse target images and fine-grained topology maps for thenext stage.
Finally, hand-object synthesis considers the appearance difference of these instances, andgenerates the target hand-object image in a split-and-combine manner.
Occlusion-aware topology modeling.
We first give an overview of the utilized model-awarerepresentation.
The hand and object are represented via MANO [30] and YCB [3, 40] models,respectively.
Both MANO and YCB provide triangulated meshes, which can densely depict thestructure of hand and object.
Specifically, the hand-object joint representation contains N v verticesand N f faces.
Its mesh H ∈ R N v × 3 represents the vertex coordinates.
The inherent topology
P ∈ R N f × 3 × 2 is organized as a vertex triplet, in which each unit is recorded as corresponding vertexcoordinates aligning the desired plane.
s , t and u are notations for the source, target and unifiedsurface space, respectively.
Embedded with the pre-defined topology, we unravel the surface of hand and object model to buildthe unified space.
In this space, the same representation is binded with the same mesh face, ignoringthe pose configuration.
This space is adopted to perform mapping from the source to the target, whileinserting the pre-known object texture.
Firstly, we leverage the source pose to map its correspondingimage appearance to this unified space in an occlusion-aware manner as follows, [Equation]
[Equation]
where T u ← s ( x, y ) denotes the flow from the source image to the unified space.
F u ( x, y ) denotes theface index belonging to the ( x, y ) location in the surface space, and W u ( x, y ) represents the relative
HO-Gen.
[Figure]
Occlusion-Aware Topology Modeling
[Figure]
𝑶𝑶 𝑢𝑢←𝑠𝑠 ( 𝑥𝑥 [Figure] 𝑦𝑦
[Figure]
�𝑰𝑰 𝑢𝑢 𝑻𝑻 𝑢𝑢←𝑠𝑠
[Figure]
𝑥𝑥
[Figure]
𝑦𝑦
[Figure]
weighted position in this face.
Besides, occlusion should be jointly computed to locate the visibletexture as follows,
[Equation]
The visible texture from the source image is mapped to our unified surface space as follows,
[Equation]
(3) where I u aligns our surface space.
⊙ and Warp( · ) represents the element-wise multiplication andwarping function, respectively.
Notably, the object in the source image inevitably contains theoccluded region, which is insufficient for the target generation.
Therefore, we utilize the pre-storedobject texture to replace the original object region in I u , resulting new texture image ˆ I u .
Finally, wecompute the flow performing the mapping between the unified space to the target image.
[Equation]
After that, the target image I t is generated via sampling ˆ I u under the guidance of T t ← u as follows,
[Equation]
I t represents the coarse target hand-object image.
Meanwhile, to provide more guidance for the nextstage, the fine-grained topology map Y t is concurrently generated as follows.
[Equation]
where Bary( · ) computes the barycenter of the corresponding face in the surface space.
Hand-object synthesis.
Considering hand and object exhibit different properties, we design thehand-object generator for target image synthesis in a split-and-combine manner.
As illustrated in
Figure
The Background Stream inpaints the background cropped from the source image with the hand-objectforeground mask.
The Object Stream takes the rendered object images as input and transfers its styleto the real image domain.
To make the stream aware of object structure information, we adopt thespatially-adaptive normalization (SPADE)
[26] to insert the object topology map.
The Hand Stream deals with the hand part that is not occluded by the object.
We first synthesize acoarse image of the visible part in the target posture by warping the hand in the source image with thepose transformation flow.
After that, the coarse image is refined by the U-structure network [31] in thehand stream.
During refinement, similar to the Object Stream , the hand topology map is modulatedinto the network with SPADE.
Meanwhile, we extract the multi-scale features from the hand-objectgenerator that reconstructs the source image, and integrates them into the generation process with theattention sampler [29].
The three streams separately process three instances with different properties, i.e. , background, objectand hand, and merge their results with the fusion module.
With the features from the last layerof three streams, our framework further learns two fusion masks, i.e. , the hand mask M h and thehand-object mask M f , which indicate the area belonging to the unoccluded hand the hand-objectforeground, respectively.
Regarding these two mask, the fusion module merges the the three-streamresults to the final generated results I as follows,
[Equation]
where I b ,
I o , I h are the generated results of the background, object and hand stream, respectively.
⊙refers to the element-wise multiplication.
3.3 Objective Functions
We design a discriminator to train our HOGAN in an adversarial learning manner.
The adversarialloss constrains the distribution of the generated images with that of the real images, which improvesthe visual performance of generated images.
With the discriminator D ( · ) , the adversarial loss isformulated as follows,
[Equation]
where the x f and x r are the distribution of the generated and real images, respectively.
c refers tothe combination of target hand and object posture information.
Besides the adversarial loss, we regularize our HOGAN with the reconstruction loss on the sourceimage and the perceptual loss on the generated image, which is formulated as follows,
[Equation]
where x s and x t are the generated source and target images while ˆ x s and ˆ x t are the ground-truthsource and target images, respectively.
f i ( · ) is the feature extractor from the i -th layer of a pre-trained
VGG network [36].
The overall loss is the summation of three objective functions as follows,
[Equation]
where λ 1 and λ 2 are the weighting parameters to balance the objective functions.
Experiment
In this section, we first introduce the experiment setup, including datasets, implementation details andevaluation metrics.
Then we elaborate the baseline methods and make comparisons with them bothquantitatively and qualitatively.
After that, we conduct ablation studies to highlight the importantcomponents in our framework.
Furthermore, we explore more applications of our HOIG task.
Source Image
Target Posture
GesGAN + OBJ
MG2T + OBJ
HOGAN
Ground-truth
4.1 Experiment Setup
Datasets.
We evaluate our method on two large-scale datasets with annotated hand-object meshrepresentation, i.e., HO3Dv3 [14] and DexYCB [6].
HO3Dv3 is captured in the real-world setting.
Itcontains 10 different subjects performing various fine-grained manipulation on one among 10 objectsfrom YCB models [3].
The training and testing set contain 58,148 and 13,938 images, respectively.
DexYCB is recorded in the controlled environment, with 10 subjects manipulating one among 20objects.
In our experiment, we choose the frames containing interaction between hand and object,with 33,562 and 8,554 images for training and testing, respectively.
Implementation details.
The whole framework is implemented on PyTorch and we perform experiments on 4 NVIDIA RTX 3090.
The Adam optimizer is adopted and the training lasts 30 epochs.
Weset the batch size to 8 in our experiment.
The learning rate is set as 2e-4 for the first 15 epochs andlinearly decays to 2e-6 till the end.
The hyperparameter λ [-
-]1 and λ 2 are set to 10 and 10, respectively.
Evaluation metrics.
We design an evaluation protocol to measure the hand-object interaction imagegeneration quality both quantitatively and qualitatively.
In quantitative evaluation, we measure boththe fidelity and structure preservation of generated images.
For the image fidelity, we adopt the [Table] [Table] widely-used Fréchet Inception Distance (FID) [10] and Learned Perceptual Similarity (LPIPS)
[41]metrics. To evaluate the posture preservation of hand and object in the generated images, we utilizean off-the-shelf hand-object pose estimator [21] to report AUC and PA-MPJPE for hand pose, and
ADD-0.1D for object pose.
PA-MPJPE represents the joint mean error after Procrustes alignment,and AUC denotes the area under the 3D PCK curve.
ADD-0.1D denotes the percentage of averageobject vertices error within 10% of object diameter.
In qualitative evaluation, we conduct a user study to evaluate the visual performance of our methodand two baselines.
There are 20 volunteers participating in this study.
In the study, the volunteers areasked to select the most high-fidelity generated results among our method and two baselines.
Thepercent of generated image preferred is recorded as User Preference Ratio (UPR).
4.2 Baselines
Since there exists no work on this task, we present two baselines from most related single-handgeneration, i.e. , GestureGAN [38] and MG2T [17], with a few modifications.
These methods aremainly modified with the object condition information involved as follows.
GestureGAN + OBJ.
GestureGAN translates the single-hand source image to the target posture ina cycle-consistent manner.
The source image is fed into the network along with the target posturerepresented by sparse keypoints.
In GestureGAN + OBJ, we concatenate the object rendered resultwith the target posture and feed them into the network.
The modified framework is trained with thesame architecture design and objective functions as the previous GestureGAN.
MG2T + OBJ.
MG2T is also a single-hand generation framework with hand prior incorporated.
In MG2T + OBJ, we maintain the hand processing module in MG2T and further extend the objectmodeling to involve the object posture information.
The rendered result of the object is merged intothe foreground branch to produce the hand-object translation results.
4.3 Comparison with Baselines
We present quantitative results to evaluate the effectiveness of our method on HO3Dv3 and DexYCBdatasets.
Firstly , we study the fidelity of generated images on two datasets.
As shown in Table 1,our method surpasses two baselines on all metrics with a notable gain.
Secondly , it is essential forhand-object interaction image generation to maintain the posture information of the target hand andobject.
Therefore, we carefully analyze the posture preservation of generated images.
With theoff-the-shelf network [21], we estimate the hand and object pose in the generated images and compute the error with the target pose annotation.
As shown in Table 2, it can be observed that our methodexhibits superior performance for hand and objects of different categories over two other baselines.
Furthermore, we perform qualitative comparisons with baselines.
From Figure 3, we observe that theimages generated by our method exhibit better visual performance compared with previous methods.
Under the complex scenes, where the hand and object are highly interacting, our method can generateimages with more reasonable spatial relationship, which significantly outperforms baseline methods.
We also conduct a user study to evaluate subjective visual performance.
The voting results arereported as the USP metric in Table 1.
It is observed that our method is preferred by over 70 percentagainst two competitors.
4.4 Ablation Studies
We perform ablative experiments to highlight several important components, i.e. , the hand and objecttopology, source feature transfer and split-and-combine generation.
[Table]
Hand and object topology.
In HOGAN, we modulate both hand and object topology into the generatorto provide detailed geometric information of handand object.
In Table 3, we verify its effectiveness bycomparing it with non-topology variants.
It can beobserved that HOGAN (the first row) achieves thebest performance over two other variants.
Withouteither hand or object topology, the framework suffersperformance degradation, e.g. 6.0 and 6.5 on FID,0.012 and 0.015 on LPIPS.
[Table]
Source feature transfer.
To further enhance the refinement procedure of the coarse target hand image, we transfer multi-layers features from the source hand-object generator according to the transformation flow.
In Table 4,we analyze the importance of the source feature transfer.
The None variant refers to the framework without sourcefeature transfer.
The Bilinear and Attention variants refer to the framework with different samplers for featuretransferring, i.e. , a bilinear sampler and an attention sampler, respectively.
It can be seen that the source featuretransfer provides useful cues for target image generation and the attention sampler achieves the bestperformance among three settings.
[Table]
Split-and-combine Generation.
Considering hand andobject exhibit different properties, we involve a split-andcombine (S&C) strategy in HOGAN.
In this setting, wecompare HOGAN with a parameter-comparable baselinemodel without separately generating hand and object.
In
Table 5, it is observed that HOGAN outperforms its variant (without S&C) by 2.9 FID and 0.036 LPIPS gain, whichdemonstrates this strategy benefits the quality of generatedimages when involving instances with different properties.
4.5 Application
In this subsection, we further explore several interesting applications on our HOGAN, i.e. , objecttexture editing and real-world generation.
In object texture editing, we alter the object texture withcharacters, i.e. , “NeurIPS” and “2022”, and generate the images conditioned on the edited textures.
From Figure 4 (a), it is observed that our generated images well preserve the edited characters on theobject texture.
Furthermore, we take the hand image from the real scene as the source image to testour HOGAN framework pre-trained on HO3Dv3 dataset.
As shown in Figure 4 (b), the generatedimages both maintain the source image appearance and meet the target posture condition.
Theseapplications demonstrate the generalization of HOGAN on hands and objects for the real-worldscenario.
Source Image
Source Image
Target Posture Generated Image
[Figure]
Limitations and Future Work
Our framework considers the complex spatial relationship and different appearance properties betweenhand and object.
The effectiveness of our framework has been validated in the Experiment section.
There still exist limitations in our framework.
Specifically, it utilizes the dense mesh representation asthe pose condition, which inevitably contains misalignment with the RGB image plane even manuallyannotated.
This issue mainly disturbs coarse image synthesis and our framework mainly resorts tothe Hand-Object Synthesis stage for further refinement.
Another way to mitigate this issue is to adopta more effective hand-object pose estimation method for better pose representation.
Our explored HOIG task is of broad research interest to the community.
When hand pose estimationturns from the isolated hand to the interacting hand-object scenario, we advocate the image generationcommunity to draw more attention to this new HOIG task.
This task involves generation of cooccurring instances under complex occlusion conditions.
The advance in HOIG can also inspireother related human-centric image generation tasks.
We outline the future works as follows.
More suitable representations can be explored to serve as a condition during generation.
Moreadvanced techniques can be combined into the framework to enhance the generation quality. 3) Moreapplications such as HOPE data augmentation can be explored to fertilize this new HOIG task.
Conclusion
In this paper, we make the first attempt to explore a novel task, namely hand-object interactionimage generation.
This task brings new challenges since it involves generating two co-occurringinstances under complex interaction conditions.
To deal with the challenges of this task, we proposethe HOGAN framework.
It explicitly considers the self- and mutual occlusion among hand and objectand generates the target image in a split- and -combine manner
For comprehensive comparisons , we present baselines adopted from related single hand generation and evaluate the generated imagesfrom multiple perspectives, including fidelity and structure preservation
Extensive experiments on two datasets demonstrate our method outperforms baselines both quantitatively and qualitatively.
Broader Impact.
Our research aims to benefit application scenarios like virtual / augmented realityand online shopping, both of which impact the daily life of millions of users. The capability ofgenerating the realistic hand-object image under the given interaction condition better optimizes theuser experience. Furthermore, the research advance in this challenging task can provide useful cuesin solving other complex human-centric generation scenarios. On the other side, our frameworkinvolves generating non-existent images, which may be misused in fake content creation.
This mayspread misinformation and lead to other privacy issues, which is opposite to our intent.
A variety ofregulatory and technical measures have been proposed to address this issue.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We have clearly stated the contributions and scope inthe Abstract and Introduction.
(b) Did you describe the limitations of your work?
[Yes] We describe the limitations in theSection 5.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss the potential negative societal impacts in Section 6.(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We include all the necessary code, instructions and environment needed in the Supplementary Materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] The important training details are specified in the Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] In the Supplementary Material, we show the error bars ofour main experiments in the form of variance.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We report the type of resources used in the experiments in Section 4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] In Section 4.1, we cite corresponding papers for the asserts we use.
(b) Did you mention the license of the assets?
[Yes] HO3Dv3 is publicly available without stating its license, and DexYCB follows the license of CC BY-NC 4.0.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] In the supplementary material, we provide the complete experiment code, the necessary commands and the corresponding environment for generating the training
dataset, training the model, and testing the model.
d) Did you discuss whether and how consent was obtained from people whose datayou’re using
curating Yes ] The training datasets HO3Dv3 and DexYCB are publiclyavailable.
We cite the paper that provides the dataset and the model.
The real-worldimages used in Section 4.5 are captured and granted by the authors themselves.
e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Yes
] The datasets used in the experiments arepublicly available and we have removed personally identifiable information or offensivecontent The real -world
image
used in
Section 4.5 are captured and granted from the authors.5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the
full text of instructions given to participants and screenshots ifapplicable [Yes
We include the full text of
instructions given to the participantsduring the user study.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
N/A In the
user study
, the
participants are asked to score our
experimental results
There is no personal information involved and the participants
remain anonymous throughout.
(c) Did you include the estimated hourly wage paid to participants and the total
amount
spent on participant compensation?
Yes ] All participants for the
user
study arevoluntary and
we have estimated the hourly wage in our local region and have paidthem a gift of equal value.
Besides, the time spent by each participant is estimatedaround 10 minutes.