Hand-Object Interaction Image Generation
Anonymous Author(s)
AffiliationAddress email
Abstract
In this work, we are dedicated to a new task, i e., hand-object interaction imagegeneration, which aims to conditionally generate the hand-object image under thegiven hand, object and their interaction status.
This task is challenging and researchworthy in many potential application scenarios, such as AR/VR games and onlineshopping, etc.
To address this problem, we propose a novel HOGAN framework,which utilizes the expressive model-aware hand-object representation and leveragesits inherent topology to build the unified surface space.
In this space, we explicitlyconsider the complex self- and mutual occlusion during interaction.
During finalimage synthesis, we consider different characteristics of hand and object andgenerate the target image in a split-and-combine manner.
For evaluation, we build acomprehensive protocol to access both the fidelity and structure preservation of thegenerated image.
Extensive experiments on two large-scale datasets, i e., HO3Dv3and DexYCB, demonstrate the effectiveness and superiority of our framework bothquantitatively and qualitatively.
Introduction
As a crucial step for analyzing human actions, hand-object interaction understanding is researchworthy in a broad range of applications related to virtual or augmented reality.
Current workslargely focus on hand-object pose estimation (HOPE) [16, 19, 21], which aims to capture thepose configuration of the given hand-object image.
In contrast, its inverse counterpart is seldomconsidered.
In this work, we aim to explore this novel task and term it Hand-Object Interaction image
Generation (HOIG).
Its objective is to generate the interacting hand-object image under the guidanceof the target posture, while preserving the appearance of the source image, as illustrated in Figure 1.
This HOIG task is of both application and research value to the community.
On one hand, HOIGcan be potentially applied to many scenarios, such as AR/VR games, online shopping, etc.
On theother hand, HOIG explicitly involves simultaneous generation of two instances (hand and object)with high interaction relationship.
These characteristics thus bring new challenges.
1) It requiresto process the complex self- and mutual occlusion between the interacting hand and object.
2) Itinvolves image translation of co-occurring instances, where different appearance characteristics ofthe hand and object need to be considered.
The effectiveness of Generative Adversarial Networks (GANs) has been validated in realistic imagesynthesis, such as human, face and hand [8, 27, 29, 39].
These GAN-based synthesis methods can beconditioned on different input information, such as simple-to-draw sketches, 2D sparse keypoints anddense semantic masks, etc.
Among them, GestureGAN [39] resolves the isolated hand generation,which is mostly related to our task.
It utilizes 2D sparse hand keypoints as the condition and attemptsto generate the target hand image according to the optical flow learned from the source and targetconditions.
However, these methods do not consider the generation of two interacting instances,which cannot meet the requirement of the HOIG task.
[Figure]
Info.
In this work, we propose the HOGAN framework to deal with the challenges of this novel task.
HOGAN utilizes the expressive model-aware representation as the condition and leverages its inherentstructure topology to build the unified hand-object surface space.
In this space, complex self- andmutual occlusion among hand and object are explicitly modeled.
Specifically, the visible parts of handand object are mapped to the target image plane, along with their corresponding fine-grained topologymap.
Meanwhile, the transformation flow is computed between the source and target.
These middleresults provide abundant information for final image synthesis.
During synthesis, our frameworkconsiders appearance difference between hand and object and generates the target hand-object imagein a split-and-combine manner.
To systematically explore this task, we build the comparison baselines via adopting representativemethods from the most related single hand generation task with a few modifications.
The imagegeneration quality is carefully evaluated from multiple perspectives, including the fidelity (FID and
LPIPS), structure preservation (AUC, PA-MPJPE and ADD-0.1D) and subjective user study.
Our contributions are summarized as follows,
• To our best knowledge, we are the first to explore the novel task named the hand-object interactionimage generation, i.e., conditionally generating the hand-object image under the given hand, object
and their interaction status.
This task is of board research and application value to the community.
• To deal with the challenges of this task, we propose the HOGAN framework, which considers
the
hand-object occlusion and generates the target image in a split-and-combine manner.
• To systematically explore this task, we present comparison baselines from related single-hand
generation.
Besides, the comprehensive metrics are chosen to evaluate the both fidelity and
structure preservation of the generated image.
Extensive experiments on two datasets demonstrate
the effectiveness and superiority of our method over baselines.
Related Work
In this section, we will briefly review the related topics, including pose-guided image synthesis andhand-object interaction.
Pose-guided image synthesis.
Pose-guided image synthesis is a conditional generation task, whichaims to generate an image under the condition of the target pose while preserving the identity ofthe source image.
This problem involves instances with rigid parts such as bodies [1, 5, 11, 12, 13,20, 29, 36, 34, 38, 43], faces [8, 18, 24, 28, 35], and hands [17, 22, 39, 40], and can be utilized invarious scenarios, such as image animation, face reenactment, and sign language production, etc.
Inrecent work, Ren et al.
[29] generate person images under target posture with a differential globalflow local-attention framework in a multi-scale manner.
Deng et al.
[8] utilize 3DMM [2] whichparameterizes pose and shape to disentangle posture representation for face generation.
Hu et al.
[17]incorporate hand prior for pose-guided hand image synthesis instead of 2D joint representation.
However, previous methods mainly focus on single-object pose translation problems.
Different fromthem, the novel problem we formulate, i.e. , HOIG, involves generation of the co-occurring subjects,which brings new characteristics worth exploring.
Hand-object interaction.
Most current works on hand-object interaction focus on simultaneouslyestimating hand-object pose aligning the given image [4, 7, 9, 15, 16, 19, 21, 23, 25, 32].
To betterdepict hand-object interaction, they resort to dense triangle meshes with the pre-defined topologyas the representations, which are produced by the MANO hand model [30] and the known objectmodel [3, 41].
Hasson et al.
[16] leverage physical constraints to better estimate hand and objectmeshes.
Cao et al.
[4] propose a optimization-based method, which leverages 2D image cues and3D contact priors for hand-object interaction estimation.
Liu et al.
[21] further boost the estimationperformance by semi-supervised learning with the assistance of in-the-wild hand-object videos.
Toour best knowledge, there exists no work on the inverse task of hand-object interaction estimation.
Inthis work, we aim to explore this novel task and term it hand-object interaction image generation.
Methodology
In this section, we first discuss our problem formulation.
Then we elaborate the architecture of ourproposed HOGAN framework.
Finally, we introduce the loss function during optimization.
3.1 Problem Formulation
Hand-object interaction image generation is a conditional generation task.
Its objective is to generatethe interacting hand-object image under the target pose condition, while preserving the sourceappearance.
Specifically, the object is well-modeled with texture known.
Resolving this task ischallenging, since it is non-trivial to understand the complex interacting relationship between handand object during image generation.
We summarize the main challenges as follows.
Firstly , it requires modeling the occlusion betweenco-occurring instances.
In the interacting hand-object scenario, complex self- and mutual occlusionusually occur.
Since the occlusion leads to more transformation complexity between the sourceand target, the occluded regions should be located and identified, which eases the final synthesis.
Secondly , it needs to take into account different characteristics of two instances during generation.
Specifically, hand is articulated and encounters self-occlusion among joints, while object is usuallyrigid with fine-grained texture.
The framework should generate realistic appearance of co-occurringinstances, jointly with reasonable manipulation between them.
3.2 HOGAN
Overview.
As illustrated in Figure 2, our framework first utilizes the expressive model-awarerepresentation as the pose condition.
Then we leverage the pre-defined structure in the model tobuild the unified space to perform occlusion-aware topology modeling.
This modeling identifiesthe occluded regions, and provides the coarse target images and fine-grained topology maps for thenext stage.
Finally, hand-object synthesis considers the appearance difference of these instances, andgenerates the target hand-object image in a split-and-combine manner.
Occlusion-aware topology modeling.
We first give an overview of the utilized model-awarerepresentation.
The hand and object are represented via MANO [30] and YCB [3, 41] models,respectively.
Both MANO and YCB provide triangulated meshes, which can densely depict thestructure of hand and object.
Specifically, the hand-object joint representation contains N v verticesand N f faces.
Its mesh H ∈ R N v × 3 represents the vertex coordinates.
The inherent topology
P ∈ R N f × 3 × 2 is organized as a vertex triplet, in which each unit is recorded as corresponding vertexcoordinates aligning the desired plane.
s , t and u are notations for the source, target and unifiedsurface space, respectively.
Embedded with the pre-defined topology, we unravel the surface of hand and object model to buildthe unified space.
In this space, the same representation is binded with the same mesh face, ignoringthe pose configuration.
This space is adopted to perform mapping from the source to the target, whileinserting the pre-known object texture.
Firstly, we leverage the source pose to map its correspondingimage appearance to this unified space in an occlusion-aware manner as follows,
[Equation]
where T u ← s ( x, y ) denotes the flow from the source image to the unified space.
F u ( x, y ) denotes theface index belonging to the ( x, y ) location in the surface space, and W u ( x, y ) represents the relative
HO-Gen.
[Figure]
Occlusion-Aware Topology Modeling
[Figure]
𝑶𝑶 𝑢𝑢←𝑠𝑠 ( 𝑥𝑥
[Figure]
𝑦𝑦
[Figure]
�𝑰𝑰 𝑢𝑢 𝑻𝑻 𝑢𝑢←𝑠𝑠
[Figure]
𝑥𝑥
[Figure]
𝑦𝑦
[Figure]
weighted position in this face.
Besides, occlusion should be jointly computed to locate the visibletexture as follows,
[Equation]
The visible texture from the source image is mapped to our unified surface space as follows,
[Equation]
(3) where I u aligns our surface space.
⊙ and Warp( · ) represents the element-wise multiplication andwarping function, respectively.
Notably, the object in the source image inevitably contains theoccluded region, which is insufficient for the target generation.
Therefore, we utilize the pre-storedobject texture to replace the original object region in I u , resulting new texture image ˆ I u .
Finally, wecompute the flow performing the mapping between the unified space to the target image.
[Equation]
After that, the target image I t is generated via sampling ˆ I u under the guidance of T t ← u as follows,
[Equation]
I t represents the coarse target hand-object image.
Meanwhile, to provide more guidance for the nextstage, the fine-grained topology map Y t is concurrently generated as follows.
[Equation]
where Bary( · ) computes the barycenter of the corresponding face in the surface space.
Hand-object synthesis.
Considering hand and object exhibit different properties, we design thehand-object generator for target image synthesis in a split-and-combine manner.
As illustrated in
Figure
The Background Stream inpaints the background cropped from the source image with the hand-objectforeground mask.
The Object Stream takes the rendered object images as input and transfers its styleto the real image domain.
To make the stream aware of object structure information, we adopt thespatially-adaptive normalization (SPADE)
[26] to insert the object topology map.
The Hand Stream deals with the hand part that is not occluded by the object.
We first synthesize acoarse image of the visible part in the target posture by warping the hand in the source image with thepose transformation flow.
After that, the coarse image is refined by the U-structure network [31] in thehand stream.
During refinement, similar to the Object Stream , the hand topology map is modulatedinto the network with SPADE.
Meanwhile, we extract the multi-scale features from the hand-objectgenerator that reconstructs the source image, and integrates them into the generation process with theattention sampler [29].
The three streams separately process three instances with different properties, i.e. , background, objectand hand, and merge their results with the fusion module.
With the features from the last layer ofthree streams, our framework further utilizes two convolutional layers to learn two fusion masks,i.e. , the hand mask M h and the hand-object mask M f , which indicate the area belonging to theunoccluded hand the hand-object foreground, respectively.
Regarding these two mask, the fusionmodule merges the the three-stream results to the final generated results I as follows,
[Equation]
) where I b , I o ,
I h are the generated results of the background, object and hand stream, respectively.
⊙refers to the element-wise multiplication.
3.3 Objective Functions
We design a discriminator to train our HOGAN in an adversarial learning manner.
The adversarialloss constrains the distribution of the generated images with that of the real images, which improvesthe visual performance of generated images.
With the discriminator D ( · ) , the adversarial loss isformulated as follows,
[Equation]
where the x f and x r are the distribution of the generated and real images, respectively.
c refers tothe combination of target hand and object posture information.
Besides the adversarial loss, we regularize our HOGAN with the reconstruction loss on the sourceimage and the perceptual loss on the generated image, which is formulated as follows,
[Equation]
where x s and x t are the generated source and target images while ˆ x s and ˆ x t are the ground-truthsource and target images, respectively.
f i ( · ) is the feature extractor from the i -th layer of a pre-trained
VGG-19 network [37].
The overall loss is the summation of three objective functions as follows,
[Equation]
where λ 1 and λ 2 are the weighting parameters to balance the objective functions.
Experiment
In this section, we first introduce the experiment setup, including datasets, implementation details,and evaluation metrics.
Then we elaborate the baseline methods and make comparison with themboth quantitatively and qualitatively.
After that, we conduct ablation study to highlight the importantcomponents in our framework.
Furthermore, we explore more applications of our HOIG task.
Source Image
Target Posture
GesGAN + OBJ
MG2T + OBJ
HOGAN
Ground-truth
4.1 Experiment Setup
Datasets.
We evaluate our method on two large-scale datasets with annotated hand-object meshrepresentation, i.e., HO3Dv3 [14] and DexYCB [6].
HO3Dv3 is captured in the real-world setting.
Itcontains 10 different subjects performing various fine-grained manipulation on one among 10 objectsfrom YCB models [3].
The training and testing set contain 58,148 and 13,938 images, respectively.
DexYCB is recorded in the controlled environment, with 10 subjects manipulating one among 20objects.
In our experiment, we choose the frames containing interaction between hand and object,with 33,562 and 8,554 images for training and testing, respectively.
Implementation details.
The whole framework is implemented on PyTorch and we perform experiments on 4 NVIDIA RTX 3090.
All U-Nets are trained from scratch.
The hand-object generator istrained in an end-to-end manner and all the networks are being trained simultaneously.
The Adamoptimizer is adopted and the training lasts 30 epochs.
We set the batch size to 8 in our experiment.
The learning rate is set as 2e-4 for the first 15 epochs and linearly decays to 2e-6 till the end.
Thehyperparameter λ {+
+} and λ 2 are set to 10 and 10, respectively. [Table] [Table]
Evaluation metrics.
We design an evaluation protocol to measure the hand-object interaction imagegeneration quality both quantitatively and qualitatively.
In quantitative evaluation, we measure boththe fidelity and structure preservation of generated images.
For the image fidelity, we adopt thewidely-used Fréchet Inception Distance (FID) [10] and Learned Perceptual Similarity (LPIPS) [42]metrics.
To evaluate the posture preservation of hand and object in the generated images, we utilizean off-the-shelf hand-object pose estimator [21] to report AUC and PA-MPJPE for hand pose,
ADD- for object pose
PA-MPJPE represents the joint mean error after alignment
In qualitative evaluation, we conduct a user study to evaluate the visual performance of our methodand two baselines.
There are 20 volunteers participating in this study.
In the study, the volunteers areasked to select the most high-fidelity generated results among our method and two baselines.
Thepercent of generated image preferred is recorded as User Preference Ratio (UPR).
4.2 Baselines
Since there exists no work on this task, we present two baselines from most related single-handgeneration, i.e. , GestureGAN [39] and MG2T [17], with a few modifications.
These methods aremainly modified with the object condition information involved as follows.
GestureGAN + OBJ.
GestureGAN translates the single-hand source image to the target posture ina cycle-consistent manner.
The source image is fed into the network along with the target posturerepresented by sparse keypoints.
In GestureGAN + OBJ, we concatenate the object rendered resultwith the target posture and feed them into the network.
The modified framework is trained with thesame architecture design and objective functions as the previous GestureGAN.
MG2T + OBJ.
MG2T is also a single-hand generation framework with hand prior incorporated.
In MG2T + OBJ, we maintain the hand processing module in MG2T and further extend the objectmodeling to involve the object posture information.
The rendered result of the object is merged intothe foreground branch to produce the hand-object translation results.
4.3 Comparison with Baselines
We present quantitative results to evaluate the effectiveness of our method on HO3Dv3 and DexYCBdatasets.
Firstly , we study the fidelity of generated images on two datasets.
As shown in Table 1,our method surpasses two baselines on all metrics with a notable gain.
Secondly , it is essential forhand-object interaction image generation to maintain the posture information of the target hand andobject.
Therefore, we carefully analyze the posture preservation of generated images.
With the off-the-shelf network [21], we estimate the hand and object pose in the generated images and computethe error with the target pose annotation.
As shown in Table 2, it can be observed that our methodexhibits superior performance for hand and objects of different categories over two other baselines.
Furthermore, we perform qualitative comparisons with baselines.
From Figure 3, we observe that theimages generated by our method exhibit better visual performance compared with previous methods.
Under the complex scenes, where the hand and object are highly interacting, our method can generateimages with more reasonable spatial relationship, which significantly outperforms baseline methods.
We also conduct a user study to evaluate subjective visual performance.
The voting results arereported as the USP metric in Table 1.
It is observed that our method is preferred by over 70 percentagainst two competitors.
4.4 Ablation Studies
We perform ablative experiments to highlight several important components, i.e. , the hand and objecttopology, source feature transfer and split-and-combine generation.
[Table]
Hand and object topology.
In HOGAN, we modulate both hand and object topology into the generatorto provide detailed geometric information of handand object.
In Table 3, we verify its effectiveness bycomparing it with non-topology variants.
It can beobserved that HOGAN (the first row) achieves thebest performance over two other variants.
Withouteither hand or object topology, the framework suffersperformance degradation, e.g. 6.0 and 6.5 on FID,0.012 and 0.015 on LPIPS.
[Table]
feature transfer.
To further enhance the refinement procedure of the coarse target hand image, we transfermulti-layers features from the source hand-object generator according to the transformation flow.
In Table 4, weanalyze the importance of the source feature transfer.
The None variant refers to the framework without source feature transfer.
The Bilinear and Attention variants refer tothe framework with different samplers for feature transferring, i.e. , a bilinear sampler and an attention sampler,respectively.
It can be seen that the source feature transfer provides useful cues for target image generation and the attention sampler achieves the bestperformance among three settings.
[Table]
Split-and-combine Generation.
Considering hand andobject exhibit different properties, we involve a split-andcombine (S&C) strategy in HOGAN.
In this setting, wecompare HOGAN with a parameter-comparable baselinemodel without separately generating hand and object.
In
Table 5, it is observed that HOGAN outperforms its variant (without S&C) by 2.9 FID and 0.036 LPIPS gain, whichdemonstrates this strategy benefits the quality of generatedimages when involving instances with different properties.
4.5 Application
In this subsection, we further explore several interesting applications on our HOGAN, i.e. , objecttexture editing and real-world generation.
In object texture editing, we alter the object texture withcharacters, i.e. , “NeurIPS” and “2022”, and generate the images conditioned on the edited textures.
From Figure 4 (a), it is observed that our generated images well preserve the edited characters on theobject texture.
Furthermore, we take the hand image from the real scene as the source image to testour pre-trained HOGAN framework.
As shown in Figure 4 (b), the generated images both maintainthe source image appearance and meet
the
[Figure]
[Table] Based on our explored task, it is also of great importancefor synthetic data creation [33]. We validate the effectiveness of synthetic data creation in the aspect of boosting HOPE performance as shown in Table 6.
Specifically, weadopt the backbone [21].
“Aug” represents the backboneis trained on both HO3D and our synthesized data. It canbe observed that "Aug" outperforms the baseline (without Aug) under all metrics, especially in the object.  Limitations and Future Work
Our framework considers the complex spatial relationship and different appearance properties betweenhand and object.
The effectiveness of our framework has been validated in Sec. 4.
There stillexist limitations in our framework.
Specifically, it utilizes the dense mesh representation as thepose condition, which inevitably contains misalignment with the RGB image plane even manuallyannotated.
This issue mainly disturbs coarse image synthesis and our framework mainly resorts tothe Hand-Object Synthesis stage for further refinement.
Another way to mitigate this issue is to adopta more effective hand-object pose estimation method for better pose representation.
Our explored HOIG task is of broad research interest to the community.
When hand pose estimationturns from the isolated hand to the interacting hand-object scenario, we advocate the image generationcommunity to draw more attention to this new HOIG task.
This task involves generation of cooccurring instances under complex occlusion conditions.
The advance in HOIG can also inspireother related human-centric image generation tasks.
We outline the future works as follows.
More suitable representations can be explored to serve as a condition during generation.
Moreadvanced techniques can be combined into the framework to enhance the generation quality.
Moreapplications can be explored to fertilize this new HOIG task.
Conclusion
In this paper, we make the first attempt to explore a novel task, namely hand-object interactionimage generation.
This task brings new challenges since it involves generating two co-occurringinstances under complex interaction conditions.
To deal with the challenges of this task, we proposethe HOGAN framework.
It explicitly considers the self- and mutual occlusion among hand and objectand generates the target image in a split-and-combine manner.
For comprehensive comparisons, wepresent baselines adopted from related single hand generation and evaluate the generated imagesfrom multiple perspectives, including fidelity and structure preservation.
Extensive experiments ontwo datasets demonstrate our method outperforms baselines both quantitatively and qualitatively.
Checklist
For all authors...(
a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions
and scope
Yes We have clearly stated the contributions and scope inthe Abstract and Introduction.
b) Did you describe the limitations of your work?
[Yes] We describe the limitations in the Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss
the potential negative societal impacts in Section A of the Supplement. (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments
[Table] (a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)?
[Yes] We include
[Table]
all the necessary code, instructions and environment needed in the Supplementary
[Table]
Materials.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] The important training details are specified in the Section 4.1. [Table]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] In the Supplementary Material, we show the error bars of [Table] our main experiments in the form of variance.
We choose the hyper-parameter by our
experience.
For the number of epochs, we keep it consistent among all methods for fair
comparison. Grid search may bring better performance. [Table] (d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] We report the type of resources
used in the experiments in Section 4.
If
you are using existing
assets (
e
g.,
code , data, models) or curating/releasing new assets..
(a) If your work uses existing assets , did you cite the
creators?
[Yes]
In Section , we
cite corresponding papers for the asserts we use.
(b) Did you mention the license of the assets?
[Yes] HO3Dv3 is publicly available without
stating its license, and DexYCB follows the license of CC BY-NC 4.0.
(c) Did you include any new assets either in the supplemental material or as a URL? Yes]
In the supplementary material we provide
the complete
experiment code,
the necessary commands and the corresponding environment for generating the trainingdataset, training
the model and testing the model
(d) Did you discuss
whether and how consent was obtained from people whose datayou’re
using/
curating
[Yes
] The training datasets HO3Dv3 and DexYCB are publiclyavailable
We
cite the paper that provides the dataset and the model.
The real-worldimages used in
Section
are captured and granted
by the authors themselves.
(e) Did you discuss whether the data you are using curating contains
personally
identifiable information or offensive content?
[Yes
] The datasets used in the experiments
are publicly available and we have removed personally identifiable information or offensive
content.
The real-world image used in Section are captured and granted from the
authors. If you used
crowdsourcing or conducted research with human subjects (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[Yes] We include the full text of instructions given to the
participants during the user study.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] Our IRB application has been granted by
our institution.
The qualitative study is outsourced to the hired participants, without any author of this paper. We hire the participants in person and we randomly hire the participants in our college.
In the user study, the participants are asked to score our
experimental results.
There is no personal information involved and the participants
remain anonymous throughout.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes] All participants for the user study are
voluntary and we have estimated the hourly wage in our local region and have paid them a gift of equal value.
Besides, the time spent by each participant is estimated around 10 minutes.