Original: I NTERACTIVE P ORTRAIT H ARMONIZATION
Revised: I NTERACTIVE P ORTRAIT H ARMONIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Current image harmonization methods consider the entire background as the guidance for harmonization.
Revised: Current image harmonization methods consider the entire background as the guidance for harmonization.

Original: However, this may limit the capability for user to choose any specific object/person in the background to guide the harmonization.
Revised: However, this may limit the capability for user to choose any specific object/person in the background to guide the harmonization.

Original: To enable flexible interaction between user and harmonization, we introduce interactive harmonization, a new setting where the harmonization is performed with respect to a selected region in the reference image instead of the entire background.
Revised: To enable flexible interaction between user and harmonization, we introduce interactive harmonization, a new setting where the harmonization is performed with respect to a selected region in the reference image instead of the entire background.

Original: A new flexible framework that allows users to pick certain regions of the background image and use it to guide the harmonization is proposed.
Revised: A new flexible framework that allows users to pick certain regions of the background image and use it to guide the harmonization is proposed.

Original: Inspired by professional portrait harmonization users, we also introduce a new luminance matching loss to optimally match the color/luminance conditions between the composite foreground and select reference region.
Revised: Inspired by professional portrait harmonization users, we also introduce a new luminance matching loss to optimally match the color/luminance conditions between the composite foreground and select reference region.

Original: This framework provides more control to the image harmonization pipeline achieving visually pleasing portrait edits.
Revised: This framework provides more control to the image harmonization pipeline achieving visually pleasing portrait edits.

Original: Furthermore, we also introduce a new dataset carefully curated for validating portrait harmonization.
Revised: Furthermore, we also introduce a new dataset carefully curated for validating portrait harmonization.

Original: Extensive experiments on both synthetic and real-world datasets show that the proposed approach is efficient and robust compared to previous harmonization baselines, especially for portraits.
Revised: Extensive experiments on both synthetic and real-world datasets show that the proposed approach is efficient and robust compared to previous harmonization baselines, especially for portraits.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the increasing demand of virtual social gathering and conferencing in our lives, image harmonization techniques become essential components to make the virtual experience more engaging and pleasing.
Revised: With the increasing demand of virtual social gathering and conferencing in our lives, image harmonization techniques become essential components to make the virtual experience more engaging and pleasing.

Original: For example, if you cannot join a wedding or birthday party physically but still want to be in the photo, the first option would be to edit yourself into the image.
Revised: For example, if you cannot join a wedding or birthday party physically but still want to be in the photo, the first option would be to edit yourself into the image.

Original: Directly compositing yourself into the photo would not look realistic without matching the color/luminance conditions.
Revised: Directly compositing yourself into the photo would not look realistic without matching the color/luminance conditions.

Original: One possible solution to make the composition image more realistic is to leverage existing image harmonization methods
Revised: One possible solution to make the composition image more realistic is to leverage existing image harmonization methods

Original: Cong et al.
Revised: Cong et al.

Original: (2020; 2021); Cun & Pun (2020);
Revised: (2020; 2021); Cun & Pun (2020);

Original: Ling et al.
Revised: Ling et al.

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: (2021); Guo et al.
Revised: (2021); Guo et al.

Original: (2021b)
Revised: (2021b)

Original: ; Tsai et al.
Revised: ; Tsai et al.

Original: Most previous works focus on a more general image harmonization setup, where the goal is to match a foreground object to a new background scene without too much focus on highly retouched portrait.
Revised: Most previous works focus on a more general image harmonization setup, where the goal is to match a foreground object to a new background scene without too much focus on highly retouched portrait.

Original: However, when we conduct surveys among professional composition Photoshop/Affinity 1 users, we realized that portrait harmonization is the most common task of image editing in real-world scenario and professional settings.
Revised: However, when we conduct surveys among professional composition Photoshop/Affinity 1 users, we realized that portrait harmonization is the most common task of image editing in real-world scenario and professional settings.

Original: This makes portrait harmonization the most important use case of image harmonization.
Revised: This makes portrait harmonization the most important use case of image harmonization.

Original: We note that previous harmonization works have not focused on addressing portrait harmonization on real-world data.
Revised: We note that previous harmonization works have not focused on addressing portrait harmonization on real-world data.

Original: In this work, we aim to explore a better solution to obtain realistic and visually pleasing portrait harmonization for real-world high-resolution edited images.
Revised: In this work, we aim to explore a better solution to obtain realistic and visually pleasing portrait harmonization for real-world high-resolution edited images.

Original: One common question that pops up when we demo existing image harmonization workflow to these professional users is: ‘ How could we choose a certain person as reference when we do harmonization with existing workflow ? ’.
Revised: One common question that pops up when we demo existing image harmonization workflow to these professional users is: ‘ How could we choose a certain person as reference when we do harmonization with existing workflow ? ’.

Original: The workflow design of existing state-of-the-art harmonization methods Cong et al.
Revised: The workflow design of existing state-of-the-art harmonization methods Cong et al.

Original: Ling et al.
Revised: Ling et al.

Original: ; Guo et al.
Revised: ; Guo et al.

Original: (2021b) limits the capability for user to choose any person/region as reference during the harmonization process.
Revised: (2021b) limits the capability for user to choose any person/region as reference during the harmonization process.

Original: These frameworks are designed such that they just take in the composite image and foreground mask as input thus offering no specific way to help the user to guide the harmonization.
Revised: These frameworks are designed such that they just take in the composite image and foreground mask as input thus offering no specific way to help the user to guide the harmonization.

Original: Certain frameworks such as
Revised: Certain frameworks such as

Original: Bargain-Net Cong et al.
Revised: Bargain-Net Cong et al.

Original: (2021) have the flexibility to be tweaked and converted to serve interactive harmonization.
Revised: (2021) have the flexibility to be tweaked and converted to serve interactive harmonization.

Original: However, from our experiments we find that they are not robust enough to perform realistic portrait harmonization.
Revised: However, from our experiments we find that they are not robust enough to perform realistic portrait harmonization.

Original: [Figure]
Revised: [Figure]

Original: Furthermore, professional portraits are mostly shot at studios where screens constitute the background.
Revised: Furthermore, professional portraits are mostly shot at studios where screens constitute the background.

Original: These screens offer little to no information in harmonizing a new composite when edited into the photo.
Revised: These screens offer little to no information in harmonizing a new composite when edited into the photo.

Original: This causes current harmonization methods to produce unstable results (see first row of Figure 1) as they have not been trained to perform harmonization with screens as background.
Revised: This causes current harmonization methods to produce unstable results (see first row of Figure 1) as they have not been trained to perform harmonization with screens as background.

Original: Also, portraits which are captured by everyday users usually contain a background of spatially varying luminance characteristics.
Revised: Also, portraits which are captured by everyday users usually contain a background of spatially varying luminance characteristics.

Original: Using the entire background to guide harmonization here can result in undesirable outcomes (see second row of Figure 1) as harmonization depends on the location where the foreground is composited to.
Revised: Using the entire background to guide harmonization here can result in undesirable outcomes (see second row of Figure 1) as harmonization depends on the location where the foreground is composited to.

Original: To this end, we introduce Interactive harmonization - a more realistic and flexible setup where the user can guide the harmonization.
Revised: To this end, we introduce Interactive harmonization - a more realistic and flexible setup where the user can guide the harmonization.

Original: The user can choose specific regions in the background as a reference to harmonize the composite foreground.
Revised: The user can choose specific regions in the background as a reference to harmonize the composite foreground.

Original: From a professional stand-point, this is a much-needed feature as it could enable a lot of control for editors.
Revised: From a professional stand-point, this is a much-needed feature as it could enable a lot of control for editors.

Original: In this work, we propose a new interactive portrait harmonization framework that has the flexibility to take in a reference region provided by the user.
Revised: In this work, we propose a new interactive portrait harmonization framework that has the flexibility to take in a reference region provided by the user.

Original: We use an encoder-decoder based harmonization network which takes in the foreground composite region as input and a style encoder which takes in the reference region as its input.
Revised: We use an encoder-decoder based harmonization network which takes in the foreground composite region as input and a style encoder which takes in the reference region as its input.

Original: The reference region to guide the harmonization is selected by the user and can be a person, and object or even just some part of the background.
Revised: The reference region to guide the harmonization is selected by the user and can be a person, and object or even just some part of the background.

Original: However, it can be noted that in portrait editing it is very common to choose another person in the picture as reference to obtain an effective harmonization.
Revised: However, it can be noted that in portrait editing it is very common to choose another person in the picture as reference to obtain an effective harmonization.

Original: We also use a style encoder that extracts the style code of the reference region and injects it to the harmonized foreground.
Revised: We also use a style encoder that extracts the style code of the reference region and injects it to the harmonized foreground.

Original: We carefully align the style information with foreground while also preserving the spatial content using adaptive instance normalization Huang & Belongie (2017) layers at decoder.
Revised: We carefully align the style information with foreground while also preserving the spatial content using adaptive instance normalization Huang & Belongie (2017) layers at decoder.

Original: To make the harmonization look more realistic it is important to optimally match the luminance, color, and other appearance information between the reference and foreground.
Revised: To make the harmonization look more realistic it is important to optimally match the luminance, color, and other appearance information between the reference and foreground.

Original: To match these characteristics in manual photo editing, professional photography users usually match statistics of the highest (highlight), lowest (shadow) and average (mid-tone) of luminance points between the composite foreground image and the reference region.
Revised: To match these characteristics in manual photo editing, professional photography users usually match statistics of the highest (highlight), lowest (shadow) and average (mid-tone) of luminance points between the composite foreground image and the reference region.

Original: Hence, we propose a new luminance matching loss that is inspired by professional users 2 .
Revised: Hence, we propose a new luminance matching loss that is inspired by professional users 2 .

Original: In the proposed loss, we match the highlight, mid-tone and shadow points between the reference region and foreground composite region.
Revised: In the proposed loss, we match the highlight, mid-tone and shadow points between the reference region and foreground composite region.

Original: Current publicly available datasets are developed only for background harmonization problem.
Revised: Current publicly available datasets are developed only for background harmonization problem.

Original: So, we curate a new IntHarmony dataset to enable training for interactive harmonization utilizing augmentations that would work well for general as well as portrait harmonization.
Revised: So, we curate a new IntHarmony dataset to enable training for interactive harmonization utilizing augmentations that would work well for general as well as portrait harmonization.

Original: We also introduce a new real-world testing data PortraitTest with ground-truth annotations collected from professional experts to benchmark the methods.
Revised: We also introduce a new real-world testing data PortraitTest with ground-truth annotations collected from professional experts to benchmark the methods.

Original: Codes, datasets and pretrained models will be made public after the review process.
Revised: Codes, datasets and pretrained models will be made public after the review process.

Original: In summary, the following are the key contributions of this work:
Revised: In summary, the following are the key contributions of this work:

Original: • We are the first to introduce Interactive Harmonization and propose a new framework which has the flexibility to solve it.
Revised: • We are the first to introduce Interactive Harmonization and propose a new framework which has the flexibility to solve it.

Original: • We propose a new luminance matching loss to better match the foreground appearance to the reference region.
Revised: • We propose a new luminance matching loss to better match the foreground appearance to the reference region.

Original: • We curate a new synthetic training dataset as well as a professionally annotated benchmark test for portrait harmonization and demonstrate state-of-the-art results on it.
Revised: • We curate a new synthetic training dataset as well as a professionally annotated benchmark test for portrait harmonization and demonstrate state-of-the-art results on it.

Original: • We show that performing interactive harmonization using our proposed method results in visually pleasing portrait edits.
Revised: • We show that performing interactive harmonization using our proposed method results in visually pleasing portrait edits.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Classical methods for image harmonization perform color and tone matching by matching global statistics Reinhard et al.
Revised: Classical methods for image harmonization perform color and tone matching by matching global statistics Reinhard et al.

Original: ; Ling et al.
Revised: ; Ling et al.

Original: There have also been methods which try to apply gradient domain methods to transfer the color and tone P´erez et al.
Revised: Methods like P´erez et al.

Original: ; Tao et al.
Revised: ; Tao et al.

Original: Zhu at al.
Revised: (2010) try to apply gradient domain methods to transfer the color and tone .

Original: Zhu et al.
Revised: Zhu et al.

Original: (2015) proposed the first convolutional network-based method to improve realism of composite images.
Revised: (2015) proposed the first convolutional network-based method to improve realism of composite images.

Original: The first end-to-end deep learning-based image harmonization method was proposed by Tsai et al.Tsai et al.
Revised: The first end-to-end deep learning-based image harmonization method was proposed by Tsai et al.

Original: (2017) where an encoder-decoder based convolutional network was used.
Revised: (2017) where an encoder-decoder based convolutional network was used.

Original: Following that Cun et al.
Revised: Following that Cun &

Original: Cun & Pun (2020) proposed an attention-based module to improve harmonization.
Revised: Pun (2020) proposed an attention-based module to improve harmonization.

Original: Dove-Net, proposed by Cong et al.Cong et al.
Revised: Dove-Net, proposed by Cong et al.

Original: (2020) used a Generative Adversarial Network (GAN)-based method with an additional domain verification discriminator to verify if the foreground and background of a given image belongs to the same image.
Revised: (2020) used a Generative Adversarial Network (GAN)-based method with an additional domain verification discriminator to verify if the foreground and background of a given image belongs to the same image.

Original: Cong et al.

Original: Cong et al.
Revised: Cong et al.

Original: (2020) also proposed a public dataset, iHarmony4 for image harmonization.
Revised: (2020) also proposed a public dataset, iHarmony for image harmonization.

Original: Following that, Cong et al.Cong et al.
Revised: Following that, Cong et al.

Original: (2021) proposed Bargain-Net which uses a domain extractor to extract information about the background to make the harmonized foreground more consistent with the background.
Revised: (2021) proposed Bargain-Net which uses a domain extractor to extract information about the background to make the harmonized foreground more consistent with the background.

Original: Stfiiuk et al.

Original: Sofiiuk et al.
Revised: Sofiiuk et al.

Original: (2021) proposed a new architecture involving pre-trained classification models to improve harmonization.
Revised: (2021) proposed a new architecture involving pre-trained classification models to improve harmonization.

Original: Attention-based feature modulation for harmonization was proposed in Hao et al.
Revised: Attention-based feature modulation for harmonization was proposed in Hao et al.

Original: Ling et al.

Original: Ling et al.
Revised: Ling et al.

Original: (2021) proposed RainNet which introduces a region-aware adaptive instance normalization (RAIN) module to make the style between the background and the composite foreground consistent.
Revised: (2021) proposed RainNet which introduces a region-aware adaptive instance normalization (RAIN) module to make the style between the background and the composite foreground consistent.

Original: Guo et al.
Revised: Guo et al.

Original: Guo et al.

Original: (2021b) proposed intrinsic image harmonization where reflectance and illumination are disentangled and harmonized separately.
Revised: (2021b) proposed intrinsic image harmonization where reflectance and illumination are disentangled and harmonized separately.

Original: It can be noted that photo-realistic style transfer methods Li et al.
Revised: It can be noted that photo-realistic style transfer methods Li et al.

Original:  Luan et al.
Revised: ; Luan et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: (2019) when adopted for harmonization do not perform well as they require similar layout as explained in Jiang et al.Jiang et al.
Revised: (2019) when adopted for harmonization do not perform well as they require similar layout as explained in Jiang et al.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2021) proposed a self-supervised framework to solve image harmonization.
Revised: (2021) proposed a self-supervised framework to solve image harmonization.

Original: Recently, Guo et al.
Revised: Recently, Guo et al.

Original: Guo et al.

Original: (2021a) proposed an image harmonization framework using transformers.
Revised: (2021a) proposed an image harmonization framework using transformers.

Original: Portrait relighting methods Zhou et al.
Revised: Portrait relighting methods Zhou et al.

Original: ; Pandey et al.
Revised: ; Pandey et al.

Original: (2021) have also been explored to produce realistic composite images for a desired scene.
Revised: (2021) have also been explored to produce realistic composite images for a desired scene.

Original:  I NTERACTIVE H ARMONIZATION
Revised:  I NTERACTIVE H ARMONIZATION

Original: Setting:
Revised: Setting:

Original: Given an input composite image C , the goal is to generate a harmonized image H that looks realistic.
Revised: Given an input composite image C , the goal is to generate a harmonized image H that looks realistic.

Original: Composite image corresponds to the edited image where a new person is introduced into the scene.
Revised: Composite image corresponds to the edited image where a new person is introduced into the scene.

Original: This newly introduced region is termed as composite foreground region F and the scene it was introduced into is termed as background B .
Revised: This newly introduced region is termed as composite foreground region F and the scene it was introduced into is termed as background B .

Original: In general harmonization, we use the background B to harmonize F .
Revised: In general harmonization, we use the background B to harmonize F .

Original: In interactive harmonization , we propose using a specific region R of the background B to guide harmonizing F .
Revised: In interactive harmonization , we propose using a specific region R of the background B to guide harmonizing F .

Original: Note that the region R is a subset of B i.e ( R ∈ B ).
Revised: Note that the region R is a subset of B i.e ( R ∈ B ).

Original: R can be any region pertaining to the background that the user wants F to be consistent with.
Revised: R can be any region pertaining to the background that the user wants F to be consistent with.

Original: For portrait images, an easy way to perform realistic harmonization would be to select the person/object in the reference portrait as R to guide the edited person/object F .
Revised: For portrait images, an easy way to perform realistic harmonization would be to select the person/object in the reference portrait as R to guide the edited person/object F .

Original: This will make sure that that the luminance conditions of the reference portrait image is consistent with that of the newly edited-in portrait.
Revised: This will make sure that that the luminance conditions of the reference portrait image is consistent with that of the newly edited-in portrait.

Original: However, we do not hardly constrain the region to be only a portrait in the background, it can also be objects or even a part of the scene.
Revised: However, we do not hardly constrain the region to be only a portrait in the background, it can also be objects or even a part of the scene.

Original: Please note that portraits here do not only correspond to images containing people, it can also contain objects.
Revised: Please note that portraits here do not only correspond to images containing people, it can also contain objects.

Original: A photo becomes a portrait when certain subjects are the main focus of the picture.
Revised: A photo becomes a portrait when certain subjects are the main focus of the picture.

Original: [Figure]

Original: Dataset Curation: Publicly available datasets like iHarmony4 Cong et al.
Revised: Dataset Curation: Publicly available datasets like iHarmony4 Cong et al.

Original: (2020) were proposed for background harmonization and do not provide any reference region information.
Revised: (2020) were proposed for background harmonization and do not provide any reference region information.

Original: So, we curate  a synthetic dataset and also introduce a real-world portrait harmonization dataset for validating.
Revised: So, we curate [Figure] a synthetic dataset and also introduce a real-world portrait harmonization dataset for validating.

Original: 1) IntHarmony:
Revised: 1) IntHarmony:

Original: This newly curated synthetic dataset specifies an additional reference region to guide the harmonization.
Revised: This newly curated synthetic dataset specifies an additional reference region to guide the harmonization.

Original: IntHarmony has the following information for each data instance: composite image, ground truth, foreground mask of the composite foreground and a guide mask which provides information about the reference region to guide the harmonization.
Revised: IntHarmony has the following information for each data instance: composite image, ground truth, foreground mask of the composite foreground and a guide mask which provides information about the reference region to guide the harmonization.

Original: IntHarmony is built on top of MS-COCO dataset Lin et al.
Revised: IntHarmony is built on top of MS-COCO dataset Lin et al.

Original: We make use of the instance masks provided in
Revised: We make use of the instance masks provided in

Original: MS-COCO to simulate foreground and reference regions.
Revised: MS-COCO to simulate foreground and reference regions.

Original: First, we select a random instance mask to pick the foreground region.
Revised: First, we select a random instance mask to pick the foreground region.

Original: The selected foreground region is then augmented using a wide set of meaningful augmentations focusing on luminance, contrast and color.
Revised: The selected foreground region is then augmented using a wide set of meaningful augmentations focusing on luminance, contrast and color.

Original: We use all the augmentations proposed in Jiang et al.
Revised: We use all the augmentations proposed in Jiang et al.

Original: (2021) to create the composite.
Revised: (2021) to create the composite.

Original: More details on this can be found in the supplementary document.
Revised: More details on this can be found in the supplementary document.

Original: This augmented foreground is composited with the original image to get the composite image ( I ).
Revised: This augmented foreground is composited with the original image to get the composite image ( I ).

Original: Another random instance mask is used to get the reference guide mask.
Revised: Another random instance mask is used to get the reference guide mask.

Original: The original image is considered as the ground truth.
Revised: The original image is considered as the ground truth.

Original: The number of training images in IntHarmony is 118287 and 959 images are allocated for testing.
Revised: The number of training images in IntHarmony is 118287 and 959 images are allocated for testing.

Original: Note that the instance masks and the augmentations are chosen at random to induce more generalizabilty to the network.
Revised: Note that the instance masks and the augmentations are chosen at random to induce more generalizabilty to the network.

Original: IntHarmony consists of both objects, backgrounds and people and is not very focused for portraits.
Revised: IntHarmony consists of both objects, backgrounds and people and is not very focused for portraits.

Original: We use this dataset to help the network learn the interactive way to perform harmonization as we have a reference region to access.
Revised: We use this dataset to help the network learn the interactive way to perform harmonization as we have a reference region to access.

Original: We also use a similar dataset creation scheme on Multi-Human People Parsing (MHP) dataset Zhao et al.
Revised: We also use a similar dataset creation scheme on Multi-Human People Parsing (MHP) dataset Zhao et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: ; Nie et al.
Revised: ; Nie et al.

Original: (2017) to synthesize another data further finetune the models for portrait harmonization.
Revised: (2017) to synthesize another data further finetune the models for portrait harmonization.

Original: Note thatthe MHP dataset contains only images withpeople.
Revised: Note that MHP dataset contains only images with

Revised: people.

Original: 2) PortraitTest:
Revised: PortraitTest

Original:  To validate our performance for real-time applications, we introduce a real-world test data consisting of high resolution composite portrait images.
Revised: : To validate our performance for real-time applications, we introduce a real-world test data consisting of high resolution composite portrait images.

Original: The ground truth is annotated with the help of multiple professionals using PhotoShop curve adjustment layers to change brightness, hue, saturation layer and contrast of the composite foreground to make it look more natural and pleasing.
Revised: The ground truth is annotated with the help of multiple professionals using PhotoShop curve adjustment layers to change brightness, hue, saturation layer and contrast of the composite foreground to make it look more natural and pleasing.

Original: The number of images in this test dataset is 50 and the composite foreground is of varied luminance conditions, contrast and color.
Revised: The number of images in this test dataset is 50 and the composite foreground is of varied luminance conditions, contrast and color.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we first explain the details of the framework we propose to solve interactive harmonization.
Revised: In this section, we first explain the details of the framework we propose to solve interactive harmonization.

Original: We then give the training details and the loss functions we introduce for this task.
Revised: We then give the training details and the loss functions we introduce for this task.

Original: Network Details: An overview of the proposed framework is shown in Figure 2.
Revised: Network Details: An overview of the proposed framework is shown in Figure 2.

Original: It consists of two sub-networks: a harmonizer network and a style encoder network.
Revised: It consists of two sub-networks: a harmonizer network and a style encoder network.

Original: Harmonization Network is an encoder-decoder architecture which takes in two inputs: composite foreground image ( I ), a foreground mask and a style code ( ϕ ) extracted from the style encoder which is of dimension 1 × D .
Revised: Harmonization Network is an encoder-decoder architecture which takes in two inputs: composite foreground image ( I ), a foreground mask and a style code ( ϕ ) extracted from the style encoder which is of dimension 1 × D .

Original: The encoder is built using 4 convolutional blocks where each conv block has a conv layer that increases the number of channels and performs downsampling, followed by a set of residual blocks.
Revised: The encoder is built using 4 convolutional blocks where each conv block has a conv layer that increases the number of channels and performs downsampling, followed by a set of residual blocks.

Original: The latent features from the encoder have a spatial dimension of ( H/ 16) ×
Revised: The latent features from the encoder have a spatial dimension of ( H/ 16) ×

Original: ( W/ 16) .
Revised: ( W/ 16) .

Original: The decoder is also built using 4 conv blocks where each block has an upsampling layer and a set of residual blocks similar to the encoder.
Revised: The decoder is also built using 4 conv blocks where each block has an upsampling layer and a set of residual blocks similar to the encoder.

Original: In addition, we use adaptive instance norm (AdaIN) layers in each conv block that takes in the style code ( ϕ ) and the features from previous conv block in decoder as their input.
Revised: In addition, we use adaptive instance norm (AdaIN) layers in each conv block that takes in the style code ( ϕ ) and the features from previous conv block in decoder as their input.

Original: Thus, the decoder uses the style code extracted from the reference object to guide the harmonization.
Revised: Thus, the decoder uses the style code extracted from the reference object to guide the harmonization.

Original: The output from decoder is the harmonized foreground image.
Revised: The output from decoder is the harmonized foreground image.

Original: It is then composited with the original reference image to get the complete harmonized output.
Revised: It is then composited with the original reference image to get the complete harmonized output.

Original: Style Encoder takes in the reference region chosen by the user as the input.
Revised: Style Encoder takes in the reference region chosen by the user as the input.

Original: The style encoder consists of a series of partial convolutional layers Liu et al.
Revised: The style encoder consists of a series of partial convolutional layers Liu et al.

Original: (2018) to downsample the input image, extract meaningful features and output a style code.
Revised: (2018) to downsample the input image, extract meaningful features and output a style code.

Original: The partial convolution layers take in both the reference image and guide mask as their output.
Revised: The partial convolution layers take in both the reference image and guide mask as their output.

Original: The latent embedding is fed through a average pooling layer to obtain the 1D style code ( ϕ ).
Revised: The latent embedding is fed through a average pooling layer to obtain the 1D style code ( ϕ ).

Original: Training Details: Training the network for interactive harmonization is performed stage-wise.
Revised: Training Details: Training the network for interactive harmonization is performed stage-wise.

Original: In the first stage, we train the network to perform background harmonization, where the training data of iHarmony4 Cong et al.
Revised: In the first stage, we train the network to perform background harmonization, where the training data of iHarmony4 Cong et al.

Original: (2020) is used.
Revised: (2020) is used.

Original: The input to the harmonizer network is the masked out foreground image concatenated with the style code of the background extracted from the style encoder.
Revised: The input to the harmonizer network is the masked out foreground image concatenated with the style code of the background extracted from the style encoder.

Original: In the next stage, we finetune our network for interactive harmonization on IntHarmony.
Revised: In the next stage, we finetune our network for interactive harmonization on IntHarmony.

Original: Here, the input to the style encoder is the reference region.
Revised: Here, the input to the style encoder is the reference region.

Original: Thus, the network here is trained in such a way that a reference region guides the harmonization which is also due to the loss functions we introduce (see Section 4.3).
Revised: Thus, the network here is trained in such a way that a reference region guides the harmonization which is also due to the loss functions we introduce (see Section 4.3).

Original: Finally, we further fine-tune the network on the augmented MHP dataset for portrait harmonization to be used in professional settings.
Revised: Finally, we further fine-tune the network on the augmented MHP dataset for portrait harmonization to be used in professional settings.

Original: The input to the harmonizer network is the masked out foreground image (rather than entire composite image) concatenated with the style code of the background extracted from the style encoder.
Revised: The input to the harmonizer network is the masked out foreground image (rather than entire composite image) concatenated with the style code of the background extracted from the style encoder.

Original: From our experiments, we realized that using the entire composite as input to the harmonizer network does not result in optimal training.
Revised: From our experiments, we realized that using the entire composite as input to the harmonizer network does not result in optimal training.

Original: This happens because the style code information bleeds out to the background of the composite, reducing its influence to the foreground.
Revised: This happens because the style code information bleeds out to the background of the composite, reducing its influence to the foreground.

Original: Masking out the background makes sure that the style code affects only the composite foreground and not the entire image.
Revised: Masking out the background makes sure that the style code affects only the composite foreground and not the entire image.

Original: Loss Functions: We use a combination of the proposed luminance matching loss, consistency loss, harmonization loss, and triplet losses for training our framework.
Revised: Loss Functions: We use a combination of the proposed luminance matching loss, consistency loss, harmonization loss, and triplet losses for training our framework.

Original: Luminance Matching loss: The main objective of interactive portrait harmonization is to make sure that the harmonized foreground matches the appearance of the reference region.
Revised: Luminance Matching loss: The main objective of interactive portrait harmonization is to make sure that the harmonized foreground matches the appearance of the reference region.

Original: To achieve this, we introduce three new losses - highlight, mid-tone and shadow losses, which are used to match the highlight, mid-tone and shadow between the reference region and foreground region to be harmonized.
Revised: To achieve this, we introduce three new losses - highlight, mid-tone and shadow losses, which are used to match the highlight, mid-tone and shadow between the reference region and foreground region to be harmonized.

Original: We define these losses as follows:
Revised: We define these losses as follows:

Original: [Equation]
Revised: [Equation]

Original: where H corresponds to the harmonized image and ˆ I corresponds to the ground truth.
Revised: where H corresponds to the harmonized image and R corresponds to the reference region.

Original: These losses try to match the intensities of the foreground with the reference region at the highest, lowest and average luminescence points thus matching the contrast.
Revised: These losses try to match the intensities of the foreground with the reference region at the highest, lowest and average luminescence points thus matching the contrast.

Original: For the highest and lowest points, we choose the 90 th and 10 th percentile respectively to improve stability.
Revised: For the highest and lowest points, we choose the 90 th and 10 th percentile respectively to improve stability.

Original: We define luminance matching loss ( L LM ) as the sum of these 3 losses:
Revised: We define luminance matching loss ( L LM ) as the sum of these 3 losses:

Original: [Equation]
Revised: [Equation]

Original: Luminance matching loss ( L LM ) is illustrated in Figure 3.
Revised: Luminance matching loss ( L LM ) is illustrated in Figure 3.

Original: Note that L LM is an main addition to our training methodology which helps to make sure that the statistics between the reference and foreground are matched.
Revised: Note that L LM is an main addition to our training methodology which helps to make sure that the statistics between the reference and foreground are matched.

Original: This loss penalizes the network if the statistics of the harmonized foreground and the reference region are not matched.
Revised: This loss penalizes the network if the statistics of the harmonized foreground and the reference region are not matched.

Original: Consistency loss: We also introduce a consistency loss L consis between style codes of the harmonized foreground and the reference region to penalize the network whenever the style code of the reference region and harmonized foreground region are different from each other.
Revised: Consistency loss: We also introduce a consistency loss L consis between style codes of the harmonized foreground and the reference region to penalize the network whenever the style code of the reference region and harmonized foreground region are different from each other.

Original: The consistency loss is defined as:
Revised: The consistency loss is defined as:

Original: [Equation]
Revised: [Equation]

Revised: Harmonization loss: We also use a generic harmonization loss which is an L1-loss between the prediction and the ground-truth as follows:

Revised: [Equation]

Revised: where H corresponds to the harmonized image and ˆ I represents the ground truth.

Revised: This loss tries to minimize the distance between the prediction and input thus making sure the prediction

Original: [Figure]
Revised: [Figure]

Original: Highlight Matching
Revised: Highlight Matching

Original: Midtone Matching
Revised: Midtone Matching

Original: Shadow Matching
Revised: Shadow Matching

Original: F o r e g r ound
Revised: F o r e g r ound

Original: R e g i on
Revised: R e g i on

Original: R e f e r e n ce
Revised: R e f e r e n ce

Original: R
Revised: R

Original: e
Revised: e

Original: g
Revised: g

Original: i on
Revised: i on

Original: Harmonization loss: We also use a generic harmonization loss which is an L1-loss between the prediction and the ground-truth as follows:

Original: [Equation]

Original: This loss tries to minimize the distance between the prediction and input thus making sure the prediction is always consistent in appearance with that of the ground truth.

Original: Triplet losses: We introduce two triplet losses similar to Cong et al.
Revised: Triplet losses: We introduce two triplet losses similar to Cong et al.

Original: based on the style codes of the composite foreground ( c ), harmonized foreground ( h ), real foreground ( r ) (ground truth) and that of the reference region ( b ).
Revised: based on the style codes of the composite foreground ( c ), harmonized foreground ( h ), real foreground ( r ) (ground truth) and that of the reference region ( b ).

Original: The first triplet loss is as follows:
Revised: The first triplet loss is as follows:

Original: [Equation]
Revised: [Equation]

Original: This loss tries to make the style code generated by the style encoder for harmonized foreground ( h ) to be closer to that of style code of reference region ( b ) while being far away from the style code of composite foreground ( c ).
Revised: This loss tries to make the style code generated by the style encoder for harmonized foreground ( h ) to be closer to that of style code of reference region ( b ) while being far away from the style code of composite foreground ( c ).

Original: m here corresponds to the margin.
Revised: m here corresponds to the margin.

Original: We define another triplet loss which forces the real foreground style code ( r ) to be close to harmonized foreground style code ( h ) while making sure the style code of harmonized foreground style code ( h ) is far away from composite foreground ( c ):
Revised: We define another triplet loss which forces the real foreground style code ( r ) to be close to harmonized foreground style code ( h ) while making sure the style code of harmonized foreground style code ( h ) is far away from composite foreground ( c ):

Original: [Equation]
Revised: [Equation]

Original: The total loss used to train the network is as follows:
Revised: The total loss used to train the network is as follows:

Original: [Equation]
Revised: [Equation]

Original: where α , β and λ are the factors that control the contribution the triplet losses have in the total loss.
Revised: where α , β and λ are the factors that control the contribution the triplet losses have in the total loss.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Implementation Details: Our framework is developed in Pytorch Paszke et al.
Revised: Implementation Details: Our framework is developed in Pytorch Paszke et al.

Original: and the training is done using NVIDIA RTX 8000 GPUs.
Revised: and the training is done using NVIDIA RTX 8000 GPUs.

Original: The training is done stage-wise as explained in the previous section.
Revised: The training is done stage-wise as explained in the previous section.

Original: We use an Adam optimizer Kingma & Ba (2014) with a learning rate of 10 − 4 , 10 − 5 , 10 − 6 at each stage respectively.
Revised: We use an Adam optimizer Kingma & Ba (2014) with a learning rate of 10 − 4 , 10 − 5 , 10 − 6 at each stage respectively.

Original: The batch size is set equal to 48.
Revised: The batch size is set equal to 48.

Original: The images are resized to 256 × 256 while training.
Revised: The images are resized to 256 × 256 while training.

Original: During inference, to get the high resolution images back, we use a polynomial fitting like in Afifi et al.
Revised: During inference, to get the high resolution images back, we use a polynomial fitting like in Afifi et al.

Original: We composite the high resolution harmonized foreground with the reference image to get the high resolution harmonized image.
Revised: We composite the high resolution harmonized foreground with the reference image to get the high resolution harmonized image.

Original: Comparison with State-of-the-art: We compare our proposed framework with recent harmonization networks like Dove-Net Cong et al.
Revised: Comparison with State-of-the-art: We compare our proposed framework with recent harmonization networks like Dove-Net Cong et al.

Original: Bargain-Net Cong et al.
Revised: Bargain-Net Cong et al.

Original: (2021), and Rain-Net Ling et al.
Revised: (2021), and Rain-Net Ling et al.

Original: We note that Bargain-Net has a domain extractor branch that directly takes in the background of the image to make the harmonization consistent with the background.
Revised: We note that Bargain-Net has a domain extractor branch that directly takes in the background of the image to make the harmonization consistent with the background.

Original: In addition to comparing with the original model, we also re-train Bargain-Net in a interactive fashion similar to our proposed approach for fair comparison.
Revised: In addition to comparing with the original model, we also re-train Bargain-Net in a interactive setup similar to our proposed approach for fair comparison.

Original: Here, we feed in the reference region as the input to the domain extractor and use similar protocols as our method.
Revised: Here, we feed in the reference region as the input to the domain extractor and use similar protocols as our method.

Original: We call this configuration BargainNet (R).
Revised: We call this configuration Bargain-Net (R).

Original: Note that Bargain-Net (R) is trained with the same data and stage-wise training as our method for fair comparison.
Revised: Note that Bargain-Net (R) is trained with the same data and stage-wise training as our method for fair comparison.

Original: We explain why the other frameworks cannot be converted for interactive harmonization in the supplementary material.
Revised: We explain why the other frameworks cannot be converted for interactive harmonization in the supplementary material.

Original: We use publicly available weights of Dove-Net, Bargain-Net and Rain-Net to get their predictions on PortraitTest and IntHarmony.
Revised: We use publicly available weights of Dove-Net, Bargain-Net and Rain-Net to get their predictions on PortraitTest and IntHarmony.

Original: We denote our approach as IPH (Interactive Portrait Harmonization).
Revised: We denote our approach as IPH (Interactive Portrait Harmonization).

Original: While testing with IntHarmony, Bargain-Net [Table] are trained on the training data of IntHarmony.
Revised: While testing with IntHarmony, Bargain-Net (R) and IPH are trained on the training data of IntHarmony.

Original: While testing with PortraitTest, we compare with two more configurations: Bargain-Net (R) (++) and IPH (++) which are further finetuned on augmented portrait images from the MHP dataset as explained in Section 3.3.
Revised: While testing with PortraitTest, we compare with two more configurations: Bargain-Net (R) (++) and IPH (++) which are further finetuned on augmented portrait images from the MHP dataset as explained in Section 3.3.

Original: As IntHarmony actually contains objects, people and natural scenes, validating on it helps us prove that the interactive feature helps in even normal harmonization.
Revised: As IntHarmony actually contains objects, people and natural scenes, validating on it helps us prove that the interactive feature helps in even normal harmonization.

Original: Validating on PortraitTest helps us prove if our method will work well  on real-world high resolution portrait images.
Revised: Validating on PortraitTest helps us prove if our method will work well [Table] on real-world high resolution portrait images.

Original: It also helps us understand where our method stands when compared to professional editors.
Revised: It also helps us understand where our method stands when compared to professional editors.

Original: Referenced Quality Metrics: We use referenced quality metrics like PSNR, SSIM and MSE to quantitatively compare the performance of IPH with the previous methods.
Revised: Referenced Quality Metrics: We use referenced quality metrics like PSNR, SSIM and CIE Delta to quantitatively compare the performance of IPH with the previous methods.

Original: Table 1 consists of the performance comparison on both the PortraitTest and IntHarmony test datasets.
Revised: Table 1 consists of the performance comparison on both the PortraitTest and IntHarmony test datasets.

Original: We note that IPH achieves the best performance in terms of PSNR, MSE and SSIM when compared to the previous harmonization networks.
Revised: We note that IPH achieves the best performance in terms of PSNR, CIE Delta and SSIM when compared to the previous harmonization networks.

Original: IPH outperforms all previous methods across all metrics in both datasets.
Revised: IPH outperforms all previous methods across all metrics in both datasets.

Original: It can be noted that BargainNet (R)/ BargainNet (R) (++) do not perform as well as
Revised: It can be noted that BargainNet (R)/ BargainNet (R) (++) do not perform as well as

Original: IPH/IPH (++) even though they are trained with the same pipeline and training conditions of IPH.
Revised: IPH/IPH (++) even though they are trained with the same pipeline and training conditions of IPH.

Original: Prior methods fail as they do not consider arbitrary regions to guide the harmonization and also don’t account for luminance matching.
Revised: Prior methods fail as they do not consider arbitrary regions to guide the harmonization and also don’t account for luminance matching.

Original: We get the best performance as the proposed approach is more effective at improving the performance compared to the previous frameworks.
Revised: We get the best performance as the proposed approach is more effective at improving the performance compared to the previous frameworks.

Original: Visual Quality Comparison: In Figure 4, we show results on composite portraits from PortraitTest dataset.
Revised: Visual Quality Comparison: In Figure 4, we show results on composite portraits from PortraitTest dataset.

Original: It can be seen that the harmonized foreground generated by generic harmonization methods like Dove-Net, Bargain-Net, and Rain-Net is inconsistent with color and luminance conditions when compared to the reference portrait.
Revised: It can be seen that the harmonized foreground generated by generic harmonization methods like Dove-Net, Bargain-Net, and Rain-Net is inconsistent with color and luminance conditions when compared to the reference portrait.

Original: Also, the predictions of Bargain-Net and Rain-Net fail to harmonize the local lighting conditions present in the composite as they fail to explicitly match the highlight, mid-tone and shadow.
Revised: Also, the predictions of Bargain-Net and Rain-Net fail to harmonize the local lighting conditions present in the composite as they fail to explicitly match the highlight, mid-tone and shadow.

Original: Bargain-Net (R) (++) performs interactive harmonization however it still fails to match the contrast of the reference.
Revised: Bargain-Net (R) (++) performs interactive harmonization however it still fails to match the contrast of the reference.

Original: Our method harmonizes the composite well even considering the local luminance conditions and is close to the annotations done by a professional.
Revised: Our method harmonizes the composite well even considering the local luminance conditions and is close to the annotations done by a professional.

Original: User Study: Wealso conduct a human subjective review to compare the performance of IPH with other methods.
Revised: User Studies: We conduct two different human subjective reviews to compare the performance of IPH with other methods.

Original: We select 15 images from the PortraitTest dataset and present the predictions of Dove-Net, Bargain-Net (R) (++), Rain-Net and IPH (++) on a screen.
Revised: First, we select 15 images from the PortraitTest dataset and present the predictions of Dove-Net, Bargain-Net (R) (++), Rain-Net and IPH (++) on a screen.

Original: We ask 30 subjects to independently rank the predictions from 1 to 3 based on the visual quality considering if the color and luminance conditions of all the portraits in the image look consistent.
Revised: We asksubjects to independently rank the predictions from 1 to 3 based on the visual quality considering if the color and luminance conditions of all the portraits in the image look consistent.

Original: We give scores 3,2, and 1 for ranks 1,2 and 3 respectively.
Revised: We give scores 3,2, and 1 for ranks 1,2 and 3 respectively.

Original: The mean scores of each method has been tabulated in Table 2.
Revised: Additionally, we also conduct a pair-wise A/B user study. Here, we show the input image and the output of two methods to a user. We then collect these votes and report the preference of the users. We apply the Thurstone Case V analysis to these votes to get the preference values.

Revised: We also report the Bradley-Terry (BT) score .

Original: We note that our  methodachieves the best score which is consistent with the observation from performance metrics comparison.
Revised: All these scores have been tabulated in Table 2 where our proposed method is observed to be desired the most. [Table] [Figure] [Table]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Ablation Study: We conduct an ablation study to analyse the importance of each component of our proposed method.
Revised: Ablation Study: We conduct an ablation study to analyse the importance of each component of our proposed method.

Original: We evaluate these models on the PortraitTest dataset.
Revised: We evaluate these models on the PortraitTest dataset.

Original: We start with just the
Revised: We start with just the

Original: Harmonizer Network (H-Net) without AdaIN layers trained for generic harmonization.
Revised: Harmonizer Network (H-Net) without AdaIN layers trained for generic harmonization.

Original: Then we add the style encoder and AdaIN layers in the decoder and train for interactive harmonization (H-Net+[Figure] [Table] [Table]
Revised: Then we add the style encoder and AdaIN layers in the decoder and train for interactive harmonization (H-Net+

Original: SE).
Revised: SE).

Original: All of these methods are trained using the losses L harmonization , L consis and L triplet .
Revised: All of these methods are trained using the losses L harmonization , L consis and L triplet .

Original: Then we add the novel losses introduced in this work - L highlight , L mid − tone , and L shadow one by one.
Revised: Then we add the novel losses introduced in this work - L highlight , L mid − tone , and L shadow individually.

Original: From Table [Table] it can be observed that all the new components introduced in this work are important and play a key role in achieving better performance.
Revised: From Table 3, it can be observed that all the new components introduced in this work are important and play a key role in achieving better performance.

Original: Making the framework interactive by using the style encoder and injecting the style code of reference region to guide the harmonization is sufficient the push the performance above previous methods.
Revised: Making the framework interactive by using the style encoder and injecting the style code of reference region to guide the harmonization is sufficient the push the performance above previous methods.

Original: Furthermore, the proposed luminance matching loss helps improve the performance by a significant margin proving that matching the highlight, mid-tone and shadow of the foreground and reference help obtain better harmonization.
Revised: Furthermore, the proposed luminance matching loss helps improve the performance by a significant margin proving that matching the highlight, mid-tone and shadow of the foreground and reference help obtain better harmonization.

Original: Robustness to chosen reference region: One key advantage our framework has over generic harmonization is that it is more flexible and robust.
Revised: Robustness to chosen reference region: One key advantage our framework has over generic harmonization is that it is more flexible and robust.

Original: We show that IPH is robust to different regions chosen by the user.
Revised: We show that IPH is robust to different regions chosen by the user.

Original: We note that the output harmonization adjusts itself with the select region chosen from the background.
Revised: We note that the output harmonization adjusts itself with the select region chosen from the background.

Original: This is very useful in cases where the luminance conditions vary across different parts of the background image.
Revised: This is very useful in cases where the luminance conditions vary across different parts of the background image.

Original: We illustrate this in Figure 10.
Revised: We illustrate this in Figure 10.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we proposed a new framework called interactive portrait harmonization where the user has the flexibility of choosing a specific reference region to guide the harmonization.
Revised: In this work, we proposed a new framework called interactive portrait harmonization where the user has the flexibility of choosing a specific reference region to guide the harmonization.

Original: We showed that this helps the user obtain more realistic harmonization results while providing more control and flexibility to the compositing workflow.
Revised: We showed that this helps the user obtain more realistic harmonization results while providing more control and flexibility to the compositing workflow.

Original: We also proposed a new luminance matching loss to carefully match the appearance between the foreground composite and reference region.
Revised: We also proposed a new luminance matching loss to carefully match the appearance between the foreground composite and reference region.

Original: In addition, we introduced two datasets: a synthetic IntHarmony dataset for training and a real-world PortraiTest dataset for testing.
Revised: In addition, we introduced two datasets: a synthetic IntHarmony dataset for training and a real-world PortraiTest dataset for testing.

Original: Extensive experiments and analysis showed that IPH performs better and is more robust and useful to solve real-world portrait harmonization problems in practical settings compared to the previous methods.
Revised: Extensive experiments and analysis showed that IPH performs better and is more robust and useful to solve real-world portrait harmonization problems in practical settings compared to the previous methods.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: Converting user-guided to automatic?
Revised: Converting user-guided to automatic?

Original: Please note that although we provide the flexibility for the user to point to a reference region to guide the harmonization, our framework can be made completely automatic.
Revised: Please note that although we provide the flexibility for the user to point to a reference region to guide the harmonization, our framework can be made completely automatic.

Original: We can make use of saliency detection models Pang et al.
Revised: We can make use of saliency detection models Pang et al.

Original: (2020) to get the segmentation mask of the most salient object/person in the background and use that as the reference region to guide the portrait harmonization.
Revised: (2020) to get the segmentation mask of the most salient object/person in the background and use that as the reference region to guide the portrait harmonization.

Original: In fact, we can make this the initial outcome for portrait harmonization and get feedback from the user if they are interested to select a different region or a sub-part of the current region selected to guide the harmonization.
Revised: In fact, we can make this the initial outcome for portrait harmonization and get feedback from the user if they are interested to select a different region or a sub-part of the current region selected to guide the harmonization.

Original: Different Objects Harmonization:
Revised: Different Objects Harmonization:

Original: In IPH, the input to the style encoder does not have any constraint.
Revised: In IPH, the input to the style encoder does not have any constraint.

Original: It need not even be a region present in the background image.
Revised: It need not even be a region present in the background image.

Original: Any region from any image can be fed as the input to the style encoder thus making the framework more flexible.
Revised: Any region from any image can be fed as the input to the style encoder thus making the framework more flexible.

Original: This can be useful in the context of style transfer and re-luminance applications.
Revised: This can be useful in the context of style transfer and re-luminance applications.

Original: Figure 6 illustrates the harmonization result when the composite foreground is harmonized with a totally different objects from the different reference images.
Revised: Figure 6 illustrates the harmonization result when the composite foreground is harmonized with a totally different objects from the different reference images.

Original: [Figure]
Revised: [Figure]

Original: Number of Parameters:
Revised: Number of Parameters:

Original: In terms of parameters, IPH is lighter compared to the previous
Revised: In terms of parameters, IPH is lighter compared to the previous

Original: IPH has 27 M parameters compared to Bargain-Net’s 58 M , Dove-Net’s 54 M and Rain-Net’s 54 M number of parameters.
Revised: IPH has 27 M parameters compared to Bargain-Net’s 58 M , Dove-Net’s 54 M and Rain-Net’s 54 M number of parameters.

Original: Tweaking from Harmonization to Color Transfer: We show that the appearance of the harmonized image can be easily controlled using the style code.
Revised: Tweaking from Harmonization to Color Transfer: We show that the appearance of the harmonized image can be easily controlled using the style code.

Original: If the style code is made to learn the match color instead of harmonizing the image, we show that our framework can be used to perform color transfer.
Revised: If the style code is made to learn the match color instead of harmonizing the image, we show that our framework can be used to perform color transfer.

Original: We make two simple changes in our framework to obtain this: (1) increase color augmentations (2) change L consis loss to make the style codes of the harmonized foreground and the reference region to be far apart from each other; we could obtain style encoder which impart an aggressive color change.
Revised: We make two simple changes in our framework to obtain this: (1) increase color augmentations (2) change L consis loss to make the style codes of the harmonized foreground and the reference region to be far apart from each other; we could obtain style encoder which impart an aggressive color change.

Original: We term the style code extracted from this style encoder as ψ and the original style code as ϕ .
Revised: We term the style code extracted from this style encoder as ψ and the original style code as ϕ .

Original: Now to control the appearance, we just blend these color codes extracted from these two encoders using different rations.
Revised: Now to control the appearance, we just blend these color codes extracted from these two encoders using different rations.

Original: The new style code γ would be
Revised: The new style code γ would be

Original: [Equation]
Revised: [Equation]

Original: where r 1 and r 2 are the blending ratios.
Revised: where r 1 and r 2 are the blending ratios.

Original: In Figure 7, we show the change in appearance for different combinations of r 1 and r 2 .
Revised: In Figure 7, we show the change in appearance for different combinations of r 1 and r 2 .

Original: This shows that IPH can be easily tweaked to make it interactive resulting in realistic harmonization depending on the user’s choice of r 1 and r 2 .
Revised: This shows that IPH can be easily tweaked to make it interactive resulting in realistic harmonization depending on the user’s choice of r 1 and r 2 .

Original: Limitations: Although interactive portrait harmonization outperforms previous harmonization methods, there are still some limitations.
Revised: Limitations: Although interactive portrait harmonization outperforms previous harmonization methods, there are still some limitations.

Original: Our method works fairly well even for in-the-wild composite portraits consisting of people.
Revised: Our method works fairly well even for in-the-wild composite portraits consisting of people.

Original: However, its performance decreases if we test it on portraits consisting of objects.
Revised: However, its performance decreases if we test it on portraits consisting of objects.

Original: This can be understood as it is still difficult for the style encoder to extract meaningful style code if the object material is peculiar.
Revised: This can be understood as it is still difficult for the style encoder to extract meaningful style code if the object material is peculiar.

Original: For example, if the reference object chosen has a shiny surface or is of a unique material, it is difficult for the style encoder to extract a meaningful style code capturing the texture/material reflectance of the object and use it for harmonization.
Revised: For example, if the reference object chosen has a shiny surface or is of a unique material, it is difficult for the style encoder to extract a meaningful style code capturing the texture/material reflectance of the object and use it for harmonization.

Original: Bargain-Net and Rain-Net: We note that Bargain-Net Cong et al.
Revised: Bargain-Net and Rain-Net: We note that Bargain-Net Cong et al.

Original: and Rain-Net Ling et al.
Revised: and Rain-Net Ling et al.

Original: (2021) specifically focus on using the background to guide the harmonization.
Revised: (2021) specifically focus on using the background to guide the harmonization.

Original: Bargain-Net has a domain extractor to extract the features of background while Rain-Net proposes a RAIN module which normalizes the background features with the foreground.
Revised: Bargain-Net has a domain extractor to extract the features of background while Rain-Net proposes a RAIN module which normalizes the background features with the foreground.

Original: We tried to make both of these networks interactive by replacing the background mask with a guide mask to select the reference region.
Revised: We tried to make both of these networks interactive by replacing the background mask with a guide mask to select the reference region.

Original: With the pre-trained weights, while changing the guide mask we did not see any significant change in terms of performance metrics nor visual quality for both Bargain-Net and Rain-Net.
Revised: With the pre-trained weights, while changing the guide mask we did not see any significant change in terms of performance metrics nor visual quality for both Bargain-Net and Rain-Net.

Original: If we retrain the network to take in the guide mask instead of background mask, we noted a small improvement in results for Bargain-Net as reported in the main paper.
Revised: If we retrain the network to take in the guide mask instead of background mask, we noted a small improvement in results for Bargain-Net as reported in the main paper.

Original: For Rain-Net, we did not see any improvement.
Revised: For Rain-Net, we did not see any improvement.

Original: This might be because RAIN module uses the masks in the feature space to normalize and that might not be an optimal way to actually get meaningful features to inject to the foreground.
Revised: This might be because RAIN module uses the masks in the feature space to normalize and that might not be an optimal way to actually get meaningful features to inject to the foreground.

Original: Training it in an interactive way in fact is not robust at all and we get a small decrease in performance.
Revised: Training it in an interactive way in fact is not robust at all and we get a small decrease in performance.

Original: We also not that for these setups changing the guide mask to background mask does not bring out much effect in the harmonized output which makes it clear that these frameworks cannot be converted to serve interactive harmonization efficiently.
Revised: We also not that for these setups changing the guide mask to background mask does not bring out much effect in the harmonized output which makes it clear that these frameworks cannot be converted to serve interactive harmonization efficiently.

Original: Augmentations: To generate the composite image in the synthetic dataset IntHarmony, we use a set of augmentations.
Revised: Augmentations: To generate the composite image in the synthetic dataset IntHarmony, we use a set of augmentations.

Original: For each image, these augmentations are chosen at random.
Revised: For each image, these augmentations are chosen at random.

Original: To account for appearance changes, we use a brightness and contrast augmentation.
Revised: To account for appearance changes, we use a brightness and contrast augmentation.

Original: We also use color jitter to enforce hue augmentation.
Revised: We also use color jitter to enforce hue augmentation.

Original: In addition, we also use a gamma transformation.
Revised: In addition, we also use a gamma transformation.

Original: Apart from these, we also use 3D LUT augmentations as proposed in Jiang et al.
Revised: Apart from these, we also use 3D LUT augmentations as proposed in Jiang et al.

Original: We also generate local masks and enforce augmentations to cover local lighting effects.
Revised: We also generate local masks and enforce augmentations to cover local lighting effects.

Original: Here, we apply augmentations such as soft lighting, dodge, grain merging and grain extraction.
Revised: Here, we apply augmentations such as soft lighting, dodge, grain merging and grain extraction.

Original: Note that all the above set of augmentations are considered as a set and for each image a random augmentation is selected and applied on a composite foreground.
Revised: Note that all the above set of augmentations are considered as a set and for each image a random augmentation is selected and applied on a composite foreground.

Original: Potential Negative Societal Impacts: The proposed method may have negative social impacts if people are not using this model properly.
Revised: Potential Negative Societal Impacts: The proposed method may have negative social impacts if people are not using this model properly.

Original: As this work focuses on making an edited image look realistic, it can be used to create realistic fake images.
Revised: As this work focuses on making an edited image look realistic, it can be used to create realistic fake images.

Original: Realistic fake images should be contained as it can act as a tool for harassment and fraud.
Revised: Realistic fake images should be contained as it can act as a tool for harassment and fraud.

Original: One potential solution is gated release of trained models.
Revised: One potential solution is gated release of trained models.

Original: More Results:
Revised: More Results:

Original: In Figure 9, we show results on composite portraits from IntHarmony dataset.
Revised: In Figure 9, we show results on composite portraits from IntHarmony dataset.

Original: It can be observed that our method produces realistic harmonization results closer to ground truth when compared to previous methods.
Revised: It can be observed that our method produces realistic harmonization results closer to ground truth when compared to previous methods.

Original: The quality of predictions proves the usefulness of our proposed approach to solve the interactive harmonization problem.
Revised: The quality of predictions proves the usefulness of our proposed approach to solve the interactive harmonization problem.

Original: Please refer to the supplementary material to see many more visualizations.
Revised: Please refer to the supplementary material to see many more visualizations.

Original: In-the-Wild Portrait Harmonization Results:
Revised: In-the-Wild Portrait Harmonization Results:

Original: To further validate our method for in-the-wild photos, we randomly create composite images and check the portrait harmonization results.
Revised: To further validate our method for in-the-wild photos, we randomly create composite images and check the portrait harmonization results.

Original: We visualize these composites and the corresponding results in Figure 5.
Revised: We visualize these composites and the corresponding results in Figure 5.

Original: It can be observed that our method produces realistic harmonization results with the appearance being close to how it would have been if the person was photographed in-situ.
Revised: It can be observed that our method produces realistic harmonization results with the appearance being close to how it would have been if the person was photographed in-situ.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]