Conditional Diffusion Process for Inverse Halftoning
Abstract
Inverse halftoning is a technique used to recover realistic images from ancient prints ( e.g. , photographs, newspapers, books).
The rise of deep learning has led to the gradual incorporation of neural network designs into inverse halftoning methods.
Most of existing inverse halftoning approaches adopt the U-net architecture, which uses an encoder to encode halftone prints, followed by a decoder for image reconstruction.
However, the mainstream supervised learning paradigm with element-wise regression commonly adopted in U-net based methods has poor generalization ability in practical applications.
Specifically, when there is a large gap between the dithering patterns of the training and test halftones, the reconstructed continuous-tone images have obvious artifacts.
This is an important issue in practical applications, since the algorithms for generating halftones are ever-evolving.
Even for the same algorithm, different parameter choices will result in different halftone dithering patterns.
In this paper, we propose the first generative halftoning method in the literature, which regards the black pixels in halftones as physically moving particles, and makes the randomly distributed particles move under some certain guidance through reverse diffusion process, so as to obtain desired halftone patterns.
In particular, we propose a Conditional Diffusion model for image Halftoning (CDH), which consists of a halftone dithering process and an inverse halftoning process.
By changing the initial state of the diffusion model, our method can generate visually plausible halftones with different dithering patterns under the condition of image gray level and Laplacian prior.
To avoid introducing redundant patterns and undesired artifacts, we propose a meta-halftone guided network to incorporate blue noise guidance in the diffusion process.
In this way, halftone images subject to more diverse distributions are fed into the inverse halftoning model, which helps the model to learn a more robust mapping from halftone distributions to continuous-tone distributions, thereby improving the generalization ability to unseen samples.
Quantitative and qualitative experimental results demonstrate that the proposed method achieves state-of-the-art results
Introduction
Halftoning refers to the task of simulating the brightness change of a continuous-tone image by changing the size or frequency of halftone dots (such as ink dots).
In the last century, halftoning technology has been widely used in the printing industry to store precious image data through old newspapers, books, photographs, etc .
At the same time, inverse halftoning technology emerged to recover stored continuous-tone image from vintage materials.
The goal of the inverse halftoning technique is to minimize the loss of information in the restoration process, so that the restored print has the highest possible visual quality.
Traditional inverse halftoning methods are mainly based on image filtering ( e.g. , wavelet domain filtering (Xiong et al., 1999), edge-preserving filtering (Kite et al., 2000), SUSAN filtering (Siddiqui and Bouman, 2007)), bilateral filtering (Sun et al., 2014) and statistical learning methods ( Chen and maximum Stevenson,
With the revival of deep learning technology, inverse halftoning methods based on deep neural networks have made great progress and attracted more and more attention (Hou and Qiu, 2017; Xiao et al., 2017; Kim and Park, 2018).
The most representative method is the U-net (Ronneberger et al., 2015) based architecture, which has an encoder to learn the hidden encoding of the halftone printing, followed by a decoder to reconstruct the image (Xia and Wong, 2018; Gao et al., 2019).
However, the paradigm of supervised learning with element-wise regression commonly adopted in the U-net based methods suffers from poor generalization in practical applications.
Specifically, when there is a large gap between the dithering patterns of training and testing halftones, the restored continuous-tone images often have obvious artifacts.
This is an important issue in applications, since the algorithms used for generating halftones evolve with time.
Even for the same algorithm, different parameter choices will result in different halftone dithering patterns.
Taking frequency modulation (FM) halftoning as an example, we select 9 classical error diffusion processes with different dithering patterns, namely Floyd-Steinberg Dithering, Jarvis-Judice-Ninke Dithering, Stucki
Dithering, Atkinson Dithering, Burkes Dithering, Sierra Dithering, and several of their variants (Lau and Arce, 2018).
We train the U-net model on the halftones generated by 5 of the algorithms, and test on the halftones obtained with the rest 4 algorithms.
The experimental results are shown in Figure 1.
Artifacts can be clearly observed in the restored continuous-tone images.
To address this problem, in this paper, we propose a Conditional Diffusion model for image Halftoning (CDH), as shown in Figure 2, which consists of a halftone dithering process and an inverse halftoning process.
We regard the black pixels in halftones as physically moving particles, and make the randomly distributed particles move under some certain guidance through the reverse diffusion process, so as to obtain the desired halftone distribution.
Specifically, for the halftone dithering process, we train a conditional diffusion model to generate halftones with different dithering patterns under the condition of image gray level and Laplacian prior.
By changing the initial state of the diffusion model, it can simulate different dithering processes to generate diverse halftone images.
To avoid introducing redundant patterns and undesired artifacts during halftone generation, we propose a meta-halftone guided network to incorporate the blue noise guidance into the diffusion process.
For inverse halftoning, we train an inverse halftoning diffusion model to learn the mapping function from the halftone distribution to the continuous-tone distribution.
In this way, halftones subject to more diverse distributions are input to the inverse halftoning model, which helps the model to learn a more robust mapping and improve the generalization ability to unseen samples.
Our contributions are summarized as follows:
• This is the first work to propose a generative halftoning method, which regards the black pixels in halftones as physically moving particles, and makes the randomly distributed particles move under some certain guidance through the reverse diffusion process, so as to obtain the desired halftone dithering patterns. • To avoid introducing redundant patterns and undesired artifacts during halftone generation, we propose a meta-halftone guided network to incorporate the blue noise guidance into the halftone diffusion process.
• To obtain better generalization ability, we use the x 0 state of halftone dithering diffusion as the condition for inverse halftoning diffusion, so that the inverse halftoning diffusion model benefits from a wider range of dithering patterns and learns a more robust mapping.
We conduct experiments on the dataset consisting of 9 halftoning algorithms, and quantitative and qualitative experiments demonstrate that the proposed method achieves state-of-the-art results.
Related Work
Inverse Halftoning.
Traditional inverse halftoning methods are mainly based on image filtering ( e.g. , edge-preserving filtering (Kite et al., 2000), wavelet domain filtering (Xiong et al., 1999), bilateral filtering (Sun et al., 2014), SUSAN filtering (Siddiqui and Bouman, 2007)) and statistical learning methods ( e.g. , look-up table (Chung and Wu, 2005; Mese and Vaidyanathan, 2001), least mean square filtering (Chen and Hang, 1997), maximum a posteriori (Stevenson, 1997), dictionary learning (Zhang et al., 2018)).
For example, Sun et al.
(Sun et al., 2014) proposed to use an anisotropic Gaussian filter and an edge-preserving filter for inverse halftoning.
With the revival of deep learning technology, some researchers have tried to use deep neural networks to accomplish inverse halftoning, e.g. ,
U-net based models (Hou and Qiu, 2017; Xiao et al., 2017; Gao et al., 2019), residual learning based models (Xia and Wong, 2018) and contextual learning based models (Kim and Park, 2018).
For example, Xia et al.
(Xia and Wong, 2018) proposed a progressively residual based U-net that synthesizes the global tone and subtle details to generate inverse halftones.
However, these approaches suffer from unacceptable artifacts in the restored continuous-tone images when faced with dithering pattern gaps.
Diffusion Models.
Diffusion models are a class of deep generative models developed from nonequilibrium thermodynamics (Sohl-Dickstein et al., 2015).
They define a Markovian process for the diffusion steps, incrementally adding random noises to the original data, and then learn the reverse diffusion process to resample the data from noises (Sohl-Dickstein et al., 2015; Ho et al., 2020;
Luo and Hu, 2021).
Diffusion models are closely related to score-based generative models, which generate samples by Langevin dynamics based on estimated gradients of the data distribution (Song and Ermon, 2019, 2020; Song et al., 2020b, 2021).
Some improved techniques are proposed to help the diffusion model converge to a lower negative log-likelihood or to speed up sampling (Nichol and
Dhariwal, 2021; Dhariwal and Nichol, 2021; Song et al., 2020a).
For example, Song et al.
(Song et al., 2020a) generalizes denoising diffusion probabilistic models via a class of non-Markovian processes, which can correspond to deterministic generative processes and gives rise to implicit models that produce high quality samples much faster.
However, these diffusion models cannot be directly applied to halftone generation, since they do not take into account the blue noise properties of halftone prints, making it difficult to generate visually pleasing dithering patterns.
Method 3.1 Halftone Dithering Diffusion Conditioned on Gray
-scale and Laplacian
Prior
Traditional halftone dithering methods are mainly based on techniques of amplitude modulation (Blatner and Roth, 1993;
Campbell et al., 1966) and frequency modulation (Eschbach, 1997; Floyd, 1976;
Me¸se and Vaidyanathan, 1999).
With the development of deep learning, some modern approaches try
[Figure] [Figure]
to generate halftone images using convolutional neural networks (Kim and {+Park, 
+}
; Xia et al., 2021).
However, these methods are limited to specific dithering patterns and cannot achieve flexible halftone generation.
To address this problem, considering that the halftone prints are only composed of black and white pixels, we regard the black pixels in halftones as moving particles, and make these particles move from random Gaussian distributions to halftone distributions under reverse diffusion processes, so as to achieve a generative halftoning approach.
Given the distribution q s 1 ( x s 1 0 ) of halftone prints, the diffusion process q s 1 is a Markovian noising process (Ho et al., 2020) that gradually adds noise to x s 1 0 to obtain x s 1 1: T , where s 1 denotes the halftone dithering diffusion stage.
Specifically, at each step t , the diffusion step adds the random
Gaussian noise with a β t -controlled variance:
[Equation]
where β t ∈ (0 , 1) , t = 1 , ..., T .
With the reparameterization trick (Kingma and Welling, 2013), we can sample x s 1 t from any time step t in a closed form: x s 1 t = √
[Equation]
) , that is
[Equation]
where α t = 1 − β t , ¯ α t = (cid:81) ti =
1 α i .
In this way, we can directly derive x
s 1 t by q s 1 ( x s 1 t | x s 1 0 ) without repeatedly applying the Markovian process q s 1 and calculating q s 1 ( x s 1 t | x s 1 t − 1 ) .
In the halftone dithering scenario, we would like to obtain the halftone sample x s 1 0 via the reverse diffusion process q s 1 ( x s 1 t − 1 | x s 1 t ) from a random Gaussian distribution, i.e. , the state x s 1 T ∼ N ( 0 , I ) .
Considering that the particles tend to rely on the gradient information of images when forming halftone patterns, we use the image gray level r and the Laplacian prior l as conditions to guide the reverse diffusion process.
Formally, the reverse halftoning diffusion process is defined on the distribution p s 1 θ 1 ( x s 1 0: T | r , l ) , which is a Markov chain with start state p s 1 ( x s 1 T ) =
N ( x s 1 T ; 0 , I ) :
[Equation]
Different from previous methods like using class labels (Dhariwal and Nichol, 2021) or shape latents (Luo and Hu, 2021) as conditions for the diffusion model, we use the pixel-wise image priors
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Figure]
as conditions.
The conditions are performed in a simple way, i.e. , we concatenate the state vector x s 1 t at diffusion step t with r and l along the channel dimension to provide pixel-wise guidance:
[Equation]
6) where ⊕ denotes the concatenate operation.
Due to the properties of halftones, local dithering patterns are more important for generating high-quality halftones than global semantic information.
We also observe in experiments that excessive introduction of global information can be harmful to dithering results.
We would like to learn the model p s 1 θ 1 to approximate the conditional probabilities
q s 1 ( x s 1 t − 1 | x s 1 t ) .
With Bayes’ theorem, it has
[Equation]
and q s 1 ( x s 1 t − 1 | x s 1 t , x s 1 0 ) can be represented as a Gaussian distribution:
[Equation]
[Equation] [Equation]
The simplified objective function (Ho et al., 2020) is used to train the model during the halftone dithering diffusion process:
[Equation]
3.2 Meta-halftone Guided Network
Blue noise properties are critical for generating high-quality halftones.
It avoids noticeable lowfrequency visual artifacts in the resulting halftones by forcing random pixel dithering (Lau and Arce, 2018).
Some traditional methods have been proposed to derive blue noise dithering patterns, such as simulated annealing based methods (Sullivan et al., 1991), void-and-cluster techniques (Ulichney, 1993), power spectrum manipulation algorithms (Yao and Parker, 1994), and dither pattern ordering methods (Lau et al., 1999).
However, most of these approaches are based on statistical methods or manually designed blue-noise dithering matrices, which are difficult to directly apply to learningbased neural frameworks.
Recently, some researchers have tried to inject blue noise properties into the L1 norm for learning objectives (Xia et al., 2021), however, since the calculation of the L1 norm requires known halftone generation results, it cannot be used in generative models.
So far, how to incorporate the blue-noise dithering properties into halftone generative models remains an unexplored problem.
To address this issue, we propose a meta-halftone guided network (as shown in Figure 3), which introduces blue noise guidance into halftone dithering diffusion process and avoids dithering results containing artifacts or redundant patterns.
Formally, for step t of the halftone dithering diffusion process, we consider the state vector x s 1 t and take it as input to the meta-halftone guided network.
We first feed x s 1 t into a feature extraction network E to obtain the extracted hidden feature f ( x s 1 t ) , i.e. , f ( x s 1 t ) =
E ( x s 1 t ) .
We use the pre-trained VGG network (Simonyan and Zisserman, 2015) as E in the experiment, other networks such as InceptionNet (Szegedy et al., 2016) or ResNet (He et al., 2016) are also available.
The purpose of using pre-training is to save the consumption of computing resources and training time.
Next, we define the meta-halftone set M , which consists of a set of meta-halftone vectors m i (1 ≤ [Equation] ≤ k ) :
[Equation]
where k represents the number of meta halftones.
We obtain m i by the halftone dithering diffusion states of a set of low-frequency images.
Specifically, we construct k images I 1 , I 2 , I 3 ..., I k with a large range of low-frequency regions ( e.g. , images with constant gray levels), and obtain the halftones H 1 , H 2 , H 3 , ..., H k accordingly using conventional halftone algorithms ( e.g. , Floyd-Steinberg Algorithm (Lau and Arce, 2018)).
We first train for a certain step using the traditional diffusion model alone, where the model uses U-net architecture common in (Ho et al., 2020; Dhariwal and Nichol, 2021) without adding the proposed meta-halftone guided network.
We have
[Equation]
where the superscript h i denotes the constructed halftone sample from H i ∈
{ H 1 , H 2 , ..., H k } , and r h i and l h i can be derived from I i ∈ { I 1 , I 2 , ..., I k } accordingly.
The superscript s 1 of x h i t − 1 , x h i t are omitted here for brevity.
We can get ϵ s 1 θ 1 ( √ ¯ α t x h
i 0 + √ 1 − ¯ α t ϵ , t ) by parameterization of µ s 1 θ 1 :
[Equation]
We use m i in Equation 11 to represent the model prediction of ϵ s 1 θ 1 and construct meta-halftone set M accordingly:
[Equation]
Next, the model learns affine relationships between the hidden feature f ( x s 1 t ) and the meta-halftone set M through an affine learning layer, and then obtains affine factors g ( x s 1 t ← i ) :
[Equation]
(15) where w g ← i and b g ← i represent learnable weights and biases, respectively.
With the calculated g ( x s 1 t ← i ) , we perform a depth-wise aggregation of the meta-halftone set M to learn the refined representations on depth channels:
[Equation]
Besides depth information, spatial information is also crucial for meta-halftone guidance.
Metahalftones can provide guidance for the generation of new dithering patterns in local areas.
In light of this, we next perform a spatial-wise aggregation of the meta-halftone refined representation (cid:102) m and the dithering diffusion state vector x s 1 t , and denote the result as (cid:102) m ′ :
[Equation]
17) where ⊕ represents the element-wise concatenation in the spatial dimension, ∗ denotes the spatial convolution operation, and d represents the convolution kernel.
The output o of the meta-halftone guided network is determined by the meta-halftone guidance (cid:102) m ′ and dithering diffusion state vector x s 1 t :
[Equation]
and o is subsequently fed into U-net as in previous work (Dhariwal and Nichol, 2021;
Nichol and Dhariwal, 2021) for model predictions.
3.3 Inverse Halftoning Diffusion Conditioned on x s 1
The goal of inverse halftoning process is to learn a mapping from halftone distributions to continuoustone distributions and reduce the loss of information in the process. It is similar to halftone dithering diffusion process in Section 3.1, but with different diffusion conditions.
For the inverse halftoning diffusion process, we deal with the process q s 2 ( x s 2 1: T ′ | x s 2 0 ) that gradually adds noise to the data x s 2 0 from a continuous-tone distribution q s 2 ( x s 2 0 ) , and the process q s 2 ( x s 2 t − 1 | x s 2 t , x s 2 0 ) of gradually denoising from Gaussian noise to obtain desired samples.
We use a model p s 2 θ 2 ( x s 2 t − 1 | x s 2 t , h ) to estimate q s 2 ( x s 2 t − 1 | x s 2 t ) , which is conditioned on the halftone distribution h ( i.e. , sampling x s 1 0 from halftone dithering diffusion model):
[Equation]
Pixel-wise guidance between the state x s 2 t and the halftone condition h is applied in the inverse halftoning diffusion process:
[Equation]
where ⊕ denotes the concatenate operation.
The rest parts (such as the training objective) are similar to the halftone dithering diffusion process, and we omit them here to save space.
Experiments 4.1 Experimental Setup
Datasets.
In order to evaluate the generalization ability of the proposed model CDH to different halftoning methods, we construct a dataset with relatively strong domain shifts.
The domain shifts mainly arise from two aspects: the halftone algorithm and the image semantic.
We construct the training set and validation set based on the UTKFace dataset ( Zhang et al., 2017) and the test set based on the VOC2012 dataset (Everingham et al., 2010).
We collect 9 different halftone dithering algorithms, namely Floyd-Steinberg Dithering, Jarvis-Judice-Ninke Dithering, Stucki Dithering,
Atkinson Dithering, Burkes Dithering, Sierra Dithering and several of their variants (Lau and Arce,
We use halftones generated by 5 of the algorithms in training and validation sets, and halftones generated by remaining 4 algorithms are used as test sets.
There are 7 , 857 images in the training set, 400 images in the validation set, and 400 images in the test set
Baselines.
We choose the baselines that are commonly used in inverse halftoning tasks, which include the U-net based methods, generative adversarial network based methods, and super-resolution based methods: PRL (Xia and Wong, 2018) proposes an inverse halftoning network with progressively residual learning, which synthesizes the global tone and subtle details from halftone images in a progressive manner. Dhariwal et al.
(Dhariwal and Nichol, 2021) show that diffusion models can achieve superior image sampling quality than existing generative models, and they use an improved
U-net model architecture to achieve high-quality image synthesis.
Nichol et al. (Nichol and Dhariwal, 2021) show that with some simple modifications, denoising diffusion probabilistic models can also achieve competitive log-likelihood while maintaining high sample quality. They learn variances of reverse diffusion process using reparameterizations and a hybrid learning objective that combines the variational lower-bound with the simplified objective from (Ho et al., 2020). In additon, they improve the noise schedule from a linear noise schedule to a cosine noise schedule.
Song et al.
(Song et al., 2020a) introduces denoising diffusion implicit models, giving rise to implicit models that produce high quality samples much faster.
ESRGAN (Wang et al., 2018) is used as baselines in previous inverse halftoning work, which introduces the residual-in-residual dense block to generate realistic textures and avoid unpleasant artifacts.
GLEAN (Chan et al., 2021) is applied to the superresolution task, using pre-trained generative adversarial networks to solve ill-posed problems in image restoration.
We first smooth the halftone images with Gaussian kernels of different sizes, and then use GLEAN for image restoration.
RealESRGAN (Wang et al., 2021) is extended to a practical image restoration application, which introduces a high-order degradation modeling process to better simulate complex real-world degradations and employs a U-net discriminator with spectral normalization to increase discriminator capability. [Table]
Evaluation Metrics.
Following previous inverse halftoning work (Xia and Wong, 2018) [Table] we use two metrics to evaluate the performance of different models: Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM).
PSNR expresses the ratio between the power of peak signals and the power of noises, and SSIM evaluates the similarity between images in terms of luminance, contrast, and structure.
Implementation Details.
The image size for halftone dithering and inverse halftoning is 256 × 256 .
The channel number of input halftones is 1 .
For halftone dithering diffusion model, the learning rate is set to 0 . 0001 , and k is set to 20 .
We use 200 diffusion steps in training and testing phases.
The number of model channels is 64 and the linear noise schedule is adopted throughout diffusion.
We adopt AdamW (Loshchilov and Hutter, 2018) optimizer to train the halftone dithering diffusion model.
For inverse halftoning diffusion model, we set the learning rate to 0 . 0001 and use 800 steps in the diffusion process.
We set the number of model attention heads to 4 .
AdamW (Loshchilov and Hutter, 2018) is also used for optimization.
4.2 Experimental Results
We evaluate the performance of the proposed method CDH and baseline methods in terms of
PSNR and SSIM, and the experimental results are shown in Table 1.
We evaluate the effect of different Gaussian kernel sizes on baseline performance, where ESRGAN (lkernel), GLEAN (lkernel),
RealESRGAN (lkernel) denote the use of Gaussian kernels of size 7 , and ESRGAN, GLEAN, RealESRGAN represent the use of Gaussian kernels of size 3 , respectively.
The experimental results show that our method achieves the best performance on both metrics.
Previous methods such as PRL (Xia and Wong, 2018), due to the lack of considering the generalization ability of the model on different halftoning algorithms, show sub-optimal performance on test data with relatively strong domain shifts. Compared with traditional diffusion models (Dhariwal and Nichol, 2021;
Nichol and
Dhariwal, 2021; Song et al., 2020a), the proposed method CDH achieves better results by taking into account the blue noise characteristics in the halftone dithering process, which enables the model to learn from a more realistic and diverse halftone distribution.
Different Gaussian kernel sizes affect the performance of image restoration baselines, such as (Wang et al., 2018; Chan et al., 2021; Wang et al., 2021).
A larger Gaussian kernel erases dot dithering patterns in halftones to a greater extent, resulting in restored images with less halftone artifacts, but also loss of high frequency information.
Our method performs better than the image restoration baselines with different Gaussian kernel sizes.
This shows that simply using traditional image restoration approaches is not suitable for the inverse halftoning task, since they do not take into account diverse pixel dithering patterns unique to halftone images.
To verify the effectiveness of the proposed meta-halftone guided network, we conduct experiments to observe the effect on generated halftone dithering qualities with and without the meta-halftone guided network.
Experimental results are demonstrated in Figure 4.
From the results shown in Figure 4, it can be observed that when the meta-halftone guided network is removed, the generated
[Figure]
[Table]
halftones have more obvious dithering artifacts, resulting in poorer visual quality.
When using the meta-halftone guided network
[Table]
the visual quality of generated halftones is improved and more similar to the groundtruth.
The experimental results demonstrate the effectiveness of the proposed meta-halftone guided network.
4.3 Study the Effect of Different Halftoning Algorithms
In order to explore the performance of CDH on different halftoning algorithms
[Table]
we conduct experiments on halftones generated by 9 halftoning algorithms, namely Floyd-Steinberg Dithering,
Jarvis-Judice-Ninke Dithering, Stucki Dithering, Atkinson Dithering, Burkes Dithering, Sierra Dithering [Table] and several of their variants (Lau and Arce, 2018). The experimental results are shown in Table 2.
From the experimental results we observe that the proposed method achieves similar results on different halftoning algorithms, which also verifies good generalization abilities of our method to different halftoning algorithms.
4.4 Qualitative Results
To further verify the effectiveness of the proposed method, we also conduct qualitative experiments to compare the performance of our methods and baselines, which are illustrated in Figure 5.
From the qualitative experimental results, it can be observed that continuous-tone images generated by baseline methods still have some redundant artifacts, which are dot dithering patterns that are not completely removed during the inverse halftoning process.
In contrast, our model removes the artifacts well without introducing redundant patterns, validating the effectiveness of the proposed method. [Figure]
Conclusion
In this work, we propose a Conditional Diffusion model for image Halftoning (CDH), which consists of a halftone dithering process and an inverse halftoning process. We introduce a generative halftoning method, which regards the black pixels in halftones as physically moving particles, and makes the randomly distributed particles move under some certain guidance through the reverse diffusion process, so as to obtain the desired halftone dithering patterns.
A meta-halftone guided network is introduced to avoid redundant patterns and undesired artifacts in generated halftones.
By adopting more diverse halftones from halftone dithering diffusion process, we further improve the generalization ability of inverse halftoning model.
We conduct experiments on the dataset consisting of 9 halftoning algorithms, and quantitative and qualitative experiments demonstrate the effectiveness of the proposed method.
Acknowledgement: The research is supported by Science and Technology Innovation 2030 - New Generation
Artificial Intelligence (2020AAA0104401), Beijing Natural Science Foundation (Z190001), and Peng Cheng
Laboratory Key Research Project No.PCL2021A07.
Broader Impact
A potential negative side effect of this work is that the sample generation model may be used to generate fake images in the halftoning or inverse halftoning process for some certain purpose.
In addition, most of the images we used in the dataset are collected from the Internet and may contain some bias.
The biases are preserved in the learning of the model and may be reflected in the generated image samples ( e.g. , induce the model to produce undesired results).
An example of this work being used for unethical purposes is to first train the model with biased or discriminatory data and then induce the model to produce unfaithful results when inverse halftoning images.
This may misinterpret the original meaning of some ancient prints and mislead people.
Limitation Analysis
A limitation of our model is that the diffusion process is relatively slow and thus requires more inference time than traditional inverse halftoning models, which is limited when the model is applied to mobile devices.
One way to address this limitation is to design parallel inverse halftoning methods, such as restoring low-frequency and high-frequency components of images in parallel, thereby increasing the sampling speed.