Paper:
Neural Universal Discrete Denoiser
We present a new framework of applying deep neural networks (DNN) to devise a universal discrete denoiser. Unlike other approaches that utilize supervised learning for denoising, we do not require any additional training data. In such setting, while the ground-truth label, i.e., the clean data, is not available, we devise "pseudolabels" and a novel objective function such that DNN can be trained in a same way as supervised learning to become a discrete denoiser. We experimentally show that our resulting algorithm, dubbed as Neural DUDE, significantly outperforms the previous state-of-the-art in several applications with a systematic rule of choosing the hyperparameter, which is an attractive feature in practice.

Review:
This paper proposed a neural universal discrete denoiser. It is an extension of the universal discrete denoiser (DUDE) based on a deep neural network. The DNN can be trained using the so-called "pseudo-labels" under a novel objective function. Experimental results show some significant improvements over the conventional DUDE.
The core of the work is to use a DNN as an estimator of the empirical distribution of symbols given the context. The authors also designed a novel objective function that does not require labeled data. Not surprisingly, the DNN used in the presented Neural DUDE turns out to be a pretty good estimator with shared parameters. Significant improvements were observed.  This is an elegant piece of work, an interesting extension of the convention DUDE. I don't have problems with the Neural DUDE per se. But I have following concerns or comments.  1. Both DUDE and Neural DUDE will see a serious data sparsity problem when the alphabet becomes large. So far, a lot of application examples of DUDE are limited to very small alphabets, e.g. black-white images (|X|=2) and DNA sequencing (|X|=4). Neural DUDE uses a DNN with parameter sharing can mitigate it to some degree but I guess it will quickly run into problems when the context is long with a big alphabet. For example, the authors mentioned literature [4] and [22] in the introduction as related work using supervised training. But I wonder how the neural DUDE will perform when dealing with grayscale images as used in [4] and [22] rather than the black-white binary images used in this work? The alphabets are much bigger in [4] and [22]. So, although it is elegant in theory, it is hard to see it makes an impact to the real image denoising applications.  2. Another concern about the experiments is that the authors assume the channel is known. In other words, the channel transition matrix is known before hand. This is not a very realistic assumption in real world scenarios. Most cases, you don't know the channel and you have to estimate it. This is another crucial issue for the neural DUDE to make a real impact. How will the estimated channel transition matrix affect the performance? When the alphabet is big, the estimation of the matrix and its inversion itself may pose a problem.