Paper:
Learnable Visual Markers
We propose a new approach to designing visual markers (analogous to QR-codes, markers for augmented reality, and robotic fiducial tags) based on the advances in deep generative networks. In our approach, the markers are obtained as color images synthesized by a deep network from input bit strings, whereas another deep network is trained to recover the bit strings back from the photos of these markers. The two networks are trained simultaneously in a joint backpropagation process that takes characteristic photometric and geometric distortions associated with marker fabrication and marker scanning into account. Additionally, a stylization loss based on statistics of activations in a pretrained classification network can be inserted into the learning in order to shift the marker appearance towards some texture prototype. In the experiments, we demonstrate that the markers obtained using our approach are capable of retaining bit strings that are long enough to be practical. The ability to automatically adapt markers according to the usage scenario and the desired capacity as well as the ability to combine information encoding with artistic stylization are the unique properties of our approach. As a byproduct, our approach provides an insight on the structure of patterns that are most suitable for recognition by ConvNets and on their ability to distinguish composite patterns.

Review:
In this paper, the authors proposed a way to generate markers (images which encode information, like a QR code) by training an end-to-end network to maximize the recognizability in some marker-aid scenarios for robots. The network consists of a generator which encodes information and generates a marker, a renderer which simulates how the markers are placed in a real environment, and a recognizer responsible for information extraction. Qualitative experiments were detailed, and results under various scenarios were compared.
Briefly, the overall idea of serializing the whole encoding--environment simulation--recognition into an end-to-end network is interesting and worth exploration. The presentation is clear and comprehensive. To me, however, there is still large room to fulfill the idea and its application possibility. Specifically, more experiments should have been done to support and demonstrate the idea.  About the idea. The end-to-end learning is intuitive and proved effective, at least qualitatively, but the model structure binds the synthesizer and the recognizer together, assuming that we already know the recognizer in the first place. In real situations, the two parts are mostly fully decoupled, and this structure limits its applications. For example, different robots may vary in different computational capabilities, but they all need to keep the same recognizer network inside its memory. In other words, unlike a barcode or a QR code which includes only encoding and decoding rules of sequence/image in its protocol, this method also takes the recognizer into its "protocol". Another concern is on the rendering network. The purpose of the rendering network is to simulate the real environment as much as possible, thus directly determining the performance of the method. To ensure the differentiability of the end-to-end network, the rendering network may not perform some of the transformations, such as bending, as mentioned, and some other non-linear transformations, though these can be solved in other ways: limit the marker transformation types in the real world. Also, the rendering in the real world is far more complicated than 2D rendering(example: shadows).         One concern on texture loss. The purpose of texture loss is to incorporate aesthetics into the markers. However, the loss enables but the similarity to the original image, no patterns learned, as is shown in Figure 4. What we expected are some patterns which resemble a QR code: differs in content but follows the same style. Regarding this, the texture loss may harm the recognition accuracy, which limits its use in the real world.          About the experiments. Since there seems no universal criterion for the performance of a protocol, it is acceptable to have only qualitative results and no quantitative comparisons. However, results of existing protocols should be reported and compared, such as of a QR code and of a bar code. For example, the encoder can be replaced with a QR code encoder, and the recognizer a QR decoder (an off-the-shelf one or a trained network exactly same as the used structure in the experiments). Question: in Figure 5, what are the black boxes?          For the renderer, it would be more interesting to investigate that how to encode information can resist certain transformations, such as affine transformations only, lighting variations only, motion blurs only, optical blurs only, or even no transformations at all, where we assume that a robot can adjust its pose to locate the marker and take an image almost same as the originally generated one. This can provide perspective on how to design new protocols to reduce the effects of certain transformations.