Unite Beijing 2018: Part 2

This article contains AI-generated translation.

 Author
 Previous

今天我们继续分享米哈游技术总监贺甲在Unite Beijing 2018大会上的演讲《在Unity上实现高品质卡通渲染的效果》下篇，上篇请点击此处阅读。
Today, we continue to share the second part of the speech "Achieving High-Quality Cartoon Rendering in Unity" by He Jia, the Technical Director at miHoYo, presented at Unite Beijing 2018. If you haven't read the first part, you can find it here.

下面为演讲内容：
The following is the content of the speech:

接下来我们就来介绍一下头发的渲染。头发是卡通渲染角色较为重要且独特的部分。我们想要实现根据光源动态变化的高光和阴影渐变，并且这个实现还应具备直观的所见即所得的色彩调节能力。
Next, let's talk about hair rendering. Hair is an important and unique part of cartoon rendering for characters. We aim to achieve dynamic specular highlights and shadow gradients based on light sources, while providing intuitive color adjustments.

和皮肤的材质一样，对于头发的漫反射渲染我们同样使用了Multi-ramp的方法，而镜面反射高光我们则使用了二层高光做叠加，通过组合高低频的高光成分在一起，我们可以得到满意的结果。此外，我们还使用Glossy Map和AO纹理来进一步增强头发的质感。
Similar to the skin material, for the diffuse rendering of hair, we also use the Multi-ramp method. However, for specular reflections and highlights on hair, we use a two-layer approach. By combining high and low-frequency specular components, we can achieve satisfactory results. Additionally, we use Glossy Maps and AO textures to further enhance the texture of the hair.

头发的高光渲染使用了各向异性高光，相比普通的高光使用Normal计算光照，各项异性使用Tangent作为计算基础，因此可以使高光显示出垂直于发丝方向的形状。
The hair's specular highlights are rendered using anisotropic specular reflections. Unlike regular specular highlights, which use the Normal for lighting calculations, anisotropic reflections use the Tangent as the calculation basis, allowing the highlights to appear shaped vertically along the direction of the hair strands.

我们在制作头发模型的时候，如果模型拓扑较为复杂，UV展开较难做到全部垂直，我们也可以使用Flowmap来梳理高光的形状。
When creating hair models, if the topology is complex and it's difficult to achieve complete vertical UV unwrapping, we can also use Flowmaps to shape the specular highlights.

我们还使用Jittermap抖动贴图用来增强卡通渲染头发的质感。通过扰动切线方向来达到模拟发丝细节的高光效果。另外，通过调整Jittermap的UV scale还可以做到调整发丝的高光粗细。
We also use Jittermap, a jitter texture, to enhance the texture of cartoon-rendered hair. By perturbing the tangent direction, we can simulate the specular effect of individual hair strands. Moreover, adjusting the UV scale of the Jittermap allows us to control the thickness of the specular highlights on the hair.

这四张图分解展示了各个高光成分对渲染结果的影响。而右下角的则是最终的图像。我们可以看到，结合了低频和高频成分的高光显示，头发看起来更具表现力。
The four images shown here demonstrate the impact of each specular highlight component on the rendering result. The bottom-right image is the final result. As we can see, the combination of low-frequency and high-frequency components in the specular highlights makes the hair look more expressive.

接下来让我们看看另外一种实现Cel-shading头发高光的实现。
Next, let's take a look at another approach to achieve Cel-shading for hair highlights.

我们的目标同样是使其可以动态化，高光应根据光源和相机位置沿发丝方向移动，形状也应该在移动中有着动态的形态变化。
Our goal is to make it dynamic, where the highlights move along the hair strands based on the light source and camera position, and the shape undergoes dynamic changes while moving.

Cel-shading风格的头发高光较为独特的形态，很难用传统的高光计算方法来描述。同样我们需要使用切线方向而不是法线来进行高光计算，并且需要更为特殊的方法去表现高光形状。
The cel-shading style for hair highlights has a unique and intricate form, making it challenging to describe using traditional highlight calculation methods. Similarly, we need to use the tangent direction instead of the normal to perform the highlight calculation and require a more specialized approach to represent the highlight shape.

首先，我们要把每缕头发模型在垂直方向进行UW展开，以便高光可以沿着每根发束移动。然后从将每一缕从左侧向右侧填充0到1，用来标识动态生成的高光形状的起始和结束位置，我们使用几个曲线定义的模板来描述头发高光的基本形状，然后使用抖动噪声纹理来调制头发高光的粗细变化，
Firstly, we need to unwrap each strand of hair model in the vertical direction for UW mapping, so that the highlights can move along each hair strand. Then, we fill each strand from left to right with values from 0 to 1, indicating the starting and ending positions of the dynamically generated highlight shape. We use several templates defined by curves to describe the basic shape of hair highlights, and then modulate the thickness variation of hair highlights using jitter noise textures.

材质方面有很多参数用来控制生成图案的形状。位置、偏移、宽度、抖动比例等，通过调整这些参数，我们可以根据需要获得各种不同的形状。
Regarding the material, there are various parameters used to control the shape of the generated pattern. These parameters include position, offset, width, jitter scale, etc. By adjusting these parameters, we can obtain various shapes as needed.

我们来看另一种各向异性材质的例子：丝绸。这次我们使用了副法线方向来计算了高光反射，并使用三个高光层合成在一起获得最终的渲染效果，我们为每一层分别设置不同的颜色，以便最终材质看起来色彩层次较为丰富。
Let's look at another example of an anisotropic material: silk. This time, we use the secondary normal direction to calculate the specular reflection for silk, and we combine three specular layers to achieve the final rendering effect. We set different colors for each layer to enrich the color richness of the final material.

这里显示了在不同的视角下，丝绸各向异性高光的反射变化。
Here are the reflections of anisotropic silk highlights under different viewing angles.

我们的角色材质中还包括其它特殊的材质，例如：水晶和纱巾等半透明材质，直接使用Alpha混合不能表现出应有的质感，这就需要我们实现折射和模糊效果，这二个效果都依赖于Unity的Command buffer。
In our character materials, we also have other special materials, such as crystals and veils, which are translucent materials. Directly using alpha blending cannot achieve the desired texture, so we need to implement refraction and blur effects, both of which rely on Unity's Command Buffer.

实现折射效果时，Command buffer在渲染折射前获取已经渲染好的Backbuffer作为背景，用于折射采样, Rgb通道设置不同折射系数，分别采样三次来模拟色散效果。
To achieve the refraction effect, the Command Buffer first captures the already rendered backbuffer as the background before rendering refraction, which is used for refraction sampling. We set different refraction coefficients in the RGB channels and sample three times to simulate dispersion effects.

对于模糊效果，则是用Command buffer将Backbuffer降采样并做模糊，生成4张尺寸依次减半模糊度递增的RenderTexture，然后根据相机距离和FOV以及材质固有的模糊参数，确定模糊程度，选择对应的RenderTexture来完成模糊效果。
For the blur effect, the Command Buffer downsamples the backbuffer and applies blur, generating four render textures with decreasing sizes and increasing levels of blur. Then, based on the camera distance, field of view (FOV), and material's intrinsic blur parameters, the appropriate render texture is selected to achieve the desired level of blur.

我们还对这二者的实现做了一定的优化，不对直接对Backbuffer使用全屏模糊，把物体本身作为Proxy mesh，只处理需要画的部分。
We have also optimized the implementation of these effects by not applying fullscreen blur directly to the backbuffer. Instead, we use the object itself as a proxy mesh and only process the parts that need to be drawn.

接下来让我们来谈谈高品质勾线的方法。
Next, let's discuss the method for achieving high-quality outlines.

对于角色和动态物体我们使用Backface勾线方法，并使用顶点色对勾线的宽度进行控制，勾线本身需要连续的顶点法线才能在锐角边不会出现断层，因此我们将平滑过的法线存储在另一套顶点色里。
For characters and dynamic objects, we use the Backface outline method and control the outline width using vertex colors. The outline itself requires continuous vertex normals to avoid gaps at sharp edges, so we store smoothed normals in another set of vertex colors.

此外，我们也使用顶点色来控制勾线宽度，例如：发尖处勾线会逐渐变细，我们通过在顶点颜色填充渐变为0的值以使线条宽度逐渐过渡到零。
In addition, we also use vertex colors to control the outline width. For example, at the tips of the hair, the outline gradually becomes thinner. We achieve this by filling vertex colors with a gradient that approaches zero, making the line width transition gradually to zero.

另外，根据相机与物体之间的距离，还应有基于距离修正的勾线宽度。每种材质上也应该有对应的不同勾线颜色，所有这些功能都是高品质的勾线所必需的。
Furthermore, there should be distance-based correction for the outline width based on the distance between the camera and the object. Each material should also have its corresponding outline color. All of these features are essential for achieving high-quality outlines.

Backface勾线方法虽然可以做到较为细致的勾线还原。但也有着自身的固有缺陷，那就是不能在非边缘的尖锐折线处产生勾线。 而这些折线在硬表面模型上是很常见。
Although the Backface outline method can achieve detailed outline rendering, it has its inherent limitation, which is the inability to generate outlines at sharp creases that are not edge contours. These creases are common in hard surface models.

为了解决这个问题，我们添加一个预处理过程来提取这些边缘，并将它们保存到额外的Mesh资源中，并使用Geometry shader绘制它们。对于这些折线我们使用了和Backface法类似的调整参数，从而使它们看起来完全相同。增加了折线的绘制之后，我们可以看到右侧的图片捕获到了更多的勾线细节。
To address this issue, we add a preprocessing step to extract these creases and save them into an additional Mesh resource, which is then rendered using a Geometry shader. We adjust parameters for these creases similarly to the Backface method to make them look consistent. After adding the rendering of these creases, we can see in the image on the right that more fine details of the outlines are captured.

勾线另一种常见方法就是在图像空间中生成轮廓线。通过检测场景图像中Normal和Depth的不连续性，我们可以获得细节较为丰富的勾线。无论场景的复杂性如何，这种方法的性能都是恒定的，我们还添加了对勾线颜色的色相、明度、饱和度的调整，使勾线更为自然。
Another common method for outlines is to generate contour lines in image space. By detecting discontinuities in Normal and Depth within the scene image, we can obtain more detailed outlines. The performance of this method remains constant regardless of scene complexity. We also added adjustments for hue, brightness, and saturation of the outline color to make the outlines look more natural.

这种方法的缺点则是较难控制勾线的宽度，如果我们想实现距离相关的线宽，我们只能在几个像素的范围内调整它，因此基于图像的方法主要适用于场景轮廓渲染，对于靠近摄像头很近的物体，我们最好使用Backface的方法。
The downside of this method is that it's challenging to control the width of the outlines. If we want to achieve distance-related line widths, we can only adjust them within a few pixels' range. Therefore, image-based methods are mainly suitable for rendering scene contours. For objects that are close to the camera, it's better to use the Backface method.

最后一种做法是基于笔刷的购线方法，这在离线渲染中使用的比较多，通常分为以下几步。
The last approach is based on a brush stroke method, which is more commonly used in offline rendering. It typically involves the following steps:

* 轮廓线提取：从Mesh上提取轮廓边，主要分为Sharp Edge和Smooth Edge二种。
Contour extraction: Extract contour edges from the Mesh, mainly classified into Sharp Edges and Smooth Edges.
* 连接轮廓线：根据模型的拓补关系，将相邻的轮廓边连接成尽可能长的轮廓线。
Connecting contour edges: Based on the topological relationship of the model, connect adjacent contour edges to form the longest contour lines possible.
* 轮廓线分段：在步骤2的基础上，根据轮廓线上曲率和可见性的变化，将轮廓线在曲率或可见性的突变处分开。
Contour segmentation: Building on step 2, separate the contour lines at points of curvature or visibility changes.
* 笔触映射：将想要添加的笔触制作成纹理，根据对应的纹理坐标映射到步骤3的轮廓线上。
Brush stroke mapping: Create the desired brush stroke as a texture and map it onto the contour lines from step 3 based on corresponding texture coordinates.

这种方法可以达到更为风格化，笔触更明显的勾线方式，Pencil+ blender里Freestyle render基本都是采用类似的方法，性能开销较大，可以用于CG品质渲染，但不适合直接在游戏中使用。
This method can achieve a more stylized and pronounced outline, similar to the Freestyle render in Pencil+ Blender. However, it comes with a higher performance cost and is more suitable for CG-quality rendering rather than direct use in games.

接下来我们来看看其它特殊效果的实现，这些渲染效果在场景刻画中同样起到重要的作用。
Next, let's take a look at the implementation of other special effects, which also play an important role in scene depiction.

这是一段用来展示体积光的场景。我们可以看到，具有雾效的体积光配合bloom一起使用，场景表现出了较强层次和氛围感。
Here is a scene demonstrating volumetric lighting. As we can see, with the combination of volumetric lighting and bloom, the scene exhibits a strong sense of depth and atmosphere.

下面就来看看体积光的实现细节。
Now, let's delve into the details of volumetric lighting implementation.

我们使用Unity内置的曲线来体积光的形状，这在运行时也方便调整形状，强度参数变化同样由曲线定义。
We use Unity's built-in curves to define the shape of the volumetric lighting. This allows us to adjust the shape conveniently during runtime, and the intensity parameter is also defined by curves.

为了进一步模拟烟雾效果，我们还使用3D noise纹理来模拟动态烟雾流动的效果。Noise烟雾本身也有一些参数可调.。例如：粒度大小、尺寸比例、噪声强度、流动速度等。
To further simulate the effect of dynamic flowing smoke, we utilize a 3D noise texture. This noise texture creates the illusion of dynamic movement of the smoke. The noise-based smoke also has adjustable parameters, such as grain size, scale ratio, noise intensity, flow speed, etc.

此外，配合Cookie map还可以自定义体积光投影形状，使用Cookie map后同时也引入了高频的变化成分，这就需要对应增加采样数来减少走样，使用抖动算法可以减少采样不足导致的走样，我们实现了二种抖动方式：Bayer pattern和Blue noise， 通过实验发现Blue noise配合Temporal AA可以在较低的采样数下实现处较好的体积光效果。
Furthermore, by using a Cookie map, you can customize the shape of the volumetric lighting projection. However, using a Cookie map introduces high-frequency variations, which requires an increase in the number of samples to reduce aliasing. To mitigate undersampling artifacts, we implemented two jittering methods: Bayer pattern and Blue noise. Through experimentation, we found that Blue noise, in combination with Temporal Anti-Aliasing (TAA), can achieve better volumetric lighting effects with lower sample counts.

接下来我们来看一下使用Real-time GI的例子。
Next, let's take a look at an example using Real-time Global Illumination (GI).

在这个简单的演示场景中，我们使用Enlighten来烘焙 Real-time GI的Lightmap，然后使用动态自发光材质和体积光作为光源，我们使用AVpro插件解码视频文件，将其设置在自发光纹理上，并设置强度值为1以上。我们就可以获得一个动态且明亮的面积光源，同时要记得更新GICache，以便在运行时刻可以动态更新光照环境，当与动态体积光一起使用时，整体的照明效果看起来令人印象深刻。
In this simple demo scene, we use Enlighten to bake Real-time GI lightmaps. Then, we use dynamic emissive materials and volumetric lighting as light sources. By using the AVpro plugin to decode video files and set them as emissive textures with an intensity value above 1, we can create a dynamic and bright area light source. Don't forget to update the GICache to dynamically update the lighting environment at runtime. When used in conjunction with dynamic volumetric lighting, the overall lighting effect becomes quite impressive.

对于角色上的动态AO实现，我们使用修改过的HBAO，用于指定AO区域中颜色的饱和度和色调调整，以使加入AO后的图像颜色看起来不会变脏，通过对比图我们可以看出，在应用了AO之后，右图比左图层次感更强。
For dynamic Ambient Occlusion (AO) implementation on characters, we use a modified version of HBAO. This version allows us to specify adjustments for color saturation and hue within the AO area, ensuring that the image's colors do not appear dirty after applying AO. By comparing the images, we can see that the one with AO applied (right image) has a stronger sense of depth compared to the one without AO (left image).

我们还重新实现了适用于卡通渲染的基于图像的眩光效果，用于模拟镜头产生的鬼影和星形散射效果。这里使用与Bloom类似的方式提取的高光区域作为输入，然后进行多次不同方向上的卷积并应用色彩调制来获得最终结果。
We also re-implemented image-based glare effects suitable for cartoon-style rendering. This effect is used to simulate lens artifacts, ghosting, and star-shaped scattering. Similar to Bloom, we extract highlight areas as input and then apply multiple convolutions in different directions, along with color modulation, to achieve the final result.

下面我们来看几张CG视频中的截图和特写。
Let's take a look at some screenshots and close-ups from CG videos.

这是另一组场景截图。我们可以看到在应用了之前提到的这些渲染技术之后，整个场景可以更接近离线渲染的品质。
Here's another set of scene screenshots. After applying the rendering techniques mentioned earlier, the overall scene quality comes closer to offline rendering.

下面的图描述了上述场景中所应用到的主要渲染特性。从图中我们可以看到这些效果包括：风格化的PBR材质、卡通风格的AO、屏幕空间勾线、屏幕空间反射以及曲面细分等。
The image below describes the main rendering features applied in the above scene. From the image, we can see that these effects include stylized PBR materials, cartoon-style Ambient Occlusion (AO), screen-space outlines, screen-space reflections, and tessellation.

综合应用这些效果对于高品质的动画风格场景渲染起着重要的作用，我们的目标是在PBR的Shading基础上加入风格化的调整使其更具有表现力。
The combined application of these effects plays a crucial role in achieving high-quality animation-style scene rendering. Our goal is to add stylized adjustments on top of PBR shading to make it more expressive.

场景中的大部分材质都是基于物理的渲染。我们对PBR纹理集进行了一些风格化上的适应调整，例如：对于色彩的卡通化调整，以及对于物体材质细节的强调或省略。再结合使用图像空间的勾线来强调物体边缘，整体场景的表现就显得更接近动画风格。
Most of the materials in the scene are based on physically-based rendering (PBR). We made some stylized adjustments to the PBR texture set, such as cartoonish color adjustments and emphasizing or omitting material details. Combining the use of screen-space outlines to emphasize object edges, the overall scene appears closer to an animated style.

下图展示了这些材质在不同光照角度下的光影变化。
The image below shows the lighting and shadow variations of these materials from different angles.

下图展示了光影的变化。
The following image shows the variations in lighting and shadow.

除了场景渲染之外，我们再来看看其它一些动画渲染所涉及的内容，动画表情。
Besides scene rendering, let's take a look at other aspects involved in animation rendering, such as animated expressions.

我们使用Blendshape来制作面部表情。 眼睛，嘴巴和眉毛的表情独立为不同的部件单独制作，然后通过我们的自定义面部表情插件，来实现表情动画的及语音嘴型的自动映射。此外，我们还可以通过预定义不同的表情集合来在交互应用中驱动面部表情。
We use Blendshapes to create facial expressions. Expressions for the eyes, mouth, and eyebrows are separately created as different components. Then, through our custom facial expression plugin, we achieve automatic mapping of facial animations and speech lip sync. Additionally, we can drive facial expressions in interactive applications by using predefined sets of different expressions.

在Unity中使用Humanoid作为动画导入方式的时候，如果关节处旋转角度较大，按照动画品质的要求关节处的形状就不能令人满意。
In Unity, when using Humanoid as the animation import method, if the rotation angles at the joints are large, the shape at the joints may not meet the quality requirements for animations.

为此我们通过在建模软件中，建立了每关节修正的Blendshape导入到Unity当中来防止关节变形。我们使用一个自动控制脚本根据关节旋转角度来差值混合形状。 为了确保更好的结果，我们为每个关节分别制作了二个Blendshape，一个用于90度，另一个用于140度以补正关节变形。
To address this, we establish correction Blendshapes for each joint in the modeling software and import them into Unity to prevent joint deformations. We use an automatic control script that interpolates between the shapes based on the joint's rotation angle. To ensure better results, we create two Blendshapes for each joint, one for 90 degrees and the other for 140 degrees, to correct the joint deformation.

另外一种方法还可以使用额外的骨骼进行关节修正。这种方法更容易制作，但是对于结构细节的表现不如使用Blendshape。
Another method for joint correction involves using additional bones. This method is easier to create but may not provide the same level of detail representation as using Blendshapes.

为了可以表现更复杂的场景动态，比如流体和破碎的场景，我们可以使用Alembic格式，或者用EXR纹理作为载体从Houdini或其它DCC工具导入顶点动画资源。Houdini对EXR纹理格式导出顶点动画提供了很好的转换支持，对于Real-time的应用而言，顶点动画纹理在因为是在GPU上运行，运行效率及加载速度要快于Alembic格式。
To handle more complex dynamic scenes, such as fluid and fractured scenes, we can use the Alembic format or use EXR textures as carriers to import vertex animation resources from Houdini or other DCC tools. Houdini offers excellent support for exporting vertex animations in the EXR texture format. For real-time applications, vertex animation textures running on the GPU are more efficient and load faster than the Alembic format.

最后，我们来谈谈实时卡通渲染在今后可以继续改进和完善的地方。
Finally, let's discuss areas where real-time cartoon rendering can continue to improve and be refined in the future.

首先是实现所有类型材质完全可定制的风格化渲染，目前我们初步在人物皮肤和服装渲染中的应用了笔刷以获得笔触效果。
First, we aim to achieve fully customizable stylized rendering for all types of materials. Currently, we have applied brushes to achieve the brushstroke effect in character skin and clothing rendering.

下一步我们希望将其扩展到整个场景的渲染，比如新海诚式的场景风格，以呈现有着独特且统一的风格化动画风格渲染。另外一点是要进一步提高模型的渲染精度，我们希望可以实时呈现CG级的模型精度。可以尝试使用Geometry shader或预烘培displacement map进行动态自适应的曲面细分，相比直接导入原始高模，它可以极大减少资源导入的开销和提升运行效率。
Next, we plan to expand this approach to the entire scene rendering, such as achieving a Makoto Shinkai-style scene style, to present a unique and unified stylized animation rendering. Another point is to further improve the rendering accuracy of models. We hope to achieve real-time rendering with CG-level model precision. We can try using Geometry shaders or pre-baked displacement maps for dynamic adaptive surface subdivision. Compared to directly importing high-poly models, this approach can significantly reduce resource import overhead and improve runtime efficiency.

最后是优化整套流程解决方案，使之更易于实时调整和编辑，进一步提升运行效率以适合在游戏中使用。
Lastly, we will optimize the entire pipeline solution to make it more user-friendly for real-time adjustments and editing, further enhancing runtime efficiency to make it suitable for use in games.

好的，以上就是我们今天有关于卡通渲染要分享的主要内容，谢谢大家！
Alright, that concludes our main content on cartoon rendering that we wanted to share today. Thank you, everyone!