[Unity] Encoding Data into Pixels (Part 1): CG Shaders
Tags: unity3d, gamedev
Published: 2023-06-04 16:32:38

For a project, I want to implement a feature which allows the picking of gameobjects by color. I was first introduced to this technique [here](http://www.opengl-tutorial.org/miscellaneous/clicking-on-objects/picking-with-an-opengl-hack/).

I want pixel perfect object selection, and this approach is relatively straightforward. All it involves is painting each object in a distinct color, this color being a unique ID associated to each object. Then, we can simply see what color, and thus what object the player is pointing at.

## Ideation

We create a new *Universal Renderer Data* since we want to color each gameobject in a different color, but don't want this effect to be visible on the main camera. We want this effect to be exclusive to a secondary camera which we will use only for the color picking action.

## Setup
1. Locate the *Universal Render Pipeline Asset* used in your project. (To identify this one you can open *Project Settings* and then navigate to the *Graphics* section.)
2. Add a new field to its *Renderer List* under the *Rendering* section.
3. Create a new *Universal Renderer Data* asset.
(Create -> Rendering -> URP Universal Renderer)
4. See its details in the *Inspector*, and then click on *Add Render Feature*, selecting the option *Render Objects (Experimental)*. This one in particular allows us to apply the same material to every gameobject in our scene.
5. Inside the section for the newly created *Render Feature*, under the *Filters* section, select the *Everything* option for the *Layer Mask* field. In this case, I will want to paint every single *GameObject*.
6. Inside the section for the newly created *Render Feature*, click on *Overrides*. A *Material* field should be visible.
7. Create a new *Material*.
8. Create a new *Shader*, we will use this one later to color each gameobject in a unique color.
9. Assign the *Shader* to the newly created *Material*.
10. Assign the newly created *Material* to the *Material* field in the *Render Feature* section.
11. Assign the *Universal Renderer Data* to the previously created *Renderer List* field, created in step 2.
12. Inspect your secondary camera.
13. Select the newly created *Renderer* from the *Renderer* field in the *Rendering* section.

## Hello World Shader

If you have followed along the steps above, recreating the *Renderer*, I have added this small section that tests if you did the setup correctly.

To test the correctness of the setup, you can use the following *Shader* code.

```
Shader "Custom/UniqueColoring" {
    SubShader {
        Tags { "RenderType"="Opaque" }
        LOD 100

        Pass {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata {
                float4 vertex : POSITION;
            };

            struct v2f {
                float4 vertex : SV_POSITION;
            };

            v2f vert (appdata v) {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                return o;
            }

            fixed4 frag (v2f i) : SV_Target {
                // always return red color
                return fixed4(1.0, 0.0, 0.0, 1.0);
            }
            ENDCG
        }
    }
}
```
If everything was setup correctly, any *GameObject* you see through the second camera should now be red.

## Bits & Bits

As of right now, our *Shader* only returns the color red for each fragment, which is determined by the following line:

```
return fixed4(1.0, 0.0, 0.0, 1.0);
```

We want to store a number in at least one of the color channels, say the red one for example.

The fragment *Shader* can be defined to return either a *fixed4*, *half4*, or a *float4* data structure, which represent respectively, 4 11-bit numbers, 4 16-bit numbers, and 4 32-bit numbers. [[Source](https://docs.unity3d.com/Manual/SL-DataTypesAndPrecision.html)]

To maximize the amount of data we can encode in the color, will choose to go with our fragment *Shader* returning a *float4* data structure.

## 32-bits

The following section is very important to make sure everything works as expected.

We need to pay attention that when the fragment *Shader* returns 4 32-bit values, that any other functions these are passed to, do not make us lose any precision by converting the values to a data structure of lower precision.

Specifically, for reading colors I will be using my approach from my [last post](https://dev.to/alpenglow/unity-fast-pixel-reading-part-2-asyncgpureadback-4kgn).

The following outlines the entire pass from the *Shader* to the final reading:

1. The *Shader* returns 4 32-bit values.
2. The image of my second camera is rendered into my *Render Texture*.
3. My *Compute Shader* reads a pixel from the *Render Texture*, and stores it in a buffer.
4. A script I have reads the value from the buffer.

Something that I missed early on when developing this solution is the precision value of the color channels for the *Render Texture*, which by default, are set to a very low precision value.

## Best *Render Texture*

A *Render Texture* object has many properties for its *RenderTextureFormat* enumeration.

Here's a screenshot from [the official documentation](https://docs.unity3d.com/ScriptReference/RenderTextureFormat.html) including some of them:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/li5rmijznwdv4wudvkc2.png)

In our case we select the last option visible in the screenshot above: "ARGBFloat", making each color channel 32-bits, such that we preserve the precision level from the *Shader*.

In my case, I am not creating the *Render Texture* programmatically. I select the *R32G32B32A32_SFLOAT* option under the *Color Format* field.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3324ygz9jx99fm98az4z.png)

In any case, we need to make sure that there is no loss in precision, and that at each step we preserve the 4 32-bit values without making any change to them.

In line with another great [article](https://medium.com/@philippchristoph/pixel-perfect-selection-using-shaders-16070d3094d):
*Additional Considerations*:
1. We use point filtering, since we don't want anti-aliasing or filtering.
2. We don't need MipMaps.
3. The *Skybox* should just be a solid color, filled with 0s as can be seen below:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/puvffny8drkxv937e2e5.png)

## How much is 32-bits?
In a 32-bit system, each bit can represent two possible values: 0 or 1. Since there are 32 bits in total, we can calculate the number of possible values by raising 2 to the power of 32:

2^32 = 4,294,967,296

Therefore, there are 4,294,967,296 different values that can be represented in 32 bits.

## Conclusion
Everything should work now.
Try changing the *Shader*'s color to have something like *0.5* or *0.001* in its red color channel, to then see if this is indeed the value output in the console (assuming you are logging every reading).