Interpolating Between 2 Aspect Ratios in a Slightly Overcomplicated Way
Tags: gamedev, unity3d, il2cpp, ghidra
Published: 2024-05-06 14:45:21

The other day, I had to work on a simple Lerp calculation for a Unity project. It was about lerping (= Linear Interpolation) an offset value from `3.0f` to `4.0f`, depending on the current window aspect ratio. 

The initial specifications from the director were quite straightforward:
* In 'Vertical Mode', when the game's window is narrow, the offset should be at the lower end (`3.0f`).
* In 'Horizontal Mode', when the window is wide, the offset should be at the upper end (`4.0f`).
* To prevent the offset from becoming too extreme, the value should be capped at each end:
    * If the aspect ratio becomes smaller than `1080 * 1920` (a 9 : 16 aspect ratio), the offset should be capped at `3.0f`, and should not be smaller than the cap. 
    * If the ratio exceeds `1920 * 1080` (a 16 : 9 ratio), the offset should be capped at `4.0f`. No greater than that.
    * When the aspect ratio is 1:1, the offset value should be at 50% of the range, i.e., `(3.0 + 4.0) / 2 = 3.5`. 

By the way, users can freely adjust the window size to whatever they want, hence the need for capping.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/or0ntdidapt9eur86tof.png)

The image above summarizes and visualizes the details:
1. Each capping end and the middle screen resolutions are denoted by the yellow boxes and the yellow __'resolution' points__.
2. Since the game's window can be any size, any __resolution points__ on the quadrant should be properly mapped to a lerp factor. (`0.0f to 1.0f`)
    * If a __resolution point__ lies along the `y = x` line, such as (800 * 800), the factor must be `0.5f`.
    * If a __resolution point__ lies on or below the `y = 1080 / 1920 x` line, such as (480 * 270), the factor should be `0.0f`.
    * If it lies on or above the `y = 1920 / 1080 x` line, such as (100 * 1000), the factor should be `1.0f`.

## 1. Can't just plug in the Aspect Ratio itself.

Simply plugging in the aspect ratio to obtain the lerpFactor wouldn't suffice. While it nicely interpolates between the 2 ends, it wouldn't yield the middle value when the aspect ratio is 1:1.

```csharp
var minRatio = 1080/1920; // = 0.5625f
var maxRatio = 1920/1080; // = 1.7778f

// Capping the ratio
var aspectRatioClamped = Mathf.Clamp(aspectRatio, minRatio, maxRatio);

// Getting the lerpFactor (0 - 1)
var lerpFactor =  aspectRatioClamped - minRatio / (maxRatio - minRatio);

// NOTE: When the ratio is 1:1, the lerpFactor shall NOT be 0.5f. 
```

When the ratio is 1:1, the lerp factor would be `1 - 0.5625 / (1.7778 - 0.5625) = 0.5372`, slightly off from exactly 0.5f.

Converting the ratio into numerical form means setting the denominator to 1. In the graphical representation below, it's like obtaining the y-value of the intersection point between the line `x = 1`, and `y = ratio * x`.

For instance, if I am trying to get the lerp factor of a resolution of `960 * 640`, the aspect ratio would be `960 / 640 = 1.5`. To derive the lerp factor, I should first find the intersection point between `x = 1`, and `y = 1.5 x`, which is `(1, 1.5)`. Then, I can perform the inverse-lerp with the y-value, to get a lerp factor of `(1.5 - 0.5625) / (1.7778 - 0.5625) = 0.7714`.
Any resolution proportional to this would have the same aspect ratio of 1.5, regardless of the size, thus yielding the same Lerp factor. 

The `x = 1` line vertically slices the ratio graph; as it's not perpendicular to each capping end, it naturally divides the range where `slope < 1` and `slope >` 1 unevenly.

It's basically like performing an off-center projection (= oblique projection), so the center happens to be slightly off.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ei9vdpbxu9lzb5m8l2yd.png)

## 2. The `Atan()` way

To circumvent the unevenly divided range situation, one of the simplest approaches is to calculate the angles directly.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qdrw07szvs3cw23nq7vm.png)

* Get the angle between the resolution point to the x-axis (referred to as theta here).
* Compare and perform an `inverse-lerp` operation on the angle within the range of (alpha to beta)
* This ensures that the ranges around the midpoint are evenly divided; a resolution of 1:1 would yield a lerp factor of 0.5f.

FYI: (1920/1080 = 30 degrees, 1080/1920 = 60 degrees)

To represent this in C# code, it would appear as below:
```csharp
private float AtanLerp(Vector2 resolutionPoint)
{
    var resolutionPointAngle = Mathf.Atan2(resolutionPoint.y, resolutionPoint.x);
    resolutionPointAngle = Mathf.Clamp(resolutionPointAngle, MinAngle, MaxAngle);

    float lerpFactor = (resolutionPointAngle - MinAngle) / (MaxAngle - MinAngle);
    return lerpFactor;
}
```

The code directly calculates the angle between the `resolutionPoint` to the x-axis, then checks whether the angle lies within the angular range of the 2 capped aspect ratios.

There's no need for projection to a vertical slice line; it works as intended. This was the approach I initially took in the codebase.

## 3. The Projection Matrix Way 

After a couple of weeks from the implementation, I happened to glance over the graph I scribbled while rifling through the notebook. The graphs had somehow etched into my brain, and then I realized a function call to `atan2()` for each frame could be somewhat costly.

In reality, the performance cost of the function is virtually negligible. There are plenty of other parts that impose much heavier workloads. Nevertheless, I decided to delve into this during my free time for no reason.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/znr7uvcnxpvoe50tjiyl.png)

The major issue with the 1st method (using the aspect ratio) is that the __'projection line'__ wasn't aligned with the projection range. As long as I set the __'projection line'__ (the green one) perpendicular to the mid-range slope (the dark-blue line: `y = x`), evenly dividing the projection range (the light-blue lines), there will be no more skewed projection issue.

Once I obtain the projected point (the yellow point where the green and yellow lines intersect) from a __resolution point__, I can then calculate the lerp factor with a few floating-point arithmetic operations by comparing the projected point with the capping points (the green points).

This approach sounded quite promising. The projection could be handled with a matrix-vector multiplication, which could potentially be SIMD accelerated. Additionally, a few floating-point additions, subtractions, and divisions wouldn't hurt the performance as much as `atan2()` would.

### 3.1. Projection Matrix

I can derive a projection matrix that transforms a __resolution point__ into a __'projection point'__, utilizing the very same technique that every game engine uses, but in a way much simpler way. 

A __'projected point'__ is simply the intersection point between the projection line and the line that connects the origin point and a resolution point. 

The Latter, <u>the line connecting the origin and the resolution point</u>, could be represented as `P(t) = t * P`, where `P` is the resolution point, and `t` is a scalar parameter. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pr2qvjrvlj7fflr3nirl.png)

To get the intersection point (= `projected point`), plug in the parameterized `P(t)` to the projection line `y = -x + 2`, and then rearrange the equations to solve for `t`. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/klv393oet04do7rw5ldk.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m2enw958s3b4wpvhiefy.png)

After rearranging the equation, it becomes clear how to determine the projected point from the resolution point `P`. For each element of `P`, multiply it by 2, then divide it by `(y + x)`.

The matrix below will perform such a calculation with a homogeneous 2D vector (augmented with w = 1).

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/txvgrjlmbdwcu2xtn6g8.png)

After homogeneous division, I can finally attain the projected point. 

### 3.2. Calculating the Lerp Factor

Locating the projected point `(x, y)` is one thing, but I still need to derive the lerp factor from it. 

To put it naively, I could compare <u>the length between the projected point to a min or max projected point</u> with <u>the length between the min and max points</u>. While `Vector2.Distance()` could work, but it's quite computationally costly, defeating the purpose of the whole point.

In a case like this, the 'projection' property of the dot product can be utilized. I can take a dot product of 2 vectors. One points at the projected point from the min projected point, (the yellow dashed one, 'Projected'), and the other points from the min to the max projected point (the green one, 'Direction').

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ggd3lx5atbtrcr4vwtbq.png)

The length of the `Direction` could be computed beforehand, preferably in the constructor of a class or someplace similar. 

Then, I can calculate the length of the yellow vector, denoted as `l`. Once I divide it once more by the length of direction, then I get the lerp factor. 

```csharp
// Prepare the projection matrix upon class initialization.
// (no need to re-initialize the very same matrix for each method invocation)
private readonly UnityEngine.Matrix4x4 mat = new(
    new Vector4(2, 0, 0, 1),
    new Vector4(0, 2, 0, 1),
    new Vector4(0, 0, 1, 0),
    new Vector4(0, 0, 0, 0));

private float ProjectionLerp1(Vector2 resolutionPoint)
{
    var projectedPointInHomgen = mat * resolutionPoint;
    var projectedPoint = projectedPointInHomgen / projectedPointInHomgen.w;

    // Calc the lerp factor
    var originAlignedPoint = (Vector2)projectedPoint - MinProjectionPoint;
    var lerpFactor = Vector2.Dot(originAlignedPoint, projLineDirection) / projLineLength;
    var normalizedLerpFactor = Mathf.Clamp01(lerpFactor / projLineLength); // normalize the factor to a range of (0, 1)
    return normalizedLerpFactor;
}
```

### 3.3. The Performance wasn't Good Enough

Not only there was no performance gain, but it turns out the performance has been severely degraded compared to the prior version. 

I created a simple demo in Unity that accumulates the tick counts for each method. Every frame, it invokes each method once, counts its tick count, and accumulates it respectively. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1amawbguhqljogbvinfh.png)

Until frame number 4890, it took 31766 ticks in total to invoke the Atan version method, 42244 for the projection matrix version. (8 and 13 ticks at this particular frame)

There must be something happening under the hood. Let's take a look.

### 3.4. Examining the IL2CPP and Build Outcome

#### 3.4.1 The Atan method post-il2cpp outcome
Plain and simple. It almost looks like a direct translation from the original C# code.

It calls the `atan2f()`, `Mathf.Clamp()`, and then subsequently calls the `il2cpp_codegen_subtract` and division.

Although It may seem like to make 4 function calls, only one of the calls to `atan2f()` happens. The rest of the calls get inlined by the C++ compiler. I could peer into the final outcome (`GameAssembly.dll`) through Ghidra.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u9p4jvjr6wbrydvm7wvz.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c9l2qd2vfopmez9qtw3i.png)

(After all, `atan2f()` is the only function that gets invoked in the method)

#### 3.4.2 The Projection Matrix method post-il2cpp outcome

Looks slightly more verbose than the atan version, but it doesn't seem so bad at a glance. The function calls, mostly vector/matrix arithmetic operations, look like they're going to easily get inlined by the compiler in the end. 

Considering the `atan2f()` and the extra complexity it imposes with its nested if-else statements inside, this new version seems to have fewer instructions. Right?

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bblzii5v19a0y3i6466x.png)

Well, that wasn't the case. It has a larger instruction count, hence the degraded performance. The image below shows what it actually looks like after the C++ compiler.

As shown in the image, there's a bulky matrix initialization at the beginning of the decompiled function, followed by a function call to the `Matrix4x4.Multiply(Vector3)`, which didn't get inlined.

Overall, this projection version shows no performance advantage over the `atan2f()` version inherently.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wwkhfook8zwlztmekprp.png)

This bloated length of instructions will definitely NOT make this one faster than the previous one.

## 4. The vector way

Using a 4-by-4 matrix where a couple of simple vector arithmetic operations should do justice seems to be an overkill anyway. It could be worthwhile if it's a part of a series of affine transforms, but it isn't.

I could just get a `t` value and multiply it for each component of a resolution point.

``` csharp
private float ProjectionLerp2(Vector2 resolutionPoint)
{
    // Calc the point on the projection line
    var t = 2f / (resolutionPoint.x + resolutionPoint.y);
    var projectedPoint = t * resolutionPoint;

    // Calc the lerp factor: 
    // Project the projected point on the projection segment, to get the 't'
    // (dot product)
    var originAlignedPoint = projectedPoint - MinProjectionPoint;
    var lerpFactor = Vector2.Dot(originAlignedPoint, projLineDirection) / projLineLength;

    // Normalize the factor to a range of (0,1)
    var normalizedLerpFactor = Mathf.Clamp01(lerpFactor / projLineLength);

    return normalizedLerpFactor;
}
```

|Disassembled|Decompiled|
|:-:|:-:|
|![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ubk84p2b1te8b1e66vza.png)|![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5nya14uk4fdppwb7sw33.png)|

Compared to the previous matrix version, it looks much more lightweight. No single function call is involved. Let's count how many ticks it consumes.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/19si3vcbg5uj9tnzsvqc.png)

Seems pretty performant this time.

## 5. Calc the x position only

Then I thought: _why don't I just calculate the x-value only? I still can get the LerpFactor, and that's one less component to deal with. It could be more performant_.

The projection line is just a plain, simple monotonous 1:1 mapping function (the green one). So just by designating one x-value, the corresponding y-value gets determined automatically, as the y-value is dependent on the x-value. (the other way around also makes sense, btw) 

Simply performing the range comparison and finding the lerp factor of the yellow point on the x-axis in the green range will suffice.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v7e103jwwdyn6yj9fdqx.png)

The code would look like below:

```csharp
private float ProjectionLerp3(Vector2 resolutionPoint)
{
    // Calc only the X of the projected point
    var t = 2f / (resolutionPoint.x + resolutionPoint.y);
    var projectedPointX = t * resolutionPoint.x;

    // Just Map the x value to the projection segment range.
    var clampedProjectedPointX = Mathf.Clamp(projectedPointX, MaxProjectionPoint.x, MinProjectionPoint.x);
    var lerpFactor = (MinProjectionPoint.x - clampedProjectedPointX) / (MinProjectionPoint.x - MaxProjectionPoint.x);

    return lerpFactor;
}
```

|Disassembled|Decompiled|
|:-:|:-:|
|![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lb7m3z3w31itbbj7vkso.png)|![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fm7cqxzc3p8n6v7upxvn.png)|

Well, the instruction count didn't shrink much this time. The previous vector version itself was quite efficient enough, taking advantage of the SIMD acceleration. Leaving not enough room to squeeze.

## Overall Tick Counts

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7t3vn72fnrs4zhabe5eq.png)

Over the course of 4890 frames:
* The `atan2f()` version has accumulated 31766 ticks for its invocations, with 8 ticks during this frame.
* The Projection Matrix version has 42244 ticks so far, with 13 ticks in this frame.
* The Vector version has 16428 ticks, with 5 ticks in this frame.
* The float version has 19761 ticks, with 4 ticks in this frame.

The projection matrix version couldn't bring much performance gain because of the matrix initialization overhead and the not-inlined function call. 

The other 2 versions have successfully removed unnecessary operations, including function calls, resulting in fewer tick counts than the other previous 2 versions.

## Conclusion

* When lerp-ing between 2 aspect ratios, you can't just lerp the number themselves.
* While lerp-ing, make sure your mid-point lies in the middle.
* Could take advantage of the homogeneous coordinate and a projection matrix, to make it evenly lerp-ed.

Performance-wise:
* Matrix-to-Vector multiplication doesn't necessarily always get inlined.
* Copy-constructing a matrix can take some time.
* Take a look at what your compiler emits. Examining the code sometimes isn't sufficient.