Repository: pytorch
Issue: Improved definition for Hardsigmoid
Labels: module: nn,triaged

### 🚀 The feature, motivation and pitch

I was looking at the Hardsigmoid and could not find the origin of the formulation of the PyTorch's implementation of the Hardsigmoid, however, I have mainly seen it in the format $\max(-1, \min(1, \frac{x +1}{2}))$ (https://arxiv.org/pdf/1511.00363v3, https://paperswithcode.com/method/hard-sigmoid, https://stackoverflow.com/questions/35411194/how-is-hard-sigmoid-defined). Would it be possible to add further documentation about where this version was found?

Since it seems like the PyTorch version was just trying to better approximate the original sigmoid function, I was looking into how to best optimize the Hardsigmoid activation function to minimize the error between the two implementations and got the following results:

$$
\begin{align}
y_h &= \alpha x + \frac{1}{2} \\
y_t &= \frac{1}{1 + e^{-x}}
\end{align}
$$

We wish to optimize:

$$
\begin{align}
\int_{0}^{\infty} (y_h - y_t)^2 dx &= -\frac{4 \alpha  \left(\pi ^2 \alpha +3\right)+5}{24 \alpha }-2 \alpha  \text{Li}_2\left(-e^{\left.\frac{1}{2}\right/\alpha }\right)
\end{align}
$$

(Where $\text{Li}_2\left(\cdot\right)$ represents the polylogarithm function)

From this, getting the derivative:

$$
\begin{align}
\frac{\partial}{\partial \alpha}\left(-\frac{4 \alpha  \left(\pi ^2 \alpha +3\right)+5}{24 \alpha }-2 \alpha  \text{Li}_2\left(-e^{\left.\frac{1}{2}\right/\alpha }\right)\right) &= \frac{5}{24 \alpha ^2}-\frac{\log \left(e^{\left.\frac{1}{2}\right/\alpha }+1\right)}{\alpha }-2 \text{Li}_2\left(-e^{\left.\frac{1}{2}\right/\alpha }\right)-\frac{\pi ^2}{6}
\end{align}
$$

Where solving for the minimum ends up becoming the problem of finding the root of a univariate transcendental equation and is thus only solvable in a numerical fashion, which gets solved to be $\alpha = 0.1923312226780115801134295966889092989087208992810093155047485229577227369110521531792719079253476632$ with the location where $y_h=1$ would be when $x=\frac{1}{2 \alpha} = 5.199363816628644532311143463905539244263001782697868420503570284299935792120883736036838950460889190$

For the proof, you can verify it using Mathematica:

```mathematica
yt = 1/(1  + Exp[-x])
yh = Max[0, Min[1, (x * \[Alpha] + 1/2)]]
m = Assuming[{Element[\[Alpha], PositiveReals]}, Integrate[(yh - yt)^2, {x, 0, \[Infinity]}]]  // FullSimplify
dm = D[m, \[Alpha]] // FullSimplify
s = Solve[dm == 0, \[Alpha], PositiveReals, Assumptions -> {\[Alpha] > 0, \[Alpha] < 1}]
N[s, 100]
N[1/\[Alpha] /. s, 100]
```

This means that the updated activation function should instead be:

$$
\begin{align}
\text{Hardsigmoid}(x) &= \begin{cases}
0 & \text{if } x \leq -\alpha \\
1 & \text{if } x \ge \alpha \\
\alpha x + \frac{1}{2} & \text{otherwise} \\
\end{cases}
\end{align}
$$

Or given the that $\alpha \approx 0.2 = \frac{1}{5}$, so the more optimal version should be:

$$
\begin{align}
\text{Hardsigmoid}(x) &= \begin{cases}
0 & \text{if } x \leq -2.5 \\
1 & \text{if } x \ge 2.5 \\
\frac{x}{5} + \frac{1}{2} & \text{otherwise} \\
\end{cases}
\end{align}
$$

In the case instead, where you want to optimize for only the linear region and minimize just for that you would get the following steps:

$$
\begin{align}
\int_{0}^{\frac{1}{2 \alpha}} (y_h - y_t)^2 dx &= -\frac{\pi ^2 \alpha }{6}+\frac{7}{24 \alpha }-\log \left(e^{\left.\frac{1}{2}\right/\alpha }+1\right)-\frac{1}{2} \tanh \left(\frac{1}{4 \alpha }\right)-2 \alpha  \text{Li}_2\left(-e^{\left.\frac{1}{2}\right/\alpha
   }\right)
\end{align}
$$

(Where $\text{Li}_2\left(\cdot\right)$ represents the polylogarithm function)

From this, getting the derivative:

$$
\begin{align}
\frac{\partial}{\partial \alpha}\left(-\frac{\pi ^2 \alpha }{6}+\frac{7}{24 \alpha }-\log \left(e^{\left.\frac{1}{2}\right/\alpha }+1\right)-\frac{1}{2} \tanh \left(\frac{1}{4 \alpha }\right)-2 \alpha  \text{Li}_2\left(-e^{\left.\frac{1}{2}\right/\alpha
   }\right)\right) &= \frac{1}{24} \left(\frac{5-\frac{12}{\left(e^{\left.\frac{1}{2}\right/\alpha }+1\right)^2}}{\alpha ^2}-4 \pi ^2\right)-\frac{\log \left(e^{\left.\frac{1}{2}\right/\alpha }+1\right)}{\alpha }-2
   \text{Li}_2\left(-e^{\left.\frac{1}{2}\right/\alpha }\right)
\end{align}
$$

Where solving for the minimum ends up becoming the problem of finding the root of a univariate transcendental equation again and is thus again only solvable in a numerical fashion, which gets solved to be $\alpha = 0.2054970267158964682036128983042348752082936695690816078237062763532105813855967891118081176897595642$ with the location where $y_h=1$ would be when $x=\frac{1}{2 \alpha} = 4.866250456180657743562901938100164679280222850465078977748315073324396050720004715397234973684419269$

The Mathematica proof for this is:

```mathematica
yt = 1/(1  + Exp[-x])
yh = Max[0, Min[1, (x * \[Alpha] + 1/2)]]
m = Assuming[{Element[\[Alpha], PositiveReals]}, Integrate[(yh - yt)^2, {x, 0, 1/(2 \[Alpha])}]] // FullSimplify
dm = D[m, \[Alpha]] // FullSimplify
s = Solve[dm == 0, \[Alpha], PositiveReals, Assumptions -> {\[Alpha] > 0, \[Alpha] < 1}]
N[s, 100]
N[1/\[Alpha] /. s, 100]
```

Where this optimal value also ends up having $\alpha \approx 0.2 = \frac{1}{5}$.

(The different Hardsigmoid functions)

![desmos-graph](https://github.com/user-attachments/assets/c2e86e6b-9aa0-4070-9ca8-6d7e43b84cb4)


### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki