hase transitions in when feedback is usefu



Appendix
========

Shoehorning the dynamics to an LQG control problem {#Shoehorning-dynamics}
--------------------------------------------------

Substituting Eqs \[world-dynamics\] - \[residual\] in Eq \[noisy-residual\]. $$\begin{aligned}
    \tilde{\delta}^{t} =& {\delta}^{t}+\eta_{\rm f}^{t}\nonumber\\
    =& ({D}\ {\tilde{p}}^{t}+{E}\ o^{t})+\eta_{\rm f}^{t}\nonumber\\
    =& {D}\ {\tilde{p}}^{t}+{E}\ (C\ x^{t}+\eta_{\rm o}^{t}) +\eta_{\rm f}^{t}\nonumber\\
    =& {D}\ {\tilde{p}}^{t}+{E}\ C\ x^{t}+{E}\ \eta_{\rm o}^{t} +\eta_{\rm f}^{t}\nonumber\\
    =& {D}\ {\tilde{p}}^{t}+{E}\ C\ (A\ x^{t-1}+\eta_{\rm p}^{t})+{E}\ \eta_{\rm o}^{t} +\eta_{\rm f}^{t}\nonumber\\
    =& {D}\ {\tilde{p}}^{t}+{E}\ C\ A\ x^{t-1}+{E}\ C\ \eta_{\rm p}^{t}+{E}\ \eta_{\rm o}^{t} +\eta_{\rm f}^{t}\nonumber\\
    =& {D}\ (p^{t}+\eta_{\rm b}^{t})+{E}\ C\ A\ x^{t-1}+{E}\ C\ \eta_{\rm p}^{t}+{E}\ \eta_{\rm o}^{t} +\eta_{\rm f}^{t}\nonumber\\
    =& {E}\ C\ A\ x^{t-1}+ {D}\ p^{t}+{E}\ C\ \eta_{\rm p}^{t}+{E}\ \eta_{\rm o}^{t}+{D}\ \eta_{\rm b}^{t} +\eta_{\rm f}^{t}\label{delta-expanded}\end{aligned}$$ Combining Eq \[world-dynamics\] and Eq \[delta-expanded\], we can write the dynamics of $z^{t}$ as in Eq \[state-LQG\]. The noisy residual can now be written as a partial observation of the augmented state, as shown in Eq \[residual-LQG\]. $$\begin{aligned}
    z^{t}=&\begin{bmatrix}
x^{t} \\
\tilde{\delta}^{t}
\end{bmatrix} = \overbrace{\begin{bmatrix}
A & 0 \\
E\ C\ A & 0
\end{bmatrix}}^{{A}_{\rm aug}}\overbrace{\begin{bmatrix}
x^{t-1} \\
\tilde{\delta}^{t-1}
\end{bmatrix}}^{z^{t-1}}+\overbrace{\begin{bmatrix}
0 \\
D
\end{bmatrix}}^{{B}_{\rm aug}}p^{t} + \overbrace{\begin{bmatrix}
I & 0 & 0 & 0\\
E\ C & E & D & I
\end{bmatrix}\begin{bmatrix}
\eta_{\rm p}^{t} \\
\eta_{\rm o}^{t} \\
\eta_{\rm b}^{t} \\
\eta_{\rm f}^{t}
\end{bmatrix}}^{\eta_{\rm aug}^{t-1}}\label{state-LQG}\\
\tilde{\delta}^{t}
 =& \underbrace{\begin{bmatrix}
0 & I 
\end{bmatrix}}_{{C}_{\rm aug}}\underbrace{\begin{bmatrix}
x^{t} \\
\tilde{\delta}^{t}
\end{bmatrix}}_{{z}^{t}}\label{residual-LQG}\end{aligned}$$ Rewriting Eqs \[state-LQG\] - \[residual-LQG\] using concise notations, we get the LQG dynamics equations where prediction $p^{t}$ is treated as the control signal. $$\begin{aligned}
    z^{t}=&A_{\rm aug}\ z^{t-1}+B_{\rm aug}\ p^{t} + \eta_{\rm aug}^{t-1}\nonumber\\
\tilde{\delta}^{t}
 =& C_{\rm aug}\ z^{t}\nonumber\end{aligned}$$

Shoehorning the energy cost to LQG objective function {#Shoehorning-energycost}
-----------------------------------------------------

Substituting Eq \[prediction\] in the energy cost part of Eq \[tot-cost\], $$\begin{aligned}
    {\rm Cost}_{\rm energy} =& \lim_{T \to \infty} \frac{1}{T} \sum_{t=1}^{T}  \left(\underbrace{\mbox{$\tilde{\delta}^{t}$}^{\top} W_{\rm f}\ {\tilde{\delta}}^{t}}_{{\rm Cost}_{\rm f}}+   \underbrace{\mbox{$\tilde{p}^{t}$}^{\top} W_{\rm b}\ \tilde{p}^{t}}_{{\rm Cost}_{\rm b}} \right)\nonumber\\
    =& \lim_{T \to \infty} \frac{1}{T} \sum_{t=1}^{T}  \left(\mbox{$\tilde{\delta}^{t}$}^{\top} W_{\rm f}\ {\tilde{\delta}}^{t}+   \mbox{$(p^{t}+\eta_{\rm b}^{t})$}^{\top} W_{\rm b}\ (p^{t}+\eta_{\rm b}^{t}) \right).\nonumber\end{aligned}$$ Since feedback noise is i.i.d $\sim \mathcal{N}(0,\,\Sigma_{\rm b})\,$, above simplifies to $$\begin{aligned}
    {\rm Cost}_{\rm energy} =& \lim_{T \to \infty} \frac{1}{T} \sum_{t=1}^{T}  \left(\mbox{$\tilde{\delta}^{t}$}^{\top} W_{\rm f}\ {\tilde{\delta}}^{t}+ \mbox{$p^{t}$}^{\top} W_{\rm b}\ p^{t} + \mbox{$\eta_{\rm b}^{t}$}^{\top} W_{\rm b}\ \eta_{\rm b}^{t}\right)\nonumber\\
    =& \lim_{T \to \infty} \frac{1}{T} \sum_{t=1}^{T}  \left(\mbox{$\tilde{\delta}^{t}$}^{\top} W_{\rm f}\ {\tilde{\delta}}^{t}+ \mbox{$p^{t}$}^{\top} W_{\rm b}\ p^{t} \right)+ Tr(W_{\rm b}\ \Sigma_{\rm b})\nonumber\\
    =& \lim_{T \to \infty} \frac{1}{T} \sum_{t=1}^{T}  \left({{z}^{t}}^{\top} Q\ {z}^{t}+ \mbox{$p^{t}$}^{\top} R\ p^{t} \right)+ Tr(W_{\rm b}\ \Sigma_{\rm b}),\ \text{where}\ Q = \begin{bmatrix}
0 & 0 \\
0 & W_{\rm f}
\end{bmatrix}\ \text{\&}\ R = W_{\rm b}.\nonumber\\
\implies \min_{p} {\rm Cost}_{\rm energy} 
=& \min_{p} \lim_{T \to \infty} \frac{1}{T} \sum_{t=1}^{T}  \left({{z}^{t}}^{\top} Q\ {z}^{t}+ \mbox{$p^{t}$}^{\top} R\ p^{t} \right)\nonumber\end{aligned}$$

For a fixed <span>D</span>& <span>E</span>, minimizing the Energy cost also minimizes Inference cost {#separation-inference}
----------------------------------------------------------------------------------------------------

The LQG problem of interest is described by Eqs \[concise-state-LQG\]-\[energy-cost-LQG-obj\]. The separation principle states that the solution for LQG problem includes an estimation part and a control part, both of which jointly forms the optimal solution even if treated separately. The estimation part of LQG solution minimizes the average squared norm error ${\rm Cost}_{\rm est}$ between estimated state $\hat{z}$ and the true state $z$ as shown in Eq \[estimation-error-z\]. Note that at any time $t$, we get to observe all the evidence up until that time (i.e $\tilde{\delta}_{1:t}$). This enables the simplification from Eq \[estimation-error-z-vector\] to Eq \[estimation-error-x\], because the optimal estimate $\hat{\tilde{\delta}}^{t}$ given $\tilde{\delta^{t}}$, is $\tilde{\delta}^{t}$ itself. Hence, from Eq \[estimation-inference\],we conclude that although we define the LQG objective function to be just the energy cost, the LQG solution that minimizes the energy cost also minimizes the inference cost for a fixed $D$ and $E$. $$\begin{aligned}
   {\rm Cost}_{\rm est} =& \lim_{T \to \infty} \frac{1}{T} \sum_{t=1}^{T} (z^{t} - \hat{z}^{t})^{\top}\ (z^{t} - \hat{z}^{t})\label{estimation-error-z}\\
   =& \lim_{T \to \infty} \frac{1}{T} \sum_{t=1}^{T} (\begin{bmatrix}
x^{t}\\
\tilde{\delta}^{t}
\end{bmatrix} - {\begin{bmatrix}
\hat{x}^{t} \\
\hat{\tilde{\delta}}^{t}
\end{bmatrix}})^{\top}\ (\begin{bmatrix}
x^{t} \\
\tilde{\delta}^{t}
\end{bmatrix} - {\begin{bmatrix}
\hat{x}^{t} \\
\hat{\tilde{\delta}}^{t}
\end{bmatrix}})\label{estimation-error-z-vector}\\
=& \lim_{T \to \infty} \frac{1}{T} \sum_{t=1}^{T} (x^{t} - \hat{x}^{t})^{\top}\ (x^{t} - \hat{x}^{t})\label{estimation-error-x}\\
=& \frac{1}{w_{\rm inf}}  {\rm Cost}_{\rm inf}\label{estimation-inference}\end{aligned}$$

Solution for LQG estimation {#estimation-solution}
---------------------------

The optimal state estimate for Eqs \[concise-state-LQG\]-\[energy-cost-LQG-obj\] is given as $$\begin{aligned}
    \hat{z}^{t}=& (I-K\ C_{\rm aug})A_{\rm aug}\ \hat{z}^{t-1}+K\ \tilde{\delta}^{t}+(I-K\  C_{\rm aug})B_{\rm aug}\ p^{t}\label{update-eqn}\\
    K =& \check{\Sigma}\ C_{\rm aug}^{T}(C_{\rm aug}\  \check{\Sigma}\  C_{\rm aug}^{\top})^{-1},\nonumber\end{aligned}$$ where $\check{\Sigma}$ is the solution to the following discrete algebraic Riccati equation, with $W$ as the covariance of noise vector $\eta_{\rm aug}^{t}$. $$\begin{aligned}
    \check{\Sigma} = A_{\rm aug}\ \check{\Sigma}\ A_{\rm aug}^{\top}+W-A_{\rm aug}\ \check{\Sigma}\ C_{\rm aug}^{\top}(C_{\rm aug}\ \check{\Sigma}\  C_{\rm aug}^{\top})^{-1}C_{\rm aug}\ \check{\Sigma}\ A_{\rm aug}^{\top}\nonumber\end{aligned}$$ For brevity, we rewrite Eq \[update-eqn\] in terms of block matrics using the following substitutions, $$\begin{aligned}
    {F}_{\rm aug}=&(I-K\ C_{\rm aug})A_{\rm aug},\ {H}_{\rm aug}=(I-K\ C_{\rm aug})B_{\rm aug}\nonumber\\
    {\begin{bmatrix}
\hat{x}^{t} \\
\tilde{\delta}^{t}
\end{bmatrix}}=& \begin{bmatrix}
({F}_{\rm aug})_{11} & ({F}_{\rm aug})_{12} \\
({F}_{\rm aug})_{21} & ({F}_{\rm aug})_{22}
\end{bmatrix} {\begin{bmatrix}
\hat{x}^{t-1} \\
\tilde{\delta}^{t-1}
\end{bmatrix}}+\begin{bmatrix}
(K)_{1} \\
(K)_{2}
\end{bmatrix}\tilde{\delta}^{t}+\begin{bmatrix}
({H}_{\rm aug})_{1}\\
({H}_{\rm aug})_{2}
\end{bmatrix}p^{t},\ \text{yielding}\nonumber\end{aligned}$$ $$\begin{aligned}
    \hat{x}^{t}=& ({F}_{\rm aug})_{11}\ \hat{x}^{t-1} + ({F}_{\rm aug})_{12}\ \tilde{\delta}^{t-1}+(K)_{1}\ \tilde{\delta}^{t}+({H}_{\rm aug})_{1}\  p^{t}\label{before-deduction}\\
    \tilde{\delta}^{t}=& ({F}_{\rm aug})_{21}\ \hat{x}^{t-1} + ({F}_{\rm aug})_{22}\ \tilde{\delta}^{t-1}+(K)_{2}\ \tilde{\delta}^{t}+({H}_{\rm aug})_{2}\  p^{t}.\nonumber\end{aligned}$$ Since the states are Markovian, consequently the optimal estimates are Markovian as well. This implies that $\hat{x}^{t}$ given $\hat{x}^{t-1}$, does not depend on $\tilde{\delta}^{t-1}$. Likewise, $\tilde{\delta}^{t}$ given $\tilde{\delta}^{t}$, does not depend on $\hat{x}^{t-1}$, $\tilde{\delta}^{t-1}$, and $p^{t}$. The above two arguments can be used to deduce (also verified empirically for one dimensional case), $$\begin{aligned}
  ({F}_{\rm aug})_{12} =0,\ ({F}_{\rm aug})_{21} =0,\ ({F}_{\rm aug})_{22} =0\ (K)_{2}=I,\ ({H}_{\rm aug})_{2}=0.\nonumber\end{aligned}$$ Substituting $({F}_{\rm aug})_{12} =0$ in Eq \[before-deduction\] and comparing it with Eq \[estimate\], we get the optimal solutions for ${F}$, ${G}$, and ${H}$ in terms of ${D}$ and ${E}$, denoted as ${F}^{\prime}$, ${G}^{\prime}$, and ${H}^{\prime}$ respectively. $$\begin{aligned}
    {F}^{\prime} = ({F}_{\rm aug})_{11},\ {G}^{\prime} = (K)_{1},\ {H}^{\prime} = ({H}_{\rm aug})_{1}\nonumber\end{aligned}$$

$\mathbf{Tr(Q\ \Sigma)=0}$
--------------------------

Let $\Sigma$ be the optimal estimation error covariance matrix in estimating $z^{t}$. Since the noisy residual component $\tilde{\delta}^{t}$ of the augmented state vector $z^{t}$ is fully observable at any time point, the corresponding lower block matrix in $\Sigma$ would be zero. By definition, in Eq \[energy-cost-LQG-obj\], we know that the upper block matrix in $Q$ is zero. As a consequence, $Tr(Q\ \Sigma)$ would alwaybs be zero. $$Tr(Q\ \Sigma)=0\label{trace}$$

Solution for LQG control {#control-solution}
------------------------

The optimal LQG control for Eqs \[concise-state-LQG\]-\[energy-cost-LQG-obj\] is given as $$\begin{aligned}
    p^{t}=&{L}_{\rm aug}\ \hat{z}^{t-1}\label{control-estimate}\\
    {L}_{\rm aug}=&-(R+B_{\rm aug}^{\top}\ P\ B_{\rm aug})^{-1}B_{\rm aug}^{\top}\ P\ A_{\rm aug}\label{control-gain} \end{aligned}$$ where $P$ is the solution of the discrete algebraic Riccati equation (Eq \[DARE-2\]). $$\begin{aligned}
    P = Q+A_{\rm aug}^{\top}\ P\  A_{\rm aug}-A_{\rm aug}^{\top}\ P\ B_{\rm aug}(R + B_{\rm aug}^{\top}\ P\ B_{\rm aug})^{-1}\ B_{\rm aug}^{\top}\ P\ A_{\rm aug}\label{DARE-2}\end{aligned}$$ As shown in Eq \[state-LQG\], the last set of columns in $A_{\rm aug}$ are all zero vectors. As a consequence, since $A_{\rm aug}$ is the last multiplier in the right side of equality in Eq \[control-estimate\], the last set of columns of ${L}_{\rm aug}$ would also be zero vectors. More specifically, Eq \[control-estimate\] can be further simplified by writing ${L}_{\rm aug}$ in terms of block matrices in the following way $$\begin{aligned}
    p^{t}= {\begin{bmatrix}
({L}_{\rm aug})_{1} &
({L}_{\rm aug})_{2}
\end{bmatrix}} {\begin{bmatrix}
\hat{x}^{t-1} \\
\tilde{\delta}^{t-1}
\end{bmatrix}}=&({L}_{\rm aug})_{1}\ \hat{x}^{t-1}+({L}_{\rm aug})_{2}\ \tilde{\delta}^{t-1},\ \text{then}\ ({L}_{\rm aug})_{2}=0\nonumber\\
\implies p^{t}=&({L}_{\rm aug})_{1}\ \hat{x}^{t-1}\nonumber\end{aligned}$$ Hence, the optimal prediction is linearly related to the estimate through $({L}_{\rm aug})_{1}$. For consistency in notations, we denote $({L}_{\rm aug})_{1}$ as ${L}^{\prime}$, yielding $$\begin{aligned}
p^{t}={L}^{\prime}\ \hat{x}^{t-1},\ \text{where}\ {L}^{\prime} = ({L}_{\rm aug})_{1}.\nonumber\end{aligned}$$

Condition for useful feedback {#nofbCondder}
-----------------------------

It can be algebraically shown that the total cost is an even function in both ${D}$ and ${E}$ and hence can be written as a function of $\check{{D}}={D}^{2}$ and $\check{{E}}={E}^{2}$. This limits the function domain to non-negative values, thereby reducing the search space for numerical optimization. We empirically observe that the cost function now has just one minimum when plotted with $\check{{D}}$ and $\check{{E}}$, and that the negative of gradient at any point always directs toward this minimum. We compute the value of optimal $\check{{E}}$, denoted as $\check{{E}}^{\prime}$, when $\check{{D}}=0$. We then find the derivative of cost function w.r.t $\check{{D}}$, at $\check{{E}}^{\prime}$. If this derivative is less than 0, then we directly have optimal ${D}^{\star}$ to be non-zero. If the derivative equals 0, then ${D}^{\star}=0$. And, if the derivative is greater than 0, the net gradient would be in the negative direction of $\check{{D}}$, as gradient w.r.t $\check{{E}}$ is 0 at ($\check{{D}}=0,\check{{E}}=\check{{E}}^{\prime}$). But we know domain of $\check{{D}}$ is non-negative. Hence, we again have ${D}^{\star}=0$. The condition for the derivative to be negative is concisely written as $${U}_{\rm bn}<\Phi({U}_{\rm fn},A,\sigma_{\rm p}^{2},\frac{C^{2}}{\sigma_{\rm o}^{2}}),$$ where $\Phi$ is a complicated function involving roots of a quartic equation.

[^1]: Use footnote for providing further information about author (webpage, alternative address)—*not* for acknowledging funding agencies.

{'source': 'arXiv_out/2110'}