Task: Newton casscf hcc (issue #701) (pyscf/pyscf#1869)

org: pyscf
repo: pyscf
pr_number: 1869
issue_title: Newton casscf hcc (issue #701)
issue_body: This merge request fixes the errors in mcscf.newton_casscf.gen_g_hop in the calculation of the CI vector second derivative of the (state-averaged) energy [here](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L296-L313) as described in issue #701. The newton_casscf algorithm does not directly use the CI trial vector computed by the gradient-descent step using this function to update the result CI vector without first running the actual FCI kernel, and the coupling between the orbital and CI degrees of freedom likewise used a robust expression that had the effect of canceling the erroneous contribution, so these errors did not lead to erroneous results in the sense that converged MC-SCF calculations were always correctly converged. This fix should therefore not change the outcome of any calculations in which the MC-SCF iteration converges to the correct stationary point, and the only unittest changes are to meaningless arbitrary reference points. However, this fix does cause the function to finally _do what it says it does,_ which IMO is important. For completeness, and since my original issue page used a third-party plugin for equations that may not render correctly for everyone, I now repeat the derivation using the native markdown latex support.

The first and second derivatives of an expectation value such as the with respect to some sort of unitary transformation are obtained by expressing the unitary operator as the exponential of the corresponding antihermitian unitary group generators,

$$
E = \langle 0|e^{-\hat{G}} \hat{H} e^{\hat{G}}|0\rangle
$$

For the CI vector part of an MC-SCF wave function in pyscf.mcscf.newton_casscf.py, the Hamiltonian-vector product is functionally abstracted, so we don't need to think too hard about the form of $\hat{G}$; we can just use the basic bra-ket language and the definition of antihermiticity:

$$
\hat{G} = \sum_{I} g_I (|I\rangle\langle0| - |0\rangle\langle I|) = |\vec{g}\rangle\langle 0| - |0\rangle\langle\vec{g}|
$$

It follows after some straightforward tedium that the first and second derivatives with respect to the generators are

$$
\frac{\partial E}{\partial g_I} = \langle I|\hat{H} - E_0|0\rangle + \textrm{h.c.}
$$

$$
\frac{\partial^2 E}{\partial g_I\partial g_J} = \langle I|\hat{Q}(\hat{H}-E_0)\hat{Q}|J\rangle + \textrm{h.c.}
$$

with

$$
\hat{Q} = 1 - |0\rangle\langle 0| ; E_0 = \langle 0|\hat{H}|0\rangle
$$

Using that $\langle 0|\hat{H}-E_0|0\rangle=0$ and that for real wave functions $\langle I|0\rangle = \langle 0|I\rangle = c_I$, the CI vector for the keyframe state $|0\rangle$, the second derivative is expanded as

$$
\frac{\partial^2 E}{\partial g_I\partial g_J} = \langle I|\hat{H}-E_0|J\rangle -\frac{1}{2}(c_I\frac{\partial E}{\partial g_J} + \frac{\partial E}{\partial g_I}c_J) + \textrm{h.c.}
$$

Note that "+ h.c." in both the first and second derivatives are accounted for by factors of 2 [here](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L303) and [here](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L384) that are applied at the end of the calculation. This means that `gci` at [this](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L296C15-L296C15) point in the function stores $\frac{1}{2}\frac{\partial E}{\partial g_I}$. Hopefully it is now clear why the factor of 4 on [this](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L296C15-L296C15) line is incorrect and should be a factor of 2. For the other two erroneous factors [here](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L312-L313), we can determine the correct expression by treating the second derivative as an operator, $\langle I|\partial^2 E|J\rangle \equiv \frac{\partial^2 E}{\partial g_I\partial g_J}$:

$$
\partial^2 E|I\rangle = (\hat{H} - E_0)\ket{I} - (\hat{H} - E_0)\ket{0}\braket{0|I} - \ket{0}\braket{0|\hat{H}-E_0|I} + \mathrm{h.c.}
$$

Again, "+ h.c." is accounted for later in the function. Hopefully it is clear that factors of 2 simply should not be [here](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L312-L313).
resolved_issues: number: 701
title: Is there a formal error in the H_cc sector of newton_casscf.gen_g_hop?
body: In differentiating with respect to an infinitesimal transformation of the CI vectors, we parameterize the energy as

<a href="https://www.codecogs.com/eqnedit.php?latex=E(x)=\langle0|e^{-\hat{P}}\hat{H}e^{\hat{P}}|0\rangle" target="_blank"><img src="https://latex.codecogs.com/gif.latex?E(x)=\langle0|e^{-\hat{P}}\hat{H}e^{\hat{P}}|0\rangle" title="E(x)=\langle0|e^{-\hat{P}}\hat{H}e^{\hat{P}}|0\rangle" /></a>

with

<a href="https://www.codecogs.com/eqnedit.php?latex=\hat{P}\equiv\sum_{\vec{n}}|\vec{n}\rangle\langle0|-|0\rangle\langle\vec{n}|" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\hat{P}\equiv\sum_{\vec{n}}x_{\vec{n}}(|\vec{n}\rangle\langle0|-|0\rangle\langle\vec{n}|)" title="\hat{P}\equiv\sum_{\vec{n}}x_{\vec{n}}(|\vec{n}\rangle\langle0|-|0\rangle\langle\vec{n}|)" /></a>

The second derivative of the energy is

<a href="https://www.codecogs.com/eqnedit.php?latex=\frac{\partial^2&space;E}{\partial&space;x_{\vec{n}}\partial&space;x_{\vec{n}^\prime}}=\langle\vec{n}|\hat{Q}(\hat{H}-E_0)\hat{Q}|\vec{n}^\prime\rangle" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\frac{\partial^2&space;E}{\partial&space;x_{\vec{n}}\partial&space;x_{\vec{n}^\prime}}=\langle\vec{n}|\hat{Q}(\hat{H}-E_0)\hat{Q}|\vec{n}^\prime\rangle" title="\frac{\partial^2 E}{\partial x_{\vec{n}}\partial x_{\vec{n}^\prime}}=\langle\vec{n}|\hat{Q}(\hat{H}-E_0)\hat{Q}|\vec{n}^\prime\rangle" /></a>

with

<a href="https://www.codecogs.com/eqnedit.php?latex=\hat{Q}\equiv1-|0\rangle\langle0|" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\hat{Q}\equiv1-|0\rangle\langle0|" title="\hat{Q}\equiv1-|0\rangle\langle0|" /></a>
<a href="https://www.codecogs.com/eqnedit.php?latex=E_0\equiv\langle0|\hat{H}|0\rangle" target="_blank"><img src="https://latex.codecogs.com/gif.latex?E_0\equiv\langle0|\hat{H}|0\rangle" title="E_0\equiv\langle0|\hat{H}|0\rangle" /></a>

or, if we can forgive a bit of notational handwaving,

<a href="https://www.codecogs.com/eqnedit.php?latex=\partial^2&space;E|\vec{x}\rangle=(\hat{H}-E_0)|\vec{x}\rangle-|0\rangle\langle0|(\hat{H}-E_0)|\vec{x}\rangle-(\hat{H}-E_0)|0\rangle\langle0|\vec{x}\rangle" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\partial^2&space;E|\vec{x}\rangle=(\hat{H}-E_0)|\vec{x}\rangle-|0\rangle\langle0|(\hat{H}-E_0)|\vec{x}\rangle-(\hat{H}-E_0)|0\rangle\langle0|\vec{x}\rangle" title="\partial^2 E|\vec{x}\rangle=(\hat{H}-E_0)|\vec{x}\rangle-|0\rangle\langle0|(\hat{H}-E_0)|\vec{x}\rangle-(\hat{H}-E_0)|0\rangle\langle0|\vec{x}\rangle" /></a>

But what's programmed in newton_casscf.py appears to be

<a href="https://www.codecogs.com/eqnedit.php?latex=\partial^2&space;E|\vec{x}\rangle=(\hat{H}-E_0)|\vec{x}\rangle-2|0\rangle\langle0|(\hat{H}-E_0)|\vec{x}\rangle-2(\hat{H}-E_0)|0\rangle\langle0|\vec{x}\rangle" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\partial^2&space;E|\vec{x}\rangle=(\hat{H}-E_0)|\vec{x}\rangle-2|0\rangle\langle0|(\hat{H}-E_0)|\vec{x}\rangle-2(\hat{H}-E_0)|0\rangle\langle0|\vec{x}\rangle" title="\partial^2 E|\vec{x}\rangle=(\hat{H}-E_0)|\vec{x}\rangle-2|0\rangle\langle0|(\hat{H}-E_0)|\vec{x}\rangle-2(\hat{H}-E_0)|0\rangle\langle0|\vec{x}\rangle" /></a>

in both hop and hdiag. I would understand a factor of 2 convention difference (EDIT: actually all three of my second-derivative expressions up there should have a +h.c. that I dropped), but this isn't a factor of 2, unless I've gotten confused about what (H-E0) is. In fact, it spoils intermediate normalization (we should have <0|x>=0 at all times; the component of P along the trial vector |0> is undefined by construction):

```
import numpy
from pyscf import gto, scf, lib, fci
from pyscf.mcscf import newton_casscf, CASSCF, addons

mol = gto.Mole()
mol.verbose = lib.logger.DEBUG
mol.output = '/dev/null'
mol.atom = [
    ['H', ( 5.,-1.    , 1.   )],
    ['H', ( 0.,-5.    ,-2.   )],
    ['H', ( 4.,-0.5   ,-3.   )],
    ['H', ( 0.,-4.5   ,-1.   )],
    ['H', ( 3.,-0.5   ,-0.   )],
    ['H', ( 0.,-3.    ,-1.   )],
    ['H', ( 2.,-2.5   , 0.   )],
    ['H', ( 1., 1.    , 3.   )],
]
mol.basis = 'sto-3g'
mol.build()
mf = scf.RHF(mol)
mf.max_cycle = 3
mf.kernel()
mc = newton_casscf.CASSCF(mf, 4, 4)
mc.fcisolver = fci.direct_spin1.FCI(mol)
mc.kernel()

numpy.random.seed(1)
mo = numpy.random.random(mf.mo_coeff.shape)
ci0 = numpy.random.random((6,6))
ci0/= numpy.linalg.norm(ci0)
gall, gop, hop, hdiag = newton_casscf.gen_g_hop(mc, mo, ci0, mc.ao2mo(mo))
x = numpy.random.random(gall.size)
hx = hop (x)
print ("|hx_ci| =",numpy.linalg.norm (hx[-36:]),"<0|hx> =", ci0.ravel ().dot (hx[-36:]))
```

yields

```
|hx_ci| = 73.30773999100957 <0|hx> = 4.13369273110702
```

I think this has minimal effects on the results of calculations because 1) the extract_rotation function forcibly normalizes the new output CI vector after every Newton-CG step, and the s10 business in the H_oc sector corrects for the effect of the spurious <0|x> component on the orbitals, so all this does is attenuate CI vector evolution almost like a trust radius, and 2) the coupled-perturbed solver for the state-average analytical gradient calculation that I added has a preconditioner that enforces <0|x> = 0 post hoc. But I wanted to ask about it to see if I was missing something, especially since changing it would make at least one test fail.



Reference solution:
fix_patch: diff --git a/pyscf/fci/rdm.py b/pyscf/fci/rdm.py
index 58be01c4fe..284053c014 100644
--- a/pyscf/fci/rdm.py
+++ b/pyscf/fci/rdm.py
@@ -101,8 +101,8 @@ def make_rdm1_spin1(fname, cibra, ciket, norb, nelec, link_index=None):
         link_indexa, link_indexb = link_index
     na,nlinka = link_indexa.shape[:2]
     nb,nlinkb = link_indexb.shape[:2]
-    assert (cibra.size == na*nb)
-    assert (ciket.size == na*nb)
+    assert (cibra.size == na*nb), '{} {} {}'.format (cibra.size, na, nb)
+    assert (ciket.size == na*nb), '{} {} {}'.format (ciket.size, na, nb)
     rdm1 = numpy.empty((norb,norb))
     fn = getattr(librdm, fname)
     fn(rdm1.ctypes.data_as(ctypes.c_void_p),
diff --git a/pyscf/mcscf/newton_casscf.py b/pyscf/mcscf/newton_casscf.py
index 941def1e56..b6e4d9c352 100644
--- a/pyscf/mcscf/newton_casscf.py
+++ b/pyscf/mcscf/newton_casscf.py
@@ -293,7 +293,8 @@ def g_update(u, fcivec):
 #    h_diag[ncore:nocc,ncore:nocc] -= v_diag[:,ncore:nocc]*2
     h_diag = casscf.pack_uniq_var(h_diag)
 
-    hci_diag = [hd - ec - gc*c*4 for hd, ec, gc, c in zip (_Hdiag (h1cas_0, eri_cas), eci0, gci, ci0)]
+    # Fix intermediate normalization? - MRH 2023/09/15
+    hci_diag = [hd - ec - gc*c*2 for hd, ec, gc, c in zip (_Hdiag (h1cas_0, eri_cas), eci0, gci, ci0)]
     hci_diag = [h * w for h, w in zip (hci_diag, weights)]
     hdiag_all = numpy.hstack((h_diag*2, _pack_ci (hci_diag)*2))
 
@@ -308,9 +309,10 @@ def h_op(x):
         ci1 = _unpack_ci (x[ngorb:])
 
         # H_cc
+        # Fix intermediate normalization? - MRH 2023/09/15
         hci1 = [hc1 - c1 * ec0 for hc1, c1, ec0 in zip (_Hci (h1cas_0, eri_cas, ci1), ci1, eci0)]
-        hci1 = [hc1 - 2*(hc0 - c0*ec0)*c0.dot(c1) for hc1, hc0, c0, ec0, c1 in zip (hci1, hci0, ci0, eci0, ci1)]
-        hci1 = [hc1 - 2*c0*(hc0 - c0*ec0).dot(c1) for hc1, hc0, c0, ec0, c1 in zip (hci1, hci0, ci0, eci0, ci1)]
+        hci1 = [hc1 - (hc0 - c0*ec0)*c0.dot(c1) for hc1, hc0, c0, ec0, c1 in zip (hci1, hci0, ci0, eci0, ci1)]
+        hci1 = [hc1 - c0*(hc0 - c0*ec0).dot(c1) for hc1, hc0, c0, ec0, c1 in zip (hci1, hci0, ci0, eci0, ci1)]
 
         # H_co
         rc = x1[:,:ncore]