Issue:
Update BFD pseudo potential and bases

The existing BFD pp and bases are poor for H and alkali elements. They can be updated using the one developed by Burkatzki, Claudia, and Michael Dolg


---

Is there a formal error in the H_cc sector of newton_casscf.gen_g_hop?

In differentiating with respect to an infinitesimal transformation of the CI vectors, we parameterize the energy as

<a href="https://www.codecogs.com/eqnedit.php?latex=E(x)=\langle0|e^{-\hat{P}}\hat{H}e^{\hat{P}}|0\rangle" target="_blank"><img src="https://latex.codecogs.com/gif.latex?E(x)=\langle0|e^{-\hat{P}}\hat{H}e^{\hat{P}}|0\rangle" title="E(x)=\langle0|e^{-\hat{P}}\hat{H}e^{\hat{P}}|0\rangle" /></a>

with

<a href="https://www.codecogs.com/eqnedit.php?latex=\hat{P}\equiv\sum_{\vec{n}}|\vec{n}\rangle\langle0|-|0\rangle\langle\vec{n}|" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\hat{P}\equiv\sum_{\vec{n}}x_{\vec{n}}(|\vec{n}\rangle\langle0|-|0\rangle\langle\vec{n}|)" title="\hat{P}\equiv\sum_{\vec{n}}x_{\vec{n}}(|\vec{n}\rangle\langle0|-|0\rangle\langle\vec{n}|)" /></a>

The second derivative of the energy is

<a href="https://www.codecogs.com/eqnedit.php?latex=\frac{\partial^2&space;E}{\partial&space;x_{\vec{n}}\partial&space;x_{\vec{n}^\prime}}=\langle\vec{n}|\hat{Q}(\hat{H}-E_0)\hat{Q}|\vec{n}^\prime\rangle" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\frac{\partial^2&space;E}{\partial&space;x_{\vec{n}}\partial&space;x_{\vec{n}^\prime}}=\langle\vec{n}|\hat{Q}(\hat{H}-E_0)\hat{Q}|\vec{n}^\prime\rangle" title="\frac{\partial^2 E}{\partial x_{\vec{n}}\partial x_{\vec{n}^\prime}}=\langle\vec{n}|\hat{Q}(\hat{H}-E_0)\hat{Q}|\vec{n}^\prime\rangle" /></a>

with

<a href="https://www.codecogs.com/eqnedit.php?latex=\hat{Q}\equiv1-|0\rangle\langle0|" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\hat{Q}\equiv1-|0\rangle\langle0|" title="\hat{Q}\equiv1-|0\rangle\langle0|" /></a>
<a href="https://www.codecogs.com/eqnedit.php?latex=E_0\equiv\langle0|\hat{H}|0\rangle" target="_blank"><img src="https://latex.codecogs.com/gif.latex?E_0\equiv\langle0|\hat{H}|0\rangle" title="E_0\equiv\langle0|\hat{H}|0\rangle" /></a>

or, if we can forgive a bit of notational handwaving,

<a href="https://www.codecogs.com/eqnedit.php?latex=\partial^2&space;E|\vec{x}\rangle=(\hat{H}-E_0)|\vec{x}\rangle-|0\rangle\langle0|(\hat{H}-E_0)|\vec{x}\rangle-(\hat{H}-E_0)|0\rangle\langle0|\vec{x}\rangle" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\partial^2&space;E|\vec{x}\rangle=(\hat{H}-E_0)|\vec{x}\rangle-|0\rangle\langle0|(\hat{H}-E_0)|\vec{x}\rangle-(\hat{H}-E_0)|0\rangle\langle0|\vec{x}\rangle" title="\partial^2 E|\vec{x}\rangle=(\hat{H}-E_0)|\vec{x}\rangle-|0\rangle\langle0|(\hat{H}-E_0)|\vec{x}\rangle-(\hat{H}-E_0)|0\rangle\langle0|\vec{x}\rangle" /></a>

But what's programmed in newton_casscf.py appears to be

<a href="https://www.codecogs.com/eqnedit.php?latex=\partial^2&space;E|\vec{x}\rangle=(\hat{H}-E_0)|\vec{x}\rangle-2|0\rangle\langle0|(\hat{H}-E_0)|\vec{x}\rangle-2(\hat{H}-E_0)|0\rangle\langle0|\vec{x}\rangle" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\partial^2&space;E|\vec{x}\rangle=(\hat{H}-E_0)|\vec{x}\rangle-2|0\rangle\langle0|(\hat{H}-E_0)|\vec{x}\rangle-2(\hat{H}-E_0)|0\rangle\langle0|\vec{x}\rangle" title="\partial^2 E|\vec{x}\rangle=(\hat{H}-E_0)|\vec{x}\rangle-2|0\rangle\langle0|(\hat{H}-E_0)|\vec{x}\rangle-2(\hat{H}-E_0)|0\rangle\langle0|\vec{x}\rangle" /></a>

in both hop and hdiag. I would understand a factor of 2 convention difference (EDIT: actually all three of my second-derivative expressions up there should have a +h.c. that I dropped), but this isn't a factor of 2, unless I've gotten confused about what (H-E0) is. In fact, it spoils intermediate normalization (we should have <0|x>=0 at all times; the component of P along the trial vector |0> is undefined by construction):

```
import numpy
from pyscf import gto, scf, lib, fci
from pyscf.mcscf import newton_casscf, CASSCF, addons

mol = gto.Mole()
mol.verbose = lib.logger.DEBUG
mol.output = '/dev/null'
mol.atom = [
    ['H', ( 5.,-1.    , 1.   )],
    ['H', ( 0.,-5.    ,-2.   )],
    ['H', ( 4.,-0.5   ,-3.   )],
    ['H', ( 0.,-4.5   ,-1.   )],
    ['H', ( 3.,-0.5   ,-0.   )],
    ['H', ( 0.,-3.    ,-1.   )],
    ['H', ( 2.,-2.5   , 0.   )],
    ['H', ( 1., 1.    , 3.   )],
]
mol.basis = 'sto-3g'
mol.build()
mf = scf.RHF(mol)
mf.max_cycle = 3
mf.kernel()
mc = newton_casscf.CASSCF(mf, 4, 4)
mc.fcisolver = fci.direct_spin1.FCI(mol)
mc.kernel()

numpy.random.seed(1)
mo = numpy.random.random(mf.mo_coeff.shape)
ci0 = numpy.random.random((6,6))
ci0/= numpy.linalg.norm(ci0)
gall, gop, hop, hdiag = newton_casscf.gen_g_hop(mc, mo, ci0, mc.ao2mo(mo))
x = numpy.random.random(gall.size)
hx = hop (x)
print ("|hx_ci| =",numpy.linalg.norm (hx[-36:]),"<0|hx> =", ci0.ravel ().dot (hx[-36:]))
```

yields

```
|hx_ci| = 73.30773999100957 <0|hx> = 4.13369273110702
```

I think this has minimal effects on the results of calculations because 1) the extract_rotation function forcibly normalizes the new output CI vector after every Newton-CG step, and the s10 business in the H_oc sector corrects for the effect of the spurious <0|x> component on the orbitals, so all this does is attenuate CI vector evolution almost like a trust radius, and 2) the coupled-perturbed solver for the state-average analytical gradient calculation that I added has a preconditioner that enforces <0|x> = 0 post hoc. But I wanted to ask about it to see if I was missing something, especially since changing it would make at least one test fail.



Fix patch:
diff --git a/pyscf/fci/rdm.py b/pyscf/fci/rdm.py
index 58be01c4fe..284053c014 100644
--- a/pyscf/fci/rdm.py
+++ b/pyscf/fci/rdm.py
@@ -101,8 +101,8 @@ def make_rdm1_spin1(fname, cibra, ciket, norb, nelec, link_index=None):
         link_indexa, link_indexb = link_index
     na,nlinka = link_indexa.shape[:2]
     nb,nlinkb = link_indexb.shape[:2]
-    assert (cibra.size == na*nb)
-    assert (ciket.size == na*nb)
+    assert (cibra.size == na*nb), '{} {} {}'.format (cibra.size, na, nb)
+    assert (ciket.size == na*nb), '{} {} {}'.format (ciket.size, na, nb)
     rdm1 = numpy.empty((norb,norb))
     fn = getattr(librdm, fname)
     fn(rdm1.ctypes.data_as(ctypes.c_void_p),
diff --git a/pyscf/mcscf/newton_casscf.py b/pyscf/mcscf/newton_casscf.py
index 941def1e56..b6e4d9c352 100644
--- a/pyscf/mcscf/newton_casscf.py
+++ b/pyscf/mcscf/newton_casscf.py
@@ -293,7 +293,8 @@ def g_update(u, fcivec):
 #    h_diag[ncore:nocc,ncore:nocc] -= v_diag[:,ncore:nocc]*2
     h_diag = casscf.pack_uniq_var(h_diag)
 
-    hci_diag = [hd - ec - gc*c*4 for hd, ec, gc, c in zip (_Hdiag (h1cas_0, eri_cas), eci0, gci, ci0)]
+    # Fix intermediate normalization? - MRH 2023/09/15
+    hci_diag = [hd - ec - gc*c*2 for hd, ec, gc, c in zip (_Hdiag (h1cas_0, eri_cas), eci0, gci, ci0)]
     hci_diag = [h * w for h, w in zip (hci_diag, weights)]
     hdiag_all = numpy.hstack((h_diag*2, _pack_ci (hci_diag)*2))
 
@@ -308,9 +309,10 @@ def h_op(x):
         ci1 = _unpack_ci (x[ngorb:])
 
         # H_cc
+        # Fix intermediate normalization? - MRH 2023/09/15
         hci1 = [hc1 - c1 * ec0 for hc1, c1, ec0 in zip (_Hci (h1cas_0, eri_cas, ci1), ci1, eci0)]
-        hci1 = [hc1 - 2*(hc0 - c0*ec0)*c0.dot(c1) for hc1, hc0, c0, ec0, c1 in zip (hci1, hci0, ci0, eci0, ci1)]
-        hci1 = [hc1 - 2*c0*(hc0 - c0*ec0).dot(c1) for hc1, hc0, c0, ec0, c1 in zip (hci1, hci0, ci0, eci0, ci1)]
+        hci1 = [hc1 - (hc0 - c0*ec0)*c0.dot(c1) for hc1, hc0, c0, ec0, c1 in zip (hci1, hci0, ci0, eci0, ci1)]
+        hci1 = [hc1 - c0*(hc0 - c0*ec0).dot(c1) for hc1, hc0, c0, ec0, c1 in zip (hci1, hci0, ci0, eci0, ci1)]
 
         # H_co
         rc = x1[:,:ncore]


Test patch:
diff --git a/pyscf/mcscf/test/test_newton_casscf.py b/pyscf/mcscf/test/test_newton_casscf.py
index db4e7c15a7..f2d94c3e9a 100644
--- a/pyscf/mcscf/test/test_newton_casscf.py
+++ b/pyscf/mcscf/test/test_newton_casscf.py
@@ -88,11 +88,11 @@ def test_gen_g_hop(self):
         ci0/= numpy.linalg.norm(ci0)
         gall, gop, hop, hdiag = newton_casscf.gen_g_hop(mc, mo, ci0, mc.ao2mo(mo))
         self.assertAlmostEqual(lib.fp(gall), 21.288022525148595, 8)
-        self.assertAlmostEqual(lib.fp(hdiag), -4.6864640132374618, 8)
+        self.assertAlmostEqual(lib.fp(hdiag), -15.618395788969842, 8)
         x = numpy.random.random(gall.size)
         u, ci1 = newton_casscf.extract_rotation(mc, x, 1, ci0)
         self.assertAlmostEqual(lib.fp(gop(u, ci1)), -412.9441873541524, 8)
-        self.assertAlmostEqual(lib.fp(hop(x)), 73.358310983341198, 8)
+        self.assertAlmostEqual(lib.fp(hop(x)), 24.04569925660985, 8)
 
     def test_get_grad(self):
         # mc.e_tot may be converged to -3.6268060853430573
@@ -107,11 +107,11 @@ def test_sa_gen_g_hop(self):
         ci0 = list (ci0.reshape ((2,6,6)))
         gall, gop, hop, hdiag = newton_casscf.gen_g_hop(sa, mo, ci0, sa.ao2mo(mo))
         self.assertAlmostEqual(lib.fp(gall), 32.46973284682045, 8)
-        self.assertAlmostEqual(lib.fp(hdiag), -63.6527761153809, 8)
+        self.assertAlmostEqual(lib.fp(hdiag), -70.61862254321517, 8)
         x = numpy.random.random(gall.size)
         u, ci1 = newton_casscf.extract_rotation(sa, x, 1, ci0)
         self.assertAlmostEqual(lib.fp(gop(u, ci1)), -49.017079186126, 8)
-        self.assertAlmostEqual(lib.fp(hop(x)), 169.47893548740288, 8)
+        self.assertAlmostEqual(lib.fp(hop(x)), 136.2077988624156, 8)
 
     def test_sa_get_grad(self):
         self.assertAlmostEqual(sa.e_tot, -3.62638372957158, 7)