Repository: rdkit/rdkit — pull request #3758 (cpp)
Title: Fixes #3755

#3755 is caused by two separate problems:

1) There is an inconsistency in the exhaustiveness of the tautomer enumeration between the chiral and the non-chiral versions of serine. This is caused by a bug in the logic that checks when the enumeration should be considered as complete.
As a consequence, the enumeration for the chiral molecule is quit too early, and the hydrate tautomer is not generated. Therefore, the expected carboxylic acid tautomer is generated, and the result appears to be correct (because the competing hydrate tautomer is not generated).
Instead, for the achiral molecule the enumeration is indeed complete, the hydrate tautomer scores the same as the carboxylic acid tautomer, but the hydrate tautomer is preferred as its SMILES comes first in lexicographic order.<br/>
This PR fixes the bug in the logic, so the enumeration is now complete also for the chiral molecule case.
<ol>

* I also fixed the debug build with `VERBOSE_ENUMERATION` defined (which was failing to build)
* I moved the comments around to make them more useful during debugging
* I removed an unnecessary copying of the molecule
* I removed the `done` stack altogether

<br/>The two latter points should result in improved performance.
</ol>

2) The hydrate tautomer scores the same as the carboxylic acid tautomer. This already happened in the original Python implementation of MolVS:
<ol>


```
from rdkit import Chem
from rdkit.Chem.Draw import MolsToGridImage
from rdkit.Chem.MolStandardize import tautomer

orig_vs_expected = [
    ("OC(=O)C(N)CO", "NC(CO)C(=O)O"),
    ("C([C@@H](C(=O)O)N)O", "NC(CO)C(=O)O"),
    ("OC(=O)C(N)CN", "NCC(N)C(=O)O"),
    ("NC(=O)C(N)CO", "NC(=O)C(N)CO"),
    ("NC(=N)C(N)CO", "N=C(N)C(N)CO"),
    ("NC(=N)NC(N)CO", "N=C(N)NC(N)CO")
]

enumerator = tautomer.TautomerCanonicalizer()
orig_vs_canonical_mols = []
for orig, expected in orig_vs_expected:
    orig_mol = Chem.MolFromSmiles(orig)
    orig_vs_canonical_mols.append(orig_mol)
    canonical = enumerator.canonicalize(orig_mol)
    orig_vs_canonical_mols.append(canonical)

MolsToGridImage(orig_vs_canonical_mols, molsPerRow=2, subImgSize=(200,100))
```
![image](https://user-images.githubusercontent.com/5244385/105371916-60968080-5c05-11eb-960f-06faa6715580.png)
<br/>As you may see, there are a number of molecules which are converted to hydrates or aminals.
This PR fixes this issue by adding an additional substructure rule:
```
{"C(=hetero)-hetero", "[C](=[!#1;!#6])[!#1;!#6]", 2},
```
which gives a higher score to carboxylic acids/amides/amidines vs their reduced counterparts.
</ol>

Resolved issues:
title: Inconsistent canonical tautomer on repeated application
body: 
**Describe the bug**
Tautomer canonicalization seems to be occasionally inconsistent, repeated canonicalizations will produce different results.

**To Reproduce**

Repeatedly pick the canonical tautomer for Serine.

```python
from rdkit import Chem
from rdkit.Chem.MolStandardize import rdMolStandardize

enumerator = rdMolStandardize.TautomerEnumerator()

mol1 = Chem.MolFromSmiles('C([C@@H](C(=O)O)N)O')
mol2 = enumerator.Canonicalize(mol1)
mol3 = enumerator.Canonicalize(mol2)

print('\n'.join([
    Chem.MolToSmiles(mol1),
    Chem.MolToSmiles(mol2),
    Chem.MolToSmiles(mol3)
]))
```

Output:
```
N[C@@H](CO)C(=O)O
NC(CO)C(=O)O
NC(C=O)C(O)O
```
**Expected behavior**

I would expect mol2 to be the same as mol3.  The canonical tautomer of the canonical tautomer should still be the original canonical tautomer (unless we hit MaxTautomers or something like that, which is not occurring here).

Instead mol2 and mol3 differ in where they've put the double bond.


**Configuration (please complete the following information):**
 - RDKit version: 2020.09.3
 - OS: Ubuntu 20.04
 - Python version (if relevant): 3.7.9
 - Are you using conda? Yes
 - If you are using conda, which channel did you install the rdkit from? conda-forge


Hints:

Fix patch:
diff --git a/Code/GraphMol/MolStandardize/Tautomer.cpp b/Code/GraphMol/MolStandardize/Tautomer.cpp
index 8a39a7f7b3c..8c36786a9b8 100644
--- a/Code/GraphMol/MolStandardize/Tautomer.cpp
+++ b/Code/GraphMol/MolStandardize/Tautomer.cpp
@@ -113,6 +113,7 @@ int scoreSubstructs(const ROMol &mol) {
       {"N=O", "[#7]=,:[#8]", 2},
       {"P=O", "[#15]=,:[#8]", 2},
       {"C=hetero", "[C]=[!#1;!#6]", 1},
+      {"C(=hetero)-hetero", "[C](=[!#1;!#6])[!#1;!#6]", 2},
       {"aromatic C = exocyclic N", "[c]=!@[N]", -1},
       {"methyl", "[CX4H3]", 1},
       {"guanidine terminal=N", "[#7]C(=[NR0])[#7H0]", 1},
@@ -279,7 +280,6 @@ TautomerEnumeratorResult TautomerEnumerator::enumerate(const ROMol &mol) const {
   RWMOL_SPTR kekulized(new RWMol(mol));
   MolOps::Kekulize(*kekulized, false);
   res.d_tautomers = {{smi, Tautomer(taut, kekulized, 0, 0)}};
-  std::set<std::string> done;
   res.d_modifiedAtoms.resize(mol.getNumAtoms());
   res.d_modifiedBonds.resize(mol.getNumBonds());
   bool completed = false;
@@ -292,184 +292,191 @@ TautomerEnumeratorResult TautomerEnumerator::enumerate(const ROMol &mol) const {
   while (!completed && !bailOut) {
     // std::map automatically sorts res.d_tautomers into alphabetical order
     // (SMILES)
-    for (const auto &tautomer : res.d_tautomers) {
+    for (auto &smilesTautomerPair : res.d_tautomers) {
 #ifdef VERBOSE_ENUMERATION
-      std::cout << "Done : " << std::endl;
-      for (const auto d : done) {
-        std::cout << d << std::endl;
+      std::cout << "Current tautomers: " << std::endl;
+      for (const auto &smilesTautomerPair : res.d_tautomers) {
+        std::cout << smilesTautomerPair.first << " done "
+                  << smilesTautomerPair.second.d_done << std::endl;
       }
-      std::cout << "Looking at tautomer: " << tautomer.first << std::endl;
 #endif
       std::string tsmiles;
-      if (done.count(tautomer.first)) {
+      if (smilesTautomerPair.second.d_done) {
+#ifdef VERBOSE_ENUMERATION
+        std::cout << "Skipping " << smilesTautomerPair.first << " as already done" << std::endl;
+#endif
         continue;
-      } else {
-        // done does not contain tautomer
-        for (const auto &transform : transforms) {
-          if (bailOut) {
-            break;
-          }
-          // kmol is the kekulized version of the tautomer
-          const auto &kmol = tautomer.second.kekulized;
-          std::vector<MatchVectType> matches;
-          unsigned int matched =
-              SubstructMatch(*kmol, *(transform.Mol), matches);
+      }
+#ifdef VERBOSE_ENUMERATION
+      std::cout << "Looking at tautomer: " << smilesTautomerPair.first
+                << std::endl;
+#endif
+      // tautomer not yet done
+      for (const auto &transform : transforms) {
+        if (bailOut) {
+          break;
+        }
+        // kmol is the kekulized version of the tautomer
+        const auto &kmol = smilesTautomerPair.second.kekulized;
+        std::vector<MatchVectType> matches;
+        unsigned int matched =
+            SubstructMatch(*kmol, *(transform.Mol), matches);
 
-          if (!matched) {
-            continue;
-          }
-          ++nTransforms;
+        if (!matched) {
+          continue;
+        }
+        ++nTransforms;
 #ifdef VERBOSE_ENUMERATION
-          std::string name;
-          (transform.Mol)->getProp(common_properties::_Name, name);
-          std::cout << "kmol: " << kmol->first << std::endl;
-          std::cout << MolToSmiles(*(kmol->second)) << std::endl;
-          std::cout << "transform mol: " << MolToSmarts(*(transform.Mol))
-                    << std::endl;
+        std::string name;
+        (transform.Mol)->getProp(common_properties::_Name, name);
+        std::cout << "kmol for " << smilesTautomerPair.first << " : " << MolToSmiles(*kmol) << std::endl;
+        std::cout << "transform mol: " << MolToSmarts(*(transform.Mol))
+                  << std::endl;
 
-          std::cout << "Matched: " << name << std::endl;
+        std::cout << "Matched: " << name << std::endl;
 #endif
-          // loop over transform matches
-          for (const auto &match : matches) {
-            if (nTransforms >= d_maxTransforms) {
-              res.d_status = TautomerEnumeratorStatus::MaxTransformsReached;
-              bailOut = true;
-            } else if (res.d_tautomers.size() >= d_maxTautomers) {
-              res.d_status = TautomerEnumeratorStatus::MaxTautomersReached;
-              bailOut = true;
-            } else if (d_callback.get() && !(*d_callback)(mol, res)) {
-              res.d_status = TautomerEnumeratorStatus::Canceled;
-              bailOut = true;
-            }
-            if (bailOut) {
-              break;
-            }
-            // Create a copy of in the input molecule so we can modify it
-            // Use kekule form so bonds are explicitly single/double instead of
-            // aromatic
-            ROMOL_SPTR product(new ROMol(*kmol));
-            // Remove a hydrogen from the first matched atom and add one to the
-            // last
-            int firstIdx = match.front().second;
-            int lastIdx = match.back().second;
-            Atom *first = product->getAtomWithIdx(firstIdx);
-            Atom *last = product->getAtomWithIdx(lastIdx);
-            res.d_modifiedAtoms.set(firstIdx);
-            res.d_modifiedAtoms.set(lastIdx);
-            first->setNumExplicitHs(
-                std::max(0, static_cast<int>(first->getTotalNumHs()) - 1));
-            last->setNumExplicitHs(last->getTotalNumHs() + 1);
-            // Remove any implicit hydrogens from the first and last atoms
-            // now we have set the count explicitly
-            first->setNoImplicit(true);
-            last->setNoImplicit(true);
-            // Adjust bond orders
-            unsigned int bi = 0;
-            for (size_t i = 0; i < match.size() - 1; ++i) {
-              Bond *bond = product->getBondBetweenAtoms(match[i].second,
-                                                        match[i + 1].second);
-              ASSERT_INVARIANT(bond, "required bond not found");
-              // check if bonds is specified in tautomer.in file
-              if (!transform.BondTypes.empty()) {
-                bond->setBondType(transform.BondTypes[bi]);
-                ++bi;
-              } else {
-                Bond::BondType bondtype = bond->getBondType();
+        // loop over transform matches
+        for (const auto &match : matches) {
+          if (nTransforms >= d_maxTransforms) {
+            res.d_status = TautomerEnumeratorStatus::MaxTransformsReached;
+            bailOut = true;
+          } else if (res.d_tautomers.size() >= d_maxTautomers) {
+            res.d_status = TautomerEnumeratorStatus::MaxTautomersReached;
+            bailOut = true;
+          } else if (d_callback.get() && !(*d_callback)(mol, res)) {
+            res.d_status = TautomerEnumeratorStatus::Canceled;
+            bailOut = true;
+          }
+          if (bailOut) {
+            break;
+          }
+          // Create a copy of in the input molecule so we can modify it
+          // Use kekule form so bonds are explicitly single/double instead of
+          // aromatic
+          RWMOL_SPTR product(new RWMol(*kmol));
+          // Remove a hydrogen from the first matched atom and add one to the
+          // last
+          int firstIdx = match.front().second;
+          int lastIdx = match.back().second;
+          Atom *first = product->getAtomWithIdx(firstIdx);
+          Atom *last = product->getAtomWithIdx(lastIdx);
+          res.d_modifiedAtoms.set(firstIdx);
+          res.d_modifiedAtoms.set(lastIdx);
+          first->setNumExplicitHs(
+              std::max(0, static_cast<int>(first->getTotalNumHs()) - 1));
+          last->setNumExplicitHs(last->getTotalNumHs() + 1);
+          // Remove any implicit hydrogens from the first and last atoms
+          // now we have set the count explicitly
+          first->setNoImplicit(true);
+          last->setNoImplicit(true);
+          // Adjust bond orders
+          unsigned int bi = 0;
+          for (size_t i = 0; i < match.size() - 1; ++i) {
+            Bond *bond = product->getBondBetweenAtoms(match[i].second,
+                                                      match[i + 1].second);
+            ASSERT_INVARIANT(bond, "required bond not found");
+            // check if bonds is specified in tautomer.in file
+            if (!transform.BondTypes.empty()) {
+              bond->setBondType(transform.BondTypes[bi]);
+              ++bi;
+            } else {
+              Bond::BondType bondtype = bond->getBondType();
 #ifdef VERBOSE_ENUMERATION
-                std::cout << "Bond as double: " << bond->getBondTypeAsDouble()
-                          << std::endl;
-                std::cout << bondtype << std::endl;
+              std::cout << "Bond as double: " << bond->getBondTypeAsDouble()
+                        << std::endl;
+              std::cout << bondtype << std::endl;
 #endif
-                if (bondtype == Bond::SINGLE) {
-                  bond->setBondType(Bond::DOUBLE);
+              if (bondtype == Bond::SINGLE) {
+                bond->setBondType(Bond::DOUBLE);
 #ifdef VERBOSE_ENUMERATION
-                  std::cout << "Set bond to double" << std::endl;
+                std::cout << "Set bond to double" << std::endl;
 #endif
-                }
-                if (bondtype == Bond::DOUBLE) {
-                  bond->setBondType(Bond::SINGLE);
+              }
+              if (bondtype == Bond::DOUBLE) {
+                bond->setBondType(Bond::SINGLE);
 #ifdef VERBOSE_ENUMERATION
-                  std::cout << "Set bond to single" << std::endl;
+                std::cout << "Set bond to single" << std::endl;
 #endif
-                }
               }
-              res.d_modifiedBonds.set(bond->getIdx());
             }
-            // TODO adjust charges
-            if (!transform.Charges.empty()) {
-              unsigned int ci = 0;
-              for (const auto &pair : match) {
-                Atom *atom = product->getAtomWithIdx(pair.second);
-                atom->setFormalCharge(atom->getFormalCharge() +
-                                      transform.Charges[ci++]);
-              }
+            res.d_modifiedBonds.set(bond->getIdx());
+          }
+          // TODO adjust charges
+          if (!transform.Charges.empty()) {
+            unsigned int ci = 0;
+            for (const auto &pair : match) {
+              Atom *atom = product->getAtomWithIdx(pair.second);
+              atom->setFormalCharge(atom->getFormalCharge() +
+                                    transform.Charges[ci++]);
             }
+          }
 
-            RWMOL_SPTR wproduct(new RWMol(*product));
+          unsigned int failedOp;
+          MolOps::sanitizeMol(*product, failedOp,
+                              MolOps::SANITIZE_KEKULIZE |
+                                  MolOps::SANITIZE_SETAROMATICITY |
+                                  MolOps::SANITIZE_SETCONJUGATION |
+                                  MolOps::SANITIZE_SETHYBRIDIZATION |
+                                  MolOps::SANITIZE_ADJUSTHS);
 #ifdef VERBOSE_ENUMERATION
-            std::cout << "pre-sanitization: "
-                      << MolToSmiles(*wproduct, true, true) << std::endl;
-#endif
-            unsigned int failedOp;
-            MolOps::sanitizeMol(*wproduct, failedOp,
-                                MolOps::SANITIZE_KEKULIZE |
-                                    MolOps::SANITIZE_SETAROMATICITY |
-                                    MolOps::SANITIZE_SETCONJUGATION |
-                                    MolOps::SANITIZE_SETHYBRIDIZATION |
-                                    MolOps::SANITIZE_ADJUSTHS);
-            setTautomerStereoAndIsoHs(mol, *wproduct, res);
-            tsmiles = MolToSmiles(*wproduct, true);
-#ifdef VERBOSE_ENUMERATION
-            std::string name;
-            (transform.Mol)->getProp(common_properties::_Name, name);
-            std::cout << "Applied rule: " << name << " to " << tautomer.first
-                      << std::endl;
+          std::cout << "pre-setTautomerStereo: " << MolToSmiles(*product, true)
+                    << std::endl;
 #endif
-            if (res.d_tautomers.find(tsmiles) != res.d_tautomers.end()) {
+          setTautomerStereoAndIsoHs(mol, *product, res);
+          tsmiles = MolToSmiles(*product, true);
 #ifdef VERBOSE_ENUMERATION
-              std::cout << "Previous tautomer produced again: " << tsmiles
-                        << std::endl;
+          (transform.Mol)->getProp(common_properties::_Name, name);
+          std::cout << "Applied rule: " << name << " to " << smilesTautomerPair.first
+                    << std::endl;
 #endif
-              continue;
-            }
+          if (res.d_tautomers.find(tsmiles) != res.d_tautomers.end()) {
 #ifdef VERBOSE_ENUMERATION
-            std::cout << "New tautomer produced: " << tsmiles << std::endl;
+            std::cout << "Previous tautomer produced again: " << tsmiles
+                      << std::endl;
 #endif
-            // in addition to the above transformations, sanitzation may modify
-            // bonds, e.g. Cc1nc2ccccc2[nH]1
-            for (size_t i = 0; i < mol.getNumBonds(); i++) {
-              auto molBondType = mol.getBondWithIdx(i)->getBondType();
-              auto tautBondType = wproduct->getBondWithIdx(i)->getBondType();
-              if (molBondType != tautBondType && !res.d_modifiedBonds.test(i)) {
+            continue;
+          }
+          // in addition to the above transformations, sanitization may modify
+          // bonds, e.g. Cc1nc2ccccc2[nH]1
+          for (size_t i = 0; i < mol.getNumBonds(); i++) {
+            auto molBondType = mol.getBondWithIdx(i)->getBondType();
+            auto tautBondType = product->getBondWithIdx(i)->getBondType();
+            if (molBondType != tautBondType && !res.d_modifiedBonds.test(i)) {
 #ifdef VERBOSE_ENUMERATION
-                std::cout << "Sanitization has modified bond " << i
-                          << std::endl;
+              std::cout << "Sanitization has modified bond " << i
+                        << std::endl;
 #endif
-                res.d_modifiedBonds.set(i);
-              }
+              res.d_modifiedBonds.set(i);
             }
-            RWMOL_SPTR kekulized_product(new RWMol(*wproduct));
-            MolOps::Kekulize(*kekulized_product, false);
-            res.d_tautomers[tsmiles] = Tautomer(
-                std::move(wproduct), std::move(kekulized_product),
-                res.d_modifiedAtoms.count(), res.d_modifiedBonds.count());
-
+          }
+          RWMOL_SPTR kekulized_product(new RWMol(*product));
+          MolOps::Kekulize(*kekulized_product, false);
 #ifdef VERBOSE_ENUMERATION
-            std::cout << "Now completed: " << std::endl;
-            for (const auto &tautomer : res.d_tautomers) {
-              std::cout << tautomer.first << std::endl;
-            }
-#endif
+          auto it = res.d_tautomers.find(tsmiles);
+          if (it == res.d_tautomers.end()) {
+            std::cout << "New tautomer added as ";
+          } else {
+            std::cout << "New tautomer replaced for ";
           }
+          std::cout << tsmiles << ", taut: " << MolToSmiles(*product)
+                    << ", kek: " << MolToSmiles(*kekulized_product, true, true)
+                    << std::endl;
+#endif
+          res.d_tautomers[tsmiles] = Tautomer(
+              std::move(product), std::move(kekulized_product),
+              res.d_modifiedAtoms.count(), res.d_modifiedBonds.count());
         }
       }
-      done.insert(tautomer.first);
+      smilesTautomerPair.second.d_done = true;
     }
-    completed = (res.d_tautomers.size() <= done.size());
+    completed = true;
     size_t maxNumModifiedAtoms = res.d_modifiedAtoms.count();
     size_t maxNumModifiedBonds = res.d_modifiedBonds.count();
     for (auto it = res.d_tautomers.begin(); it != res.d_tautomers.end();) {
       auto &taut = it->second;
+      if (!taut.d_done) {
+        completed = false;
+      }
       if ((taut.d_numModifiedAtoms < maxNumModifiedAtoms ||
            taut.d_numModifiedBonds < maxNumModifiedBonds) &&
           setTautomerStereoAndIsoHs(mol, *taut.tautomer, res)) {
diff --git a/Code/GraphMol/MolStandardize/Tautomer.h b/Code/GraphMol/MolStandardize/Tautomer.h
index bf589f05060..ea6226c6b62 100644
--- a/Code/GraphMol/MolStandardize/Tautomer.h
+++ b/Code/GraphMol/MolStandardize/Tautomer.h
@@ -54,18 +54,20 @@ class Tautomer {
   friend class TautomerEnumerator;
 
  public:
-  Tautomer() : d_numModifiedAtoms(0), d_numModifiedBonds(0) {}
+  Tautomer() : d_numModifiedAtoms(0), d_numModifiedBonds(0), d_done(false) {}
   Tautomer(const ROMOL_SPTR &t, const ROMOL_SPTR &k, size_t a = 0, size_t b = 0)
       : tautomer(t),
         kekulized(k),
         d_numModifiedAtoms(a),
-        d_numModifiedBonds(b) {}
+        d_numModifiedBonds(b),
+        d_done(false) {}
   ROMOL_SPTR tautomer;
   ROMOL_SPTR kekulized;
 
  private:
   size_t d_numModifiedAtoms;
   size_t d_numModifiedBonds;
+  bool d_done;
 };
 
 typedef std::map<std::string, Tautomer> SmilesTautomerMap;


Test patch:
diff --git a/Code/GraphMol/MolStandardize/testTautomer.cpp b/Code/GraphMol/MolStandardize/testTautomer.cpp
index 85b100abfa3..a4c9213a4ca 100644
--- a/Code/GraphMol/MolStandardize/testTautomer.cpp
+++ b/Code/GraphMol/MolStandardize/testTautomer.cpp
@@ -1416,6 +1416,24 @@ void testGithub3430() {
   }
 }
 
+void testGithub3755() {
+  BOOST_LOG(rdInfoLog) << "-----------------------\n testGithub3755"
+                       << std::endl;
+  // hydrates, aminals and hemiaminals should be scored lower than
+  // carboxylic acids, amides, amidines, and guanidines
+  std::vector<std::pair<std::string, std::string>> orig_vs_expected{
+      {"OC(=O)C(N)CO", "NC(CO)C(=O)O"}, {"C([C@@H](C(=O)O)N)O", "NC(CO)C(=O)O"},
+      {"OC(=O)C(N)CN", "NCC(N)C(=O)O"}, {"NC(=O)C(N)CO", "NC(=O)C(N)CO"},
+      {"NC(=N)C(N)CO", "N=C(N)C(N)CO"}, {"NC(=N)NC(N)CO", "N=C(N)NC(N)CO"}};
+  TautomerEnumerator te;
+  for (const auto &pair : orig_vs_expected) {
+    ROMOL_SPTR orig(SmilesToMol(pair.first));
+    TEST_ASSERT(orig);
+    ROMOL_SPTR canonical(te.canonicalize(*orig));
+    TEST_ASSERT(MolToSmiles(*canonical) == pair.second);
+  }
+}
+
 int main() {
   RDLog::InitLogs();
 #if 1
@@ -1433,5 +1451,6 @@ int main() {
   testPickCanonicalCIPChangeOnChiralCenter();
   testTautomerEnumeratorResult_const_iterator();
   testGithub3430();
+  testGithub3755();
   return 0;
 }