Repository: openmm/openmm — pull request #4956 (cpp)
Title: Handle relative include paths from force fields in data directories

For `<Include>` in force field files, the user manual states that

> The file attribute gives the path of the file to include. It may be relative either to the directory containing the parent XML file (the one with the <Include> tag) or the OpenMM data directory (the one containing built in force fields).

However, this doesn't work if the including file is in a subdirectory of a data directory and points to the included file in that subdirectory using a relative path, *e.g.*, if `data/a/b.xml` includes `data/a/c.xml`, `<Include file="c.xml" />` doesn't work, so `<Include file="a/c.xml" />` is needed.  It seems like the test https://github.com/openmm/openmm/blob/2b8ad70394f38d296497215da757aa31ac36b87b/wrappers/python/openmm/app/forcefield.py#L283-L285 doesn't work here since `parentDir` is that of `file`, but if the including file is in a data directory, `f` ends up being its path instead.

This change ensures that `file` will contain a resolved path to the force field if it is in some data directory, so the relative path lookup will find files correctly.  This fixes an issue with some openmmforcefields files that rely on it working.  It also allows more cleanly reporting errors in loading force fields in data directories, since we don't actually always control such files (any package can register an entry point).

Does this behavior make sense?  If not, and includes from data directories aren't supposed to be able to do relative path lookups, I will make changes in openmmforcefields instead.  (This shouldn't affect any of the OpenMM-distributed force fields.)

Resolved issues:
title: Handle relative include paths from force fields in data directories
body: ### Issue Description

**Problem:**  
Relative file paths specified in `<Include>` tags within force field XML files do not behave as described in the user manual when the including file is located in a subdirectory of a registered data directory. Specifically, if a file in a subdirectory includes another file from the same subdirectory using a relative path, the inclusion fails unless the path is explicitly adjusted to include the subdirectory. This discrepancy causes issues for force field files that rely on relative path functionality as documented.

**Expected Behavior:**  
Relative paths in `<Include>` tags should work as described in the user manual, allowing files in subdirectories of data directories to include other files from the same subdirectory without requiring explicit adjustments to the path. This ensures consistency with the documented behavior and improves compatibility with force field files that depend on this functionality.

## IMPORTANT INSTRUCTIONS

1. If you edit a python file, you MUST run 'make PythonInstall' inside the build directory
2. If you edit a c++ file, you MUST run 'make -j4 && make install" inside the build directory

Hints:

Fix patch:
diff --git a/wrappers/python/openmm/app/forcefield.py b/wrappers/python/openmm/app/forcefield.py
index e470a5fd5f..9a7a7021af 100644
--- a/wrappers/python/openmm/app/forcefield.py
+++ b/wrappers/python/openmm/app/forcefield.py
@@ -245,29 +245,24 @@ def loadFile(self, files, resname_prefix=''):
         i = 0
         while i < len(files):
             file = files[i]
-            tree = None
-            try:
-                # this handles either filenames or open file-like objects
-                tree = etree.parse(file)
-            except IOError:
+            # this handles either filenames or open file-like objects
+            if isinstance(file, str) and not os.path.isfile(file):
                 for dataDir in _getDataDirectories():
                     f = os.path.join(dataDir, file)
                     if os.path.isfile(f):
-                        tree = etree.parse(f)
+                        file = f
                         break
+            try:
+                tree = etree.parse(file)
+            except FileNotFoundError:
+                raise ValueError('Could not locate file "%s"' % file)
             except Exception as e:
                 # Fail with an error message about which file could not be read.
-                # TODO: Also handle case where fallback to 'data' directory encounters problems,
-                # but this is much less worrisome because we control those files.
-                msg  = str(e) + '\n'
                 if hasattr(file, 'name'):
                     filename = file.name
                 else:
                     filename = str(file)
-                msg += "ForceField.loadFile() encountered an error reading file '%s'\n" % filename
-                raise Exception(msg)
-            if tree is None:
-                raise ValueError('Could not locate file "%s"' % file)
+                raise Exception('ForceField.loadFile() encountered an error reading file "%s": %s' % (filename, e))
 
             trees.append(tree)
             i += 1


Test patch:
diff --git a/wrappers/python/tests/TestForceField.py b/wrappers/python/tests/TestForceField.py
index 009811807f..fe32e92def 100644
--- a/wrappers/python/tests/TestForceField.py
+++ b/wrappers/python/tests/TestForceField.py
@@ -6,6 +6,8 @@
 import openmm.app.element as elem
 import openmm.app.forcefield as forcefield
 import math
+import shutil
+import tempfile
 import textwrap
 try:
     from cStringIO import StringIO
@@ -1254,6 +1256,23 @@ def test_Includes(self):
         self.assertTrue('spce-O' in forcefield._atomTypes)
         self.assertTrue('HOH' in forcefield._templates)
 
+    def test_IncludesFromDataDirectory(self):
+        """Test relative include paths from subdirectories of the data directory."""
+
+        oldDataDirs = forcefield._dataDirectories
+        try:
+            with tempfile.TemporaryDirectory() as tempDataDir:
+                forcefield._dataDirectories = forcefield._getDataDirectories() + [tempDataDir]
+                os.mkdir(os.path.join(tempDataDir, 'subdir'))
+                for testFileName in ['ff_with_includes.xml', 'test_amber_ff.xml']:
+                    shutil.copyfile(os.path.join('systems', testFileName), os.path.join(tempDataDir, 'subdir', testFileName))
+                ff = ForceField(os.path.join('subdir', 'ff_with_includes.xml'))
+                self.assertTrue(len(ff._atomTypes) > 10)
+                self.assertTrue('spce-O' in ff._atomTypes)
+                self.assertTrue('HOH' in ff._templates)
+        finally:
+            forcefield._dataDirectories = oldDataDirs
+
     def test_ImpropersOrdering(self):
         """Test correctness of the ordering of atom indexes in improper torsions
         and the torsion.ordering parameter.