CVE-2023-6507 (published 2023-12-08T19:15Z)

An issue was found in CPython 3.12.0 `subprocess` module on POSIX platforms. The issue was fixed in CPython 3.12.1 and does not affect other stable releases.\n\nWhen using the `extra_groups=` parameter with an empty list as a value (ie `extra_groups=[]`) the logic regressed to not call `setgroups(0, NULL)` before calling `exec()`, thus not dropping the original processes' groups before starting the new process. There is no issue when the parameter isn't used or when any value is used besides an empty list.\n\nThis issue only impacts CPython processes run with sufficient privilege to make the `setgroups` system call (typically `root`).\n\n

CVSS v3 base score: 4.9, MEDIUM (CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:N/I:H/A:N)

Weakness information:
cwe_id: NVD-CWE-noinfo
cwe_name: Insufficient Information
description: There is insufficient information about the issue to classify it; details are unkown or unspecified.
extended_description: Insufficient Information

Fix information:
repo_url: https://github.com/python/cpython
commit_details: repo_url: https://github.com/python/cpython
author: Gregory P. Smith
author_date: 2023-12-04 15:08:19-08:00
msg: gh-112334: Restore subprocess's use of `vfork()` & fix `extra_groups=[]` behavior (#112617)

Restore `subprocess`'s intended use of `vfork()` by default for performance on Linux;
also fixes the behavior of `extra_groups=[]` which was unintentionally broken in 3.12.0:

Fixed a performance regression in 3.12's :mod:`subprocess` on Linux where it
would no longer use the fast-path ``vfork()`` system call when it could have
due to a logic bug, instead falling back to the safe but slower ``fork()``.

Also fixed a security bug introduced in 3.12.0.  If a value of ``extra_groups=[]``
was passed to :mod:`subprocess.Popen` or related APIs, the underlying
``setgroups(0, NULL)`` system call to clear the groups list would not be made
in the child process prior to ``exec()``.

The security issue was identified via code inspection in the process of
fixing the first bug.  Thanks to @vain for the detailed report and
analysis in the initial bug on Github.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
repository: repo_url: https://github.com/python/cpython
repo_name: python/cpython
description: The Python programming language
repo_language: Python
owner: python
file_changes: filename: test_subprocess.py
change_type: ModificationType.MODIFY
diff: @@ -2066,8 +2066,14 @@ def test_group_error(self):
     def test_extra_groups(self):
         gid = os.getegid()
         group_list = [65534 if gid != 65534 else 65533]
+        self._test_extra_groups_impl(gid=gid, group_list=group_list)
+
+    @unittest.skipUnless(hasattr(os, 'setgroups'), 'no setgroups() on platform')
+    def test_extra_groups_empty_list(self):
+        self._test_extra_groups_impl(gid=os.getegid(), group_list=[])
+
+    def _test_extra_groups_impl(self, *, gid, group_list):
         name_group = _get_test_grp_name()
-        perm_error = False
 
         if grp is not None:
             group_list.append(name_group)
@@ -2077,11 +2083,8 @@ def test_extra_groups(self):
                     [sys.executable, "-c",
                      "import os, sys, json; json.dump(os.getgroups(), sys.stdout)"],
                     extra_groups=group_list)
-        except OSError as ex:
-            if ex.errno != errno.EPERM:
-                raise
-            perm_error = True
-
+        except PermissionError:
+            self.skipTest("setgroup() EPERM; this test may require root.")
         else:
             parent_groups = os.getgroups()
             child_groups = json.loads(output)
@@ -2092,12 +2095,15 @@ def test_extra_groups(self):
             else:
                 desired_gids = group_list
 
-            if perm_error:
-                self.assertEqual(set(child_groups), set(parent_groups))
-            else:
-                self.assertEqual(set(desired_gids), set(child_groups))
+            self.assertEqual(set(desired_gids), set(child_groups))
 
-        # make sure we bomb on negative values
+        if grp is None:
+            with self.assertRaises(ValueError):
+                subprocess.check_call(ZERO_RETURN_CMD,
+                                      extra_groups=[name_group])
+
+    # No skip necessary, this test won't make it to a setgroup() call.
+    def test_extra_groups_invalid_gid_t_values(self):
         with self.assertRaises(ValueError):
             subprocess.check_call(ZERO_RETURN_CMD, extra_groups=[-1])
 
@@ -2106,16 +2112,6 @@ def test_extra_groups(self):
                                   cwd=os.curdir, env=os.environ,
                                   extra_groups=[2**64])
 
-        if grp is None:
-            with self.assertRaises(ValueError):
-                subprocess.check_call(ZERO_RETURN_CMD,
-                                      extra_groups=[name_group])
-
-    @unittest.skipIf(hasattr(os, 'setgroups'), 'setgroups() available on platform')
-    def test_extra_groups_error(self):
-        with self.assertRaises(ValueError):
-            subprocess.check_call(ZERO_RETURN_CMD, extra_groups=[])
-
     @unittest.skipIf(mswindows or not hasattr(os, 'umask'),
                      'POSIX umask() is not available.')
     def test_umask(self):

programming_language: Python
filename: 2023-12-01-21-05-46.gh-issue-112334.DmNXKh.rst
change_type: ModificationType.ADD
diff: @@ -0,0 +1,11 @@
+Fixed a performance regression in 3.12's :mod:`subprocess` on Linux where it
+would no longer use the fast-path ``vfork()`` system call when it could have
+due to a logic bug, instead falling back to the safe but slower ``fork()``.
+
+Also fixed a second 3.12.0 potential security bug.  If a value of
+``extra_groups=[]`` was passed to :mod:`subprocess.Popen` or related APIs,
+the underlying ``setgroups(0, NULL)`` system call to clear the groups list
+would not be made in the child process prior to ``exec()``.
+
+This was identified via code inspection in the process of fixing the first
+bug.

programming_language: Markdown
filename: _posixsubprocess.c
change_type: ModificationType.MODIFY
diff: @@ -767,8 +767,10 @@ child_exec(char *const exec_array[],
 #endif
 
 #ifdef HAVE_SETGROUPS
-    if (extra_group_size > 0)
+    if (extra_group_size >= 0) {
+        assert((extra_group_size == 0) == (extra_groups == NULL));
         POSIX_CALL(setgroups(extra_group_size, extra_groups));
+    }
 #endif /* HAVE_SETGROUPS */
 
 #ifdef HAVE_SETREGID
@@ -1022,7 +1024,6 @@ subprocess_fork_exec_impl(PyObject *module, PyObject *process_args,
     pid_t pid = -1;
     int need_to_reenable_gc = 0;
     char *const *argv = NULL, *const *envp = NULL;
-    Py_ssize_t extra_group_size = 0;
     int need_after_fork = 0;
     int saved_errno = 0;
     int *c_fds_to_keep = NULL;
@@ -1103,6 +1104,13 @@ subprocess_fork_exec_impl(PyObject *module, PyObject *process_args,
         cwd = PyBytes_AsString(cwd_obj2);
     }
 
+    // Special initial value meaning that subprocess API was called with
+    // extra_groups=None leading to _posixsubprocess.fork_exec(gids=None).
+    // We use this to differentiate between code desiring a setgroups(0, NULL)
+    // call vs no call at all.  The fast vfork() code path could be used when
+    // there is no setgroups call.
+    Py_ssize_t extra_group_size = -2;
+
     if (extra_groups_packed != Py_None) {
 #ifdef HAVE_SETGROUPS
         if (!PyList_Check(extra_groups_packed)) {

programming_language: C
repo_url: https://github.com/python/cpython
commit_details: repo_url: https://github.com/python/cpython
author: Gregory P. Smith
author_date: 2023-12-08 16:18:35-08:00
msg: gh-112334: Regression test that vfork is used when expected. (#112734)

Regression test that vfork is used when expected by subprocess.

This is written integration test style, it uses strace if it is present and appears to work to find out what system call actually gets used in different scenarios.

Test coverage is added for the default behavior and that of each of the specific arguments that must disable the use of vfork.  obviously not an entire test matrix, but it covers the most important aspects.

If there are ever issues with this test being flaky or failing on new platforms, rather than try and adapt it for all possible platforms, feel free to narrow the range it gets tested on when appropriate. That is not likely to reduce coverage.
repository: repo_url: https://github.com/python/cpython
repo_name: python/cpython
description: The Python programming language
repo_language: Python
owner: python
file_changes: filename: posix-deps-apt.sh
change_type: ModificationType.MODIFY
diff: @@ -21,6 +21,7 @@ apt-get -yq install \
     libssl-dev \
     lzma \
     lzma-dev \
+    strace \
     tk-dev \
     uuid-dev \
     xvfb \

programming_language: Shell
filename: script_helper.py
change_type: ModificationType.MODIFY
diff: @@ -92,13 +92,28 @@ def fail(self, cmd_line):
 # Executing the interpreter in a subprocess
 @support.requires_subprocess()
 def run_python_until_end(*args, **env_vars):
+    """Used to implement assert_python_*.
+
+    *args are the command line flags to pass to the python interpreter.
+    **env_vars keyword arguments are environment variables to set on the process.
+
+    If __run_using_command= is supplied, it must be a list of
+    command line arguments to prepend to the command line used.
+    Useful when you want to run another command that should launch the
+    python interpreter via its own arguments. ["/bin/echo", "--"] for
+    example could print the unquoted python command line instead of
+    run it.
+    """
     env_required = interpreter_requires_environment()
+    run_using_command = env_vars.pop('__run_using_command', None)
     cwd = env_vars.pop('__cwd', None)
     if '__isolated' in env_vars:
         isolated = env_vars.pop('__isolated')
     else:
         isolated = not env_vars and not env_required
     cmd_line = [sys.executable, '-X', 'faulthandler']
+    if run_using_command:
+        cmd_line = run_using_command + cmd_line
     if isolated:
         # isolated mode: ignore Python environment variables, ignore user
         # site-packages, and don't add the current directory to sys.path

programming_language: Python
filename: test_subprocess.py
change_type: ModificationType.MODIFY
diff: @@ -1561,21 +1561,6 @@ def test_class_getitems(self):
         self.assertIsInstance(subprocess.Popen[bytes], types.GenericAlias)
         self.assertIsInstance(subprocess.CompletedProcess[str], types.GenericAlias)
 
-    @unittest.skipIf(not sysconfig.get_config_var("HAVE_VFORK"),
-                     "vfork() not enabled by configure.")
-    @mock.patch("subprocess._fork_exec")
-    def test__use_vfork(self, mock_fork_exec):
-        self.assertTrue(subprocess._USE_VFORK)  # The default value regardless.
-        mock_fork_exec.side_effect = RuntimeError("just testing args")
-        with self.assertRaises(RuntimeError):
-            subprocess.run([sys.executable, "-c", "pass"])
-        mock_fork_exec.assert_called_once()
-        self.assertTrue(mock_fork_exec.call_args.args[-1])
-        with mock.patch.object(subprocess, '_USE_VFORK', False):
-            with self.assertRaises(RuntimeError):
-                subprocess.run([sys.executable, "-c", "pass"])
-            self.assertFalse(mock_fork_exec.call_args_list[-1].args[-1])
-
 
 class RunFuncTestCase(BaseTestCase):
     def run_python(self, code, **kwargs):
@@ -3360,6 +3345,89 @@ def exit_handler():
         self.assertEqual(out, b'')
         self.assertIn(b"preexec_fn not supported at interpreter shutdown", err)
 
+    @unittest.skipIf(not sysconfig.get_config_var("HAVE_VFORK"),
+                     "vfork() not enabled by configure.")
+    @mock.patch("subprocess._fork_exec")
+    def test__use_vfork(self, mock_fork_exec):
+        self.assertTrue(subprocess._USE_VFORK)  # The default value regardless.
+        mock_fork_exec.side_effect = RuntimeError("just testing args")
+        with self.assertRaises(RuntimeError):
+            subprocess.run([sys.executable, "-c", "pass"])
+        mock_fork_exec.assert_called_once()
+        # NOTE: These assertions are *ugly* as they require the last arg
+        # to remain the have_vfork boolean. We really need to refactor away
+        # from the giant "wall of args" internal C extension API.
+        self.assertTrue(mock_fork_exec.call_args.args[-1])
+        with mock.patch.object(subprocess, '_USE_VFORK', False):
+            with self.assertRaises(RuntimeError):
+                subprocess.run([sys.executable, "-c", "pass"])
+            self.assertFalse(mock_fork_exec.call_args_list[-1].args[-1])
+
+    @unittest.skipIf(not sysconfig.get_config_var("HAVE_VFORK"),
+                     "vfork() not enabled by configure.")
+    @unittest.skipIf(sys.platform != "linux", "Linux only, requires strace.")
+    def test_vfork_used_when_expected(self):
+        # This is a performance regression test to ensure we default to using
+        # vfork() when possible.
+        strace_binary = "/usr/bin/strace"
+        # The only system calls we are interested in.
+        strace_filter = "--trace=clone,clone2,clone3,fork,vfork,exit,exit_group"
+        true_binary = "/bin/true"
+        strace_command = [strace_binary, strace_filter]
+
+        try:
+            does_strace_work_process = subprocess.run(
+                    strace_command + [true_binary],
+                    stderr=subprocess.PIPE,
+                    stdout=subprocess.DEVNULL,
+            )
+            rc = does_strace_work_process.returncode
+            stderr = does_strace_work_process.stderr
+        except OSError:
+            rc = -1
+            stderr = ""
+        if rc or (b"+++ exited with 0 +++" not in stderr):
+            self.skipTest("strace not found or not working as expected.")
+
+        with self.subTest(name="default_is_vfork"):
+            vfork_result = assert_python_ok(
+                    "-c",
+                    textwrap.dedent(f"""\
+                    import subprocess
+                    subprocess.check_call([{true_binary!r}])"""),
+                    __run_using_command=strace_command,
+            )
+            # Match both vfork() and clone(..., flags=...|CLONE_VFORK|...)
+            self.assertRegex(vfork_result.err, br"(?i)vfork")
+            # Do NOT check that fork() or other clones did not happen.
+            # If the OS denys the vfork it'll fallback to plain fork().
+
+        # Test that each individual thing that would disable the use of vfork
+        # actually disables it.
+        for sub_name, preamble, sp_kwarg, expect_permission_error in (
+                ("!use_vfork", "subprocess._USE_VFORK = False", "", False),
+                ("preexec", "", "preexec_fn=lambda: None", False),
+                ("setgid", "", f"group={os.getgid()}", True),
+                ("setuid", "", f"user={os.getuid()}", True),
+                ("setgroups", "", "extra_groups=[]", True),
+        ):
+            with self.subTest(name=sub_name):
+                non_vfork_result = assert_python_ok(
+                    "-c",
+                    textwrap.dedent(f"""\
+                    import subprocess
+                    {preamble}
+                    try:
+                        subprocess.check_call(
+                                [{true_binary!r}], **dict({sp_kwarg}))
+                    except PermissionError:
+                        if not {expect_permission_error}:
+                            raise"""),
+                    __run_using_command=strace_command,
+                )
+                # Ensure neither vfork() or clone(..., flags=...|CLONE_VFORK|...).
+                self.assertNotRegex(non_vfork_result.err, br"(?i)vfork")
+
 
 @unittest.skipUnless(mswindows, "Windows specific tests")
 class Win32ProcessTestCase(BaseTestCase):

programming_language: Python
filename: 2023-12-04-15-56-11.gh-issue-112334.FFc9Ti.rst
change_type: ModificationType.ADD
diff: @@ -0,0 +1,2 @@
+Adds a regression test to verify that ``vfork()`` is used when expected by
+:mod:`subprocess` on vfork enabled POSIX systems (Linux).

programming_language: SQL