CVE-2023-34457 (published 2023-07-05T20:15Z)

MechanicalSoup is a Python library for automating interaction with websites. Starting in version 0.2.0 and prior to version 1.3.0, a malicious web server can read arbitrary files on the client using a `<input type="file" ...>` inside HTML form. All users of MechanicalSoup\'s form submission are affected, unless they took very specific (and manual) steps to reset HTML form field values. Version 1.3.0 contains a patch for this issue.

CVSS v3 base score: 7.5, HIGH (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N)

Weakness information:
cwe_id: CWE-20
cwe_name: Improper Input Validation
description: The product receives input or data, but it does
        not validate or incorrectly validates that the input has the
        properties that are required to process the data safely and
        correctly.
extended_description: 
	   Input validation is a frequently-used technique
	   for checking potentially dangerous inputs in order to
	   ensure that the inputs are safe for processing within the
	   code, or when communicating with other components.  When
	   software does not validate input properly, an attacker is
	   able to craft the input in a form that is not expected by
	   the rest of the application. This will lead to parts of the
	   system receiving unintended input, which may result in
	   altered control flow, arbitrary control of a resource, or
	   arbitrary code execution.
	   Input validation is not the only technique for
	   processing input, however.  Other techniques attempt to
	   transform potentially-dangerous input into something safe, such
	   as filtering (CWE-790) - which attempts to remove dangerous
	   inputs - or encoding/escaping (CWE-116), which attempts to
	   ensure that the input is not misinterpreted when it is included
	   in output to another component. Other techniques exist as well
	   (see CWE-138 for more examples.)
	   Input validation can be applied to:
	   
	     raw data - strings, numbers, parameters, file contents, etc.
	     metadata - information about the raw data, such as headers or size
	   
	   Data can be simple or structured.  Structured data
	   can be composed of many nested layers, composed of
	   combinations of metadata and raw data, with other simple or
	   structured data.
	   Many properties of raw data or metadata may need
	   to be validated upon entry into the code, such
	   as:
	   
	     specified quantities such as size, length, frequency, price, rate, number of operations, time, etc.
	     implied or derived quantities, such as the actual size of a file instead of a specified size
	     indexes, offsets, or positions into more complex data structures
	     symbolic keys or other elements into hash tables, associative arrays, etc.
	     well-formedness, i.e. syntactic correctness - compliance with expected syntax 
	     lexical token correctness - compliance with rules for what is treated as a token
	     specified or derived type - the actual type of the input (or what the input appears to be)
	     consistency - between individual data elements, between raw data and metadata, between references, etc.
	     conformance to domain-specific rules, e.g. business logic 
	     equivalence - ensuring that equivalent inputs are treated the same
	     authenticity, ownership, or other attestations about the input, e.g. a cryptographic signature to prove the source of the data
	   
	   Implied or derived properties of data must often
	   be calculated or inferred by the code itself.  Errors in
	   deriving properties may be considered a contributing factor
	   to improper input validation.
	   
	   Note that "input validation" has very different
	   meanings to different people, or within different
	   classification schemes.  Caution must be used when
	   referencing this CWE entry or mapping to it.  For example,
	   some weaknesses might involve inadvertently giving control
	   to an attacker over an input when they should not be able
	   to provide an input at all, but sometimes this is referred
	   to as input validation.
	   Finally, it is important to emphasize that the
	   distinctions between input validation and output escaping
	   are often blurred, and developers must be careful to
	   understand the difference, including how input validation
	   is not always sufficient to prevent vulnerabilities,
	   especially when less stringent data types must be
	   supported, such as free-form text. Consider a SQL injection
	   scenario in which a person's last name is inserted into a
	   query. The name "O'Reilly" would likely pass the validation
	   step since it is a common last name in the English
	   language. However, this valid name cannot be directly
	   inserted into the database because it contains the "'"
	   apostrophe character, which would need to be escaped or
	   otherwise transformed. In this case, removing the
	   apostrophe might reduce the risk of SQL injection, but it
	   would produce incorrect behavior because the wrong name
	   would be recorded.
       
         

Fix information:
repo_url: https://github.com/MechanicalSoup/MechanicalSoup
commit_details: repo_url: https://github.com/MechanicalSoup/MechanicalSoup
author: Eduard Christian Dumitrescu
author_date: 2023-07-04 14:47:44-04:00
msg: Merge pull request from GHSA-x456-3ccm-m6j4

* Add suggested mitigation against malicious HTML form file input. Test cases and docs not updated.

* Make tests pass. Add test specifically for this vulnerability. Fix the `Form.new_control()` method so that it correctly sets the input value if the value is a file object.

* Add better documentation about the vulnerability remediation. Wording fix.

* tests: Add separate test to check for CVE-2023-34457.

* Revert "tests: Add separate test to check for CVE-2023-34457."

This reverts commit bd4c6d92a8a803499a447b511bc46b1ba00841d0.

* Misc cleanup

* No need to raise the error on forms with an enctype that do not
  upload files (e.g. everything besides multipart).

* Make sure that open file inputs are converted to their basename
  regardless of what kind of form it is to avoid leaking local file
  paths.

* Throw error at time when the file input is set incorrectly, rather
  than at submission time.

* Factor complex repeated logic into their own functions (i.e. for
  the multipart file input check and invalid file value check).

* Fix failing tests.

* Avoid excessive whitespace in exception message.

* Reorder 'and' operands so the slightly less expensive check comes first.

---------

Co-authored-by: Dan Hemberger <daniel.hemberger@gmail.com>
repository: repo_url: https://github.com/MechanicalSoup/MechanicalSoup
repo_name: MechanicalSoup/MechanicalSoup
description: A Python library for automating interaction with websites.
repo_language: Python
owner: MechanicalSoup
file_changes: filename: browser.py
change_type: ModificationType.MODIFY
diff: @@ -1,3 +1,4 @@
+import io
 import os
 import tempfile
 import urllib
@@ -10,7 +11,7 @@ import requests
 
 from .__version__ import __title__, __version__
 from .form import Form
-from .utils import LinkNotFoundError
+from .utils import LinkNotFoundError, is_multipart_file_upload
 
 
 class Browser:
@@ -228,18 +229,20 @@ class Browser:
 
                 # If the enctype is not multipart, the filename is put in
                 # the form as a text input and the file is not sent.
-                if tag.get("type", "").lower() == "file" and multipart:
-                    filepath = value
-                    if filepath != "" and isinstance(filepath, str):
-                        content = open(filepath, "rb")
+                if is_multipart_file_upload(form, tag):
+                    if isinstance(value, io.IOBase):
+                        content = value
+                        filename = os.path.basename(getattr(value, "name", ""))
                     else:
                         content = ""
-                    filename = os.path.basename(filepath)
-                    # If value is the empty string, we still pass it
+                        filename = os.path.basename(value)
+                    # If content is the empty string, we still pass it
                     # for consistency with browsers (see
                     # https://github.com/MechanicalSoup/MechanicalSoup/issues/250).
                     files[name] = (filename, content)
                 else:
+                    if isinstance(value, io.IOBase):
+                        value = os.path.basename(getattr(value, "name", ""))
                     data.append((name, value))
 
             elif tag.name == "button":

programming_language: Python
filename: form.py
change_type: ModificationType.MODIFY
diff: @@ -1,9 +1,10 @@
 import copy
+import io
 import warnings
 
 from bs4 import BeautifulSoup
 
-from .utils import LinkNotFoundError
+from .utils import LinkNotFoundError, is_multipart_file_upload
 
 
 class InvalidFormMethod(LinkNotFoundError):
@@ -68,6 +69,7 @@ class Form:
             i = self.form.find("input", {"name": name})
             if not i:
                 raise InvalidFormMethod("No input field named " + name)
+            self._assert_valid_file_upload(i, value)
             i["value"] = value
 
     def uncheck_all(self, name):
@@ -261,12 +263,12 @@ class Form:
             form.set("eula-checkbox", True)
 
         Example: uploading a file through a ``<input type="file"
-        name="tagname">`` field (provide the path to the local file,
+        name="tagname">`` field (provide an open file object,
         and its content will be uploaded):
 
         .. code-block:: python
 
-            form.set("tagname", path_to_local_file)
+            form.set("tagname", open(path_to_local_file, "rb"))
 
         """
         for func in ("checkbox", "radio", "input", "textarea", "select"):
@@ -300,6 +302,7 @@ class Form:
         control['value'] = value
         for k, v in kwargs.items():
             control[k] = v
+        self._assert_valid_file_upload(control, value)
         self.form.append(control)
         return control
 
@@ -383,3 +386,17 @@ class Form:
                 if subtag.string:
                     subtag.string = subtag.string.strip()
             print(input_copy)
+
+    def _assert_valid_file_upload(self, tag, value):
+        """Raise an exception if a multipart file input is not an open file."""
+        if (
+            is_multipart_file_upload(self.form, tag) and
+            not isinstance(value, io.IOBase)
+        ):
+            raise ValueError(
+                "From v1.3.0 onwards, you must pass an open file object "
+                'directly, e.g. `form["name"] = open("/path/to/file", "rb")`. '
+                "This change is to remediate a security vulnerability where "
+                "a malicious web server could read arbitrary files from the "
+                "client (CVE-2023-34457)."
+            )

programming_language: Python
filename: utils.py
change_type: ModificationType.MODIFY
diff: @@ -14,3 +14,10 @@ class LinkNotFoundError(Exception):
       StatefulBrowser).
     """
     pass
+
+
+def is_multipart_file_upload(form, tag):
+    return (
+        form.get("enctype", "") == "multipart/form-data" and
+        tag.get("type", "").lower() == "file"
+    )

programming_language: Python
filename: test_browser.py
change_type: ModificationType.MODIFY
diff: @@ -164,8 +164,9 @@ def test_enctype_and_file_submit(httpbin, enctype, submit_file, file_field):
         else:
             # Encoding doesn't allow sending the content, we expect
             # the filename as a normal text field.
-            expected_content = pic_path.encode()
-        form.find("input", {"name": "pic"})["value"] = pic_path
+            expected_content = os.path.basename(pic_path.encode())
+        tag = form.find("input", {"name": "pic"})
+        tag["value"] = open(pic_path, "rb")
 
     browser = mechanicalsoup.Browser()
     response = browser._request(form)

programming_language: Python
filename: test_stateful_browser.py
change_type: ModificationType.MODIFY
diff: @@ -391,7 +391,12 @@ def test_form_multiple():
 
 def test_upload_file(httpbin):
     browser = mechanicalsoup.StatefulBrowser()
-    browser.open(httpbin + "/forms/post")
+    url = httpbin + "/post"
+    file_input_form = f"""
+    <form method="post" action="{url}" enctype="multipart/form-data">
+        <input type="file" name="first" />
+    </form>
+    """
 
     # Create two temporary files to upload
     def make_file(content):
@@ -399,24 +404,64 @@ def test_upload_file(httpbin):
         with open(path, "w") as fd:
             fd.write(content)
         return path
-    path1, path2 = (make_file(content) for content in
-                    ("first file content", "second file content"))
+    path1 = make_file("first file content")
+    path2 = make_file("second file content")
 
-    # The form doesn't have a type=file field, but the target action
-    # does show it => add the fields ourselves, and add enctype too.
+    value1 = open(path1, "rb")
+    value2 = open(path2, "rb")
+
+    browser.open_fake_page(file_input_form)
     browser.select_form()
-    browser._StatefulBrowser__state.form.form[
-      "enctype"] = "multipart/form-data"
-    browser.new_control("file", "first", path1)
-    browser.new_control("file", "second", "")
-    browser["second"] = path2
-    browser.form.print_summary()
+
+    # Test filling an existing input and creating a new input
+    browser["first"] = value1
+    browser.new_control("file", "second", value2)
+
     response = browser.submit_selected()
     files = response.json()["files"]
     assert files["first"] == "first file content"
     assert files["second"] == "second file content"
 
 
+def test_upload_file_with_malicious_default(httpbin):
+    """Check for CVE-2023-34457 by setting the form input value directly to a
+    file that the user does not explicitly consent to upload, as a malicious
+    server might do.
+    """
+    browser = mechanicalsoup.StatefulBrowser()
+    sensitive_path = tempfile.mkstemp()[1]
+    with open(sensitive_path, "w") as fd:
+        fd.write("Some sensitive information")
+    url = httpbin + "/post"
+    malicious_html = f"""
+    <form method="post" action="{url}" enctype="multipart/form-data">
+        <input type="file" name="malicious" value="{sensitive_path}" />
+    </form>
+    """
+    browser.open_fake_page(malicious_html)
+    browser.select_form()
+    response = browser.submit_selected()
+    assert response.json()["files"] == {"malicious": ""}
+
+
+def test_upload_file_raise_on_string_input():
+    """Check for use of the file upload API that was modified to remediate
+    CVE-2023-34457. Users must now open files manually to upload them.
+    """
+    browser = mechanicalsoup.StatefulBrowser()
+    file_input_form = """
+    <form enctype="multipart/form-data">
+        <input type="file" name="upload" />
+    </form>
+    """
+    browser.open_fake_page(file_input_form)
+    browser.select_form()
+    with pytest.raises(ValueError, match="CVE-2023-34457"):
+        browser["upload"] = "/path/to/file"
+    with pytest.raises(ValueError, match="CVE-2023-34457"):
+        browser.new_control("file", "upload2", "/path/to/file")
+
+
 def test_with():
     """Test that __enter__/__exit__ properly create/close the browser."""
     with mechanicalsoup.StatefulBrowser() as browser:

programming_language: Python