CVE-2021-3828 (published 2021-09-27T13:15Z)

nltk is vulnerable to Inefficient Regular Expression Complexity

Severity: MEDIUM
CVSS v2 base score: 5.0 (AV:N/AC:L/Au:N/C:N/I:N/A:P)
CVSS v3 base score: 7.5, HIGH (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H)

Weakness information:
cwe_id: CWE-697
cwe_name: Incorrect Comparison
description: The product compares two entities in a security-relevant context, but the comparison is incorrect, which may lead to resultant weaknesses.
extended_description: 
            This Pillar covers several possibilities:
               
                  the comparison checks one factor incorrectly;
                  the comparison should consider multiple factors, but it does not check at least one of those factors at all;
                  the comparison checks the wrong factor.
               
         
         

Fix information:
repo_url: https://github.com/nltk/nltk
commit_details: repo_url: https://github.com/nltk/nltk
author: Tom Aarsen
author_date: 2021-09-25 01:14:32+02:00
msg: Resolved ReDoS vulnerability in Corpus Reader (#2816)

* Resolved ReDoS vulnerability in the Corpus Reader for the Comparative Sentence Dataset

* Solidified performance tests
repository: repo_url: https://github.com/nltk/nltk
repo_name: nltk/nltk
description: NLTK Source
repo_language: Python
owner: nltk
file_changes: filename: comparative_sents.py
change_type: ModificationType.MODIFY
diff: @@ -45,7 +45,7 @@ CLOSE_COMPARISON = re.compile(r"</cs-[1234]>")
 GRAD_COMPARISON = re.compile(r"<cs-[123]>")
 NON_GRAD_COMPARISON = re.compile(r"<cs-4>")
 ENTITIES_FEATS = re.compile(r"(\d)_((?:[\.\w\s/-](?!\d_))+)")
-KEYWORD = re.compile(r"\((?!.*\()(.*)\)$")
+KEYWORD = re.compile(r"\(([^\(]*)\)$")
 
 
 class Comparison:

programming_language: Python
filename: corpus.doctest
change_type: ModificationType.MODIFY
diff: @@ -2162,3 +2162,35 @@ access to its tuples() method
     >>> from nltk.corpus import qc
     >>> qc.tuples('test.txt')
     [('NUM:dist', 'How far is it from Denver to Aspen ?'), ('LOC:city', 'What county is Modesto , California in ?'), ...]
+
+Ensure that KEYWORD from `comparative_sents.py` no longer contains a ReDoS vulnerability.
+
+    >>> import re
+    >>> import time
+    >>> from nltk.corpus.reader.comparative_sents import KEYWORD
+    >>> sizes = {
+    ...     "short": 4000,
+    ...     "long": 40000
+    ... }
+    >>> exec_times = {
+    ...     "short": [],
+    ...     "long": [],
+    ... }
+    >>> for size_name, size in sizes.items():
+    ...     for j in range(9):
+    ...         start_t = time.perf_counter()
+    ...         payload = "( " + "(" * size
+    ...         output = KEYWORD.findall(payload)
+    ...         exec_times[size_name].append(time.perf_counter() - start_t)
+    ...     exec_times[size_name] = sorted(exec_times[size_name])[4] # Get the mean
+
+Ideally, the execution time of such a regular expression is linear
+in the length of the input. As such, we would expect exec_times["long"]
+to be roughly 10 times as big as exec_times["short"].
+With the ReDoS in place, it took roughly 80 times as long.
+For now, we accept values below 30 (times as long), due to the potential
+for variance. This ensures that the ReDoS has certainly been reduced,
+if not removed.
+
+    >>> exec_times["long"] / exec_times["short"] < 30
+    True

programming_language: Python